mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-30 18:56:16 +00:00
allaun
318db01e31
Remove EC2/RDS deployment scripts, EC2 watchdog, and RDS substrate provisioning files. Drop the legacy AWS API MCP server from all IDE configs. Replace hardcoded AWS proof-server and RDS default endpoints with localhost/env-only placeholders. Deleted: - 4-Infrastructure/infra/deploy_aws_language_proof_server.sh - 4-Infrastructure/infra/aws_language_proof_server_user_data.sh - 4-Infrastructure/infra/ec2-configuration.nix - 4-Infrastructure/infra/ec2_idle_watchdog.py - 4-Infrastructure/infra/ec2-idle-watchdog.service - 4-Infrastructure/infra/ec2-idle-watchdog.timer - 4-Infrastructure/deploy/rds-substrate/* Changed: - .mcp.json, .cursor/mcp.json, .roo/mcp.json, .vscode/mcp.json: remove aws server, make remote-lean-proof URL env-only - .vscode/settings.json: remoteLeanProof.url -> localhost - 4-Infrastructure/infra/remote_lean_proof_mcp.py: default URL -> localhost - ene-rds/ene-session-sync: default RDS_HOST -> localhost - RECOVERY.md, INFRASTRUCTURE.md, TODO_MAP.md: remove AWS/EC2 refs Build: cargo check ene-rds OK; python3 -m py_compile OK; JSON valid
155 lines
5 KiB
Markdown
155 lines
5 KiB
Markdown
# ene-session-sync
|
|
|
|
Rust daemon that syncs OpenCode chat sessions to the ENE RDS PostgreSQL cluster.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
opencode.db (SQLite) ──► OpenCodeSource ──►┐
|
|
.claw/sessions/*.jsonl ──► ClawSource ──►├── RdsSink ──► ene.chat_sessions
|
|
│ ene.chat_messages
|
|
│ ene.ingestion_receipts
|
|
Ollama embed ──► vector(768) columns (optional)
|
|
|
|
PythonBridge ──► ene_rds_wiki_layer / fractal_fold
|
|
(existing Python surfaces via subprocess)
|
|
```
|
|
|
|
### Rust surfaces (this crate)
|
|
|
|
| Module | Role |
|
|
|--------|------|
|
|
| `source.rs` | `OpenCodeSource` (SQLite/rusqlite) + `ClawSource` (JSONL) |
|
|
| `sink.rs` | `RdsSink` — tokio-postgres, idempotent upserts, DDL init |
|
|
| `embed.rs` | `Embedder` — Ollama `/api/embeddings` via reqwest |
|
|
| `normalize.rs` | OpenCode row → `ChatSession`/`ChatMessage` normalization |
|
|
| `bridge.rs` | `PythonBridge` — subprocess delegation to Python infra modules |
|
|
| `models.rs` | Shared data types |
|
|
| `main.rs` | CLI (clap) + orchestration |
|
|
|
|
### Python adaptation layer (`bridge.rs` + `bridge_wrapper.py`)
|
|
|
|
The Rust binary can delegate to any Python infra surface via:
|
|
|
|
```
|
|
Rust PythonBridge::call("ene_rds_wiki_layer", payload)
|
|
└─► python3 bridge_wrapper.py <infra_dir>
|
|
├── reads JSON request from stdin
|
|
├── imports the named module
|
|
├── calls instance.handle_request(payload)
|
|
└── writes JSON response to stdout
|
|
```
|
|
|
|
Modules currently bridgeable: `ene_rds_wiki_layer`, `ene_rds_fractal_fold`,
|
|
`ene_rds_ephemeral_node`, `ene_embedding`, and any module that exposes
|
|
`handle_request(payload: dict) -> dict`.
|
|
|
|
## RDS Schema
|
|
|
|
Tables are created/migrated automatically on first connect:
|
|
|
|
```sql
|
|
ene.chat_sessions -- one row per session
|
|
ene.chat_messages -- one row per message (UNIQUE on session_id, message_index)
|
|
ene.ingestion_receipts -- audit trail for each sync run
|
|
```
|
|
|
|
Full DDL lives in `sink.rs::init_tables()`.
|
|
|
|
## CLI
|
|
|
|
```
|
|
ene-session-sync [OPTIONS] <SUBCOMMAND>
|
|
|
|
Global options:
|
|
--db <PATH> opencode.db path (default: ~/.local/share/opencode/opencode.db)
|
|
--dsn <DSN> PostgreSQL DSN (default: RDS_* env vars)
|
|
--infra-dir <DIR> Python infra directory for bridge calls
|
|
--embed Enable Ollama embedding generation
|
|
|
|
Subcommands:
|
|
sync [--since <MS>] One-shot sync of opencode.db
|
|
watch [--interval <SECS>] Continuous poll (default 60 s)
|
|
claw-sync --sessions-dir <D> Sync .claw/sessions/ JSONL files
|
|
search <QUERY> [--limit N] [--semantic]
|
|
list [--limit N]
|
|
get <SESSION_ID>
|
|
bridge-test [MODULE] Test Python bridge for a module
|
|
init-schema Initialize RDS schema only
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Credentials come from environment variables only — never hard-code them.
|
|
|
|
| Variable | Purpose |
|
|
|----------|---------|
|
|
| `RDS_HOST` | PostgreSQL host |
|
|
| `RDS_PORT` | PostgreSQL port (default 5432) |
|
|
| `RDS_USER` | PostgreSQL user |
|
|
| `RDS_PASSWORD` / `RDS_IAM_TOKEN` | Password or IAM auth token |
|
|
| `RDS_DB` | Database name (default `postgres`) |
|
|
| `RDS_DSN` | Full libpq DSN (overrides all above) |
|
|
| `OLLAMA_HOST` | Ollama base URL (default `http://localhost:11434`) |
|
|
| `OLLAMA_EMBED_MODEL` | Embedding model (default `nomic-embed-text`) |
|
|
| `PYTHON_CMD` | Python executable for bridge (default `python3`) |
|
|
|
|
Store these in `/etc/ene-session-sync/env` or `~/.config/ene-session-sync/env`
|
|
(sourced by the systemd unit).
|
|
|
|
## Build
|
|
|
|
```bash
|
|
# From this directory:
|
|
cargo build --release
|
|
# Binary: target/release/ene-session-sync
|
|
|
|
# Or install to ~/.cargo/bin:
|
|
cargo install --path .
|
|
```
|
|
|
|
**Note:** `rusqlite` is compiled with the `bundled` feature so no system
|
|
`libsqlite3` is required.
|
|
|
|
**Note on TLS:** `tokio-postgres` is linked with `NoTls` in this build. RDS
|
|
requires SSL; use `sslmode=disable` with an SSL-terminating proxy (e.g.
|
|
`pg_bouncer` or `ssh -L`) or add `tokio-postgres-rustls` as a dependency and
|
|
swap `NoTls` in `sink.rs::connect()`.
|
|
|
|
## Systemd installation
|
|
|
|
```bash
|
|
# Copy binary
|
|
sudo cp target/release/ene-session-sync /usr/local/bin/
|
|
|
|
# Install units
|
|
sudo cp systemd/ene-session-sync.{service,timer} /etc/systemd/system/
|
|
|
|
# Create env file
|
|
sudo mkdir -p /etc/ene-session-sync
|
|
sudo tee /etc/ene-session-sync/env <<'EOF'
|
|
RDS_HOST=localhost
|
|
RDS_USER=postgres
|
|
RDS_PASSWORD=<set via env>
|
|
EOF
|
|
sudo chmod 600 /etc/ene-session-sync/env
|
|
|
|
# Enable
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now ene-session-sync.timer
|
|
|
|
# Check
|
|
systemctl status ene-session-sync.timer
|
|
journalctl -u ene-session-sync -f
|
|
```
|
|
|
|
## Python bridge setup
|
|
|
|
The bridge expects `bridge_wrapper.py` to be co-located with the infra modules.
|
|
The `--infra-dir` flag (or the directory next to the running binary) must contain
|
|
both `bridge_wrapper.py` and the Python modules.
|
|
|
|
```bash
|
|
# Test:
|
|
ene-session-sync --infra-dir /path/to/stack/4-Infrastructure/infra bridge-test ene_rds_wiki_layer
|
|
```
|