mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-31 03:05:21 +00:00
722da7018a
- source.rs: Add ClawSource adapter that reads .claw/sessions/*.jsonl (skips
LFS stubs), parses both full-JSON and JSONL-stream formats, normalises into
ChatSession/ChatMessage identical to the OpenCode path.
- models.rs: Add title/agent/model fields to ChatSession; remove unused
chrono import.
- normalize.rs: Populate title/agent/model from OpenCodeSession.
- sink.rs: Full rewrite:
- DDL adds title/agent/model columns + ALTER TABLE IF NOT EXISTS migration
path for existing clusters.
- upsert_session and upsert_messages parameter counts match placeholders
($19 and $13 respectively).
- FTS index uses COALESCE to handle NULL text_content.
- sslmode= token stripped before parse so NoTls connect doesn't reject
it; README documents the SSL upgrade path (tokio-postgres-rustls).
- with-serde_json-1 feature added to tokio-postgres dep for JSONB params.
- main.rs:
- Replace DefaultHasher sha256_text with FNV-1a 64-bit (correct hex digest).
- Add ClawSync, List, Get subcommands.
- Remove unused Context import.
- bridge.rs: Rewrite to use bridge_wrapper.py subprocess rather than inline
-c script; resolves wrapper path relative to infra_dir or CARGO_MANIFEST_DIR;
unwraps {"ok":true,"data":{}} envelope; health_check() tests python3 only.
- embed.rs: Remove unused OllamaEmbedRequest import.
- systemd/: Add ene-session-sync.service + .timer (5-min cadence, 2-min
OnBootSec, Persistent=true for catch-up after suspend).
- README.md: Architecture diagram, CLI reference, schema, build/install,
systemd install, Python bridge setup.
Generated with [Devin](https://cli.devin.ai/docs)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
155 lines
5.1 KiB
Markdown
155 lines
5.1 KiB
Markdown
# ene-session-sync
|
|
|
|
Rust daemon that syncs OpenCode chat sessions to the ENE RDS PostgreSQL cluster.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
opencode.db (SQLite) ──► OpenCodeSource ──►┐
|
|
.claw/sessions/*.jsonl ──► ClawSource ──►├── RdsSink ──► ene.chat_sessions
|
|
│ ene.chat_messages
|
|
│ ene.ingestion_receipts
|
|
Ollama embed ──► vector(768) columns (optional)
|
|
|
|
PythonBridge ──► ene_rds_wiki_layer / fractal_fold
|
|
(existing Python surfaces via subprocess)
|
|
```
|
|
|
|
### Rust surfaces (this crate)
|
|
|
|
| Module | Role |
|
|
|--------|------|
|
|
| `source.rs` | `OpenCodeSource` (SQLite/rusqlite) + `ClawSource` (JSONL) |
|
|
| `sink.rs` | `RdsSink` — tokio-postgres, idempotent upserts, DDL init |
|
|
| `embed.rs` | `Embedder` — Ollama `/api/embeddings` via reqwest |
|
|
| `normalize.rs` | OpenCode row → `ChatSession`/`ChatMessage` normalization |
|
|
| `bridge.rs` | `PythonBridge` — subprocess delegation to Python infra modules |
|
|
| `models.rs` | Shared data types |
|
|
| `main.rs` | CLI (clap) + orchestration |
|
|
|
|
### Python adaptation layer (`bridge.rs` + `bridge_wrapper.py`)
|
|
|
|
The Rust binary can delegate to any Python infra surface via:
|
|
|
|
```
|
|
Rust PythonBridge::call("ene_rds_wiki_layer", payload)
|
|
└─► python3 bridge_wrapper.py <infra_dir>
|
|
├── reads JSON request from stdin
|
|
├── imports the named module
|
|
├── calls instance.handle_request(payload)
|
|
└── writes JSON response to stdout
|
|
```
|
|
|
|
Modules currently bridgeable: `ene_rds_wiki_layer`, `ene_rds_fractal_fold`,
|
|
`ene_rds_ephemeral_node`, `ene_embedding`, and any module that exposes
|
|
`handle_request(payload: dict) -> dict`.
|
|
|
|
## RDS Schema
|
|
|
|
Tables are created/migrated automatically on first connect:
|
|
|
|
```sql
|
|
ene.chat_sessions -- one row per session
|
|
ene.chat_messages -- one row per message (UNIQUE on session_id, message_index)
|
|
ene.ingestion_receipts -- audit trail for each sync run
|
|
```
|
|
|
|
Full DDL lives in `sink.rs::init_tables()`.
|
|
|
|
## CLI
|
|
|
|
```
|
|
ene-session-sync [OPTIONS] <SUBCOMMAND>
|
|
|
|
Global options:
|
|
--db <PATH> opencode.db path (default: ~/.local/share/opencode/opencode.db)
|
|
--dsn <DSN> PostgreSQL DSN (default: RDS_* env vars)
|
|
--infra-dir <DIR> Python infra directory for bridge calls
|
|
--embed Enable Ollama embedding generation
|
|
|
|
Subcommands:
|
|
sync [--since <MS>] One-shot sync of opencode.db
|
|
watch [--interval <SECS>] Continuous poll (default 60 s)
|
|
claw-sync --sessions-dir <D> Sync .claw/sessions/ JSONL files
|
|
search <QUERY> [--limit N] [--semantic]
|
|
list [--limit N]
|
|
get <SESSION_ID>
|
|
bridge-test [MODULE] Test Python bridge for a module
|
|
init-schema Initialize RDS schema only
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Credentials come from environment variables only — never hard-code them.
|
|
|
|
| Variable | Purpose |
|
|
|----------|---------|
|
|
| `RDS_HOST` | PostgreSQL host |
|
|
| `RDS_PORT` | PostgreSQL port (default 5432) |
|
|
| `RDS_USER` | PostgreSQL user |
|
|
| `RDS_PASSWORD` / `RDS_IAM_TOKEN` | Password or RDS IAM auth token |
|
|
| `RDS_DB` | Database name (default `postgres`) |
|
|
| `RDS_DSN` | Full libpq DSN (overrides all above) |
|
|
| `OLLAMA_HOST` | Ollama base URL (default `http://localhost:11434`) |
|
|
| `OLLAMA_EMBED_MODEL` | Embedding model (default `nomic-embed-text`) |
|
|
| `PYTHON_CMD` | Python executable for bridge (default `python3`) |
|
|
|
|
Store these in `/etc/ene-session-sync/env` or `~/.config/ene-session-sync/env`
|
|
(sourced by the systemd unit).
|
|
|
|
## Build
|
|
|
|
```bash
|
|
# From this directory:
|
|
cargo build --release
|
|
# Binary: target/release/ene-session-sync
|
|
|
|
# Or install to ~/.cargo/bin:
|
|
cargo install --path .
|
|
```
|
|
|
|
**Note:** `rusqlite` is compiled with the `bundled` feature so no system
|
|
`libsqlite3` is required.
|
|
|
|
**Note on TLS:** `tokio-postgres` is linked with `NoTls` in this build. RDS
|
|
requires SSL; use `sslmode=disable` with an SSL-terminating proxy (e.g.
|
|
`pg_bouncer` or `ssh -L`) or add `tokio-postgres-rustls` as a dependency and
|
|
swap `NoTls` in `sink.rs::connect()`.
|
|
|
|
## Systemd installation
|
|
|
|
```bash
|
|
# Copy binary
|
|
sudo cp target/release/ene-session-sync /usr/local/bin/
|
|
|
|
# Install units
|
|
sudo cp systemd/ene-session-sync.{service,timer} /etc/systemd/system/
|
|
|
|
# Create env file
|
|
sudo mkdir -p /etc/ene-session-sync
|
|
sudo tee /etc/ene-session-sync/env <<'EOF'
|
|
RDS_HOST=database-1.cluster-xxxx.us-east-2.rds.amazonaws.com
|
|
RDS_USER=postgres
|
|
RDS_IAM_TOKEN=<generated by aws rds generate-db-auth-token>
|
|
EOF
|
|
sudo chmod 600 /etc/ene-session-sync/env
|
|
|
|
# Enable
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now ene-session-sync.timer
|
|
|
|
# Check
|
|
systemctl status ene-session-sync.timer
|
|
journalctl -u ene-session-sync -f
|
|
```
|
|
|
|
## Python bridge setup
|
|
|
|
The bridge expects `bridge_wrapper.py` to be co-located with the infra modules.
|
|
The `--infra-dir` flag (or the directory next to the running binary) must contain
|
|
both `bridge_wrapper.py` and the Python modules.
|
|
|
|
```bash
|
|
# Test:
|
|
ene-session-sync --infra-dir /path/to/stack/4-Infrastructure/infra bridge-test ene_rds_wiki_layer
|
|
```
|