infra(neon): decommission neon-64gb and redirect all database and LLM endpoints to nixos-laptop

This commit is contained in:
allaun 2026-07-05 16:20:22 -05:00
parent 36a78fec72
commit 5413527476
19 changed files with 173 additions and 40 deletions

View file

@ -98,10 +98,10 @@ Dynamo-style S3-compatible store written in Rust. Replaced rclone serve s3.
| Node | Tailscale IP | k3s | Garage | Zone | Disk | SSH |
|------|-------------|-----|--------|------|------|-----|
| **qfox-1** (this machine) | 100.88.57.96 | ✅ worker | ✅ 780 GiB | local | 1.8 TB NVMe | local |
| **cupfox** | 100.72.130.76 | ✅ control-plane | ✅ 69 GiB | fra | 125 GB | key OK (361395) |
| **cupfox** | 100.115.119.40 | ✅ control-plane | ✅ 69 GiB | fra | 125 GB | key OK (361395) |
| **nixos-laptop** | 100.102.173.61 | ✅ worker | ✅ 347 GiB | ord | 459 GB NVMe | key OK |
| **racknerd** | 100.80.39.40 | ✅ worker | ✅ 954 MiB | vps | 9.1 GB VPS | key OK |
| **neon-64gb** | 100.92.88.64 | ❌ rebuilt (standalone k3s) | ❌ | netcup-arm | 2 TB | key OK (allaun) |
| **neon-64gb** | 100.92.88.64 | ❌ decommissioned (rebuilt) | ❌ | netcup-arm | 2 TB | key OK (allaun) |
| **steamdeck** | 100.85.244.73 | ✅ worker | ✅ 373 GiB | gpu | 476 GB NVMe | key OK |
| rs-vps (netcup) | — | ❌ | ❌ | — | 2 TB | SSH via password |
| dracocomp | 100.100.140.27 | ❌ | ❌ | — | — | unreachable
@ -474,15 +474,15 @@ Never claim:
- `4-Infrastructure/shim/rrc_bosonic_tensor_gpu.py` — GPU-accelerated Bosonic Tensor Network Centrality: computes exp(-i*A*theta) using adaptive RK4 on GPU with `wgpu` (Vulkan) to scale N to 10000+ without CPU eigendecomposition bottleneck. Receipt: `rrc_bosonic_tensor_gpu_receipt.json`.
- `4-Infrastructure/shim/rrc_bosonic_db_buffer.py` — Asynchronous database ingestion buffer: queues, batches, and flushes PostgreSQL inserts for bosonic tensor network receipts and metrics in thread-safe worker pools.
- `4-Infrastructure/shim/pist_trace_classify_offline.py` — Offline, token-free trace classifier implementing Lean's `Semantics.PIST.Classify` color-space model and executing the local `rrc-watchdog` binary in the container.
- `4-Infrastructure/shim/lake_build_ingest.py` — Pure-I/O shim that runs `lake build <target>` and ingests the result into `ene.sessions`, `ene.packages`, `ene.receipts`, and `ene.ingest_events` on the canonical neon-64gb Postgres (`arxiv-pg` container). No admissibility logic, no Float arithmetic.
- `4-Infrastructure/shim/lake_build_ingest.py` — Pure-I/O shim that runs `lake build <target>` and ingests the result into `ene.sessions`, `ene.packages`, `ene.receipts`, and `ene.ingest_events` on the canonical nixos-laptop Postgres (`100.102.173.61`). No admissibility logic, no Float arithmetic.
- `4-Infrastructure/k3s-flake/tests/chat-verify.spec.ts` — E2E Playwright verification spec for Chat (Hermes) SSO login.
- `4-Infrastructure/k3s-flake/tests/audiobookshelf-verify.spec.ts` — E2E Playwright verification spec for Audiobookshelf SSO login.
## Canonical database locations
- **Postgres (canonical):** `arxiv-pg` podman container on **neon-64gb** (`100.92.88.64`). Databases: `arxiv` (arXiv papers + pgvector HNSW), `ene` (ENE memory substrate with full schema from `ene_substrate_schema.sql`).
- **Postgres (canonical):** PostgreSQL service on **nixos-laptop** (`100.102.173.61`). Databases: `arxiv` (arXiv papers + pgvector HNSW), `ene` (ENE memory substrate with full schema from `ene_substrate_schema.sql`).
- **Gremlin (canonical):** Azure Cosmos DB endpoint in `.env.gremlin` (`mathblob.gremlin.cosmos.azure.com`). Current graph: 44,804 vertices, 29,466 edges.
- **Local qfox Postgres:** none. Do not spin up local Postgres containers on qfox for production data; they are not replicated to neon and create confusion.
- **Local qfox Postgres:** none. Do not spin up local Postgres containers on qfox for production data; they are not replicated to nixos-laptop and create confusion.
## Compute Dispatch (WGSL → any substrate)
@ -549,7 +549,7 @@ blessed Compiler surface:
- Command: `python3 "4-Infrastructure/shim/lake_build_ingest.py" Compiler --actor opencode`
- Runs in background so `git commit` is not blocked
- Log: `~/.cache/lake-build-ingest.log`
- Destination: `ene.sessions`, `ene.packages`, `ene.receipts`, `ene.ingest_events` on neon-64gb (`arxiv-pg` container)
- Destination: `ene.sessions`, `ene.packages`, `ene.receipts`, `ene.ingest_events` on nixos-laptop (`100.102.173.61`)
To reinstall the hook on a fresh clone, append the background invocation from
`.git/hooks/post-commit` to the Git LFS post-commit hook.

View file

@ -14,7 +14,7 @@ if [[ ! -x "$BIN" ]]; then
exit 127
fi
export RDS_HOST="${RDS_HOST:-100.92.88.64}"
export RDS_HOST="${RDS_HOST:-100.102.173.61}"
export RDS_PORT="${RDS_PORT:-5432}"
export RDS_USER="${RDS_USER:-postgres}"
export RDS_DB="${RDS_DB:-postgres}"

View file

@ -19,7 +19,7 @@ from urllib.error import HTTPError
SERVER_NAME = "vikunja-mcp"
SERVER_VERSION = "0.1.0"
VIKUNJA_URL = os.environ.get("VIKUNJA_URL", "http://100.92.88.64:3456")
VIKUNJA_URL = os.environ.get("VIKUNJA_URL", "http://100.102.173.61:3456")
_token_file = os.environ.get("VIKUNJA_TOKEN_FILE", "")
VIKUNJA_TOKEN = os.environ.get("VIKUNJA_TOKEN", "")
if not VIKUNJA_TOKEN and _token_file:

View file

@ -12,7 +12,7 @@ pub struct DbConfig {
impl DbConfig {
pub fn from_env() -> Self {
let host = env::var("RDS_HOST")
.unwrap_or_else(|_| "100.92.88.64".to_string());
.unwrap_or_else(|_| "100.102.173.61".to_string());
let user = env::var("RDS_USER").unwrap_or_else(|_| "postgres".to_string());
let password = env::var("RDS_PASSWORD").unwrap_or_else(|_| "".to_string());

View file

@ -19,7 +19,7 @@ from shim.utils import sha256_text, utc_now
BATCH_SIZE = 50
TOKEN_REFRESH_SEC = 600
HOST = os.environ.get("RDS_HOST", "100.92.88.64")
HOST = os.environ.get("RDS_HOST", "100.102.173.61")
PORT = int(os.environ.get("RDS_PORT", "5432"))
USER = os.environ.get("RDS_USER", "postgres")
DB = os.environ.get("RDS_DB", os.environ.get("RDS_DBNAME", "postgres"))

View file

@ -50,7 +50,7 @@ ENDPOINTS = {
"temperature": 0.1,
},
"neon-qwen": {
"url": "http://100.92.88.64:11434/v1/chat/completions",
"url": "http://100.102.173.61:11434/v1/chat/completions",
"model": "hf.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q4_K_M",
"name": "Qwen3.6-35B-A3B (neon, CPU)",
"timeout": 3600,
@ -58,7 +58,7 @@ ENDPOINTS = {
"temperature": 0.1,
},
"neon-step": {
"url": "http://100.92.88.64:11434/v1/chat/completions",
"url": "http://100.102.173.61:11434/v1/chat/completions",
"model": "step3.7-flash",
"name": "Step-3.7-Flash (neon, CPU)",
"timeout": 3600,
@ -68,7 +68,7 @@ ENDPOINTS = {
# DeepSeek-Prover-V2-7B: purpose-built Lean 4 / math proof model (~4-5 GB Q4_K_M).
# Best for hard construction sorries; trained on Lean proof search data.
"neon-deepseek": {
"url": "http://100.92.88.64:11434/v1/chat/completions",
"url": "http://100.102.173.61:11434/v1/chat/completions",
"model": "hf.co/irmma/DeepSeek-Prover-V2-7B-Q4_K_M-GGUF:latest",
"name": "DeepSeek-Prover-V2-7B (neon, CPU)",
"timeout": 3600,

View file

@ -34,7 +34,7 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(mess
log = logging.getLogger("dataset_ingest_rds")
# Config
RDS_HOST = os.environ.get("RDS_HOST", "100.92.88.64")
RDS_HOST = os.environ.get("RDS_HOST", "100.102.173.61")
STACK_ROOT = Path(os.environ.get("STACK_ROOT", "/home/researcher/stack"))
DATA_DIR = STACK_ROOT / "shared-data" / "data" / "ingested_datasets" / "2026-05-18"

View file

@ -30,7 +30,7 @@ except ImportError:
sys.path.insert(0, os.getcwd())
import gccl_waveprobe as gw
DEFAULT_TARGET_HOST = "100.92.88.64" # neon-64gb IP
DEFAULT_TARGET_HOST = "100.102.173.61" # neon-64gb IP
# ── Manifest Generation ─────────────────────────────────────────────────────

View file

@ -215,12 +215,12 @@ def emit_markdown(report: dict, out_path: Path) -> None:
def push_to_appflowy(report: dict) -> None:
"""Push top modules to AppFloyo Cloud workspace."""
url = os.environ.get("APPFLOWY_URL", "http://100.92.88.64:8000")
url = os.environ.get("APPFLOWY_URL", "http://100.102.173.61:8000")
token = os.environ.get("APPFLOWY_TOKEN", "")
# If no token, try to get one from GoTrue
if not token:
gotrue_url = os.environ.get("GOTRUE_URL", "http://100.92.88.64:9999")
gotrue_url = os.environ.get("GOTRUE_URL", "http://100.102.173.61:9999")
email = os.environ.get("GOTRUE_ADMIN_EMAIL", "admin@researchstack.info")
password = os.environ.get("GOTRUE_ADMIN_PASSWORD", "admin123")
try:

View file

@ -9,7 +9,7 @@ lake_build_ingest.py — Run `lake build` and ingest the result into ENE on neon
This is a pure-I/O shim: it runs the build, parses the output, and writes rows to
Postgres. It contains no admissibility logic and no Float arithmetic.
Target database: arxiv-pg container on neon-64gb (Tailscale 100.92.88.64).
Target database: arxiv-pg container on neon-64gb (Tailscale 100.102.173.61).
Tables used:
ene.sessions one row per build invocation
ene.packages one row per build target/receipt
@ -46,7 +46,7 @@ from typing import Any
ROOT = Path("/home/allaun/Research Stack")
LEAN_DIR = ROOT / "0-Core-Formalism/lean/Semantics"
NEON_HOST = "100.92.88.64"
NEON_HOST = "100.102.173.61"
CONTAINER = "arxiv-pg"
DB = "ene"
PIST_CLASSIFY = LEAN_DIR / ".lake" / "build" / "bin" / "pist-classify-trace"

View file

@ -28,7 +28,7 @@ def _resolve_params() -> dict:
params["sslmode"] = v
return params
host = os.environ.get("RDS_HOST", os.environ.get("PGHOST", "100.92.88.64"))
host = os.environ.get("RDS_HOST", os.environ.get("PGHOST", "100.102.173.61"))
port = int(os.environ.get("RDS_PORT", os.environ.get("PGPORT", "5432")))
user = os.environ.get("RDS_USER", os.environ.get("PGUSER", "postgres"))
dbname = os.environ.get("RDS_DB", os.environ.get("RDS_DBNAME", os.environ.get("PGDATABASE", "postgres")))

View file

@ -17,7 +17,7 @@ from shim.utils import sha256_text, utc_now
STACK_ROOT = Path(os.environ.get("STACK_ROOT", "/home/allaun/Research Stack"))
WIKI_ROOT = Path(os.environ.get("WIKI_ROOT", str(STACK_ROOT / "6-Documentation" / "wiki")))
HOST = os.environ.get("RDS_HOST", "100.92.88.64")
HOST = os.environ.get("RDS_HOST", "100.102.173.61")
DB = os.environ.get("RDS_DB", os.environ.get("RDS_DBNAME", "postgres"))

View file

@ -32,7 +32,7 @@ TIER0_PGDUMP="/mnt/stackcache/pgdump"
TIER2_REMOTE="gdrive:research-stack-offload"
TIER1_THRESHOLD_GB=2
RDS_HOST="${RDS_HOST:-100.92.88.64}"
RDS_HOST="${RDS_HOST:-100.102.173.61}"
RDS_PORT="${RDS_PORT:-5432}"
RDS_USER="${RDS_USER:-postgres}"
RDS_DB="${RDS_DB:-postgres}"

View file

@ -39,7 +39,7 @@ export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION:-garage}"
export AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-http://localhost:3900}"
RDS_HOST="${RDS_HOST:-100.92.88.64}"
RDS_HOST="${RDS_HOST:-100.102.173.61}"
RDS_PORT="${RDS_PORT:-5432}"
RDS_USER="${RDS_USER:-postgres}"
RDS_DB="${RDS_DB:-postgres}"

View file

@ -38,7 +38,7 @@ fi
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION
export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE
RDS_HOST="${RDS_HOST:-100.92.88.64}"
RDS_HOST="${RDS_HOST:-100.102.173.61}"
RDS_PORT="${RDS_PORT:-5432}"
RDS_USER="${RDS_USER:-postgres}"
RDS_DB="${RDS_DB:-postgres}"

View file

@ -84,13 +84,13 @@ PROVIDERS: dict[str, dict] = {
"notes": "OpenRouter, deepseek-v4-flash",
},
"neon-deepseek-prover": {
"api_base": "http://100.92.88.64:11434/v1",
"api_base": "http://100.102.173.61:11434/v1",
"api_key": "",
"model": "hf.co/irmma/DeepSeek-Prover-V2-7B-Q4_K_M-GGUF",
"notes": "neon-64gb, DeepSeek-Prover-V2 7B Q4_K_M, 62GB RAM CPU-only",
},
"neon-goedel-prover": {
"api_base": "http://100.92.88.64:11434/v1",
"api_base": "http://100.102.173.61:11434/v1",
"api_key": "",
"model": "hf.co/mradermacher/Goedel-Prover-V2-8B-GGUF",
"notes": "neon-64gb, Goedel-Prover-V2 8B GGUF, 62GB RAM CPU-only",

View file

@ -5,7 +5,7 @@
#
# Environment variables (set these in cloud env config):
# BUILD_SERVER_URL=http://100.88.57.96:8765
# NEON_OLLAMA_URL=http://100.92.88.64:11434
# NEON_OLLAMA_URL=http://100.102.173.61:11434
# LAKE_WORKDIR=/workspace/0-Core-Formalism/lean/Semantics
set -e
@ -54,7 +54,7 @@ try:
print(' build_server: UP')
except: print(' build_server: DOWN (tailnet required)')
try:
r = urllib.request.urlopen('${NEON_OLLAMA_URL:-http://100.92.88.64:11434}/api/tags', timeout=5)
r = urllib.request.urlopen('${NEON_OLLAMA_URL:-http://100.102.173.61:11434}/api/tags', timeout=5)
print(' neon_ollama: UP')
except: print(' neon_ollama: DOWN (tailnet required)')
"

View file

@ -50,7 +50,7 @@ def _ensure_singleton() -> None:
# ── Resource URLs (configurable via env) ────────────────────────────────
BUILD_SERVER = os.environ.get("BUILD_SERVER_URL", "http://100.88.57.96:8765")
NEON_OLLAMA = os.environ.get("NEON_OLLAMA_URL", "http://100.92.88.64:11434")
NEON_OLLAMA = os.environ.get("NEON_OLLAMA_URL", "http://100.102.173.61:11434")
NEON_MODEL = os.environ.get("NEON_MODEL",
"hf.co/irmma/DeepSeek-Prover-V2-7B-Q4_K_M-GGUF")
LAKE_WORKDIR = os.environ.get("LAKE_WORKDIR",

155
AGENTS.md
View file

@ -58,6 +58,38 @@ fully-local Hermes-3 model on qfox-1's RTX 4070. OpenClaw is decommissioned.
api_key: "sk-local"
```
## Provider-nixos LLM Gateway (2026-07-03)
`neon-rs1000` (Tailscale `100.79.14.103`, NixOS, 4 cores, 7.8Gi RAM) runs the
FreeLLMAPI proxy + OpenCode server as the model inference gateway for the
onklaud-5 pipeline.
### FreeLLMAPI (port 3001)
- **Repo**: `tashfeenahmed/freellmapi`, cloned to `~/freellmapi`, production build
- **Dashboard**: `http://100.79.14.103:3001` — admin: `admin@researchstack.info` / `admin123`
- **Unified API key**: `freellmapi-ef5610fa456735f2bcb6205faf03f8725fe41265f129cd56`
- **Base URL**: `http://100.79.14.103:3001/v1`
- **Systemd user unit**: `freellmapi.service` — enabled, auto-restart
- **20 provider keys installed** (Google, Groq, Cerebras, NVIDIA, Mistral, OpenRouter,
GitHub, Cloudflare, Zhipu, Ollama Cloud, HuggingFace, OpenCode Zen, Kilo,
Pollinations, OVH, LLM7, AI Horde, Agnes, Reka, SiliconFlow, Routeway, BazaarLink)
- **~1.7B tokens/month aggregate**, 6.1K used (~99.99% remaining)
### OpenCode Server (port 4096)
- **Version**: 1.17.9, installed via `nix profile install nixpkgs#opencode`
- **Systemd user unit**: `opencode-serve.service` — enabled, auto-restart
- **Provider**: FreeLLMAPI via `@ai-sdk/openai-compatible` (model: `freeapi/auto`)
- **Config**: `~/.config/opencode/config.json`
### Onklaud-5 Pipeline
- **Path**: `~/onklaud-5`
- **council.py**: Patched to route through FreeLLMAPI (`localhost:3001/v1`) instead of
OpenRouter directly. Both `KIMI_MODEL` and `GLM_MODEL` set to `auto`.
- **Venv**: `~/onklaud-5/venv`, 31/32 tests pass
### Common failure modes
- `500 Internal Error` from `chat.researchstack.info` after a model swap:
@ -91,18 +123,11 @@ workstation.
failures and writes corrections to AGENTS.md. Requires `claude` CLI available
for the analysis LLM.
### Neon proxy (2026-06-19)
### DECOMMISSIONED — Neon proxy (2026-06-19)
Headroom proxy also runs on neon-64gb (netcup ARM64, NixOS) for remote agent
sessions:
- **Tailscale endpoint**: `http://100.92.88.64:8787` (also reachable as
`http://neon-64gb:8787` via MagicDNS)
- **Startup**: `/home/allaun/.headroom/headroom-proxy-start.sh` (wraps
`LD_LIBRARY_PATH` for NixOS libstdc++ compatibility)
- **Installed via**: Python venv at `~/headroom-venv/`
- **Config**: `--memory --learn --port 8787 --host 0.0.0.0`
- **Route through it**: `ANTHROPIC_BASE_URL=http://100.92.88.64:8787 claude`
[INACTIVE / DECOMMISSIONED] Headroom proxy previously ran on neon-64gb:
- **Tailscale endpoint**: `http://100.92.88.64:8787` (inactive)
- **Status**: Decommissioned. All remote/local LLM sessions route directly through qfox-1 or external providers. Do not route ANTHROPIC_BASE_URL to neon.
### Common failure modes
@ -432,6 +457,114 @@ For files using PhysicsScalar.Q16_16:
### Migration Guide
See `LEAD_Q16_16_MIGRATION_GUIDE.md` for detailed microsteps.
## Gauge Theory Research Program
Gauge theory is the most rigorous framework for mapping out math because it forces explicit
distinctions between theorems, conjectures, and empirical observations. Use it as a probe,
not as a claim.
### What Gauge Theory Demands
1. **Proven vs Conjectural**
- If you claim a correspondence, you must prove it or label it an ansatz
- Empirical correlations (r=0.41, r=-0.33, n=13, p>0.05) are data, not theorems
- Gauge theory requires: "Is this a theorem or an ansatz?"
2. **Dimensional Honesty**
- AT model is 2D; gauge confinement requires 3+1D (or 2+1D for compact U(1))
- Confinement in 2D gauge theory is trivial — this is a theorem
- Dimensional mismatches must be explicitly addressed, not papered over
3. **Category/Type Correctness**
- Frustration is boolean/Z₂
- Wilson loop is real-valued (trace of holonomy)
- Curvature is Lie-algebra-valued
- Confusing these categories is a type error
### What Gauge Theory Does NOT Give You
1. **Derivation for Free** — The Baker-Hopf coupling is an ansatz, not derived from gauge principles
2. **Uniqueness** — Other connections (arctan, Li₂, etc.) could satisfy similar constraints
3. **Predictive Power** — Correlations computed because the hypothesis suggested them
### Success Criteria
Success is NOT "SilverSight IS gauge theory."
Success IS: "We attempted to derive SilverSight structures from gauge theory.
We succeeded for X, failed for Y, and learned Z."
Specifically:
- **Attempt derivation for Baker-Hopf coupling** — proves it's gauge-theoretic or reveals it's something else
- **Attempt derivation for frustration = Wilson loop** — proves the correspondence or reveals it's just a correlation
- **Attempt derivation for Baker Λ = field strength** — proves the formal resemblance or reveals it's just a resemblance
### Failure Mode
Failure is NOT "correspondences don't hold."
Failure IS: "We didn't attempt the derivation, so we don't know what the structures actually are."
### Recommended Framing
Use gauge theory as a research program:
```
"We investigate whether SilverSight can be derived from lattice gauge theory.
We have proven X, observed Y empirically, and conjecture Z.
Here are the falsification criteria."
```
Never claim:
```
"SilverSight IS gauge theory"
```
## Fraction Rules
The Research Stack operates on fixed-point arithmetic for mathematical rigor.
Float (`ofFloat`) is forbidden in compute paths.
### Q16_16 Fraction Types
| Type | Description | Use Case |
|------|-------------|----------|
| `Q16_16` | 16.16 fixed-point | General computation |
| `Q0_16` | 0.16 fixed-point | Values in [0, 1) |
| `Q12_20` | 12.20 fixed-point | High precision fractions |
| `Q20_12` | 20.12 fixed-point | Large integer + fraction |
### Fraction Construction Rules
1. **Use canonical constructors only**
```lean
Q16_16.ofNat n -- for integers
Q16_16.ofRatio a b -- for rational numbers
Q16_16.ofRawInt x -- for already-scaled integers
```
2. **Never use `ofFloat` in compute paths**
- Allowed only at external boundary (JSON parsing, sensor input)
- Must be immediately bracketed: `ofFloat x |> toFixedPoint |> compute`
3. **Bridge pattern for legacy code**
```lean
import Semantics.PhysicsScalarBridge
-- Use PhysicsScalarBridge.add, .gt, etc.
-- Constants: PhysicsScalarBridge.one, .two, .three, .half, .quarter
```
### Fraction Comparison Pattern
```lean
-- WRONG: direct comparison with Float
if x > 0.5 then ...
-- RIGHT: fixed-point comparison
if x > Q16_16.half then ...
```
## Glossary (before reading further)
These terms appear throughout all AGENTS.md files and the codebase: