Commit graph

765 commits

Author SHA1 Message Date
cd9817b9ec feat(agent): model theorem matrix with degenerate sector analysis
Each model has strong (>0.7), mixed (0.4-0.7), and degenerate (<0.4)
sectors. For each degenerate sector, a theorem attack compensates.

Algorithm: greedy selection until all priority sectors are strong (≥0.7).

Results:
- lean_proof: DeepSeek V4 Pro (1 model covers all priority sectors)
- code_bug: Kimi K2.6 (1 model)
- architecture: Claude Opus (1 model + 2 theorem attacks)
- math_novel: DeepSeek V4 Pro (1 model)
- infra_debug: GPT 5.5 + DeepSeek Flash (2 models)
- hard_wall: Claude Opus + DeepSeek Flash (2 models + 4 attacks)

Theorem attacks for degenerate sectors:
- speed → parallel_delegation (delegate fast tasks to cheaper models)
- cost_efficiency → selective_invocation (escalation path)
- formal_proof → delegate_to_opus (can't do Lean, delegate)
- breadth → context_injection (inject codebase context)
- reasoning_depth → chain_of_thought (break into steps)
2026-06-22 16:16:13 -05:00
364d249688 feat(agent): dual quaternion model selector with mass semantic numbers
Each model = dual quaternion (real = compressive, dual = anti-compressive).
Chiral ratio χ = |real|² / (|real|² + |dual|²) determines selection.

Results at threshold 0.99:
- lean_proof: 6 models (drops GLM, DS Flash)
- code_bug: 3 models (DeepSeek + Opus + Cohere)
- architecture: 7 models (drops DS Flash)
- math_novel: 6 models (drops GLM, DS Flash)
- infra_debug: 2 models (DeepSeek + Opus only)
- hard_wall: 4 models (drops diversity tier)

Z-transform compression for model selection history included.
Dual quaternion multiplication for panel composition.
2026-06-22 16:12:58 -05:00
59db58d954 docs(agent): break-glass test list for 4D × 2 fusion validation
15 tests across 4 dimensions + 4 cross-dimension tests:
- Math (T1.1-T1.3): Q16_16 positivity, q16_merge_nz, boolMerge_preservesGap
- Proof (T2.1-T2.3): gap_factors, disjoint_factors, crossgap_factors
- Code (T3.1-T3.3): verifySpectralGap8, toPattern_merge, mergeCheck8_all
- Diversity (T4.1-T4.3): architecture, rewrite fix, counterexample search
- Cross-dim (T5.1-T5.4): chain proofs across dimensions

Estimated cost: -10 for full suite.
Success: cleanMerge_preservesGap compiles with 0 sorry.
2026-06-22 16:09:09 -05:00
77b3de121a feat(agent): break-glass as fusion of fusions (16D model)
4 dimensions × 2 models each, structured like 16D braid:
- Math: DeepSeek V4 Pro + Gemini 3.1 Pro
- Proof: Claude Opus + Cohere Command A+
- Code: GPT-5.5 + Kimi K2.6
- Diversity: GLM-5.2 + DeepSeek V4 Flash

Each dimension fuses independently, then results fuse together.
Maps to OpenRouter Fusion with 8-model panel.

Cost: -10 for full fusion, /usr/bin/bash.10-2 for single dimension.
2026-06-22 15:51:07 -05:00
ec5ade77b5 refactor(agent): break-glass is just a model selection card
OpenCode handles provider routing. The skill is a reference card
for which model to switch to. Nothing more.

DeepSeek V4 Pro: /usr/bin/bash.27/.10 (55x cheaper than Opus)
Claude Opus: 5/5 (baseline)
2026-06-22 15:49:49 -05:00
9c5e5030a5 refactor(agent): simplify break-glass to model selection guide
Hermes already handles model routing. Remove MCP server, keep skill
as a cost-optimized model selection guide.

Key insight: DeepSeek V4 Pro is 55x cheaper than Claude Opus and
nearly as good at math/code. Default to DeepSeek, escalate to Opus
only when stuck.

Removed: break_glass_mcp.py (unnecessary — Hermes handles routing)
2026-06-22 15:48:40 -05:00
e250787e8b feat(agent): 8-model Fusion panels with Cohere + GLM + hard_wall type
Revised panel selections (1-8 models per Fusion spec):
- lean_proof: 8 models (Opus, DeepSeek, GPT, Cohere, Gemini, GLM, Kimi, DS-Flash)
- code_bug: 6 models
- architecture: 8 models
- math_novel: 6 models
- infra_debug: 5 models
- hard_wall: 8 models (all, for truly stuck problems)

Added:
- Cohere Command A+ (structured output, tool-use optimized)
- GLM-5.2 (753B, genuinely different architecture)
- DeepSeek V4 Flash (fast iterations)
- hard_wall problem type (stuck/wall/impossible/tried everything)
- Budget cap per type (.50-.50)
2026-06-22 15:45:21 -05:00
7f09d0c1a3 docs(agent): break-glass problem statement — transcendental work with division-impaired brains
Defines the core tension: formal verification of novel mathematical
structures using LLMs that can't reliably do arithmetic. Documents
why panels help (diversity > quality), what makes the work
transcendental, and how the break-glass protocol addresses it.
2026-06-22 15:44:26 -05:00
2cbba82406 feat(agent): break-glass MCP agent for novel problem attacks
Last-resort multi-model panel via OpenRouter Fusion. Only invoked
when all other approaches (local model, single frontier, existing
skills) have failed on the same problem.

Architecture:
- Problem classification (lean_proof, code_bug, architecture, math_novel, infra_debug)
- Per-type model panel selection (Opus + DeepSeek + Gemini for proofs)
- Context assembly (files, build logs, AGENTS.md constraints)
- OpenRouter Fusion call (parallel panel + judge synthesis)
- Structured output (diagnosis, solution, alternatives, risks, cost)

Safety:
- Explicit 'break glass:' trigger required
- Cost gate: /usr/bin/bash.30-.50 per invocation, .00/session cap
- Rate limit: 3 invocations per session
- Audit trail via ContextStream

Files:
- 4-Infrastructure/shim/break_glass_mcp.py (MCP server)
- 6-Documentation/docs/specs/break_glass_mcp_agent.md (design spec)
- .opencode/skills/break-glass/SKILL.md (skill definition)
- .mcp.json (server registration)
2026-06-22 15:33:16 -05:00
b6ab3e5f22 refactor(silversight): move to correct location + set no-write on Research Stack
SilverSight modules moved from Semantics/Semantics/SilverSight/ to
SilverSight/SilverSight/ (separate directory, same lake project).

Changes:
- SilverSight Lean modules: Schema, WireFormat, ProductSchema,
  ProductWireFormat, Receipt, Bind → 0-Core-Formalism/lean/SilverSight/
- Imports updated: Semantics.SilverSight.* → SilverSight.*
- Namespace updated: Semantics.SilverSight → SilverSight
- Cross-project imports preserved: Semantics.FixedPoint, open Semantics.FixedPoint
- lakefile.toml: SilverSight lean_lib now uses srcDir = ../SilverSight
- AGENTS.md: READ-ONLY notice added — Research Stack is archived,
  all new formal work goes to SilverSight
- SilverSight/AGENTS.md: clean-slate operating contract

Build: SilverSight 3307 jobs, 0 errors
Build: Compiler 3314 jobs, 0 errors
2026-06-22 15:05:14 -05:00
0d387902bb docs(lean): standalone proof guide for cleanMerge_preservesGap
Self-contained document for LLM agents to close the 6 remaining
bridge sorries. Covers:
- Theorem statement and definitions
- Proof architecture (3 verified kernels + 6 bridge sorries)
- What each sorry needs and the proof strategy
- The key insight (zero/non-zero pattern only)
- The blocker (simp can't reduce List operations on 8 elements)
- Possible solutions and file context
2026-06-22 14:52:15 -05:00
56679941ed fix(lean): close canonicalize_ne_zero and canonicalize_pattern sorries
canonicalize_ne_zero: (canonicalize q != 0) = (q != 0)
  Proof: unfold canonicalize, split on if-condition,
  rw h in isTrue branch (Q16_16.one != zero = true by rfl),
  simp in isFalse branch.

canonicalize_pattern: boolPattern ∘ canonicalizeSignature = boolPattern
  Proof: unfold definitions, exhaustive match on list length (0-9+ elements),
  simp with canonicalize_ne_zero rewrites each component.

6 bridge sorries remain (down from 8). Each requires list-level reasoning
over 8-element Q16_16 lists — the simp terms from List.zip/filter/all are
too large for automatic simplification.

Build: 3314 jobs, 0 errors (Compiler surface)
2026-06-22 14:34:37 -05:00
70bdfb5c2a fix(lean): attempt canonicalize+native_decide for gap_pat_bridge
native_decide +revert fails on Q16_16 (Subtype Int, infinite type).
gapQ16_canonical (verified) proves the computation works on {zero,one}.
The bridge from arbitrary Q16_16 to canonical values requires proving
predicate invariance under canonicalization — which is the same lemma.

The fundamental chicken-and-egg: to use native_decide, we need finite
values. To get finite values, we canonicalize. To prove canonicalization
preserves predicates, we need the bridge. The bridge is the lemma we
are trying to prove.

7 bridge sorries remain, all list-induction over 8 elements.

Build: 3314 jobs, 0 errors
2026-06-22 14:12:02 -05:00
8e29ea6592 feat(lean): add gapQ16_canonical verified by native_decide
gapQ16_canonical: for all 2^8 boolean patterns, verifySpectralGap on
canonical Q16_16 bins (zero or one) = boolGapPat on the pattern.
Verified by native_decide — the first time verifySpectralGap has been
computationally verified on concrete Q16_16 values.

Three independent computational kernels now verified:
- mergeCheck_all_256: 256×256 byte pairs (native_decide)
- gap_byte_pat: 256 boolean patterns (native_decide)
- gapQ16_canonical: 256 Q16_16 patterns (native_decide)

The bridge lemmas (canonicalize_pattern, gap_preserved, etc.) remain
as sorry — each is ~20 lines of list simp on 8-element lists.

Build: 3314 jobs, 0 errors
2026-06-22 14:00:30 -05:00
67c9e72d00 fix(lean): clean up cleanMerge_preservesGap proof structure
Build: 3314 jobs, 0 errors (Compiler surface)
5 bridge sorries remain — each is list induction over 8-element Q16_16
lists showing zero/non-zero pattern ↔ byte bit positions.
2026-06-22 13:57:06 -05:00
158f10aa04 feat(lean): complete cleanMerge_preservesGap proof architecture
Proof structure (5 bridge lemmas, 2 verified by native_decide):

MERGED (0 sorry):
- mergeCheck_all_256: 256×256 byte pairs verified by native_decide
- gap_byte_pat: boolGapPat = byteGap ∘ packPat, verified by native_decide
- gap_bridge: verifySpectralGap = byteGap ∘ pack (from gap_pat + gap_byte)

BRIDGE (4 sorry — same pattern: list induction over 8 elements):
- gap_pat_bridge: activeBins indices ↔ set bits of boolPattern
- disjoint_pat_bridge: resonanceDegeneracy = 0 ↔ disjoint bits
- crossgap_pat_bridge: crossInputGap ↔ byte cross-gap
- merge_bridge: merge byte ⊆ s byte OR e byte

ASSEMBLY (1 sorry — depends on bridges + kernel):
- cleanMerge_preservesGap: rw bridges, apply kernel, done

Each bridge sorry is independently closable via list induction on
8-element lists showing zero/non-zero pattern ↔ bit positions.

Build: 3314 jobs, 0 errors (Compiler surface)
2026-06-22 13:52:41 -05:00
5da90cd1d1 docs: add Clebsch diagonal cubic claim + update GraphRank proof status
- Added silversight_claim_clebsch_diagonal_cubic to manifest (Phase 2 geometry witness: 42 nodes→scar-absence, 27 lines→strand topology)
- Updated cleanMerge_preservesGap theorem with crossInputGap hypothesis and documented TODO(lean-port) boundary for Q16_16↔Bool bridge
- Compiler surface unchanged (3314 jobs, 0 errors), full workspace verified (8604 jobs)

Build: 8604 jobs, 0 errors (lake build)
2026-06-22 13:50:59 -05:00
ac700c6460 feat(lean): add byte-level gap kernel for cleanMerge_preservesGap
A boolean is a bit. Eight bins = eight bits = one byte.

mergeCheck_all_256: closed Bool term verified by native_decide over
all 256×256 byte pairs. Zero assumptions, zero free variables, zero sorry.

The kernel proves: for any two bytes where gap(s) ∧ gap(e) ∧ disjoint ∧
crossGap all hold, gap(s OR e) also holds. This is the complete
computational proof of the spectral gap merge property at the bit level.

The remaining sorry is the Q16_16→byte bridge: converting SpectralSignature
bins to/from Nat bytes and showing the predicates correspond. This is
standard list-level reasoning over 8 elements (unzip, map, filter, all).

Build: 3314 jobs, 0 errors (Compiler surface)
2026-06-22 13:48:39 -05:00
f6cf32e2a7 fix(lean): add crossInputGap + strengthen cleanMerge_preservesGap
Theorem restated with crossInputGap hypothesis (no active bin from s
adjacent to active bin from e). Without this, the theorem is false:
counterexample s=[1,0,1,0,...] e=[0,0,0,1,0,1,...] has merge with
adjacent active bins despite resonanceDegeneracy=0.

Added to Spectrum.lean:
- crossInputGap: cross-input adjacency check on adjacent bin pairs

Added to GraphRank.lean:
- boolGap8: boolean gap check on 8 explicit values
- Complete mathematical proof sketch in docstring (4 steps)

The sorry remains: the Q16_16↔Bool bridge for 8-element lists requires
list-level induction on activeBins/verifySpectralGap/resonanceDegeneracy/
crossInputGap. The boolean kernel (boolGap8) is defined and ready for
native_decide verification once the bridge is automated.

Build: 3314 jobs, 0 errors (Compiler surface)
2026-06-22 13:35:26 -05:00
e3a1d50ba9 fix(lean): analyze cleanMerge_preservesGap — theorem false as stated
Discovery: the theorem with only resonanceDegeneracy=0 is FALSE.
Counterexample: s=[1,0,1,0,...] e=[0,0,0,1,0,1,...] — no overlap,
both gap-valid, but merge has adjacent active bins at positions 2,3.

Missing hypothesis: cross-input gap (no active bin from s adjacent to
active bin from e). This holds in Sidon-basis context (powers-of-2
labels ensure min separation ≥ 2).

The sorry is now documented with:
- Exact counterexample
- Required cross-input gap hypothesis
- Proof path when cross-gap is available
- Boolean model (Fin 8 → Bool) verification strategy

Build: 3314 jobs, 0 errors (Compiler surface)
2026-06-22 13:23:45 -05:00
32d61c6bfa fix(lean): improve cleanMerge_preservesGap proof sketch
The sorry remains but now has a concrete proof strategy:
- Boolean abstraction: convert Q16_16 bins to Bool (zero vs non-zero)
- Prove gap-preservation on finite 2^8 boolean model (native_decide)
- Lift to Q16_16 via contrapositive: merge active → input active

Blocker: native_decide can't handle free List Bool variables;
needs Fin 8 → Bool representation or custom tactic for the lift.
See TODO(lean-port) marker in theorem docstring.
2026-06-22 13:06:27 -05:00
73db623848 feat(silversight): Phase 1 core + claim promotions + theorem triage
Phase 1 core (7 new Lean files, 3307 jobs, 0 errors):
- Schema.lean: Schema class with 6 type instances
- WireFormat.lean: WireFormat structure + Layout enum
- ProductSchema.lean: Schema (α × β) instance
- ProductWireFormat.lean: Row-major pair encoders with roundTrip proofs
- Receipt.lean: Receipt structure + GateType enum
- Bind.lean: bindReceipt composition + 8 theorems
- SilverSight.lean: root import module

Claim promotions (3 claims → VERIFIED):
- silversight_claim_q16_unified: FixedPoint.lean 12 theorems, 0 sorry
- silversight_claim_eigensolid_convergence: BraidEigensolid.lean:175
- silversight_claim_receipt_invertible: BraidEigensolid.lean:249

Theorem triage (3,057 theorems classified):
- TIER_1_CORE: 118 theorems (6 modules, direct port)
- TIER_2_FOUNDATION: 1,591 theorems (121 modules)
- TIER_3_EXTENSION: 1,084 theorems (291 modules)
- TIER_4_QUARANTINE: 137 theorems (15 modules)
- TIER_5_EXCLUDED: 127 theorems (18 modules)

Build: SilverSight 3307 jobs, 0 errors, 0 sorries
2026-06-22 13:00:29 -05:00
b072c99bf6 feat(infra): populate all remaining ENE tables from extraction data
Populates 11 previously-empty ENE tables:
- wiki_pages/revisions/categories/links from 677 wiki markdown files
- crossing_weights (7,040) from braid-strand crossing matrices
- nspace_kv (13,892) manifold coordinates from lean concepts
- routes (5,000) from concept relation graph
- fractal_manifolds/nodes/graph_entities from lean module hierarchy
- vectors (10,000) pseudo-embeddings from concept text
2026-06-22 12:39:57 -05:00
cfdc934757 feat(infra): populate prover_state and prover_instances from lean_concepts
3,057 theorems/lemmas → ene.prover_state (2,966 after dedup)
3,047 verified proofs → ene.prover_instances
Also fixes ON CONFLICT DO NOTHING and FK bypass for bulk load.
2026-06-22 12:32:20 -05:00
b453be6d09 chore: finalize cleanup (no changes) 2026-06-22 05:09:51 -05:00
772a2acc4d fix(scripts): pipe SQL via stdin for neon-64gb psql
Avoid shell-escaping issues by using podman exec -i and passing SQL
via stdin instead of command line interpolation.
2026-06-22 05:08:05 -05:00
aefd06dc84 chore(cleanup): remove shared-data archive lean_binned from repo
These directories were tracked despite gitignore patterns. Removing them
frees ~120 GB (shared-data), ~1.5 GB (archive), and ~7 GB (lean_binned).
They are either offloaded to Garage S3/Google Drive or superseded
proof artifacts.
2026-06-22 05:01:33 -05:00
513d2484a3 fix(infra): escape Gremlin properties and fix ENE shell escaping
- distribute_extraction.py: add gremlin_escape() for special chars,
  add --batch-size async mode with submitAsync()
- load_cornfield.py: pipe SQL via stdin to avoid shell metachar issues
- populate_ene_tables.py: same stdin pipe fix for neon SSH calls
2026-06-22 04:20:34 -05:00
5cfa8f0898 chore(cleanup): remove ingested corpus artifacts and scratch scripts
- Delete 4-Infrastructure/shim/lean_corpus/ (downloaded third-party corpus).
- Delete shared-data/data/blockchain_corpus/ receipts (already ingested/offloaded).
- Delete scratch/ bulk-download scripts (superseded by arxiv_crossref_stream.py).
- Clean __pycache__ directories outside .venv.
- Also fix populate_ene_tables.py to pipe SQL via stdin instead of shell
  escaping for neon-64gb psql execution.
2026-06-22 04:00:32 -05:00
e02786e310 feat(scripts): Gremlin escaping + ENE table population from extraction
- distribute_extraction.py: escape strings and batch Gremlin vertex upserts.
- populate_ene_tables.py: load extraction JSONs and map concepts into ENE
tables on neon-64gb (packages, receipts, sidon_labels, braid_strands,
eigensolid_snapshots, gossip_surface_nodes/edges, relations, sessions,
ingest_events).
2026-06-22 03:52:06 -05:00
5e7200c84f feat(infra): add optional --rrc alignment to lake_build_ingest.py
When --rrc is given, the shim runs pist-classify-trace on the trace fixture
and rrc-watchdog for RRC alignment, then writes an additional
rrc_alignment receipt into ene.receipts alongside the build receipt.
2026-06-22 03:46:58 -05:00
94b04ae321 docs: add rebase rules manifest and SilverSight claims manifest 2026-06-22 03:28:53 -05:00
ea34d8a38b chore(gitignore): ignore local opencode config file 2026-06-22 03:26:34 -05:00
44688e29da chore(submodule): bump research-compute-fabric to mark RDS refs dead 2026-06-22 03:23:53 -05:00
32d1962412 feat(opencode): enable local headroom MCP server 2026-06-22 03:23:33 -05:00
1a44f890c8 fix(infra): make lake_build_ingest.py duplicate-safe
Sets content_hash to NULL for build_log packages so the partial unique

index ene_pkg_hash_idx does not prevent multiple ingestions of the same

build output. Each build session now gets its own package row. Also makes

receipt/ingest_event inserts idempotent with ON CONFLICT DO NOTHING.
2026-06-22 03:14:23 -05:00
779f4fe413 feat(data): add DB sync scripts and infra tracking
Add concept extraction/orchestration scripts under scripts/ and infra
metadata in infra.json. Generated extraction outputs live under extraction/
and are regenerated from scripts; add extraction/ to .gitignore.
2026-06-22 03:14:09 -05:00
762c88057a docs(infra): document build-log automation in AGENTS.md
Adds a Build-log automation section noting the post-commit hook runs lake_build_ingest.py Compiler in the background, inserting into ENE on neon-64gb. Includes log path and reinstall note.
2026-06-22 02:37:00 -05:00
f3d11359c6 docs(infra): document canonical database locations and lake build ingest shim
Adds canonical database location notes to 4-Infrastructure/AGENTS.md:
- Postgres canonical instance is the arxiv-pg container on neon-64gb.
- Gremlin canonical instance is the Azure Cosmos DB endpoint.
- Explicitly warns against spinning up local Postgres on qfox for
  production data.

Also adds `lake_build_ingest.py` to the stack-solidification anchors.
2026-06-22 02:22:34 -05:00
27efd54f6d feat(infra): add lake build ingestion shim for ENE database
Adds 4-Infrastructure/shim/lake_build_ingest.py, a pure-I/O shim that:
- runs `lake build <target>` in the Semantics directory,
- parses job count, error count, warnings, and status,
- inserts a session, package, receipt, and ingest_event into the
  ENE database on neon-64gb (arxiv-pg container).

The script contains no admissibility logic and no Float arithmetic.
Tested with `lake build Semantics.NKHodgeFAMM` — 8316 jobs, 0 errors,
session d1ea53db2185df87dd41497da12101f5.
2026-06-22 02:20:17 -05:00
69988453bc chore(infra): stage working tree modifications
Build: 8604 jobs, 0 errors (lake build)
2026-06-22 01:23:17 -05:00
1f7ec15f12 feat(lean): add logarithmic viscosity coordinates to NKHodgeFAMM
Adds logViscosityRatio, log_viscosity_monotone, and ν_eff_monotone
to Semantics/NKHodgeFAMM.lean section 6b. The adaptive viscosity law
ν_eff = ν₀*(1+μ) is multiplicative in ν₀ and additive in scar density μ;
taking λ = log(ν_eff/ν₀) = log(1+μ) turns the multiplicative feedback into
an additive coordinate. This gives nlinarith a direct handle on viscosity
monotonicity and connects the module to Kritchevsky's "Everything Is
Logarithms" framing (SilverSight CITATION.cff).

Also marks a few pre-existing unused variables with underscores to silence
the linter.

Build: 8316 jobs, 0 errors (lake build Semantics.NKHodgeFAMM)
2026-06-22 01:21:58 -05:00
8079ca3a20 docs(infra): document bosonic tensor network regime discovery and fix dim formula
- Updates CHANGELOG.md with 2026-06-21 entry describing the two-regime
  phase transition observed on neon-64gb: bosonic MC (p<=4) vs
  distinguishable approximation (p>=5) in rrc_bosonic_tensor_network.py.
- Adds BMCTE, SBOE, LPMCE, effective support, and observable-estimation
  vs exact-simulation terms to 6-Documentation/docs/GLOSSARY.md.
- Fixes Hilbert-dimension summary in rrc_bosonic_tensor_network.py to use
  C(N+p-1, p) instead of C(N, p).

No core logic changed; documentation and summary-output correction only.
2026-06-22 01:11:17 -05:00
b2b94b93c8 chore: configure WOLFRAM_APP_ID for wolfram-mcp
- Updated .mcp.json with WOLFRAM_APP_ID env var (fallback to hardcoded HYJE3R3R63)
- Toolchain manifest documents all tools used in hachimoji citation pipeline

Build: 3314 jobs, 0 errors (lake build)
2026-06-22 01:11:17 -05:00
040b0ef61f docs(infra): document hybrid arxiv search infrastructure
Added embed_arxiv.py, hybrid_search.sql, and jaccard_hybrid.py to the
Stack-Solidification Anchors section. Replaced slow Jaccard matcher with
397x faster static-retrieval-mrl-en-v1 model + pgvector HNSW index.

Build: 3314 jobs, 0 errors (lake build Compiler)
2026-06-22 01:11:17 -05:00
88db8987b0 feat(infra): add hybrid arxiv search with pgvector HNSW + pg_trgm
- embed_arxiv.py: batch embedding of 700k arxiv abstracts using
  static-retrieval-mrl-en-v1 (1024-dim model)
- hybrid_search.sql: RRF merger of trigram + vector search, 22ms query time
- jaccard_hybrid.py: 48 cornfield concepts matched to arxiv papers

Results: 480 hybrid citations loaded (vector-only matches for abstract concepts)
Build: 3314 jobs, 0 errors (lake build Compiler)
2026-06-22 01:11:17 -05:00
d28e9de6ca fix(infra): optimize drive compression memory usage and download skip
Check file sizes before loading bytes into memory and stream large files
directly into the tarball to prevent OOM errors on Steam Deck. Skip downloading
files that already exist locally.

Build: 0 errors (py_compile)
2026-06-22 01:11:17 -05:00
0149a5c93d feat(infra): add MCP-only Drive GCCL compression shim
Adds 4-Infrastructure/shim/mcp_drive_gccl_compress.py:
- downloads a Google Drive folder via @piotr-agier/google-drive-mcp
- GCCL-delta-compresses contents into .tar.zst + JSON manifest
- uploads archive + manifest back to Drive
- optionally trashes the original folder
- handles MCP listFolder pagination

Updates 4-Infrastructure/AGENTS.md with the new shim anchor.

No rclone or egress-heavy operations; all Drive I/O goes through MCP.
2026-06-22 01:11:17 -05:00
2b3d255a40 feat(infra): add fast Jaccard matcher using /shm keyword index
Pre-built keyword index on /shm (272 MB, 111k keywords, 700k papers).
Runs in 39 seconds vs 10+ minutes timeout for SQL-based approach.

Usage:
  # Build index (one-time, ~2 min)
  ssh neon-64gb 'python3 /tmp/build_keyword_index.py'

  # Run matcher (39 seconds)
  ssh neon-64gb 'python3 /tmp/jaccard_fast.py'

/shm setup:
  - tmpfs mount (32 GB), persistent in fstab
  - /shm/arxiv_texts.tsv (559 MB) — raw paper texts
  - /shm/keyword_index.json (272 MB) — inverted index
2026-06-22 01:11:17 -05:00
5a8179fe38 feat(infra): add Jaccard matcher for arxiv paper discovery
Finds related arxiv papers for cornfield concepts via keyword overlap.
Uses /shm/arxiv_texts.tsv (559 MB) for fast local matching.
pg_trgm GIN indexes on neon-64gb for Postgres queries.

Results: 240 candidate citations for 48 concepts.
Loaded into concept_citations table (relation_type='candidate').
2026-06-22 01:11:17 -05:00