From 040b0ef61f9145b1b074e980918e591e78b237ca Mon Sep 17 00:00:00 2001 From: allaun Date: Sun, 21 Jun 2026 22:32:51 -0500 Subject: [PATCH] docs(infra): document hybrid arxiv search infrastructure Added embed_arxiv.py, hybrid_search.sql, and jaccard_hybrid.py to the Stack-Solidification Anchors section. Replaced slow Jaccard matcher with 397x faster static-retrieval-mrl-en-v1 model + pgvector HNSW index. Build: 3314 jobs, 0 errors (lake build Compiler) --- 4-Infrastructure/AGENTS.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/4-Infrastructure/AGENTS.md b/4-Infrastructure/AGENTS.md index b98946bb..6b70fd98 100644 --- a/4-Infrastructure/AGENTS.md +++ b/4-Infrastructure/AGENTS.md @@ -315,6 +315,9 @@ python3 4-Infrastructure/storage/storage_agent.py --loop --interval 900 - `4-Infrastructure/shim/mcp_drive_gccl_compress.py` — MCP-only Google Drive folder compressor: downloads a Drive folder via `@piotr-agier/google-drive-mcp`, GCCL-delta-compresses contents into a `.tar.zst` archive with JSON manifest, uploads archive + manifest back to Drive, optionally trashes the original. Optimized to skip existing local files and stream large files (>100MB) directly to prevent OOM. No rclone. - `4-Infrastructure/shim/rrc_arxiv_kernel_refine.py` — RRC arXiv kernel refinement: title+abstract keyword search against arxiv_papers for unmatched equations +- `4-Infrastructure/shim/embed_arxiv.py` — Batch embedding of 700k+ arxiv abstracts using static-retrieval-mrl-en-v1 (1024-dim) into pgvector +- `4-Infrastructure/shim/hybrid_search.sql` — Hybrid SQL function: trigram + pgvector HNSW RRF search (22ms query time) +- `4-Infrastructure/shim/jaccard_hybrid.py` — Hybrid matcher: embeds 48 cornfield concepts, queries hybrid_search, outputs JSON matches - `4-Infrastructure/infra/lean_lsp_mcp_wrapper.py` — Python wrapper for `lean-lsp-mcp` to fix the schema of `lean_diagnostic_messages` for compatibility with strict validators like Moonshot/Kimi API. - `4-Infrastructure/shim/stack_solidification_audit.py` - `4-Infrastructure/shim/stack_fail_closure_register.py`