|
|
2b3d255a40
|
feat(infra): add fast Jaccard matcher using /shm keyword index
Pre-built keyword index on /shm (272 MB, 111k keywords, 700k papers).
Runs in 39 seconds vs 10+ minutes timeout for SQL-based approach.
Usage:
# Build index (one-time, ~2 min)
ssh neon-64gb 'python3 /tmp/build_keyword_index.py'
# Run matcher (39 seconds)
ssh neon-64gb 'python3 /tmp/jaccard_fast.py'
/shm setup:
- tmpfs mount (32 GB), persistent in fstab
- /shm/arxiv_texts.tsv (559 MB) — raw paper texts
- /shm/keyword_index.json (272 MB) — inverted index
|
2026-06-22 01:11:17 -05:00 |
|
