SilverSight/docs/SPECTRAL_CODEBOOK_GENERATOR.md

Spectral Codebook Generator

Date: 2026-07-01 Module: python/spectral_codebook.py Output: data/spectral_codebook.json Tests: tests/test_spectral_codebook.py (15 tests)

Implements the "Next Steps" of docs/SPECTRAL_CODEBOOK_ANALYSIS.md: a gap-aware spectral codebook over the 250 8×8 integer braid adjacency matrices in formal/SilverSight/PIST/Matrices250.lean, with an encode/decode round trip and an exact, float-free fingerprint per matrix.

Usage

python3 python/spectral_codebook.py                   # build + report + write JSON
python3 python/spectral_codebook.py --check-manifold  # also run the 278-row corpus
python3 python/spectral_codebook.py --gap-factor 3 --min-support 10 --no-write
python3 tests/test_spectral_codebook.py               # or python3 -m pytest tests/test_spectral_codebook.py

Pure stdlib (fractions.Fraction for exactness). NumPy is used only as an optional fast path for the spectral radius; without it, a Durand–Kerner root finder runs on the exact integer characteristic polynomial (verified to agree with NumPy to 4dp in the test suite).

API: build_codebook() returns a Codebook with encode(matrix) → (codeword, index), decode(codeword, index) → equation_id, codeword_of(equation_id), cluster_table(), and collision_report().

What changed vs. the original concept

1. Exact char-poly fingerprint replaces float λ as primary key. Each matrix gets the 8 exact integer coefficients of det(λI − M), computed by Faddeev–LeVerrier in exact rational arithmetic (integrality of every coefficient is asserted; Cayley–Hamilton p(M) = 0 is checked in integer arithmetic in the tests). This is strictly finer than 4dp λ — see the measured collision numbers below — and involves no floats at all.

2. Power-iteration λ is demoted to a traceability field. The analysis doc's 9 "mid band / edge-of-chaos" values in (0.5, 1) are power-iteration non-convergence artifacts: all 9 matrices have exact spectral radius ρ = 1.0 (peripheral spectra — several eigenvalues of modulus 1, including complex pairs). The C1 "edge-of-chaos" cluster of the analysis doc therefore does not exist; those matrices belong to the ρ = 1 class. spectral_radius in the new JSON is the exact max root modulus of the char poly; the legacy estimate is kept as lambda_power_iteration. Measured: 18 of 250 entries have |λ_pi − ρ| > 1e-3.

3. Dedupe before gap analysis. The corpus has only 238 distinct matrices over 250 ids (11 groups of byte-identical duplicates, 23 ids involved). Gap statistics and cluster support are computed over distinct matrices; duplicates are listed per entry (identical_matrix_ids) and in the collision report.

4. Sample-size guard on gap detection. Gaps > 3× median gap propose boundaries (30 candidates), but a segment must hold ≥ 10 distinct matrices to stand as a cluster; smaller segments merge into the neighbor across the smaller gap. Points isolated above a suppressed gap with < 10 occupancy are flagged sparse_tail: true (outliers, not clusters): the region above λ ≈ 7.66 holds only 10 ids (and only 4 above λ = 11 — {11.68, 16.56, 16.99, 16.99}), far too few to assert cluster structure.

Measured numbers (250 ids / 238 distinct matrices)

Fingerprint uniqueness / collisions

Fingerprint	Unique values	Ids uniquely identified	Collision groups (ids)
λ (exact, 4dp)	179	164 / 250	15 (86 ids)
char poly (exact ℤ⁸)	196	183 / 250	13 (67 ids)
raw matrix bytes	238	227 / 250	11 (23 ids)

• Exact duplicate matrices: 11 groups, 23 ids (12 ids are copies of another id's matrix). No encoder can separate these; encode returns the canonical (first-sorted) entry and the JSON lists the twins.
• Cospectral non-identical groups: 6 — same char poly, different matrices. The largest is the nilpotent class x⁸ (35 ids / 31 distinct matrices, all ρ = 0); the others are -1,0,… (5 ids), 0,0,-1,0,… (6 ids), -1,-1,0,-1,… (3 ids), and two pairs.
• The 4dp-λ count differs from the analysis doc's 190 because the 9 artifact values collapse onto ρ = 1.0 once computed exactly.
• Neither λ nor the char poly is injective — the round-trip decode key is the within-cluster rank index, and all collision classes are explicit in data/spectral_codebook.json (collisions.charpoly_collision_classes, collisions.duplicate_matrix_classes).

Cluster table (gap factor 3.0, min support 10)

Median gap 0.021671 → threshold 0.065013; 30 candidate boundaries, 8 kept: [0.5, 1.0744, 1.4599, 2.7251, 3.3598, 3.7311, 4.3264, 6.1266]; sparse-tail cutoff λ = 7.6568.

Codeword	Ids	Distinct	λ range	Note
C0	35	31	0.0	nilpotent class (char poly x⁸)
C1	20	20	[1.0, 1.000003]	peripheral-unit class (incl. the 9 mis-measured)
C2	13	13	[1.1487, 1.4253]
C3	79	77	[1.4945, 2.6919]	bulk
C4	29	28	[2.7583, 3.2910]
C5	15	14	[3.4287, 3.6970]
C6	22	20	[3.7651, 4.2889]
C7	19	18	[4.3639, 5.8789]
C8	18	17	[6.3743, 16.9915]	10 sparse-tail outliers above 7.6568

The three structurally defensible regions are ρ = 0 (nilpotent), ρ = 1 (peripheral unit), and the ρ > 1 bulk; the finer C2–C8 boundaries are gap-supported but data-driven, and the high tail is outlier territory.

Round trip

decode ∘ encode verified over all 250 equations: bijective on (codeword, index) pairs; encode(matrix) recovers the exact equation id for all 227 ids with unique matrices and a byte-identical matrix for the 23 duplicate-group ids.

278-row manifold (formal/SilverSight/RRC/Q16_16Manifold.lean)

278 rows resolve to 250 unique equation ids — the 28 extra rows are repeated ids reusing matrices already in the codebook (238 distinct matrices, no id lacks a matrix). Rerunning gap quantization on the 278-row corpus reproduces the 250-matrix boundaries exactly: no new gaps, no new clusters, all extra rows fall inside existing clusters.

Notes for the Lean side

• Identity layer vs. similarity layer. ClassifyN.hashMatrix is a positional base-5 hash, but corpus entries reach 9, so injectivity is not provable (positional carries can collide). Using base ≥ max_entry + 1 (e.g. 10) makes injectivity trivial to prove. Recommended split: the positional hash (base ≥ 10) is the identity key; the exact char-poly fingerprint is the similarity/clustering key (invariant under permutation-relabelling of strands, unlike the hash).
• Threshold mismatch. ClassifyN.lean defines signalThreshold = 98304 (1.5 in Q16.16) and oberthHighThreshold = 262144 (4.0), but docs/SPECTRAL_CODEBOOK_ANALYSIS.md claims the post-fix thresholds are 0.5/1.0. The code and the analysis doc disagree; this generator matches neither silently — it emits the gap-derived boundaries above and flags the discrepancy here for resolution.

Neon data layer (neon-64gb)

python/spectral_codebook_db.py syncs the codebook into the ENE schema on the neon-64gb Postgres ($NEON_PG, default postgres://postgres:postgres@100.92.88.64:5432/research_stack — same convention as scripts/auto/auto_pipeline.py).

python3 python/spectral_codebook_db.py                  # DRY RUN (default): summary + sample SQL, writes nothing
python3 python/spectral_codebook.py --sync-db           # same dry run from the generator CLI
python3 python/spectral_codebook_db.py --verify-schema  # + READ-ONLY column check against the live DB
python3 python/spectral_codebook_db.py --emit-sql       # write data/spectral_codebook_sync.sql (no DB contact)
python3 python/spectral_codebook_db.py --apply          # actually upsert (psycopg2, or falls back to --emit-sql)

The default is always a dry run; only an explicit --apply writes to the database. --apply has not been run — ene.rrc_predictions is untouched as of this writing. Live-DB access (even read-only --verify-schema) requires the user's go-ahead per the standing "ask before any DB work" rule; schema compatibility was instead verified offline against scripts/auto/ene_schema.sql (tested).

Landing table: ene.rrc_predictions (empty — natural target)

One flat, SQL-typed row per equation (deterministic uuid5 ids, so reruns upsert idempotently):

Column	Value
id	uuid5(namespace, equation_id)
equation_id	e.g. rrc_eq_01ab6e9c32652d06
proxy_pred	gap-aware cluster codeword C0–C8
exact_pred	shape from exact λ under the current ClassifyN thresholds (1.5/4.0 Q16.16, integer semantics mirrored)
matrix_hash	charpoly=<c1..c8>;pos10=<n> — exact char-poly (similarity key) + base-10 positional hash (identity key; injective, entries ≤ 9)
confidence	1.0 for unique char-poly fingerprints; 1/k in a k-way collision class

Dry-run counts: 250 rows — per cluster C0=35, C1=20, C2=13, C3=79, C4=29, C5=15, C6=22, C7=19, C8=18; per shape LogogramProjection=69, SignalShapedRouteCompiler=131, CognitiveLoadField=50; confidence 1.0 for 183 rows, <1.0 for 67.

Spark path

The row shape is deliberately flat so the Spark cluster on neon-64gb (master spark://100.92.88.64:7077, podman spark-worker, JDBC postgresql-42.7.5) can read it directly, the same JDBC pattern the auto-pipeline Spark analysis uses for ene.scars (11 rows) and ene.routes (5 rows):

df = (spark.read.format("jdbc")
      .option("url", "jdbc:postgresql://100.92.88.64:5432/research_stack")
      .option("dbtable", "ene.rrc_predictions")
      .option("user", "postgres").option("password", "postgres")
      .option("driver", "org.postgresql.Driver")
      .load())
df.groupBy("proxy_pred", "exact_pred").count().orderBy("proxy_pred").show()

Table map & staleness findings

Table	Rows	Status
ene.rrc_classifications	120	STALE — all spectral_radius values are 0.3–0.85, i.e. old flat-regime power-iteration artifacts, classified 2026-07-01 05:11 before the exact-eigenvalue correction (this codebook shows real ρ ∈ {0} ∪ {1} ∪ (1, 17]; the corpus has nothing in (0, 1)). Equation ids there are lean:formal/... paths, not rrc_eq_* hashes. Recommend re-classification via this codebook and reconciling the id conventions.
ene.rrc_predictions	0	Empty — landing table for this sync (matrix_hash column already fits the fingerprint).
ene.shape_predictions	100	Existing shape model output; exact_pred here can serve as cross-check.
ene.shape_ground_truth	0	Empty.
ene.eigensolid_snapshots	0	Empty (root_hash, crossing_matrix, sidon_slack, residual_series, …) — future home for full spectral profiles.
ene.braid_strands, ene.receipts, ene.sidon_labels	0	Empty.
ene.scars / ene.routes	11 / 5	Read by the Spark cluster via JDBC (auto-pipeline --spark).

(Row counts and staleness ranges as enumerated by the coordinating agent on 2026-07-01; re-check with --verify-schema + read-only SELECTs before acting on them.)

arxiv-pg citation layer

A separate Postgres runs in the arxiv-pg container on neon-64gb (arxiv DB with pgvector, concept_citations). It has no published port — access is podman-exec only. It is the citation/grounding layer for equation provenance; this sync does not (and cannot) connect to it.

Output schema (data/spectral_codebook.json)

• header: schema (spectral_codebook_v2), quantization (factor, min support, median gap, kept/candidate/suppressed boundaries, sparse-tail cutoff), clusters (table above), collisions (all counts and explicit collision classes), manifold_278 (when --check-manifold).
• entries[]: equation_id, codeword, index, spectral_radius (exact), lambda_power_iteration (legacy), charpoly (8 exact ints), density, frobenius, trace, optional sparse_tail, optional identical_matrix_ids.