SilverSight/docs/SPECTRAL_CODEBOOK_GENERATOR.md

Spectral Codebook Generator

Date: 2026-07-01 Module: python/spectral_codebook.py Output: data/spectral_codebook.json Tests: tests/test_spectral_codebook.py (15 tests)

Implements the "Next Steps" of docs/SPECTRAL_CODEBOOK_ANALYSIS.md: a gap-aware spectral codebook over the 250 8×8 integer braid adjacency matrices in formal/SilverSight/PIST/Matrices250.lean, with an encode/decode round trip and an exact, float-free fingerprint per matrix.

Usage

python3 python/spectral_codebook.py                   # build + report + write JSON
python3 python/spectral_codebook.py --check-manifold  # also run the 278-row corpus
python3 python/spectral_codebook.py --gap-factor 3 --min-support 10 --no-write
python3 tests/test_spectral_codebook.py               # or python3 -m pytest tests/test_spectral_codebook.py

Pure stdlib (fractions.Fraction for exactness). NumPy is used only as an optional fast path for the spectral radius; without it, a Durand–Kerner root finder runs on the exact integer characteristic polynomial (verified to agree with NumPy to 4dp in the test suite).

API: build_codebook() returns a Codebook with encode(matrix) → (codeword, index), decode(codeword, index) → equation_id, codeword_of(equation_id), cluster_table(), and collision_report().

What changed vs. the original concept

1. Exact char-poly fingerprint replaces float λ as primary key. Each matrix gets the 8 exact integer coefficients of det(λI − M), computed by Faddeev–LeVerrier in exact rational arithmetic (integrality of every coefficient is asserted; Cayley–Hamilton p(M) = 0 is checked in integer arithmetic in the tests). This is strictly finer than 4dp λ — see the measured collision numbers below — and involves no floats at all.

2. Power-iteration λ is demoted to a traceability field. The analysis doc's 9 "mid band / edge-of-chaos" values in (0.5, 1) are power-iteration non-convergence artifacts: all 9 matrices have exact spectral radius ρ = 1.0 (peripheral spectra — several eigenvalues of modulus 1, including complex pairs). The C1 "edge-of-chaos" cluster of the analysis doc therefore does not exist; those matrices belong to the ρ = 1 class. spectral_radius in the new JSON is the exact max root modulus of the char poly; the legacy estimate is kept as lambda_power_iteration. Measured: 18 of 250 entries have |λ_pi − ρ| > 1e-3.

3. Dedupe before gap analysis. The corpus has only 238 distinct matrices over 250 ids (11 groups of byte-identical duplicates, 23 ids involved). Gap statistics and cluster support are computed over distinct matrices; duplicates are listed per entry (identical_matrix_ids) and in the collision report.

4. Sample-size guard on gap detection. Gaps > 3× median gap propose boundaries (30 candidates), but a segment must hold ≥ 10 distinct matrices to stand as a cluster; smaller segments merge into the neighbor across the smaller gap. Points isolated above a suppressed gap with < 10 occupancy are flagged sparse_tail: true (outliers, not clusters): the region above λ ≈ 7.66 holds only 10 ids (and only 4 above λ = 11 — {11.68, 16.56, 16.99, 16.99}), far too few to assert cluster structure.

Measured numbers (250 ids / 238 distinct matrices)

Fingerprint uniqueness / collisions

Fingerprint	Unique values	Ids uniquely identified	Collision groups (ids)
λ (exact, 4dp)	179	164 / 250	15 (86 ids)
char poly (exact ℤ⁸)	196	183 / 250	13 (67 ids)
raw matrix bytes	238	227 / 250	11 (23 ids)

• Exact duplicate matrices: 11 groups, 23 ids (12 ids are copies of another id's matrix). No encoder can separate these; encode returns the canonical (first-sorted) entry and the JSON lists the twins.
• Cospectral non-identical groups: 6 — same char poly, different matrices. The largest is the nilpotent class x⁸ (35 ids / 31 distinct matrices, all ρ = 0); the others are -1,0,… (5 ids), 0,0,-1,0,… (6 ids), -1,-1,0,-1,… (3 ids), and two pairs.
• The 4dp-λ count differs from the analysis doc's 190 because the 9 artifact values collapse onto ρ = 1.0 once computed exactly.
• Neither λ nor the char poly is injective — the round-trip decode key is the within-cluster rank index, and all collision classes are explicit in data/spectral_codebook.json (collisions.charpoly_collision_classes, collisions.duplicate_matrix_classes).

Cluster table (gap factor 3.0, min support 10)

Median gap 0.021671 → threshold 0.065013; 30 candidate boundaries, 8 kept: [0.5, 1.0744, 1.4599, 2.7251, 3.3598, 3.7311, 4.3264, 6.1266]; sparse-tail cutoff λ = 7.6568.

Codeword	Ids	Distinct	λ range	Note
C0	35	31	0.0	nilpotent class (char poly x⁸)
C1	20	20	[1.0, 1.000003]	peripheral-unit class (incl. the 9 mis-measured)
C2	13	13	[1.1487, 1.4253]
C3	79	77	[1.4945, 2.6919]	bulk
C4	29	28	[2.7583, 3.2910]
C5	15	14	[3.4287, 3.6970]
C6	22	20	[3.7651, 4.2889]
C7	19	18	[4.3639, 5.8789]
C8	18	17	[6.3743, 16.9915]	10 sparse-tail outliers above 7.6568

The three structurally defensible regions are ρ = 0 (nilpotent), ρ = 1 (peripheral unit), and the ρ > 1 bulk; the finer C2–C8 boundaries are gap-supported but data-driven, and the high tail is outlier territory.

Round trip

decode ∘ encode verified over all 250 equations: bijective on (codeword, index) pairs; encode(matrix) recovers the exact equation id for all 227 ids with unique matrices and a byte-identical matrix for the 23 duplicate-group ids.

278-row manifold (formal/SilverSight/RRC/Q16_16Manifold.lean)

278 rows resolve to 250 unique equation ids — the 28 extra rows are repeated ids reusing matrices already in the codebook (238 distinct matrices, no id lacks a matrix). Rerunning gap quantization on the 278-row corpus reproduces the 250-matrix boundaries exactly: no new gaps, no new clusters, all extra rows fall inside existing clusters.

Notes for the Lean side

• Identity layer vs. similarity layer. ClassifyN.hashMatrix is a positional base-5 hash, but corpus entries reach 9, so injectivity is not provable (positional carries can collide). Using base ≥ max_entry + 1 (e.g. 10) makes injectivity trivial to prove. Recommended split: the positional hash (base ≥ 10) is the identity key; the exact char-poly fingerprint is the similarity/clustering key (invariant under permutation-relabelling of strands, unlike the hash).
• Threshold mismatch. ClassifyN.lean defines signalThreshold = 98304 (1.5 in Q16.16) and oberthHighThreshold = 262144 (4.0), but docs/SPECTRAL_CODEBOOK_ANALYSIS.md claims the post-fix thresholds are 0.5/1.0. The code and the analysis doc disagree; this generator matches neither silently — it emits the gap-derived boundaries above and flags the discrepancy here for resolution.

Output schema (data/spectral_codebook.json)

• header: schema (spectral_codebook_v2), quantization (factor, min support, median gap, kept/candidate/suppressed boundaries, sparse-tail cutoff), clusters (table above), collisions (all counts and explicit collision classes), manifold_278 (when --check-manifold).
• entries[]: equation_id, codeword, index, spectral_radius (exact), lambda_power_iteration (legacy), charpoly (8 exact ints), density, frobenius, trace, optional sparse_tail, optional identical_matrix_ids.