SilverSight/docs/SPECTRAL_CODEBOOK_GENERATOR.md

# Spectral Codebook Generator

**Date:** 2026-07-01
**Module:** `python/spectral_codebook.py`
**Output:** `data/spectral_codebook.json`
**Tests:** `tests/test_spectral_codebook.py` (15 tests)

Implements the "Next Steps" of `docs/SPECTRAL_CODEBOOK_ANALYSIS.md`: a
gap-aware spectral codebook over the 250 8×8 integer braid adjacency
matrices in `formal/SilverSight/PIST/Matrices250.lean`, with an
encode/decode round trip and an exact, float-free fingerprint per matrix.

## Usage

```sh
python3 python/spectral_codebook.py                   # build + report + write JSON
python3 python/spectral_codebook.py --check-manifold  # also run the 278-row corpus
python3 python/spectral_codebook.py --gap-factor 3 --min-support 10 --no-write
python3 tests/test_spectral_codebook.py               # or python3 -m pytest tests/test_spectral_codebook.py
```

Pure stdlib (`fractions.Fraction` for exactness). NumPy is used only as an
optional fast path for the spectral radius; without it, a Durand–Kerner
root finder runs on the exact integer characteristic polynomial (verified
to agree with NumPy to 4dp in the test suite).

API: `build_codebook()` returns a `Codebook` with
`encode(matrix) → (codeword, index)`, `decode(codeword, index) → equation_id`,
`codeword_of(equation_id)`, `cluster_table()`, and `collision_report()`.

## What changed vs. the original concept

1. **Exact char-poly fingerprint replaces float λ as primary key.** Each
   matrix gets the 8 exact integer coefficients of det(λI − M), computed by
   Faddeev–LeVerrier in exact rational arithmetic (integrality of every
   coefficient is asserted; Cayley–Hamilton p(M) = 0 is checked in integer
   arithmetic in the tests). This is strictly finer than 4dp λ — see the
   measured collision numbers below — and involves no floats at all.

2. **Power-iteration λ is demoted to a traceability field.** The analysis
   doc's 9 "mid band / edge-of-chaos" values in (0.5, 1) are power-iteration
   **non-convergence artifacts**: all 9 matrices have exact spectral radius
   ρ = 1.0 (peripheral spectra — several eigenvalues of modulus 1,
   including complex pairs). The C1 "edge-of-chaos" cluster of the analysis
   doc therefore does not exist; those matrices belong to the ρ = 1 class.
   `spectral_radius` in the new JSON is the exact max root modulus of the
   char poly; the legacy estimate is kept as `lambda_power_iteration`.
   Measured: 18 of 250 entries have |λ_pi − ρ| > 1e-3.

3. **Dedupe before gap analysis.** The corpus has only **238 distinct
   matrices** over 250 ids (11 groups of byte-identical duplicates, 23 ids
   involved). Gap statistics and cluster support are computed over distinct
   matrices; duplicates are listed per entry (`identical_matrix_ids`) and
   in the collision report.

4. **Sample-size guard on gap detection.** Gaps > 3× median gap propose
   boundaries (30 candidates), but a segment must hold ≥ 10 distinct
   matrices to stand as a cluster; smaller segments merge into the neighbor
   across the smaller gap. Points isolated above a suppressed gap with
   < 10 occupancy are flagged `sparse_tail: true` (outliers, not clusters):
   the region above λ ≈ 7.66 holds only 10 ids (and only 4 above λ = 11 —
   {11.68, 16.56, 16.99, 16.99}), far too few to assert cluster structure.

## Measured numbers (250 ids / 238 distinct matrices)

### Fingerprint uniqueness / collisions

| Fingerprint | Unique values | Ids uniquely identified | Collision groups (ids) |
|---|---|---|---|
| λ (exact, 4dp) | 179 | 164 / 250 | 15 (86 ids) |
| char poly (exact ℤ⁸) | **196** | **183 / 250** | 13 (67 ids) |
| raw matrix bytes | 238 | 227 / 250 | 11 (23 ids) |

- Exact duplicate matrices: **11 groups, 23 ids** (12 ids are copies of
  another id's matrix). No encoder can separate these; `encode` returns the
  canonical (first-sorted) entry and the JSON lists the twins.
- **Cospectral non-identical groups: 6** — same char poly, different
  matrices. The largest is the nilpotent class x⁸ (35 ids / 31 distinct
  matrices, all ρ = 0); the others are `-1,0,…` (5 ids), `0,0,-1,0,…`
  (6 ids), `-1,-1,0,-1,…` (3 ids), and two pairs.
- The 4dp-λ count differs from the analysis doc's 190 because the 9
  artifact values collapse onto ρ = 1.0 once computed exactly.
- **Neither λ nor the char poly is injective** — the round-trip decode key
  is the within-cluster rank index, and all collision classes are explicit
  in `data/spectral_codebook.json` (`collisions.charpoly_collision_classes`,
  `collisions.duplicate_matrix_classes`).

### Cluster table (gap factor 3.0, min support 10)

Median gap 0.021671 → threshold 0.065013; 30 candidate boundaries, 8 kept:
`[0.5, 1.0744, 1.4599, 2.7251, 3.3598, 3.7311, 4.3264, 6.1266]`;
sparse-tail cutoff λ = 7.6568.

| Codeword | Ids | Distinct | λ range | Note |
|---|---|---|---|---|
| C0 | 35 | 31 | 0.0 | nilpotent class (char poly x⁸) |
| C1 | 20 | 20 | [1.0, 1.000003] | peripheral-unit class (incl. the 9 mis-measured) |
| C2 | 13 | 13 | [1.1487, 1.4253] | |
| C3 | 79 | 77 | [1.4945, 2.6919] | bulk |
| C4 | 29 | 28 | [2.7583, 3.2910] | |
| C5 | 15 | 14 | [3.4287, 3.6970] | |
| C6 | 22 | 20 | [3.7651, 4.2889] | |
| C7 | 19 | 18 | [4.3639, 5.8789] | |
| C8 | 18 | 17 | [6.3743, 16.9915] | 10 sparse-tail outliers above 7.6568 |

The three structurally defensible regions are ρ = 0 (nilpotent), ρ = 1
(peripheral unit), and the ρ > 1 bulk; the finer C2–C8 boundaries are
gap-supported but data-driven, and the high tail is outlier territory.

### Round trip

`decode ∘ encode` verified over all 250 equations: bijective on
(codeword, index) pairs; `encode(matrix)` recovers the exact equation id
for all 227 ids with unique matrices and a byte-identical matrix for the
23 duplicate-group ids.

### 278-row manifold (`formal/SilverSight/RRC/Q16_16Manifold.lean`)

278 rows resolve to **250 unique equation ids** — the 28 extra rows are
repeated ids reusing matrices already in the codebook (238 distinct
matrices, no id lacks a matrix). Rerunning gap quantization on the 278-row
corpus reproduces the 250-matrix boundaries exactly: **no new gaps, no new
clusters, all extra rows fall inside existing clusters.**

## Notes for the Lean side

- **Identity layer vs. similarity layer.** `ClassifyN.hashMatrix` is a
  positional base-5 hash, but corpus entries reach **9**, so injectivity is
  not provable (positional carries can collide). Using base ≥ max_entry + 1
  (e.g. 10) makes injectivity trivial to prove. Recommended split: the
  positional hash (base ≥ 10) is the *identity* key; the exact char-poly
  fingerprint is the *similarity/clustering* key (invariant under
  permutation-relabelling of strands, unlike the hash).
- **Threshold mismatch.** `ClassifyN.lean` defines `signalThreshold =
  98304` (1.5 in Q16.16) and `oberthHighThreshold = 262144` (4.0), but
  `docs/SPECTRAL_CODEBOOK_ANALYSIS.md` claims the post-fix thresholds are
  0.5/1.0. The code and the analysis doc disagree; this generator matches
  neither silently — it emits the gap-derived boundaries above and flags
  the discrepancy here for resolution.

## Output schema (`data/spectral_codebook.json`)

- header: `schema` (`spectral_codebook_v2`), `quantization` (factor,
  min support, median gap, kept/candidate/suppressed boundaries,
  sparse-tail cutoff), `clusters` (table above), `collisions` (all counts
  and explicit collision classes), `manifold_278` (when `--check-manifold`).
- `entries[]`: `equation_id`, `codeword`, `index`, `spectral_radius`
  (exact), `lambda_power_iteration` (legacy), `charpoly` (8 exact ints),
  `density`, `frobenius`, `trace`, optional `sparse_tail`,
  optional `identical_matrix_ids`.