SilverSight/docs/SPECTRAL_CODEBOOK_GENERATOR.md

# Spectral Codebook Generator

**Date:** 2026-07-01
**Module:** `python/spectral_codebook.py`
**Output:** `data/spectral_codebook.json`
**Tests:** `tests/test_spectral_codebook.py` (43 tests)

Implements the "Next Steps" of `docs/SPECTRAL_CODEBOOK_ANALYSIS.md`: a
gap-aware spectral codebook over the 250 8×8 integer braid adjacency
matrices in `formal/SilverSight/PIST/Matrices250.lean`, with an
encode/decode round trip and an exact, float-free fingerprint per matrix.

## Usage

```sh
python3 python/spectral_codebook.py                   # build + report + write JSON
python3 python/spectral_codebook.py --check-manifold  # also run the 278-row corpus
python3 python/spectral_codebook.py --gap-factor 3 --min-support 10 --no-write
python3 python/spectral_codebook.py --cartan-floor    # + Fisher/Δ=17/1792 floor report
python3 tests/test_spectral_codebook.py               # or python3 -m pytest tests/test_spectral_codebook.py
```

Pure stdlib (`fractions.Fraction` for exactness). NumPy is used only as an
optional fast path for the spectral radius; without it, a Durand–Kerner
root finder runs on the exact integer characteristic polynomial (verified
to agree with NumPy to 4dp in the test suite).

API: `build_codebook()` returns a `Codebook` with
`encode(matrix) → (codeword, index)`, `decode(codeword, index) → equation_id`,
`codeword_of(equation_id)`, `cluster_table()`, and `collision_report()`.

## What changed vs. the original concept

1. **Exact char-poly fingerprint replaces float λ as primary key.** Each
   matrix gets the 8 exact integer coefficients of det(λI − M), computed by
   Faddeev–LeVerrier in exact rational arithmetic (integrality of every
   coefficient is asserted; Cayley–Hamilton p(M) = 0 is checked in integer
   arithmetic in the tests). This is strictly finer than 4dp λ — see the
   measured collision numbers below — and involves no floats at all.

2. **Power-iteration λ is demoted to a traceability field.** The analysis
   doc's 9 "mid band / edge-of-chaos" values in (0.5, 1) are power-iteration
   **non-convergence artifacts**: all 9 matrices have exact spectral radius
   ρ = 1.0 (peripheral spectra — several eigenvalues of modulus 1,
   including complex pairs). The C1 "edge-of-chaos" cluster of the analysis
   doc therefore does not exist; those matrices belong to the ρ = 1 class.
   `spectral_radius` in the new JSON is the exact max root modulus of the
   char poly; the legacy estimate is kept as `lambda_power_iteration`.
   Measured: 18 of 250 entries have |λ_pi − ρ| > 1e-3.

3. **Dedupe before gap analysis.** The corpus has only **238 distinct
   matrices** over 250 ids (11 groups of byte-identical duplicates, 23 ids
   involved). Gap statistics and cluster support are computed over distinct
   matrices; duplicates are listed per entry (`identical_matrix_ids`) and
   in the collision report.

4. **Sample-size guard on gap detection.** Gaps > 3× median gap propose
   boundaries (30 candidates), but a segment must hold ≥ 10 distinct
   matrices to stand as a cluster; smaller segments merge into the neighbor
   across the smaller gap. Points isolated above a suppressed gap with
   < 10 occupancy are flagged `sparse_tail: true` (outliers, not clusters):
   the region above λ ≈ 7.66 holds only 10 ids (and only 4 above λ = 11 —
   {11.68, 16.56, 16.99, 16.99}), far too few to assert cluster structure.

## Measured numbers (250 ids / 238 distinct matrices)

### Fingerprint uniqueness / collisions

| Fingerprint | Unique values | Ids uniquely identified | Collision groups (ids) |
|---|---|---|---|
| λ (exact, 4dp) | 179 | 164 / 250 | 15 (86 ids) |
| char poly (exact ℤ⁸) | **196** | **183 / 250** | 13 (67 ids) |
| raw matrix bytes | 238 | 227 / 250 | 11 (23 ids) |

- Exact duplicate matrices: **11 groups, 23 ids** (12 ids are copies of
  another id's matrix). No encoder can separate these; `encode` returns the
  canonical (first-sorted) entry and the JSON lists the twins.
- **Cospectral non-identical groups: 6** — same char poly, different
  matrices. The largest is the nilpotent class x⁸ (35 ids / 31 distinct
  matrices, all ρ = 0); the others are `-1,0,…` (5 ids), `0,0,-1,0,…`
  (6 ids), `-1,-1,0,-1,…` (3 ids), and two pairs.
- The 4dp-λ count differs from the analysis doc's 190 because the 9
  artifact values collapse onto ρ = 1.0 once computed exactly.
- **Neither λ nor the char poly is injective** — the round-trip decode key
  is the within-cluster rank index, and all collision classes are explicit
  in `data/spectral_codebook.json` (`collisions.charpoly_collision_classes`,
  `collisions.duplicate_matrix_classes`).

### Cluster table (gap factor 3.0, min support 10)

Median gap 0.021671 → threshold 0.065013; 30 candidate boundaries, 8 kept:
`[0.5, 1.0744, 1.4599, 2.7251, 3.3598, 3.7311, 4.3264, 6.1266]`;
sparse-tail cutoff λ = 7.6568.

| Codeword | Ids | Distinct | λ range | Note |
|---|---|---|---|---|
| C0 | 35 | 31 | 0.0 | nilpotent class (char poly x⁸) |
| C1 | 20 | 20 | [1.0, 1.000003] | peripheral-unit class (incl. the 9 mis-measured) |
| C2 | 13 | 13 | [1.1487, 1.4253] | |
| C3 | 79 | 77 | [1.4945, 2.6919] | bulk |
| C4 | 29 | 28 | [2.7583, 3.2910] | |
| C5 | 15 | 14 | [3.4287, 3.6970] | |
| C6 | 22 | 20 | [3.7651, 4.2889] | |
| C7 | 19 | 18 | [4.3639, 5.8789] | |
| C8 | 18 | 17 | [6.3743, 16.9915] | 10 sparse-tail outliers above 7.6568 |

The three structurally defensible regions are ρ = 0 (nilpotent), ρ = 1
(peripheral unit), and the ρ > 1 bulk; the finer C2–C8 boundaries are
gap-supported but data-driven, and the high tail is outlier territory.

### Round trip

`decode ∘ encode` verified over all 250 equations: bijective on
(codeword, index) pairs; `encode(matrix)` recovers the exact equation id
for all 227 ids with unique matrices and a byte-identical matrix for the
23 duplicate-group ids.

### 278-row manifold (`formal/SilverSight/RRC/Q16_16Manifold.lean`)

278 rows resolve to **250 unique equation ids** — the 28 extra rows are
repeated ids reusing matrices already in the codebook (238 distinct
matrices, no id lacks a matrix). Rerunning gap quantization on the 278-row
corpus reproduces the 250-matrix boundaries exactly: **no new gaps, no new
clusters, all extra rows fall inside existing clusters.**

## Φ-corkscrew and Cartan layers (f/Cartan integration batch)

Four layers added on top of the fingerprints, each labelled **exact**
(integer arithmetic, Lean-provable) or **decorative** (placement/reporting
only, carries no information):

### 1. Integer spiral-index packing — EXACT

`pack_spiral_index` / `unpack_spiral_index` with corpus-wide parameters
recorded in the JSON header (`phi_corkscrew.packing`): offset = max|c| over
all fingerprint coefficients (164 234 on this corpus), base = 2·offset + 1
(328 469). Digits are offset-encoded (`c + offset ∈ [0, base)`), so
`spiral_index = Σ (cᵢ + offset)·baseⁱ` is the base-B positional numeral of
the fingerprint — injective on fingerprints by uniqueness of positional
representation, inverted exactly by `unpack_spiral_index`, and verified by
round trip over all 250 matrices in the test suite. It inherits charpoly's
non-injectivity on *matrices* (cospectral classes share one index), so
identity still requires the within-cluster index.

This **supersedes** the float phinary packing that `integration_sprint.py`
used historically (`phinary += c·φ⁻ⁱ` then truncation — not injective).
The integer path in `integration_sprint.phi_corkscrew_index` is fixed the
same way (offset-encoded digits; its unoffset signed packing collapsed
e.g. `(-1, 1)` with `(1, 0)` in base 2), but its base is per-call — for
corpus-comparable indices use the codebook's recorded (base, offset).

### 2. f(n) layout — DECORATIVE

`f(n) = (√n·cos(nψ), √n·sin(nψ))`, ψ = 2π/φ². Injective because 1/φ² is
irrational; low-discrepancy by the three-distance theorem;
information-neutral (everything decodes from `spiral_index` alone). Each
entry stores `layout.radius_sq` (= the spiral index, exact int) and
`layout.angle_frac` (fraction of a full turn). Angles are computed in
`decimal` arithmetic at full precision because spiral indices reach
~10⁴⁴ ≫ 2⁵², where float64 retains no fractional bits of n/φ². Verified
against `VERIFICATION_LOG.md` V007: f(20121) = (−137.80079576,
−33.64432624). Measured minimum pairwise angular separation across the
corpus: ≈ 4.64 × 10⁻⁶ of a turn (positive, as irrationality requires).

### 3. Cartan ∆-floor quantization (`--cartan-floor`) — EXACT ∆, reported floor

∆ = **17/1792** exactly (λ_min of the Cartan crossing blocks,
`CartanConnection.lean:70`, `docs/cartan_fingerprint.md`), adopted as the
resolution floor on the Fisher metric of Δ₇:
`d_F(p, q) = 2·arccos(Σ√(pᵢqᵢ))` with `pᵢ = |cᵢ|/Σ|cⱼ|` (all-zero
fingerprint → uniform, by convention). Measured on this corpus: **13
fingerprint pairs fall below ∆**, merging 196 fingerprints into **187
∆-resolution codewords**, vs **9 clusters** from the 3×-median-gap λ rule.
Both rules are emitted side by side; neither replaces the other.

Caveat, asserted in the tests: the simplex projection destroys sign and
scale, so distinct fingerprints can collide on Δ₇ (this corpus has a pair
at Fisher distance exactly 0). The Fisher/∆ layer is a **similarity**
metric, not an identity key.

### 4. Torus-winding saturation fix — EXACT

`pist_braid_bridge.TorusWinding` now carries `a_exact`/`b_exact` (plain ℤ,
no clamp) alongside the legacy Q16.16 raws, which silently saturate above
32767 (i.e. for spiral_index > 65535 — and codebook indices reach ~10⁴⁴).
`torus_to_spiral_index` prefers the exact fields, making the round trip
lossless for every index (regression-tested at n = 10⁹); legacy windings
without exact fields keep the old Q16 behaviour. The matching Lean type in
`BraidEigensolid.lean` needs the same widening (Int fields alongside
Q16_16) before this layer can be mirrored formally — noted in the
dataclass docstring, Lean file intentionally untouched.

## Notes for the Lean side

- **Identity layer vs. similarity layer.** `ClassifyN.hashMatrix` is a
  positional base-5 hash, but corpus entries reach **9**, so injectivity is
  not provable (positional carries can collide). Using base ≥ max_entry + 1
  (e.g. 10) makes injectivity trivial to prove. Recommended split: the
  positional hash (base ≥ 10) is the *identity* key; the exact char-poly
  fingerprint is the *similarity/clustering* key (invariant under
  permutation-relabelling of strands, unlike the hash).
- **Threshold mismatch.** `ClassifyN.lean` defines `signalThreshold =
  98304` (1.5 in Q16.16) and `oberthHighThreshold = 262144` (4.0), but
  `docs/SPECTRAL_CODEBOOK_ANALYSIS.md` claims the post-fix thresholds are
  0.5/1.0. The code and the analysis doc disagree; this generator matches
  neither silently — it emits the gap-derived boundaries above and flags
  the discrepancy here for resolution.

## Neon data layer (neon-64gb)

`python/spectral_codebook_db.py` syncs the codebook into the ENE schema on
the neon-64gb Postgres (`$NEON_PG`, default
`postgres://postgres:postgres@100.92.88.64:5432/research_stack` — same
convention as `scripts/auto/auto_pipeline.py`).

```sh
python3 python/spectral_codebook_db.py                  # DRY RUN (default): summary + sample SQL, writes nothing
python3 python/spectral_codebook.py --sync-db           # same dry run from the generator CLI
python3 python/spectral_codebook_db.py --verify-schema  # + READ-ONLY column check against the live DB
python3 python/spectral_codebook_db.py --emit-sql       # write data/spectral_codebook_sync.sql (no DB contact)
python3 python/spectral_codebook_db.py --apply          # actually upsert (psycopg2, or falls back to --emit-sql)
```

The default is always a dry run; only an explicit `--apply` writes to the
database. `--apply` has **not** been run — `ene.rrc_predictions` is
untouched as of this writing. Live-DB access (even read-only
`--verify-schema`) requires the user's go-ahead per the standing "ask
before any DB work" rule; schema compatibility was instead verified
offline against `scripts/auto/ene_schema.sql` (tested).

### Landing table: `ene.rrc_predictions` (empty — natural target)

One flat, SQL-typed row per equation (deterministic uuid5 ids, so reruns
upsert idempotently):

| Column | Value |
|---|---|
| `id` | uuid5(namespace, equation_id) |
| `equation_id` | e.g. `rrc_eq_01ab6e9c32652d06` |
| `proxy_pred` | gap-aware cluster codeword `C0`–`C8` |
| `exact_pred` | shape from exact λ under the **current** ClassifyN thresholds (1.5/4.0 Q16.16, integer semantics mirrored) |
| `matrix_hash` | `charpoly=<c1..c8>;pos10=<n>` — exact char-poly (similarity key) + base-10 positional hash (identity key; injective, entries ≤ 9) |
| `confidence` | 1.0 for unique char-poly fingerprints; 1/k in a k-way collision class |

Dry-run counts: 250 rows — per cluster C0=35, C1=20, C2=13, C3=79, C4=29,
C5=15, C6=22, C7=19, C8=18; per shape LogogramProjection=69,
SignalShapedRouteCompiler=131, CognitiveLoadField=50; confidence 1.0 for
183 rows, <1.0 for 67.

### Spark path

The row shape is deliberately flat so the Spark cluster on neon-64gb
(master `spark://100.92.88.64:7077`, podman spark-worker, JDBC
`postgresql-42.7.5`) can read it directly, the same JDBC pattern the
auto-pipeline Spark analysis uses for `ene.scars` (11 rows) and
`ene.routes` (5 rows):

```python
df = (spark.read.format("jdbc")
      .option("url", "jdbc:postgresql://100.92.88.64:5432/research_stack")
      .option("dbtable", "ene.rrc_predictions")
      .option("user", "postgres").option("password", "postgres")
      .option("driver", "org.postgresql.Driver")
      .load())
df.groupBy("proxy_pred", "exact_pred").count().orderBy("proxy_pred").show()
```

### Table map & staleness findings

| Table | Rows | Status |
|---|---|---|
| `ene.rrc_classifications` | 120 | **STALE** — all `spectral_radius` values are 0.3–0.85, i.e. old flat-regime power-iteration artifacts, classified 2026-07-01 05:11 *before* the exact-eigenvalue correction (this codebook shows real ρ ∈ {0} ∪ {1} ∪ (1, 17]; the corpus has nothing in (0, 1)). Equation ids there are `lean:formal/...` paths, not `rrc_eq_*` hashes. **Recommend re-classification via this codebook** and reconciling the id conventions. |
| `ene.rrc_predictions` | 0 | Empty — landing table for this sync (`matrix_hash` column already fits the fingerprint). |
| `ene.shape_predictions` | 100 | Existing shape model output; `exact_pred` here can serve as cross-check. |
| `ene.shape_ground_truth` | 0 | Empty. |
| `ene.eigensolid_snapshots` | 0 | Empty (root_hash, crossing_matrix, sidon_slack, residual_series, …) — future home for full spectral profiles. |
| `ene.braid_strands`, `ene.receipts`, `ene.sidon_labels` | 0 | Empty. |
| `ene.scars` / `ene.routes` | 11 / 5 | Read by the Spark cluster via JDBC (auto-pipeline `--spark`). |

(Row counts and staleness ranges as enumerated by the coordinating agent
on 2026-07-01; re-check with `--verify-schema` + read-only SELECTs before
acting on them.)

### arxiv-pg citation layer

A separate Postgres runs in the `arxiv-pg` container on neon-64gb (arxiv
DB with pgvector, `concept_citations`). It has **no published port** —
access is podman-exec only. It is the citation/grounding layer for
equation provenance; this sync does not (and cannot) connect to it.

### Gremlin graph layer (Cosmos DB mathblob/concepts)

`python/spectral_codebook_gremlin.py` loads the codebook into the Azure
Cosmos DB Gremlin graph used by the Research Stack module/concept loaders
(credentials in `Research Stack/.env.gremlin`, same conventions as
`load_module_graph.py`). Dry-run by default; `--apply` upserts.

Model: 1 `codebook` provenance vertex, 9 `codeword` cluster vertices, 13
`fingerprint` vertices (cospectral collision classes only), 250 `equation`
vertices (ids verbatim; spiral_index stored as string — it exceeds int64),
with `has_codeword` / `in_cluster` / `shares_fingerprint` /
`within_cartan_floor(fisher_distance)` edges. First applied 2026-07-02:
273 vertices + 339 edges, all ok; the 250 equation vertices already
existed in the graph and were refreshed in place with exact values.

Spark-side availability of the synced rows was verified the same day by a
client-mode PySpark 3.5.5 job (JDK 17 via nix, driver on the tailnet,
`spark.driver.host` set) reading `ene.rrc_predictions` through JDBC from
the spark://100.92.88.64:7077 cluster: 250 rows, C0–C8 and shape counts
matching the sync exactly. CouchDB (:5984) remains unresponsive — nothing
loaded there.

## Output schema (`data/spectral_codebook.json`)

- header: `schema` (`spectral_codebook_v2`), `phi_corkscrew` (packing
  base/offset + layout semantics and measured min angular separation),
  `quantization` (factor, min support, median gap,
  kept/candidate/suppressed boundaries, sparse-tail cutoff), `clusters`
  (table above), `collisions` (all counts and explicit collision classes),
  `manifold_278` (when `--check-manifold`), `cartan_floor` (when
  `--cartan-floor`: sub-Δ Fisher pairs, Δ-resolution codeword count,
  rule comparison).
- `entries[]`: `equation_id`, `codeword`, `index`, `spectral_radius`
  (exact), `lambda_power_iteration` (legacy), `charpoly` (8 exact ints),
  `density`, `frobenius`, `trace`, `spiral_index` (exact int packing of
  charpoly), `layout` (`radius_sq`, `angle_frac`), optional `sparse_tail`,
  optional `identical_matrix_ids`.