docs: correct spectral codebook analysis per independent review

Corrections:
- Mid-band was power-iteration artifact (all 20 peripheral matrices have exact rho=1.0)
- 190 unique lambda → 180 unique rho (exact) + 192 unique char polynomials
- 7.57 bits was a count → 1.257 bits Shannon band entropy
- 7-cluster table fabricated → 3 active bands (null/peripheral/bulk)
- 12 duplicate matrices in 250-row corpus (238 distinct)

Preserved:
- Density-rho correlation 0.9806 (confirmed exact)
- Top spectral gap 4.88 at rho=[11.68, 16.56] (confirmed exact)
- Cartan gap 17/1792 as principled distinguishability floor
- F function orthogonal to Cartan structure
- Characteristic polynomial > spectral radius as codebook key

Known bugs documented:
- Power iteration non-convergence (22 matrices, >1e-3 error)
- Phinary packing not injective (float accumulation)
- Torus winding saturates at n>=65536 (Q16.16 clamp)

docs/SPECTRAL_CODEBOOK_ANALYSIS.md

@@ -1,157 +1,171 @@
 # Spectral Codebook Analysis
 
-**Date:** 2026-07-01
+**Date:** 2026-07-01 (revised after independent review)
 **Source:** 250 equations × 8×8 braid adjacency matrices from `rrc_pist_predictions_250_v1.json`
-**Method:** Power iteration (300 iterations) on raw integer matrices
+**Method:** Exact eigenvalues via numpy (characteristic polynomial), NOT power iteration
+
+## ⚠️ Corrections (2026-07-01 review)
+
+The original analysis used power iteration (300 iterations) on raw integer matrices. This introduced systematic errors:
+
+1. **The "mid-band" (9 matrices, λ ∈ [0.67, 0.95]) does not exist.** All 20 peripheral matrices have exact spectral radius ρ = 1.0. Power iteration fails on matrices with eigenvalues on the unit circle at 60° angles (peripheral spectrum non-convergence).
+2. **The 7-cluster table was incorrect.** Only 4 data points exist above λ=11, not ~25.
+3. **7.57 bits was a count, not Shannon entropy.** True band entropy is 1.257 bits.
+4. **12 matrices are exact duplicates.** The 250-row corpus contains 238 distinct matrices.
+5. **Characteristic polynomials (192 unique) outperform spectral radius (180 unique) as a codebook key.**
+6. **Power iteration in `MatrixN.lean` inherits the same bug** — 22 of 250 matrices disagree with exact eigenvalues by >1e-3.
 
 ## Executive Summary
 
-The 250-equation corpus contains **190 distinguishable spectral states** (at 4dp resolution), yielding **7.57 bits of information per matrix** from spectral radius alone. The eigensolid is a near-perfect topological fingerprint: density–λ correlation = 0.9807.
+The 250-equation corpus contains **180 distinguishable spectral radii** (at 10dp) and **192 unique characteristic polynomials**. The eigensolid is a near-perfect topological fingerprint: density–ρ correlation = 0.9806.
 
-The spectral space has natural phase boundaries defined by large gaps. The largest gap (4.88 units) separates "moderate complexity" equations (λ ≤ 11.7) from "high complexity" equations (λ ≥ 16.6). These boundaries are the natural codeword boundaries for a gap-aware spectral codebook.
+The real structure has 3 natural bands (not 5): null (ρ=0), peripheral-unit (ρ=1), and bulk (ρ>1). The Cartan gap Δ = 17/1792 ≈ 0.00949 is the principled distinguishability floor.
 
 ## Corpus Statistics
 
 | Metric | Value |
 |--------|-------|
 | Total matrices | 250 |
-| Unique spectral radii (4dp) | 190 |
-| Bits per matrix | 7.57 |
-| λ range | [0.0, 16.99] |
-| λ mean | 2.75 |
+| Distinct matrices | 238 |
+| Unique spectral radii (10dp) | 180 |
+| Unique characteristic polynomials | 192 |
+| ρ range | [0.0, 16.99] |
+| ρ mean | 2.75 |
+| Density–ρ correlation | 0.9806 |
 
-## Band Distribution
+## Band Distribution (exact eigenvalues)
 
-| Band | λ range | Count | Density range | Frobenius range |
-|------|---------|-------|---------------|-----------------|
-| zero | 0.0 | 35 | [0.0, 0.11] | [0.0, 2.6] |
-| low | (0, 0.5) | 0 | — | — |
-| mid | [0.5, 1.0) | 9 | [0.06, 0.14] | [2.0, 3.0] |
-| high | [1.0, 4.0) | 156 | [0.03, 0.45] | [1.4, 7.3] |
-| very_high | [4.0, ∞) | 50 | [0.28, 1.97] | [5.5, 22.0] |
+| Band | ρ range | Count | Interpretation |
+|------|---------|-------|----------------|
+| null | 0.0 | 35 | Nilpotent — no operator interaction |
+| peripheral | 1.0 | 20 | Unit spectral radius — Cartan boundary |
+| bulk | (1.0, 4.0] | 165 | Moderate operator density |
+| high-bulk | (4.0, ∞) | 50 | Dense operator graphs |
 
-Band entropy: **1.459 bits** (max 2.322 for 5 bands).
+Band entropy: **1.257 bits** (max 2.322 for 5 bands, but only 3 active).
 
 ### Interpretation
 
-- **zero band (35 matrices):** Empty or near-empty operator graphs. No edges → no spectral content. These are the "unclassified" equations (HoldForUnlawfulOrUnderspecifiedShape).
-- **mid band (9 matrices):** The "edge of chaos" equations. Sparse but non-trivial structure. These are the most compressible — complex enough to encode signal, simple enough to decode losslessly.
-- **high band (156 matrices):** The bulk of the corpus. Moderate operator graph density. Each matrix is distinguishable by its spectral radius.
-- **very_high band (50 matrices):** Dense operator graphs with high spectral radii. These equations have many interacting terms.
+- **null band (35 matrices):** Empty or near-empty operator graphs. ρ=0 means the adjacency matrix is nilpotent. These are the "unclassified" equations.
+- **peripheral band (20 matrices):** All have exact ρ=1.0. These are unitary-like — the operator graph has spectral content on the unit circle. The Cartan gap Δ = 17/1792 separates these from the null band.
+- **bulk band (165 matrices):** The main body. ρ > 1 means the operator graph has amplifying modes. Each matrix is distinguishable by its spectral radius.
+- **high-bulk band (50 matrices):** ρ > 4. Dense operator graphs with strong amplification.
 
-## Correlations
+## Correlations (exact eigenvalues)
 
 | Pair | Pearson r |
 |------|-----------|
-| λ ↔ density | 0.9807 |
-| λ ↔ Frobenius | 0.9817 |
-| density ↔ Frobenius | 0.9824 |
+| ρ ↔ density | 0.9806 |
 
-All three correlations are >0.98. The spectral radius, edge density, and Frobenius norm are near-interchangeable measures of equation complexity. The eigensolid C(x*) compresses all three into a single point on the information manifold.
+The spectral radius and edge density are near-interchangeable. ρ alone captures 96% of the variance (r² = 0.9616). The codebook can be 1-dimensional.
 
-**Implication:** You don't need all three. λ alone captures 96% of the variance (r² = 0.9618). The codebook can be 1-dimensional.
+**Key difference from power-iteration results:** The exact correlation is nearly identical (0.9806 vs 0.9807), confirming that the correlation is real even though the individual λ values were wrong for 22 matrices.
 
-## Spectral Gaps (Natural Phase Boundaries)
+## Spectral Gaps (exact eigenvalues)
 
-The top 10 gaps in the sorted spectral radius distribution:
+The top 5 gaps in the sorted spectral radius distribution:
 
-| Rank | Gap size | Lower λ | Upper λ | Interpretation |
+| Rank | Gap size | Lower ρ | Upper ρ | Interpretation |
 |------|----------|---------|---------|----------------|
 | 1 | **4.8807** | 11.68 | 16.56 | Major complexity wall |
 | 2 | **1.5862** | 10.10 | 11.68 | Secondary wall |
-| 3 | 0.8895 | 9.21 | 10.10 | Tertiary wall |
-| 4 | 0.7051 | 8.50 | 9.21 | Quaternary wall |
-| 5 | **0.6667** | 0.00 | 0.67 | Zero → non-zero transition |
-| 6 | 0.5706 | 7.76 | 8.33 | — |
-| 7 | 0.4954 | 5.88 | 6.37 | — |
-| 8 | 0.4293 | 16.56 | 16.99 | Top of range |
-| 9 | 0.4092 | 5.13 | 5.54 | — |
-| 10 | 0.3951 | 6.53 | 6.92 | — |
+| 3 | **1.0000** | 0.00 | 1.00 | Null → peripheral transition |
+| 4 | **0.8895** | 9.21 | 10.10 | Tertiary wall |
+| 5 | **0.7051** | 8.50 | 9.21 | Quaternary wall |
 
-### Natural Cluster Boundaries
+**Note:** The null→peripheral gap (exactly 1.0) is the third largest gap, not the fifth as in the power-iteration analysis.
 
-Using a 3× median gap threshold, the significant breaks define **7 natural clusters:**
+## The Cartan Gap
 
-| Cluster | λ range | Count | Character |
-|---------|---------|-------|-----------|
-| C0 | 0.0 | 35 | Empty / null |
-| C1 | [0.67, 1.0) | 9 | Edge-of-chaos |
-| C2 | [1.0, 5.1) | ~90 | Moderate |
-| C3 | [5.1, 8.5) | ~60 | Dense |
-| C4 | [8.5, 11.7) | ~25 | Very dense |
-| C5 | [11.7, 16.6) | ~25 | Ultra-dense |
-| C6 | [16.6, 17.0) | ~6 | Maximum |
+The Cartan gap Δ = 17/1792 ≈ 0.00949 (proven exact in `CartanConnection.lean:70`) is the minimum eigenvalue of the crossing blocks — the resolution floor of the braid operator on Δ₇.
 
-The boundaries at 0.67, 5.1, 8.5, and 11.7 are the sharpest. These are the natural "phase transitions" in the eigensolid.
+This provides a **principled, Lean-proven distinguishability floor**:
+
+- Two spectral radii separated by less than Δ are **provably indistinguishable** by the operator dynamics
+- The current statistical heuristic (3× median gap) should be replaced by Δ as the minimum inter-codeword distance
+- This transforms the codebook from "looks like a gap in this sample" to "provably resolvable by the operator"
 
 ## Encoding Capacity
 
 ### Spectral radius alone
-- 190 unique values → **7.57 bits/equation**
-- At 250 equations: ~55% codebook utilization (190/250)
+- 180 unique values (10dp) → **7.49 bits/equation**
+- But 85 matrices are in collision groups (15 groups with identical ρ)
 
-### Full spectral profile (λ + density + trace + Frobenius + degree spread)
-- Estimated: **10–12 bits/equation** (higher-dimensional codebook)
-- But correlation >0.98 means the extra dimensions add ~2–4 bits, not 5+
+### Characteristic polynomial
+- 192 unique polynomials → **7.59 bits/equation**
+- 74 matrices in 16 cospectral groups (same polynomial, different matrix)
+- Better discriminant than ρ alone: +12 unique keys
 
-### Gap-aware quantization
-- Place codeword boundaries at spectral gaps → maximizes inter-codeword distance
-- 7 clusters → **2.8 bits/equation** at cluster level
-- Within-cluster: additional **4–5 bits** from sub-cluster λ values
-- Total: **~7 bits/equation**, but with guaranteed minimum distance between codewords
+### Band-level encoding
+- 3 active bands → **1.257 bits** at band level
+- Within-band: additional bits from ρ or polynomial
 
-## Connection to SilverSight Pipeline
+### Conditional on density
+- ρ adds only **~2.2 bits** beyond raw edge count
+- Most of the "signal" is edge count, not spectral structure
+
+## λ Collisions (exact)
+
+| Metric | Count |
+|--------|-------|
+| Collision groups (same ρ at 10dp) | 15 |
+| Matrices in collisions | 85 |
+| Cospectral groups (same char poly) | 16 |
+| Matrices in cospectral groups | 74 |
+
+**Implication:** The characteristic polynomial is a better codebook key than spectral radius alone. 192 vs 180 unique keys, and the polynomial is exact (integer coefficients) while ρ is a floating-point approximation.
+
+## The F Function (byte frequency) and Cartan Orthogonality
+
+The F function (byte-frequency vector) is **orthogonal** to the Cartan spectral structure:
+
+| | F (byte frequency) | Cartan (spectral) |
+|---|---|---|
+| What it sees | Surface characters: 'a', '+', '=', '1' | Operator graph topology |
+| After normalization | All vars → 'V', all digits → 'N' | Sidon addresses preserved |
+| Sensitivity | 6 unique vectors | 180 unique spectral radii |
+| Correlation with ρ | r ≈ 0.00 | Deterministic |
+
+This orthogonality is **by design**: the normalization in F erases surface variation to expose structural invariants. The Cartan structure captures the algebraic content. They measure different things.
+
+**Where F matters:** 85 matrices collide on ρ (15 groups). F distinguishes 10 of those 16 groups — it's a tiebreaker for Cartan degeneracies.
+
+**The f(n) corkscrew function** is information-neutral — it's an injective spiral encoding from the spectral index. It contributes geometry (golden-angle low-discrepancy layout for nearest-neighbor decoding) but no new information. The contribution is at the torus transition: the winding pair (a,b) ∈ H₁(T²) is a genuine homological invariant.
+
+## Pipeline Architecture
 
 ```
 Equation text
-    → tokenize → 8×8 strand adjacency matrix
-    → powerIteration → spectral radius λ
+    → tokenize → 8×8 strand adjacency matrix (exact integers)
+    → characteristic polynomial (exact, integer coefficients)
+    → spectral radius ρ (from polynomial, not power iteration)
     → spectralRadiusToColor → shape classification
     → determineAlignment → alignment score
     → ncDerived → witness strength
     → JSON receipt
 ```
 
-The spectral radius λ is the **single number** that determines classification. Everything downstream — shape, alignment score, receipt validity — flows from λ.
+### Recommended encoding hierarchy
 
-### Before fix (thresholds 1.5/4.0)
-All 250 matrices had λ < 1.5 (in the old Q16_16 computation), so all classified as "LogogramProjection". The eigensolid was flat. No encoding possible.
+1. **Identity:** base-B integer hash of matrix (injectivity provable for B ≥ 10)
+2. **Similarity:** exact characteristic polynomial (192 unique, integer-only)
+3. **Layout:** f(n) spiral with integer packing (no phinary floats)
+4. **Distinguishability floor:** Cartan Δ = 17/1792 on Δ₇
 
-### After fix (thresholds 0.5/1.0)
-- 35 matrices → LogogramProjection (λ < 0.5)
-- 9 matrices → SignalShapedRouteCompiler (0.5 ≤ λ < 1.0)
-- 206 matrices → CognitiveLoadField (λ ≥ 1.0)
+## Known Bugs (from review)
 
-The classification space is now **3 distinct regions** with 190 distinguishable states within them.
-
-## Implications for Compression
-
-The core claim — "gaps are the encoding" — is now testable:
-
-1. **The spectral gaps encode structural transitions.** A gap of 4.88 between λ=11.68 and λ=16.56 means no equation exists in that range. That absence is information — it's a natural delimiter between "moderate" and "extreme" complexity classes.
-
-2. **The zero band is a null codeword.** 35 equations with λ=0 are structurally empty. They encode "no operator interaction" — 1 bit of information (present/absent), plus which specific equation it is from the 35.
-
-3. **The mid band is the compression sweet spot.** 9 matrices with λ ∈ [0.67, 0.95] are the most information-dense per edge: they have non-trivial structure but low density. Each one is fully distinguishable.
-
-4. **The density–λ correlation means the codebook is self-consistent.** If you encode by λ, decode by density, you get the same answer (r=0.98). The encoding is robust to measurement noise.
-
-## Next Steps
-
-1. **Build the spectral codebook:** Cluster the 250 matrices by full spectral profile, using gap-aware boundaries. Output: codeword → equation list mapping.
-
-2. **Implement gap-aware quantization:** Replace the current binary threshold (0.5/1.0) with the 7 natural cluster boundaries. This gives a 7-symbol alphabet with guaranteed minimum distance.
-
-3. **Cross-verify with Lean:** The `ClassifyN.hashTable8` is the proxy classifier (hash → shape). Extend it to use the spectral codebook instead of a fixed hash table.
-
-4. **Test on the full 278-row manifold:** The 250 matrices in Matrices250.lean are a subset. The full manifold has 278 fixture rows — run the same analysis on all of them.
-
-5. **Connect to the chaos game:** The `chaos_game` function in `silversight_engine.py` walks toward nearest attractor. The spectral codebook defines the attractor basins. Verify that the chaos game converges to the correct codeword for each equation.
+1. **Power iteration non-convergence:** 22 of 250 matrices have ρ values wrong by >1e-3. The Lean `powerIteration` in `MatrixN.lean` inherits this. Fix: use characteristic polynomial instead.
+2. **Phinary packing not injective:** `integration_sprint.py:269` uses float accumulation + truncation. Fix: replace with positional integer packing of charpoly coefficients.
+3. **Torus winding saturates at n ≥ 65536:** `pist_braid_bridge.py` stores `n//2` as Q16.16 which clamps at 32768. Fix: use integer winding counts.
+4. **120 stale DB rows:** `ene.rrc_classifications` on neon-64gb was written under the old buggy power-iteration regime. Fix: re-classify with exact eigenvalues.
 
 ## Files
 
-- `formal/SilverSight/PIST/ClassifyN.lean` — spectral thresholds (updated)
+- `formal/SilverSight/PIST/ClassifyN.lean` — spectral thresholds
 - `formal/SilverSight/PIST/Matrices250.lean` — 250 8×8 matrices
-- `formal/SilverSight/PIST/SpectralN.lean` — spectral profile computation
+- `formal/SilverSight/PIST/SpectralN.lean` — spectral profile computation (uses power iteration — needs fix)
+- `formal/SilverSight/PIST/CartanConnection.lean` — Cartan gap proof (Δ = 17/1792)
+- `formal/CoreFormalism/CharacterTransform.lean` — Sidon → Z₂⁴ → Cartan chain
 - `formal/SilverSight/RRC/Q16_16Manifold.lean` — 278-row fixture corpus
 - `python/silversight_engine.py` — Python implementation of the pipeline
+- `data/spectral_codebook_raw.json` — 250-entry raw spectral data (from power iteration — needs regeneration)