SilverSight/docs/SPECTRAL_CODEBOOK_ANALYSIS.md

# Spectral Codebook Analysis

**Date:** 2026-07-01 (revised after independent review)
**Source:** 250 equations × 8×8 braid adjacency matrices from `rrc_pist_predictions_250_v1.json`
**Method:** Exact eigenvalues via numpy (characteristic polynomial), NOT power iteration

## ⚠️ Corrections (2026-07-01 review)

The original analysis used power iteration (300 iterations) on raw integer matrices. This introduced systematic errors:

1. **The "mid-band" (9 matrices, λ ∈ [0.67, 0.95]) does not exist.** All 20 peripheral matrices have exact spectral radius ρ = 1.0. Power iteration fails on matrices with eigenvalues on the unit circle at 60° angles (peripheral spectrum non-convergence).
2. **The 7-cluster table was incorrect.** Only 4 data points exist above λ=11, not ~25.
3. **7.57 bits was a count, not Shannon entropy.** True band entropy is 1.257 bits.
4. **12 matrices are exact duplicates.** The 250-row corpus contains 238 distinct matrices.
5. **Characteristic polynomials (192 unique) outperform spectral radius (180 unique) as a codebook key.**
6. **Power iteration in `MatrixN.lean` inherits the same bug** — 22 of 250 matrices disagree with exact eigenvalues by >1e-3.

## Executive Summary

The 250-equation corpus contains **180 distinguishable spectral radii** (at 10dp) and **192 unique characteristic polynomials**. The eigensolid is a near-perfect topological fingerprint: density–ρ correlation = 0.9806.

The real structure has 3 natural bands (not 5): null (ρ=0), peripheral-unit (ρ=1), and bulk (ρ>1). The Cartan gap Δ = 17/1792 ≈ 0.00949 is the principled distinguishability floor.

## Corpus Statistics

| Metric | Value |
|--------|-------|
| Total matrices | 250 |
| Distinct matrices | 238 |
| Unique spectral radii (10dp) | 180 |
| Unique characteristic polynomials | 192 |
| ρ range | [0.0, 16.99] |
| ρ mean | 2.75 |
| Density–ρ correlation | 0.9806 |

## Band Distribution (exact eigenvalues)

| Band | ρ range | Count | Interpretation |
|------|---------|-------|----------------|
| null | 0.0 | 35 | Nilpotent — no operator interaction |
| peripheral | 1.0 | 20 | Unit spectral radius — Cartan boundary |
| bulk | (1.0, 4.0] | 165 | Moderate operator density |
| high-bulk | (4.0, ∞) | 50 | Dense operator graphs |

Band entropy: **1.257 bits** (max 2.322 for 5 bands, but only 3 active).

### Interpretation

- **null band (35 matrices):** Empty or near-empty operator graphs. ρ=0 means the adjacency matrix is nilpotent. These are the "unclassified" equations.
- **peripheral band (20 matrices):** All have exact ρ=1.0. These are unitary-like — the operator graph has spectral content on the unit circle. The Cartan gap Δ = 17/1792 separates these from the null band.
- **bulk band (165 matrices):** The main body. ρ > 1 means the operator graph has amplifying modes. Each matrix is distinguishable by its spectral radius.
- **high-bulk band (50 matrices):** ρ > 4. Dense operator graphs with strong amplification.

## Correlations (exact eigenvalues)

| Pair | Pearson r |
|------|-----------|
| ρ ↔ density | 0.9806 |

The spectral radius and edge density are near-interchangeable. ρ alone captures 96% of the variance (r² = 0.9616). The codebook can be 1-dimensional.

**Key difference from power-iteration results:** The exact correlation is nearly identical (0.9806 vs 0.9807), confirming that the correlation is real even though the individual λ values were wrong for 22 matrices.

## Spectral Gaps (exact eigenvalues)

The top 5 gaps in the sorted spectral radius distribution:

| Rank | Gap size | Lower ρ | Upper ρ | Interpretation |
|------|----------|---------|---------|----------------|
| 1 | **4.8807** | 11.68 | 16.56 | Major complexity wall |
| 2 | **1.5862** | 10.10 | 11.68 | Secondary wall |
| 3 | **1.0000** | 0.00 | 1.00 | Null → peripheral transition |
| 4 | **0.8895** | 9.21 | 10.10 | Tertiary wall |
| 5 | **0.7051** | 8.50 | 9.21 | Quaternary wall |

**Note:** The null→peripheral gap (exactly 1.0) is the third largest gap, not the fifth as in the power-iteration analysis.

## The Cartan Gap

The Cartan gap Δ = 17/1792 ≈ 0.00949 (proven exact in `CartanConnection.lean:70`) is the minimum eigenvalue of the crossing blocks — the resolution floor of the braid operator on Δ₇.

This provides a **principled, Lean-proven distinguishability floor**:

- Two spectral radii separated by less than Δ are **provably indistinguishable** by the operator dynamics
- The current statistical heuristic (3× median gap) should be replaced by Δ as the minimum inter-codeword distance
- This transforms the codebook from "looks like a gap in this sample" to "provably resolvable by the operator"

## Encoding Capacity

### Spectral radius alone
- 180 unique values (10dp) → **7.49 bits/equation**
- But 85 matrices are in collision groups (15 groups with identical ρ)

### Characteristic polynomial
- 192 unique polynomials → **7.59 bits/equation**
- 74 matrices in 16 cospectral groups (same polynomial, different matrix)
- Better discriminant than ρ alone: +12 unique keys

### Band-level encoding
- 3 active bands → **1.257 bits** at band level
- Within-band: additional bits from ρ or polynomial

### Conditional on density
- ρ adds only **~2.2 bits** beyond raw edge count
- Most of the "signal" is edge count, not spectral structure

## λ Collisions (exact)

| Metric | Count |
|--------|-------|
| Collision groups (same ρ at 10dp) | 15 |
| Matrices in collisions | 85 |
| Cospectral groups (same char poly) | 16 |
| Matrices in cospectral groups | 74 |

**Implication:** The characteristic polynomial is a better codebook key than spectral radius alone. 192 vs 180 unique keys, and the polynomial is exact (integer coefficients) while ρ is a floating-point approximation.

## The F Function (byte frequency) and Cartan Orthogonality

The F function (byte-frequency vector) is **orthogonal** to the Cartan spectral structure:

| | F (byte frequency) | Cartan (spectral) |
|---|---|---|
| What it sees | Surface characters: 'a', '+', '=', '1' | Operator graph topology |
| After normalization | All vars → 'V', all digits → 'N' | Sidon addresses preserved |
| Sensitivity | 6 unique vectors | 180 unique spectral radii |
| Correlation with ρ | r ≈ 0.00 | Deterministic |

This orthogonality is **by design**: the normalization in F erases surface variation to expose structural invariants. The Cartan structure captures the algebraic content. They measure different things.

**Where F matters:** 85 matrices collide on ρ (15 groups). F distinguishes 10 of those 16 groups — it's a tiebreaker for Cartan degeneracies.

**The f(n) corkscrew function** is information-neutral — it's an injective spiral encoding from the spectral index. It contributes geometry (golden-angle low-discrepancy layout for nearest-neighbor decoding) but no new information. The contribution is at the torus transition: the winding pair (a,b) ∈ H₁(T²) is a genuine homological invariant.

## Pipeline Architecture

```
Equation text
    → tokenize → 8×8 strand adjacency matrix (exact integers)
    → characteristic polynomial (exact, integer coefficients)
    → spectral radius ρ (from polynomial, not power iteration)
    → spectralRadiusToColor → shape classification
    → determineAlignment → alignment score
    → ncDerived → witness strength
    → JSON receipt
```

### Recommended encoding hierarchy

1. **Identity:** base-B integer hash of matrix (injectivity provable for B ≥ 10)
2. **Similarity:** exact characteristic polynomial (192 unique, integer-only)
3. **Layout:** f(n) spiral with integer packing (no phinary floats)
4. **Distinguishability floor:** Cartan Δ = 17/1792 on Δ₇

## Known Bugs (from review)

1. **Power iteration non-convergence:** 22 of 250 matrices have ρ values wrong by >1e-3. The Lean `powerIteration` in `MatrixN.lean` inherits this. Fix: use characteristic polynomial instead.
2. **Phinary packing not injective:** `integration_sprint.py:269` uses float accumulation + truncation. Fix: replace with positional integer packing of charpoly coefficients.
3. **Torus winding saturates at n ≥ 65536:** `pist_braid_bridge.py` stores `n//2` as Q16.16 which clamps at 32768. Fix: use integer winding counts.
4. **120 stale DB rows:** `ene.rrc_classifications` on neon-64gb was written under the old buggy power-iteration regime. Fix: re-classify with exact eigenvalues.

## Files

- `formal/SilverSight/PIST/ClassifyN.lean` — spectral thresholds
- `formal/SilverSight/PIST/Matrices250.lean` — 250 8×8 matrices
- `formal/SilverSight/PIST/SpectralN.lean` — spectral profile computation (uses power iteration — needs fix)
- `formal/SilverSight/PIST/CartanConnection.lean` — Cartan gap proof (Δ = 17/1792)
- `formal/CoreFormalism/CharacterTransform.lean` — Sidon → Z₂⁴ → Cartan chain
- `formal/SilverSight/RRC/Q16_16Manifold.lean` — 278-row fixture corpus
- `python/silversight_engine.py` — Python implementation of the pipeline
- `data/spectral_codebook_raw.json` — 250-entry raw spectral data (from power iteration — needs regeneration)