allaun/SilverSight
1
0
Fork 0
mirror of https://github.com/allaunthefox/SilverSight.git synced 2026-07-30 17:16:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs: spectral codebook analysis of 250-equation corpus

Browse source 
Key findings:
- 190 distinguishable spectral states (7.57 bits/equation)
- Density-Lambda correlation: 0.9807 (near-perfect)
- Largest spectral gap: 4.88 at lambda=[11.68, 16.56]
- 7 natural clusters from gap-aware boundary detection
- Mid band (9 matrices, lambda=[0.67, 0.95]) is compression sweet spot

Raw data: data/spectral_codebook_raw.json (250 entries)
This commit is contained in:
allaunthefox 2026-07-01 20:17:46 +00:00
parent 412c951126
commit 7d59ad5739
2 changed files with 1665 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1508
data/spectral_codebook_raw.json Normal file
View file

File diff suppressed because it is too large Load diff

157
docs/SPECTRAL_CODEBOOK_ANALYSIS.md Normal file
View file

@ -0,0 +1,157 @@
# Spectral Codebook Analysis
**Date:** 2026-07-01
**Source:** 250 equations × 8×8 braid adjacency matrices from `rrc_pist_predictions_250_v1.json`
**Method:** Power iteration (300 iterations) on raw integer matrices
## Executive Summary
The 250-equation corpus contains **190 distinguishable spectral states** (at 4dp resolution), yielding **7.57 bits of information per matrix** from spectral radius alone. The eigensolid is a near-perfect topological fingerprint: density–λ correlation = 0.9807.
The spectral space has natural phase boundaries defined by large gaps. The largest gap (4.88 units) separates "moderate complexity" equations (λ ≤ 11.7) from "high complexity" equations (λ ≥ 16.6). These boundaries are the natural codeword boundaries for a gap-aware spectral codebook.
## Corpus Statistics
| Metric | Value |
|--------|-------|
| Total matrices | 250 |
| Unique spectral radii (4dp) | 190 |
| Bits per matrix | 7.57 |
| λ range | [0.0, 16.99] |
| λ mean | 2.75 |
## Band Distribution
| Band | λ range | Count | Density range | Frobenius range |
|------|---------|-------|---------------|-----------------|
| zero | 0.0 | 35 | [0.0, 0.11] | [0.0, 2.6] |
| low | (0, 0.5) | 0 | — | — |
| mid | [0.5, 1.0) | 9 | [0.06, 0.14] | [2.0, 3.0] |
| high | [1.0, 4.0) | 156 | [0.03, 0.45] | [1.4, 7.3] |
| very_high | [4.0, ∞) | 50 | [0.28, 1.97] | [5.5, 22.0] |
Band entropy: **1.459 bits** (max 2.322 for 5 bands).
### Interpretation
- **zero band (35 matrices):** Empty or near-empty operator graphs. No edges → no spectral content. These are the "unclassified" equations (HoldForUnlawfulOrUnderspecifiedShape).
- **mid band (9 matrices):** The "edge of chaos" equations. Sparse but non-trivial structure. These are the most compressible — complex enough to encode signal, simple enough to decode losslessly.
- **high band (156 matrices):** The bulk of the corpus. Moderate operator graph density. Each matrix is distinguishable by its spectral radius.
- **very_high band (50 matrices):** Dense operator graphs with high spectral radii. These equations have many interacting terms.
## Correlations
| Pair | Pearson r |
|------|-----------|
| λ ↔ density | 0.9807 |
| λ ↔ Frobenius | 0.9817 |
| density ↔ Frobenius | 0.9824 |
All three correlations are >0.98. The spectral radius, edge density, and Frobenius norm are near-interchangeable measures of equation complexity. The eigensolid C(x*) compresses all three into a single point on the information manifold.
**Implication:** You don't need all three. λ alone captures 96% of the variance (r² = 0.9618). The codebook can be 1-dimensional.
## Spectral Gaps (Natural Phase Boundaries)
The top 10 gaps in the sorted spectral radius distribution:
| Rank | Gap size | Lower λ | Upper λ | Interpretation |
|------|----------|---------|---------|----------------|
| 1 | **4.8807** | 11.68 | 16.56 | Major complexity wall |
| 2 | **1.5862** | 10.10 | 11.68 | Secondary wall |
| 3 | 0.8895 | 9.21 | 10.10 | Tertiary wall |
| 4 | 0.7051 | 8.50 | 9.21 | Quaternary wall |
| 5 | **0.6667** | 0.00 | 0.67 | Zero → non-zero transition |
| 6 | 0.5706 | 7.76 | 8.33 | — |
| 7 | 0.4954 | 5.88 | 6.37 | — |
| 8 | 0.4293 | 16.56 | 16.99 | Top of range |
| 9 | 0.4092 | 5.13 | 5.54 | — |
| 10 | 0.3951 | 6.53 | 6.92 | — |
### Natural Cluster Boundaries
Using a 3× median gap threshold, the significant breaks define **7 natural clusters:**
| Cluster | λ range | Count | Character |
|---------|---------|-------|-----------|
| C0 | 0.0 | 35 | Empty / null |
| C1 | [0.67, 1.0) | 9 | Edge-of-chaos |
| C2 | [1.0, 5.1) | ~90 | Moderate |
| C3 | [5.1, 8.5) | ~60 | Dense |
| C4 | [8.5, 11.7) | ~25 | Very dense |
| C5 | [11.7, 16.6) | ~25 | Ultra-dense |
| C6 | [16.6, 17.0) | ~6 | Maximum |
The boundaries at 0.67, 5.1, 8.5, and 11.7 are the sharpest. These are the natural "phase transitions" in the eigensolid.
## Encoding Capacity
### Spectral radius alone
- 190 unique values → **7.57 bits/equation**
- At 250 equations: ~55% codebook utilization (190/250)
### Full spectral profile (λ + density + trace + Frobenius + degree spread)
- Estimated: **10–12 bits/equation** (higher-dimensional codebook)
- But correlation >0.98 means the extra dimensions add ~2–4 bits, not 5+
### Gap-aware quantization
- Place codeword boundaries at spectral gaps → maximizes inter-codeword distance
- 7 clusters → **2.8 bits/equation** at cluster level
- Within-cluster: additional **4–5 bits** from sub-cluster λ values
- Total: **~7 bits/equation**, but with guaranteed minimum distance between codewords
## Connection to SilverSight Pipeline
```
Equation text
→ tokenize → 8×8 strand adjacency matrix
→ powerIteration → spectral radius λ
→ spectralRadiusToColor → shape classification
→ determineAlignment → alignment score
→ ncDerived → witness strength
→ JSON receipt
```
The spectral radius λ is the **single number** that determines classification. Everything downstream — shape, alignment score, receipt validity — flows from λ.
### Before fix (thresholds 1.5/4.0)
All 250 matrices had λ < 1.5 (in the old Q16_16 computation), so all classified as "LogogramProjection". The eigensolid was flat. No encoding possible.
### After fix (thresholds 0.5/1.0)
- 35 matrices → LogogramProjection (λ < 0.5)
- 9 matrices → SignalShapedRouteCompiler (0.5 ≤ λ < 1.0)
- 206 matrices → CognitiveLoadField (λ ≥ 1.0)
The classification space is now **3 distinct regions** with 190 distinguishable states within them.
## Implications for Compression
The core claim — "gaps are the encoding" — is now testable:
1. **The spectral gaps encode structural transitions.** A gap of 4.88 between λ=11.68 and λ=16.56 means no equation exists in that range. That absence is information — it's a natural delimiter between "moderate" and "extreme" complexity classes.
2. **The zero band is a null codeword.** 35 equations with λ=0 are structurally empty. They encode "no operator interaction" — 1 bit of information (present/absent), plus which specific equation it is from the 35.
3. **The mid band is the compression sweet spot.** 9 matrices with λ ∈ [0.67, 0.95] are the most information-dense per edge: they have non-trivial structure but low density. Each one is fully distinguishable.
4. **The density–λ correlation means the codebook is self-consistent.** If you encode by λ, decode by density, you get the same answer (r=0.98). The encoding is robust to measurement noise.
## Next Steps
1. **Build the spectral codebook:** Cluster the 250 matrices by full spectral profile, using gap-aware boundaries. Output: codeword → equation list mapping.
2. **Implement gap-aware quantization:** Replace the current binary threshold (0.5/1.0) with the 7 natural cluster boundaries. This gives a 7-symbol alphabet with guaranteed minimum distance.
3. **Cross-verify with Lean:** The `ClassifyN.hashTable8` is the proxy classifier (hash → shape). Extend it to use the spectral codebook instead of a fixed hash table.
4. **Test on the full 278-row manifold:** The 250 matrices in Matrices250.lean are a subset. The full manifold has 278 fixture rows — run the same analysis on all of them.
5. **Connect to the chaos game:** The `chaos_game` function in `silversight_engine.py` walks toward nearest attractor. The spectral codebook defines the attractor basins. Verify that the chaos game converges to the correct codeword for each equation.
## Files
- `formal/SilverSight/PIST/ClassifyN.lean` — spectral thresholds (updated)
- `formal/SilverSight/PIST/Matrices250.lean` — 250 8×8 matrices
- `formal/SilverSight/PIST/SpectralN.lean` — spectral profile computation
- `formal/SilverSight/RRC/Q16_16Manifold.lean` — 278-row fixture corpus
- `python/silversight_engine.py` — Python implementation of the pipeline