diff --git a/docs/cmix_epigenetic_analysis.md b/docs/cmix_epigenetic_analysis.md new file mode 100644 index 00000000..b1779b26 --- /dev/null +++ b/docs/cmix_epigenetic_analysis.md @@ -0,0 +1,173 @@ +# cmix Weight Matrix Analysis: Compression Shape as Epigenetic Landscape + +## The cmix Architecture + +cmix (fx2-cmix) is the current Hutter Prize record holder for enwik9 compression: +- **Total size**: 110,793,128 bytes (including 441,463-byte decompressor) +- **Previous record**: 112,578,322 bytes +- **Improvement**: 1.585% +- **Decompression time**: 65 hours (4 vCPU, 16GB RAM) + +## Model Inventory (461 models) + +The 461 models are the "genes" of the compressor: + +| Model Type | Count | Role | Epigenetic analog | +|---|---|---|---| +| FXCM (context-mapped) | 431 | Primary predictors | Housekeeping genes (bulk of expression) | +| Match models | 10 | Sequence matching | Repeat-element silencing | +| Indirect (nonstationary) | 15 | Adaptive context | Histone modification | +| Indirect (run-map) | 1 | Run-length context | CpG methylation | +| Direct | 1 | Direct byte context | Promoter accessibility | +| Bracket | 1 | Nesting structure | Chromatin looping | +| PPMD | 1 | Context-tree model | Enhancer element | +| Byte mixer (LSTM) | 1 | Neural prediction | Transcription factor | +| **Total** | **461** | | | + +## Weight Matrix Structure (the "Genome") + +### Layer 0: 23 mixers × 461 inputs +- 23 context-dependent mixers, each with 461 weight parameters +- Each mixer has a per-context hash map (up to 10,000 contexts) +- Total active parameters: ~106M floats (424MB) +- This is the **primary genome** — the compression shape + +### Layer 1: 1 mixer × 25 inputs +- 23 layer-0 outputs + 2 auxiliary (fxcm + byte_mixer) +- Single mixer with per-context weights +- This is the **epigenetic controller** — which layer-0 mixers to trust + +### The 23 Layer-0 Mixer Contexts + +Each mixer operates on a different context (the "regulatory region"): + +| # | Context | Learning rate | Role | +|---|---|---|---| +| 1 | mx9 | 0.005 | General context | +| 2 | mx10 | 0.0005 | Slow adaptation | +| 3 | mx11 | 0.005 | General context | +| 4 | mx12 | 0.0005 | Slow adaptation | +| 5 | mx13 | 0.005 | General context | +| 6 | mxx | 0.001 | Cross-model | +| 7 | recent_bytes[2] | 0.002 | Local history | +| 8 | line_break | 0.0007 | Structural | +| 9 | longest_match | 0.0005 | Match quality | +| 10 | mx19cxt | 0.002 | Extended context | +| 11 | auxiliary | 0.0005 | Auxiliary feedback | +| 12 | mx18 | 0.001 | Special context | +| 13 | mx7 | 0.001 | Special context | +| 14 | wordscxt | 0.005 | Word-level | +| 15 | b2streamcxt | 0.001 | Byte-2 stream | +| 16 | mx5 | 0.001 | Special context | +| 17 | mx6 | 0.005 | Special context | +| 18 | b3streamcxt | 0.001 | Byte-3 stream | +| 19 | mx8 | 0.001 | Special context | +| 20 | mx17 | 0.005 | Special context | +| 21 | mx16 | 0.005 | Special context | +| 22 | mx14 | 0.005 | Special context | +| 23 | mx15 | 0.005 | Special context | + +## The Compression Shape in Math Space + +The 23 × 461 weight matrix W defines the compression shape. +Since it's non-square, the SVD gives: + +``` +W = U Σ V^T +``` + +Where: +- U: 23 × 23 orthogonal matrix (which mixers cooperate) +- Σ: 23 singular values (the "active genes" — which directions matter) +- V: 461 × 23 matrix (which models contribute to each direction) + +The 23 singular values ARE the eigenvalues of the compression shape. +They tell us: +- **High singular values**: "overexpressed genes" — models that contribute most +- **Near-zero singular values**: "silenced genes" — redundant or useless models +- **The spectrum shape**: the "epigenetic landscape" — how the compressor + distributes its prediction capacity across models + +## Epigenetic Mapping + +| Compression concept | Epigenetic concept | +|---|---| +| 461 models | 461 genes | +| 23 mixers | 23 regulatory regions | +| Layer-1 mixer | Master regulator | +| Learning rate | Methylation rate | +| Context hash map | Chromatin accessibility | +| Weight update | Histone modification | +| Weight decay | DNA methylation decay | +| Article reordering | Chromatin looping | +| NLP stemmer | Transcription factor | +| Dictionary | Reference genome | +| LSTM | Neural crest prediction | + +## How to Extract the Weight Matrix + +To compute the SVD, we need the actual trained weight matrix from a +cmix run on enwik8. The process: + +1. Modify cmix to dump the layer-0 weight matrix after compression +2. Run cmix on enwik8 (100MB, ~10 hours) +3. Extract the 23 × 461 matrix (one per context, average over contexts) +4. Compute SVD → 23 singular values +5. The singular value spectrum IS the compression shape equation + +## The "Equation" of the Compression Shape + +The characteristic polynomial of W^T W (461 × 461) gives the equation. +Its roots are the squared singular values — the eigenvalues of the +compression shape. + +For a 23 × 461 matrix, W^T W has at most 23 nonzero eigenvalues. +These 23 values describe the entire compression capacity. + +Storing these 23 values (instead of the full weight matrix) would be +the "equation" — analogous to how the minimal polynomial of a matrix +captures its eigenvalue structure from n+1 coefficients instead of n² entries. + +## Alternative Expression Patterns + +If the SVD reveals that 5 singular values capture 95% of the energy, +then 5 mixers suffice — the other 18 are redundant. This means: +- The compression shape lives in a 5-dimensional subspace +- Alternative configurations in this subspace may compress better +- The "locality" to search is the neighborhood of the current + weight matrix in this 5D subspace + +This is the "map its locality for other combinations" step: +- Current cmix: a point in 23D weight space +- SVD reduces to 5D principal subspace +- Search the 5D neighborhood for better compression + +## Connection to SilverSight + +The SilverSight pipeline's universal pipeline already implements: +- QR decomposition (O-AMMR): the SVD is the symmetric version +- Chiral hachimoji LUT: encodes the eigenvalue spectrum +- GCCL Admit: validates that a weight configuration is lawful +- AngrySphinx: budgets the search through weight space +- MMR: tracks the compression history as Mountains + +The cmix weight matrix IS the manifold. The SVD IS the eigenvalue +decomposition. The chiral hachimoji LUT IS the receipt. The epigenetic +framing tells us what to look for: which "genes" (models) are +overexpressed (high singular value), which are silenced (near-zero), +and what alternative "expression patterns" (nearby weight matrices) +might compress better. + +## Next Steps + +1. **Extract weights**: modify cmix to dump layer-0 weight matrix +2. **Compute SVD**: get the 23 singular values +3. **Encode as hachimoji LUT**: 23 coefficients × 30 bases = 690 bytes +4. **Map in math space**: the singular value spectrum is the "equation" +5. **Search neighborhood**: vary weights along top-5 singular directions +6. **Measure**: does the variation improve compression? + +The epigenetic framing predicts: the compression shape is low-rank +(most models are redundant), and the active subspace is small +(5-10 dimensions out of 23). If confirmed, the search space for +better compressors is much smaller than the full 23 × 461 space.