mirror of
https://github.com/allaunthefox/SilverSight.git
synced 2026-07-31 01:25:21 +00:00
Add cmix weight matrix analysis: compression shape as epigenetic landscape
Documents the cmix (fx2-cmix) architecture from the epigenetic perspective: - 461 models = 461 genes - 23 layer-0 mixers = 23 regulatory regions - Layer-1 mixer = master regulator - Weight matrix (23 × 461) = the compression genome - SVD of weight matrix = the compression shape equation - Singular values = active genes (high) vs silenced (low) - Learning rates = methylation rates - Context hash map = chromatin accessibility The key prediction: the compression shape is low-rank. If 5 singular values capture 95% of energy, then: - 5 mixers suffice (18 are redundant) - The search space for better compressors is 5D, not 23 × 461 - Alternative expression patterns (nearby weight matrices) may compress better Next steps: extract trained weights from cmix, compute SVD, encode as hachimoji LUT, map in math space, search neighborhood.
This commit is contained in:
openresearch
parent
7b87d1f350
commit
4910e3877c
1 changed files with 173 additions and 0 deletions
173
docs/cmix_epigenetic_analysis.md
Normal file
173
docs/cmix_epigenetic_analysis.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
# cmix Weight Matrix Analysis: Compression Shape as Epigenetic Landscape
|
||||
|
||||
## The cmix Architecture
|
||||
|
||||
cmix (fx2-cmix) is the current Hutter Prize record holder for enwik9 compression:
|
||||
- **Total size**: 110,793,128 bytes (including 441,463-byte decompressor)
|
||||
- **Previous record**: 112,578,322 bytes
|
||||
- **Improvement**: 1.585%
|
||||
- **Decompression time**: 65 hours (4 vCPU, 16GB RAM)
|
||||
|
||||
## Model Inventory (461 models)
|
||||
|
||||
The 461 models are the "genes" of the compressor:
|
||||
|
||||
| Model Type | Count | Role | Epigenetic analog |
|
||||
|---|---|---|---|
|
||||
| FXCM (context-mapped) | 431 | Primary predictors | Housekeeping genes (bulk of expression) |
|
||||
| Match models | 10 | Sequence matching | Repeat-element silencing |
|
||||
| Indirect (nonstationary) | 15 | Adaptive context | Histone modification |
|
||||
| Indirect (run-map) | 1 | Run-length context | CpG methylation |
|
||||
| Direct | 1 | Direct byte context | Promoter accessibility |
|
||||
| Bracket | 1 | Nesting structure | Chromatin looping |
|
||||
| PPMD | 1 | Context-tree model | Enhancer element |
|
||||
| Byte mixer (LSTM) | 1 | Neural prediction | Transcription factor |
|
||||
| **Total** | **461** | | |
|
||||
|
||||
## Weight Matrix Structure (the "Genome")
|
||||
|
||||
### Layer 0: 23 mixers × 461 inputs
|
||||
- 23 context-dependent mixers, each with 461 weight parameters
|
||||
- Each mixer has a per-context hash map (up to 10,000 contexts)
|
||||
- Total active parameters: ~106M floats (424MB)
|
||||
- This is the **primary genome** — the compression shape
|
||||
|
||||
### Layer 1: 1 mixer × 25 inputs
|
||||
- 23 layer-0 outputs + 2 auxiliary (fxcm + byte_mixer)
|
||||
- Single mixer with per-context weights
|
||||
- This is the **epigenetic controller** — which layer-0 mixers to trust
|
||||
|
||||
### The 23 Layer-0 Mixer Contexts
|
||||
|
||||
Each mixer operates on a different context (the "regulatory region"):
|
||||
|
||||
| # | Context | Learning rate | Role |
|
||||
|---|---|---|---|
|
||||
| 1 | mx9 | 0.005 | General context |
|
||||
| 2 | mx10 | 0.0005 | Slow adaptation |
|
||||
| 3 | mx11 | 0.005 | General context |
|
||||
| 4 | mx12 | 0.0005 | Slow adaptation |
|
||||
| 5 | mx13 | 0.005 | General context |
|
||||
| 6 | mxx | 0.001 | Cross-model |
|
||||
| 7 | recent_bytes[2] | 0.002 | Local history |
|
||||
| 8 | line_break | 0.0007 | Structural |
|
||||
| 9 | longest_match | 0.0005 | Match quality |
|
||||
| 10 | mx19cxt | 0.002 | Extended context |
|
||||
| 11 | auxiliary | 0.0005 | Auxiliary feedback |
|
||||
| 12 | mx18 | 0.001 | Special context |
|
||||
| 13 | mx7 | 0.001 | Special context |
|
||||
| 14 | wordscxt | 0.005 | Word-level |
|
||||
| 15 | b2streamcxt | 0.001 | Byte-2 stream |
|
||||
| 16 | mx5 | 0.001 | Special context |
|
||||
| 17 | mx6 | 0.005 | Special context |
|
||||
| 18 | b3streamcxt | 0.001 | Byte-3 stream |
|
||||
| 19 | mx8 | 0.001 | Special context |
|
||||
| 20 | mx17 | 0.005 | Special context |
|
||||
| 21 | mx16 | 0.005 | Special context |
|
||||
| 22 | mx14 | 0.005 | Special context |
|
||||
| 23 | mx15 | 0.005 | Special context |
|
||||
|
||||
## The Compression Shape in Math Space
|
||||
|
||||
The 23 × 461 weight matrix W defines the compression shape.
|
||||
Since it's non-square, the SVD gives:
|
||||
|
||||
```
|
||||
W = U Σ V^T
|
||||
```
|
||||
|
||||
Where:
|
||||
- U: 23 × 23 orthogonal matrix (which mixers cooperate)
|
||||
- Σ: 23 singular values (the "active genes" — which directions matter)
|
||||
- V: 461 × 23 matrix (which models contribute to each direction)
|
||||
|
||||
The 23 singular values ARE the eigenvalues of the compression shape.
|
||||
They tell us:
|
||||
- **High singular values**: "overexpressed genes" — models that contribute most
|
||||
- **Near-zero singular values**: "silenced genes" — redundant or useless models
|
||||
- **The spectrum shape**: the "epigenetic landscape" — how the compressor
|
||||
distributes its prediction capacity across models
|
||||
|
||||
## Epigenetic Mapping
|
||||
|
||||
| Compression concept | Epigenetic concept |
|
||||
|---|---|
|
||||
| 461 models | 461 genes |
|
||||
| 23 mixers | 23 regulatory regions |
|
||||
| Layer-1 mixer | Master regulator |
|
||||
| Learning rate | Methylation rate |
|
||||
| Context hash map | Chromatin accessibility |
|
||||
| Weight update | Histone modification |
|
||||
| Weight decay | DNA methylation decay |
|
||||
| Article reordering | Chromatin looping |
|
||||
| NLP stemmer | Transcription factor |
|
||||
| Dictionary | Reference genome |
|
||||
| LSTM | Neural crest prediction |
|
||||
|
||||
## How to Extract the Weight Matrix
|
||||
|
||||
To compute the SVD, we need the actual trained weight matrix from a
|
||||
cmix run on enwik8. The process:
|
||||
|
||||
1. Modify cmix to dump the layer-0 weight matrix after compression
|
||||
2. Run cmix on enwik8 (100MB, ~10 hours)
|
||||
3. Extract the 23 × 461 matrix (one per context, average over contexts)
|
||||
4. Compute SVD → 23 singular values
|
||||
5. The singular value spectrum IS the compression shape equation
|
||||
|
||||
## The "Equation" of the Compression Shape
|
||||
|
||||
The characteristic polynomial of W^T W (461 × 461) gives the equation.
|
||||
Its roots are the squared singular values — the eigenvalues of the
|
||||
compression shape.
|
||||
|
||||
For a 23 × 461 matrix, W^T W has at most 23 nonzero eigenvalues.
|
||||
These 23 values describe the entire compression capacity.
|
||||
|
||||
Storing these 23 values (instead of the full weight matrix) would be
|
||||
the "equation" — analogous to how the minimal polynomial of a matrix
|
||||
captures its eigenvalue structure from n+1 coefficients instead of n² entries.
|
||||
|
||||
## Alternative Expression Patterns
|
||||
|
||||
If the SVD reveals that 5 singular values capture 95% of the energy,
|
||||
then 5 mixers suffice — the other 18 are redundant. This means:
|
||||
- The compression shape lives in a 5-dimensional subspace
|
||||
- Alternative configurations in this subspace may compress better
|
||||
- The "locality" to search is the neighborhood of the current
|
||||
weight matrix in this 5D subspace
|
||||
|
||||
This is the "map its locality for other combinations" step:
|
||||
- Current cmix: a point in 23D weight space
|
||||
- SVD reduces to 5D principal subspace
|
||||
- Search the 5D neighborhood for better compression
|
||||
|
||||
## Connection to SilverSight
|
||||
|
||||
The SilverSight pipeline's universal pipeline already implements:
|
||||
- QR decomposition (O-AMMR): the SVD is the symmetric version
|
||||
- Chiral hachimoji LUT: encodes the eigenvalue spectrum
|
||||
- GCCL Admit: validates that a weight configuration is lawful
|
||||
- AngrySphinx: budgets the search through weight space
|
||||
- MMR: tracks the compression history as Mountains
|
||||
|
||||
The cmix weight matrix IS the manifold. The SVD IS the eigenvalue
|
||||
decomposition. The chiral hachimoji LUT IS the receipt. The epigenetic
|
||||
framing tells us what to look for: which "genes" (models) are
|
||||
overexpressed (high singular value), which are silenced (near-zero),
|
||||
and what alternative "expression patterns" (nearby weight matrices)
|
||||
might compress better.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Extract weights**: modify cmix to dump layer-0 weight matrix
|
||||
2. **Compute SVD**: get the 23 singular values
|
||||
3. **Encode as hachimoji LUT**: 23 coefficients × 30 bases = 690 bytes
|
||||
4. **Map in math space**: the singular value spectrum is the "equation"
|
||||
5. **Search neighborhood**: vary weights along top-5 singular directions
|
||||
6. **Measure**: does the variation improve compression?
|
||||
|
||||
The epigenetic framing predicts: the compression shape is low-rank
|
||||
(most models are redundant), and the active subspace is small
|
||||
(5-10 dimensions out of 23). If confirmed, the search space for
|
||||
better compressors is much smaller than the full 23 × 461 space.
|
||||
Loading…
Add table
Reference in a new issue