mirror of
https://github.com/allaunthefox/SilverSight.git
synced 2026-07-31 01:25:21 +00:00
docs: record LLM recoverable drop analysis (same conservation law)
LLMs 'drop data recoverably' via four mechanisms: 1. Residual stream (accumulate, never drop — workspace not compressor) 2. Superposition (pack N features into d<N, exact only when k-sparse) 3. Attention (soft retrieval, KV eviction = explicitly lossy) 4. Quantization (drop bits, recover approximately) Law: recoverable ⟺ sparse/redundant. Same wall as every compression branch. Dense/random data → recovery fails → entropy floor. Pipeline connection: - QR decomposition (O-AMMR) IS compressed sensing - GW ringdown at 30dB: k=5 sparse, d=9 < RIP bound → lossy → 1.5x - Order-2 PPM: 256 contexts packing 65K transitions → interference on dense data → 3.088 b/B residual - Mass number = honesty tag for what was kept vs lost (receipt) Doctrine consistency: 'MassNumber = recoverability RECEIPT' confirmed. LLM superposition = same mechanism, same RIP bound, same lossy floor.
This commit is contained in:
parent
8935cc9eaa
commit
abf8329921
1 changed files with 56 additions and 0 deletions
|
|
@ -114,3 +114,59 @@ One rule: you can move bits between columns, never beat K(data).
|
|||
Everything that "compresses" is either base conversion (bijection,
|
||||
no gain) or arithmetic coding (needs model, ship cost). The clever
|
||||
geometry buys nothing over boring xz/LPC.
|
||||
|
||||
## LLM Recoverable Drop: Same Law, Different Substrate
|
||||
|
||||
### How LLMs Actually "Drop Data Recoverably"
|
||||
|
||||
1. **Residual stream** = accumulate, don't overwrite. No bytes saved —
|
||||
the data was never dropped. The stream is a workspace, not a compressor.
|
||||
|
||||
2. **Superposition** = pack N features into d < N dimensions via
|
||||
near-orthogonal directions (Anthropic's Toy Models of Superposition).
|
||||
Recovery is exact only when k-sparse: d ≥ k·log(N/k) (RIP bound).
|
||||
Past that → interference → lossy.
|
||||
|
||||
3. **Attention** = soft retrieval, keeps everything. KV-cache eviction
|
||||
(StreamingLLM/beacons) is explicitly lossy — evicted tokens gone,
|
||||
reconstructed only approximately.
|
||||
|
||||
4. **Quantization** = drop precision bits, recover approximately.
|
||||
Lossy, bounded by weight tolerance.
|
||||
|
||||
### The Law Under All of It
|
||||
|
||||
Recoverable ⟺ sparse/redundant. Every mechanism recovers the
|
||||
structured part and loses the noise part. Feed it dense/random data
|
||||
and recovery fails — same wall as every compression branch measured.
|
||||
|
||||
### Connection to Pipeline Measurements
|
||||
|
||||
- GW ringdown at 30dB: signal is k=5 sparse in N=10,000 samples.
|
||||
RIP bound: d ≥ 5·log(2000) ≈ 37.5. Our d=9 polynomial is BELOW
|
||||
the bound → lossy recovery, residual = noise → 1.5x (ties LPC).
|
||||
|
||||
- Order-2 PPM on enwik8: 256×256 co-occurrence packs N=65,536
|
||||
transitions into d=256 contexts. Sparse text = good recovery.
|
||||
Dense Wikipedia = interference = 3.088 b/B residual.
|
||||
|
||||
- QR decomposition (O-AMMR) IS compressed sensing. Eigenvalue
|
||||
spectrum IS the sparse feature set. Golden spiral IS RIP-compliant
|
||||
recovery. But it's LOSSY — the residual is lost.
|
||||
|
||||
### The Honest Split
|
||||
|
||||
- **Lossless**: dead. K(data) is conserved across every substrate.
|
||||
- **Lossy recoverability**: real, useful, must be judged vs
|
||||
JPEG/Opus at equal distortion. The QR projection recovers the
|
||||
dominant structure (sparse part), loses the residual (noise part).
|
||||
The mass number IS the honesty tag for what was kept vs lost.
|
||||
|
||||
### Final Consistency Check
|
||||
|
||||
The doctrine says: "MassNumber is the admissibility / recoverability
|
||||
RECEIPT projected from SemanticMass." The measurement confirms:
|
||||
- As lossless compressor: dead (base conversion, 1.00x)
|
||||
- As lossy projection: real (recovers sparse structure, loses noise)
|
||||
- As receipt: exactly what the spec says (records what was kept)
|
||||
- LLM superposition: same mechanism, same RIP bound, same lossy floor
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue