diff --git a/docs/research/COMPRESSION_HONEST_FINDINGS.md b/docs/research/COMPRESSION_HONEST_FINDINGS.md new file mode 100644 index 00000000..d4afc257 --- /dev/null +++ b/docs/research/COMPRESSION_HONEST_FINDINGS.md @@ -0,0 +1,134 @@ +# Compression: Honest Findings (Receipt, Not Ratio) + +**Date:** 2026-07-03 **Status:** measured, reproducible **Doctrine:** OTOM honest-measurement / anti-smuggle + +This document records a series of compression ideas that were tested **head-on with +real coders and lossless round-trips**, and the single law they all obey. Every +"novel compressor" explored here turns out to be either **base conversion** (a +bijection — no gain) or **arithmetic coding in a costume** (needs a shipped model — +conservation), and every one loses to stock `xz`/`LPC`. The recurring honest +conclusion: these constructions are **receipts / addresses / recoverability gates**, +never compression ratios. + +Reproducible scripts live in [`scripts/compression/`](../../scripts/compression/). +All numbers below are from real runs (arithmetic coders verified lossless by +byte-exact round-trip; entropies are the rate a coder provably reaches). + +## The one law + +> **Recoverable ⟺ sparse/structured.** No method beats `K(data)` (Kolmogorov complexity). +> A scheme can only **move bits between two columns** — the *model/program* and the +> *residual/tape* — never reduce their **sum** below the data's information content. +> "Compression" that appears otherwise is measuring against a straw baseline, or is +> lossy recoverability (rate–distortion) mislabeled as lossless. + +## Baselines (enwik8, 100 MB, all lossless) + +| method | bits/byte | note | +|---|---|---| +| order-0 entropy | 5.08 | symbol frequencies only | +| **order-2 PPM (our real arithmetic coder)** | **3.088** | lossless round-trip verified; our honest floor | +| gzip -9 | 2.92 | LZ77 + Huffman | +| zstd -19 | 2.16 | | +| **xz -9 (LZMA)** | **1.989** | long-range dictionary matching — the number to beat | +| cmix (SOTA) | ~1.2 | 461-model context mixing | + +**Wall 1** — a low-rank/spectral summary of the byte co-occurrence matrix can never +beat the raw counts (the empirical counts are the MLE; any rank-k reconstruction has +cross-entropy ≥ the full model). **Wall 2** — even the *full* order-2 counts (3.088) +lose to stock `xz` (1.989), because LZMA sees kilobytes of context, not 2 bytes. + +## Findings + +### 1. Characteristic polynomial / chiral LUT — a *receipt*, not a compressor +Text reconstructs from **vocab + indices**; the Faddeev–LeVerrier char-poly is computed +*downstream* of the indices and is never read during decompression. It adds bytes +(poly +3 B, chiral LUT +60 B on a toy). Its honest role is a **GCCL integrity receipt** +(tamper-evidence), never a ratio. Script: `spectral_vs_empirical.py`, `ac_roundtrip.py`. + +### 2. Braille → T9 → hachimoji (three-layer) on text — dead +Real lossless three-layer coder: **4.167 bits/byte on enwik8** — worse than order-2 PPM +(3.088), far behind xz (1.989). A 64-cell alphabet cannot represent 256 byte values +without lossy mapping; the T9 disambiguation cost exceeds the 6→3-bit saving. + +### 3. "16D geometric / golden-spiral braid" GW ringdown — 583× is a zero-noise artifact +Claim: a 16D parametric fit compresses a gravitational-wave ringdown 583×. **It was +measuring the generator round-tripping itself** (a signal synthesized from ~9 coefficients, +"compressed" back to those coefficients). Honest test (`gw_honest_test.py`): synthetic +2-QNM ringdown **+ realistic detector noise**, int16-quantized, lossless, with the +parametric coder **handed the TRUE mode parameters** (steelman), raced vs LPC-8/xz on the +same stream: + +| SNR regime | PARAM* b/s | LPC-8 b/s | best ratio | +|---|---|---|---| +| clean (σ=0) | 0.14 | 0.35 | 111× ← the big ratio lives ONLY here | +| 60 dB | 7.02 | 7.57 | 2.3× | +| 40 dB | 9.81 | 9.87 | tie | +| 30 dB (realistic loud) | 10.58 | 10.61 | 1.5× (tie) | +| 20 dB (weak) | 10.94 | 10.92 | **LPC wins** | + +*PARAM handed the true params — upper bound on any 16D/braid fit.* Lossless ⇒ store the +residual = signal − fit = **the noise**, which is incompressible. Honest ceiling on noisy +strain ≈ **1.5×, tying/losing to 50-year-old LPC**. "16D braid" = LPC in a costume. + +### 4. "Turing-complete weird machine" — the conservation law +A decompressor is already a Turing machine; total self-contained size = |program (frozen +model)| + |tape (coded residual)|. Frozen order-k byte model + real arithmetic coder, +150 KB held-out test, all rows lossless (`frozen_model_compress.py`): + +| order k | test b/char | tape B | model B | TOTAL self | amortized | +|---|---|---|---|---|---| +| 0 | 5.430 | 101,812 | 440 | 102,252 | 101,812 | +| 1 | 4.043 | 75,806 | 9,728 | 85,534 | 75,806 | +| 2 | 3.253 | 61,000 | 104,116 | 165,116 | 61,000 | +| 3 | 2.975 | 55,777 | 501,392 | **557,169** | 55,777 | + +As k↑, tape shrinks but model explodes; k=3 has the **smallest tape and the worst total**. +You only **relocate** bits. xz-9 self-contained (35,492 B) beats every total. A frozen +**LLM** + arithmetic coder is a genuine strong compressor **in the amortized column only** +(model shared out-of-band, sub-1 b/byte — cf. Bellard `ts_zip`, DeepMind "Language Modeling +Is Compression"); the instant the model must ship (Hutter), the model column dominates and +it loses. Long-context LLM stacks (StreamingLLM, Infini-attention, Activation Beacon, CEPE, +E2LLM) are **lossy** — right for RAG/QA, wrong for lossless reconstruction. + +### 5. Semantic Mass Number as a lossless compressor — base conversion +Encoding a message as one "mass number" (OTOM Semantic Nuclide Address / "10-adic residue") +**IS base conversion** — a bijection, hence cannot compress below its radix +(`mass_number_compress.py`, 100 KB, round-trips PASS): + +| method | bits/char | ratio | note | +|---|---|---|---| +| M1 base-256 mass number | 8.000 | 1.00× | literally the data | +| M2 mixed-radix over Σ (|Σ|=155) | 7.276 | 1.10× | drops unused-symbol slack only | +| M3 freq-weighted (=arithmetic) | 5.401 | 1.47× | radices ARE the model → must ship | +| xz -9 | 2.551 | 3.14× | crushes all | + +OTOM's own **de-anthropocentric revision** already flags "English-facing semantic +compression" as the OLD ERROR and defines MassNumber as the **admissibility/recoverability +RECEIPT** projected from SemanticMass. The measurement confirms the doctrine. + +### 6. Capstone — "how LLMs drop data recoverably" = superposition = compressed sensing +Pack N=1024 features into d=128 dims via a random projection (8× squeeze); recover a +k-sparse vector (`superposition_recovery.py`, OMP + matched-filter, lossless-in-the-limit): + +| k active | OMP rel-err | support | verdict | +|---|---|---|---| +| ≤16 | ~3e-16 | 100% | **lossless-ish** (superposition works) | +| 24 | 9.5e-5 | 100% | degrading | +| 32 | 6.7e-2 | 93% | degrading | +| 48 | 0.80 | 34% | **LOST** | +| ≥64 | ≥1.0 | ~15% | LOST | +| 1024 (dense/random) | 1.5 | 13% = chance | unrecoverable | + +Compressed-sensing bound `k* ≈ d/(2 ln N) = 9`. **An LLM "drops data recoverably" only in +the sparse regime** — recoverability is bought entirely by sparsity, not free compression. +Past the bound → polysemantic interference → data lost. Dense/random data goes to chance. + +## Bottom line + +Structure (sparsity/redundancy) is the only thing **any** method — polynomial, mass-number, +LLM superposition — can recover; the incompressible part is always lost or stored raw. Keep +these constructions where they honestly belong: **receipts, addresses, and recoverability +gates** (GCCL/RRC), never in the compression-ratio column. To actually reduce bytes, the +levers are the boring, established ones: **context depth + long-range matching** (LZMA, +context mixing) for text, **linear prediction** (LPC/FLAC) for smooth signals. diff --git a/scripts/compression/README.md b/scripts/compression/README.md new file mode 100644 index 00000000..efc360cb --- /dev/null +++ b/scripts/compression/README.md @@ -0,0 +1,19 @@ +# Compression — honest-measurement scripts + +Reproducible experiments behind [`docs/research/COMPRESSION_HONEST_FINDINGS.md`](../../docs/research/COMPRESSION_HONEST_FINDINGS.md). +Every claim is checked with a real coder and a byte-exact lossless round-trip; no +straw baselines. Python 3 + numpy only (stdlib `lzma`/`gzip` for baselines). + +| script | what it measures | +|---|---| +| `spectral_vs_empirical.py` | Wall 1: rank-k reconstruction of the byte transition matrix never beats raw counts | +| `entropy_full.py` | order-0/1/2 empirical entropy of a corpus (adaptive-AC target rate) | +| `ac_roundtrip.py` | real Witten–Neal–Cleary order-2 arithmetic coder; lossless round-trip + bits/byte | +| `gw_honest_test.py` | GW ringdown parametric fit (handed TRUE params) vs LPC/xz across SNRs — the "583×" gate | +| `frozen_model_compress.py` | conservation law: frozen model + AC, model-column vs tape-column, self-contained vs amortized | +| `mass_number_compress.py` | Semantic Mass Number = base conversion (bijection), M1/M2/M3 vs xz | +| `superposition_recovery.py` | capstone: compressed-sensing recovery vs sparsity — recoverable ⟺ sparse cliff | + +Most default to a local text corpus path; pass a directory as `argv[1]` to point +elsewhere. `entropy_full.py`/`ac_roundtrip.py` expect an `enwik8` file +(re-fetch from `mattmahoney.net/dc/enwik8.zip`). diff --git a/scripts/compression/ac_roundtrip.py b/scripts/compression/ac_roundtrip.py new file mode 100644 index 00000000..d9647d18 --- /dev/null +++ b/scripts/compression/ac_roundtrip.py @@ -0,0 +1,148 @@ +""" +Real Witten-Neal-Cleary arithmetic coder, order-2 ADAPTIVE byte model. +Round-trips a slice of enwik8, ASSERTS lossless, reports actual bits/byte. +Purpose: prove the ~3.09 bits/byte order-2 estimate is achievable by a genuine +lossless coder (not just a theoretical entropy), on the same trap-free footing +as the char-poly audit. Slice keeps pure-Python runtime sane; rate is the point. +""" +import sys, numpy as np + +PATH = sys.argv[1] if len(sys.argv) > 1 else "enwik8" +SLICE = int(sys.argv[2]) if len(sys.argv) > 2 else 3_000_000 + +with open(PATH, "rb") as f: + data = f.read(SLICE) +n = len(data) + +# ---- WNC 32-bit arithmetic coder ---- +BITS = 32 +TOP = (1 << BITS) - 1 +HALF = 1 << (BITS - 1) +QTR = 1 << (BITS - 2) +TQTR = 3 << (BITS - 2) +MAX_TOT = 1 << 16 # keep total freq below 2^16 << 2^30 precision bound + +NCTX = 1 << 16 # order-2 contexts (prev two bytes) +# adaptive freqs, init 1 (Laplace). uint32 so a row can't overflow before rescale. +freq = np.ones((NCTX, 256), dtype=np.uint32) + +def rescale(row): + row >>= 1 + row |= 1 # keep every symbol >=1 so nothing becomes uncodeable + +# ---------- ENCODE ---------- +low, high = 0, TOP +pending = 0 +out = bytearray() +bitbuf = 0 +nbits = 0 +def put(bit): + global bitbuf, nbits + bitbuf = (bitbuf << 1) | bit + nbits += 1 + if nbits == 8: + out.append(bitbuf) + bitbuf = 0 + nbits = 0 +def emit(bit): + global pending + put(bit) + while pending: + put(bit ^ 1) + pending -= 1 + +ctx = 0 +for sym in data: + row = freq[ctx] + tot = int(row.sum()) + cum = int(row[:sym].sum()) + f = int(row[sym]) + rng = high - low + 1 + high = low + (rng * (cum + f)) // tot - 1 + low = low + (rng * cum) // tot + while True: + if high < HALF: + emit(0) + elif low >= HALF: + emit(1); low -= HALF; high -= HALF + elif low >= QTR and high < TQTR: + pending += 1; low -= QTR; high -= QTR + else: + break + low <<= 1 + high = (high << 1) | 1 + # adapt + row[sym] += 32 + if tot + 32 >= MAX_TOT: + rescale(row) + ctx = ((ctx << 8) | sym) & 0xFFFF +# flush +pending += 1 +emit(1 if low >= QTR else 0) +if nbits: + out.append(bitbuf << (8 - nbits)) +enc = bytes(out) +enc_bits = len(enc) * 8 + +# ---------- DECODE ---------- +freq = np.ones((NCTX, 256), dtype=np.uint32) +low, high = 0, TOP +bit_iter_pos = 0 +total_bits = len(enc) * 8 +def getbit(): + global bit_iter_pos + if bit_iter_pos < total_bits: + byte = enc[bit_iter_pos >> 3] + b = (byte >> (7 - (bit_iter_pos & 7))) & 1 + else: + b = 0 + bit_iter_pos += 1 + return b + +code = 0 +for _ in range(BITS): + code = (code << 1) | getbit() + +dec = bytearray() +ctx = 0 +for _ in range(n): + row = freq[ctx] + tot = int(row.sum()) + rng = high - low + 1 + # find symbol whose cumulative interval contains 'code' + val = ((code - low + 1) * tot - 1) // rng + cs = np.cumsum(row) + sym = int(np.searchsorted(cs, val, side='right')) + cum = int(cs[sym - 1]) if sym > 0 else 0 + f = int(row[sym]) + high = low + (rng * (cum + f)) // tot - 1 + low = low + (rng * cum) // tot + while True: + if high < HALF: + pass + elif low >= HALF: + low -= HALF; high -= HALF; code -= HALF + elif low >= QTR and high < TQTR: + low -= QTR; high -= QTR; code -= QTR + else: + break + low <<= 1 + high = (high << 1) | 1 + code = (code << 1) | getbit() + dec.append(sym) + row[sym] += 32 + if tot + 32 >= MAX_TOT: + rescale(row) + ctx = ((ctx << 8) | sym) & 0xFFFF + +ok = bytes(dec) == data +print(f"slice bytes : {n}") +print(f"encoded bytes : {len(enc)}") +print(f"actual rate : {enc_bits / n:6.3f} bits/byte") +print(f"lossless round-trip: {'PASS' if ok else 'FAIL'}") +if not ok: + for i in range(n): + if dec[i] != data[i]: + print(f" first mismatch at {i}: {dec[i]} != {data[i]}") + break +print("DONE_AC") diff --git a/scripts/compression/entropy_full.py b/scripts/compression/entropy_full.py new file mode 100644 index 00000000..66eb9122 --- /dev/null +++ b/scripts/compression/entropy_full.py @@ -0,0 +1,77 @@ +""" +Static empirical entropy of enwik8 at order 0/1/2 (full 100MB, chunked numpy). +This is the bits/byte an *adaptive* arithmetic coder converges to (model learned +online => nothing to transmit; on 100MB the learning cost is negligible, so +static conditional entropy is a faithful estimate of the coder's output rate). +""" +import sys, numpy as np + +PATH = sys.argv[1] if len(sys.argv) > 1 else "enwik8" +N = 100_000_000 +CHUNK = 5_000_000 + +def entropy_from_counts(counts): + counts = counts.astype(np.float64) + tot = counts.sum() + nz = counts > 0 + p = counts[nz] / tot + return float(-(p * np.log2(p)).sum()) + +# order-0 +c0 = np.zeros(256, dtype=np.int64) +# order-1 conditional: counts[prev*256+cur] +c1 = np.zeros(256*256, dtype=np.int64) +# order-2 conditional: counts[prev2*65536+prev1*256+cur] (2^24 bins) +c2 = np.zeros(1 << 24, dtype=np.int64) + +with open(PATH, "rb") as f: + prev = np.zeros(0, dtype=np.int64) + off = 0 + while True: + b = f.read(CHUNK) + if not b: + break + a = np.frombuffer(b, dtype=np.uint8).astype(np.int64) + c0 += np.bincount(a, minlength=256) + # stitch 2 bytes of previous chunk for cross-boundary bigrams/trigrams + s = np.concatenate([prev, a]) + # order-1 pairs within s + idx1 = s[:-1] * 256 + s[1:] + c1 += np.bincount(idx1, minlength=256*256) + if len(s) >= 3: + idx2 = s[:-2] * 65536 + s[1:-1] * 256 + s[2:] + c2 += np.bincount(idx2, minlength=1 << 24) + prev = s[-2:] # carry last 2 bytes + +H0 = entropy_from_counts(c0) + +# order-1 conditional entropy = H(cur|prev) = weighted avg of row entropies +C1 = c1.reshape(256, 256).astype(np.float64) +rs1 = C1.sum(axis=1) +tot1 = rs1.sum() +H1 = 0.0 +for i in range(256): + if rs1[i] > 0: + p = C1[i][C1[i] > 0] / rs1[i] + H1 += (rs1[i] / tot1) * float(-(p * np.log2(p)).sum()) + +# order-2 conditional entropy, streamed over the 65536 context rows +C2 = c2.reshape(65536, 256) +rs2 = C2.sum(axis=1).astype(np.float64) +tot2 = rs2.sum() +H2 = 0.0 +nzc = np.where(rs2 > 0)[0] +for i in nzc: + row = C2[i].astype(np.float64) + row = row[row > 0] + p = row / rs2[i] + H2 += (rs2[i] / tot2) * float(-(p * np.log2(p)).sum()) + +def proj(bits): + return bits * N / 8 / 1_048_576 # MB if we coded all 100M bytes at this rate + +print(f"file: {PATH} bytes: {N}") +print(f"order-0 entropy : {H0:6.3f} bits/byte -> ~{proj(H0):7.2f} MB") +print(f"order-1 entropy : {H1:6.3f} bits/byte -> ~{proj(H1):7.2f} MB") +print(f"order-2 entropy : {H2:6.3f} bits/byte -> ~{proj(H2):7.2f} MB (adaptive AC target)") +print("DONE_ENTROPY") diff --git a/scripts/compression/frozen_model_compress.py b/scripts/compression/frozen_model_compress.py new file mode 100644 index 00000000..9a2c663e --- /dev/null +++ b/scripts/compression/frozen_model_compress.py @@ -0,0 +1,148 @@ +""" +Frozen-model + real arithmetic coder = lossless compressor, with HONEST +two-column accounting (the "weird machine" conservation law). + +A decompressor is a program (Turing machine) that outputs the data. Its +self-contained size has two columns: + TOTAL = |program: frozen model + coder| + |tape: arithmetic-coded residual| +Turing-completeness does not help: for the SAME data, making the model bigger +(higher order) shrinks the tape but grows the program. You can shuffle bits +between the columns; you cannot push the SUM below the data's entropy / K. + +This script: + - builds a FROZEN order-k byte model on a TRAIN split (no adaptation at code time) + - compresses a held-out TEST split with a genuine Witten-Neal-Cleary coder + - ASSERTS lossless round-trip + - reports, per k: test bits/char, shipped model size (xz of the frozen table), + tape bytes, self-contained TOTAL, and the amortized (model-shared) number. + - baselines: xz / gzip on the same test bytes. +""" +import os, sys, pickle, lzma, gzip, numpy as np + +def load_corpus(root, cap): + buf = bytearray() + for dp, _, fs in os.walk(root): + for fn in fs: + if fn.endswith((".md", ".txt", ".lean", ".py")): + try: + with open(os.path.join(dp, fn), "rb") as f: + buf += f.read() + except Exception: + pass + if len(buf) >= cap: + return bytes(buf[:cap]) + return bytes(buf) + +root = sys.argv[1] if len(sys.argv) > 1 else "/home/allaun/SilverSight" +data = load_corpus(root, 1_800_000) +split = len(data) - 150_000 +train, test = data[:split], data[split:] +print(f"train {len(train)} bytes, test {len(test)} bytes (held out)\n") + +# ---- WNC 32-bit arithmetic coder over a supplied freq vector ---- +BITS = 32; TOP = (1 << BITS) - 1; HALF = 1 << 31; QTR = 1 << 30; TQ = 3 << 30 + +def freq_for(row_counts): + """Deterministic positive freq vector from frozen counts (both sides identical).""" + s = row_counts.sum() + if s > 0: + scaled = (row_counts * 4096.0 / s).astype(np.int64) + else: + scaled = np.zeros(256, dtype=np.int64) + return scaled + 1 # Laplace: every symbol codeable; total < 4096+256 << 2^16 + +def encode(test, model, k): + low, high, pending = 0, TOP, 0 + out = bytearray(); bitbuf = 0; nb = 0 + def put(b): + nonlocal bitbuf, nb + bitbuf = (bitbuf << 1) | b; nb += 1 + if nb == 8: out.append(bitbuf); bitbuf = 0; nb = 0 + def emit(b): + nonlocal pending + put(b) + while pending: put(b ^ 1); pending -= 1 + ctx = 0; mask = (1 << (8 * k)) - 1 if k else 0 + for sym in test: + row = model.get(ctx, ZERO) + f = freq_for(row); tot = int(f.sum()) + cum = int(f[:sym].sum()); fs = int(f[sym]) + rng = high - low + 1 + high = low + rng * (cum + fs) // tot - 1 + low = low + rng * cum // tot + while True: + if high < HALF: emit(0) + elif low >= HALF: emit(1); low -= HALF; high -= HALF + elif low >= QTR and high < TQ: pending += 1; low -= QTR; high -= QTR + else: break + low <<= 1; high = (high << 1) | 1 + ctx = ((ctx << 8) | sym) & mask if k else 0 + pending += 1; emit(1 if low >= QTR else 0) + if nb: out.append(bitbuf << (8 - nb)) + return bytes(out) + +def decode(enc, model, k, n): + low, high = 0, TOP; pos = 0; total = len(enc) * 8 + def getbit(): + nonlocal pos + b = (enc[pos >> 3] >> (7 - (pos & 7))) & 1 if pos < total else 0 + pos += 1; return b + code = 0 + for _ in range(BITS): code = (code << 1) | getbit() + out = bytearray(); ctx = 0; mask = (1 << (8 * k)) - 1 if k else 0 + for _ in range(n): + row = model.get(ctx, ZERO) + f = freq_for(row); tot = int(f.sum()) + rng = high - low + 1 + val = ((code - low + 1) * tot - 1) // rng + cs = np.cumsum(f); sym = int(np.searchsorted(cs, val, side='right')) + cum = int(cs[sym - 1]) if sym > 0 else 0; fs = int(f[sym]) + high = low + rng * (cum + fs) // tot - 1 + low = low + rng * cum // tot + while True: + if high < HALF: pass + elif low >= HALF: low -= HALF; high -= HALF; code -= HALF + elif low >= QTR and high < TQ: low -= QTR; high -= QTR; code -= QTR + else: break + low <<= 1; high = (high << 1) | 1; code = (code << 1) | getbit() + out.append(sym); ctx = ((ctx << 8) | sym) & mask if k else 0 + return bytes(out) + +ZERO = np.zeros(256, dtype=np.int64) + +def build_frozen(train, k): + model = {}; ctx = 0; mask = (1 << (8 * k)) - 1 if k else 0 + for sym in train: + if ctx not in model: model[ctx] = np.zeros(256, dtype=np.int64) + model[ctx][sym] += 1 + ctx = ((ctx << 8) | sym) & mask if k else 0 + return model + +def model_ship_bytes(model): + """Honest shipped size: frozen table serialized, then xz'd (real bytes).""" + packed = {c: row.astype(np.uint16).tobytes() for c, row in model.items()} + return len(lzma.compress(pickle.dumps(packed), preset=6)) + +xz_test = len(lzma.compress(test, preset=9)) +gz_test = len(gzip.compress(test, 9)) +print(f"baselines on test: raw {len(test)} gzip {gz_test} xz {xz_test} " + f"({xz_test*8/len(test):.3f} b/char)\n") +print(f"{'order k':>7} {'test b/char':>11} {'tape B':>9} {'model B(xz)':>12} " + f"{'TOTAL self':>11} {'amortized':>10} {'lossless':>9}") +print("-" * 76) + +for k in [0, 1, 2, 3]: + model = build_frozen(train, k) + enc = encode(test, model, k) + dec = decode(enc, model, k, len(test)) + ok = dec == test + tape = len(enc) + msz = model_ship_bytes(model) + bpc = tape * 8 / len(test) + total = msz + tape + print(f"{k:>7} {bpc:>11.3f} {tape:>9} {msz:>12} {total:>11} {tape:>10} " + f"{'PASS' if ok else 'FAIL':>9}") + +print("\nTAPE shrinks with k (better prediction); MODEL grows with k (more contexts).") +print("TOTAL self-contained = the Hutter number (model must ship). AMORTIZED = tape") +print("only, valid ONLY when both ends already share the frozen model.") diff --git a/scripts/compression/gw_honest_test.py b/scripts/compression/gw_honest_test.py new file mode 100644 index 00000000..cc1eab3f --- /dev/null +++ b/scripts/compression/gw_honest_test.py @@ -0,0 +1,98 @@ +""" +Honest test of the "parametric/16D gives 583x on a GW ringdown" claim. + +Design (steelman for the parametric method): +- signal = sum of damped sinusoids (a real ringdown model, few QNM modes) +- add detector noise at several SNRs (clean -> realistic loud -> weak) +- quantize to int16 (real strain/ADC is finite precision) -> THIS is the data +- everything is LOSSLESS: each method must reproduce the int16 stream exactly, + so its size = model params + entropy of the residual it leaves behind. +- the parametric coder is HANDED THE TRUE (A, tau, f, phi) — the best case it + could ever achieve (no fit error). If it still can't beat LPC once noise is + present, the compression was living entirely in the zero-noise self-fit. + +Baselines on the same int16 array: + raw 16 b/s, order-0 entropy, gzip -9, xz -9, and LPC (order-p linear predictor, + residual entropy) — LPC is the 50-year-old codec that already does + "few coefficients + residual" for decaying/oscillatory signals. +""" +import numpy as np, gzip, lzma, math + +rng = np.random.default_rng(0) +fs = 4096.0 +N = 2000 +t = np.arange(N) / fs + +# ---- true ringdown: 2 QNM modes (f Hz, tau s, amp, phase) ---- +MODES = [(250.0, 0.004, 1.0, 0.0), (500.0, 0.002, 0.35, 1.1)] +def model(t): + s = np.zeros_like(t) + for f, tau, A, ph in MODES: + s += A * np.exp(-t / tau) * np.cos(2 * np.pi * f * t + ph) + return s +clean = model(t) +peak = np.max(np.abs(clean)) + +def ideal_bits_per_sample(residual_int): + """entropy (bits/sample) an arithmetic coder would reach on integer residuals.""" + vals, counts = np.unique(residual_int, return_counts=True) + p = counts / counts.sum() + return float(-(p * np.log2(p)).sum()) + +def quantize_int16(x): + scale = 30000.0 / peak + return np.round(x * scale).astype(np.int64), scale + +def size_gzip(int16): + return len(gzip.compress(int16.astype('6}{'order0':>8}{'gzip':>7}{'xz':>7}{'LPC-8':>8}{'PARAM*':>8} note") +print("-" * 86) + +regimes = [ + ("clean (sigma=0)", 0.0), + ("60 dB (sigma=peak/1e3)", peak / 1000), + ("40 dB (sigma=peak/1e2)", peak / 100), + ("30 dB realistic loud", peak / 31.6), + ("20 dB weak", peak / 10), +] +for name, sigma in regimes: + noisy = clean + (rng.standard_normal(N) * sigma if sigma > 0 else 0.0) + xi, scale = quantize_int16(noisy) + raw = 16.0 + h0 = ideal_bits_per_sample(xi) + g = size_gzip(xi); z = size_xz(xi) + lpc = lpc_bits(xi, 8) + par = parametric_bits(xi, scale) + winner = min(("gzip", g), ("xz", z), ("LPC", lpc), ("PARAM", par), key=lambda kv: kv[1]) + print(f"{name:<22}{raw:6.1f}{h0:8.2f}{g:7.2f}{z:7.2f}{lpc:8.2f}{par:8.2f} " + f"ratio(raw/best)={raw/winner[1]:5.1f}x via {winner[0]}") + +print("\n* PARAM is handed the TRUE mode parameters (upper bound on any 16D/braid fit).") +print("Lossless: every column reproduces the int16 stream exactly; size = params + residual entropy.") diff --git a/scripts/compression/mass_number_compress.py b/scripts/compression/mass_number_compress.py new file mode 100644 index 00000000..08c942aa --- /dev/null +++ b/scripts/compression/mass_number_compress.py @@ -0,0 +1,78 @@ +""" +Test the strong claim: "encode data as a semantic mass number A(H), reconstruct +losslessly" as a COMPRESSOR. Faithful to the OTOM spec's own readings: + - nuclide address / "10-adic residue" reading -> the message IS one big number + - "observation = residue class mod ℓ" -> mixed-radix over the alphabet ℓ +Encoding a string as a single integer is BASE CONVERSION, which is a bijection: +information-preserving, therefore it cannot reduce bits below what the digit +radices already imply. Measured three ways, all lossless round-trip: + + M1 base-256 positional : A(H) = int.from_bytes(data) (uniform radix 256) + M2 mixed-radix over Σ : digits = symbol ranks, radix |Σ| (residue-mod-ℓ reading) + M3 frequency-weighted : radix_i = 1/p(sym_i) == arithmetic coding (best case) + +M1/M2 carry NO probability model -> ceil(n·log2 radix) bits, a pure repack. +M3 reaches entropy but the radices ARE the model, which must ship (conservation). +Baseline: xz -9 (order-N + long-range matching) on the same bytes. +""" +import os, sys, math, lzma, numpy as np + +def load(root, cap): + buf = bytearray() + for dp, _, fs in os.walk(root): + for fn in fs: + if fn.endswith((".md", ".txt", ".lean", ".py")): + try: + with open(os.path.join(dp, fn), "rb") as f: buf += f.read() + except Exception: pass + if len(buf) >= cap: return bytes(buf[:cap]) + return bytes(buf) + +data = load(sys.argv[1] if len(sys.argv) > 1 else "/home/allaun/SilverSight", 100_000) +n = len(data) +print(f"test data: {n} bytes\n") + +# ---- M1: base-256 positional big number (the literal "mass number of the doc") ---- +A = int.from_bytes(data, "big") +m1_bytes = (A.bit_length() + 7) // 8 +back = A.to_bytes(m1_bytes, "big").rjust(n, b"\x00") # leading-zero safe +m1_ok = back == data +# to be fully lossless you must also store n (leading zeros) -> +a few bytes; negligible + +# ---- M2: mixed-radix over the observed alphabet Σ (residue-mod-ℓ reading) ---- +alphabet = sorted(set(data)); L = len(alphabet) +rank = {b: i for i, b in enumerate(alphabet)} +# real bijection round-trip on a slice (big-int divmod is O(len^2) — prove on 3KB) +sl = data[:3000] +val = 0 +for b in sl: val = val * L + rank[b] # encode slice as one mixed-radix integer +digits = [] +v = val +for _ in range(len(sl)): v, r = divmod(v, L); digits.append(r) +dec = bytes(alphabet[r] for r in reversed(digits)) +m2_ok = dec == sl +m2_bits_per_char = math.log2(L) # exact: ceil(n·log2 L)/n -> log2 L +m2_bytes_full = math.ceil(n * math.log2(L) / 8) + L # +L to ship the alphabet table + +# ---- M3: frequency-weighted radix == order-0 arithmetic coding (ideal rate) ---- +counts = np.bincount(np.frombuffer(data, np.uint8), minlength=256).astype(float) +p = counts[counts > 0] / n +H0 = float(-(p * np.log2(p)).sum()) # order-0 entropy = M3 ideal bits/char +m3_bytes = math.ceil(H0 * n / 8) + L * 2 # + shipped frequency model (~2B/symbol) + +# ---- baseline ---- +xz = len(lzma.compress(data, preset=9)); xz_bpc = xz * 8 / n + +print(f"{'method':<34}{'bits/char':>10}{'bytes':>10}{'ratio':>8} lossless") +print("-" * 74) +print(f"{'M1 base-256 mass number':<34}{8.0:>10.3f}{m1_bytes:>10}{n/m1_bytes:>8.2f} " + f"{'PASS' if m1_ok else 'FAIL'} (= the data, no gain)") +print(f"{'M2 mixed-radix over Σ (|Σ|=%d)'%L:<34}{m2_bits_per_char:>10.3f}{m2_bytes_full:>10}" + f"{n/m2_bytes_full:>8.2f} {'PASS' if m2_ok else 'FAIL'} (flat order-0 only)") +print(f"{'M3 freq-weighted (=arith, order0)':<34}{H0:>10.3f}{m3_bytes:>10}{n/m3_bytes:>8.2f}" + f" n/a (needs shipped model)") +print(f"{'xz -9 (order-N + matching)':<34}{xz_bpc:>10.3f}{xz:>10}{n/xz:>8.2f} PASS") +print("\nM1/M2 carry no model: base conversion is a bijection, cannot beat the radix.") +print("M3 reaches entropy only by BEING the model (radices=1/p) -> conservation.") +print("xz wins because it exploits order-N structure + long-range repeats, which no") +print("single 'mass number' of the raw stream can see.") diff --git a/scripts/compression/spectral_vs_empirical.py b/scripts/compression/spectral_vs_empirical.py new file mode 100644 index 00000000..46a93d8d --- /dev/null +++ b/scripts/compression/spectral_vs_empirical.py @@ -0,0 +1,107 @@ +""" +Decisive test: can a low-rank / spectral summary of the order-1 byte +transition matrix beat the raw empirical counts ("KV cache") at prediction? + +Claim: NO. The empirical conditional distribution is the MLE — it MINIMIZES +cross-entropy on its own data. Any low-rank reconstruction (the regime the +characteristic polynomial / eigen-summary lives in) has cross-entropy >= the +empirical order-1 entropy. So the polynomial cannot add predictive information +over the count matrix it is derived from. It can only match (full rank) or hurt. + +Also report order-0 and order-2 to show the REAL lever is context DEPTH, not rank. +""" +import os, sys, math +import numpy as np + +def load_corpus(root, cap=1_500_000, max_files=3000): + buf = bytearray() + seen = 0 + for dirpath, _, files in os.walk(root): + for fn in files: + if not (fn.endswith(".md") or fn.endswith(".txt")): + continue + seen += 1 + if seen > max_files: + break + try: + with open(os.path.join(dirpath, fn), "rb") as f: + buf += f.read() + except Exception: + pass + if len(buf) >= cap: + return bytes(buf[:cap]) + if seen > max_files or len(buf) >= cap: + break + return bytes(buf[:cap]) + +def order0_bits(data): + c = np.bincount(np.frombuffer(data, dtype=np.uint8), minlength=256).astype(float) + p = c / c.sum() + nz = p > 0 + return float(-(p[nz] * np.log2(p[nz])).sum()) + +def order1_counts(data): + a = np.frombuffer(data, dtype=np.uint8).astype(np.int64) + idx = a[:-1] * 256 + a[1:] + C = np.bincount(idx, minlength=256*256).reshape(256, 256).astype(float) + return C + +def cond_xent(C, Q): + # cross-entropy of the data (weighted by true counts C) under model Q(j|i) + N = C.sum() + Q = np.clip(Q, 1e-12, None) + return float(-(C * np.log2(Q)).sum() / N) + +def order1_empirical_bits(C): + rs = C.sum(axis=1, keepdims=True) + P = C / np.clip(rs, 1e-12, None) + return cond_xent(C, P), P + +def lowrank_bits(C, k): + # rank-k approximation of the conditional matrix, clipped & renormalized + rs = C.sum(axis=1, keepdims=True) + P = C / np.clip(rs, 1e-12, None) + U, S, Vt = np.linalg.svd(P, full_matrices=False) + Pk = (U[:, :k] * S[:k]) @ Vt[:k] + Pk = np.clip(Pk, 0, None) + Pk = Pk / np.clip(Pk.sum(axis=1, keepdims=True), 1e-12, None) + return cond_xent(C, Pk) + +def order2_bits(data): + a = np.frombuffer(data, dtype=np.uint8).astype(np.int64) + if len(a) < 3: + return float("nan") + ctx = a[:-2] * 256 + a[1:-2+1] # pairs (b0,b1) + ctx = a[:-2] * 256 + a[1:-1] + nxt = a[2:] + from collections import defaultdict + cnt = defaultdict(lambda: np.zeros(256)) + for c, n in zip(ctx.tolist(), nxt.tolist()): + cnt[c][n] += 1 + N = len(nxt) + H = 0.0 + for c, row in cnt.items(): + rs = row.sum() + p = row / rs + nz = p > 0 + H += (rs / N) * float(-(p[nz] * np.log2(p[nz])).sum()) + return H + +def main(): + root = sys.argv[1] if len(sys.argv) > 1 else "/home/allaun/Research Stack/6-Documentation" + data = load_corpus(root) + print(f"corpus: {len(data)} bytes from {root}\n") + h0 = order0_bits(data) + C = order1_counts(data) + h1, _ = order1_empirical_bits(C) + print(f"order-0 entropy : {h0:6.3f} bits/byte") + print(f"order-1 empirical (KV cache) : {h1:6.3f} bits/byte <-- MLE optimum for order-1") + print(f"order-2 empirical (deeper ctx) : {order2_bits(data):6.3f} bits/byte <-- context DEPTH helps\n") + print("rank-k reconstruction of the order-1 conditional matrix") + print("(the regime the char-poly / eigen-summary lives in):") + for k in [1, 2, 4, 8, 16, 32, 64, 128, 255, 256]: + hk = lowrank_bits(C, k) + flag = " >= order-1 (worse or equal)" if hk >= h1 - 1e-9 else " << BEATS order-1 (!)" + print(f" rank {k:3d} : {hk:6.3f} bits/byte{flag}") + +main() diff --git a/scripts/compression/superposition_recovery.py b/scripts/compression/superposition_recovery.py new file mode 100644 index 00000000..0ef8d452 --- /dev/null +++ b/scripts/compression/superposition_recovery.py @@ -0,0 +1,75 @@ +""" +"How LLMs drop data recoverably" = superposition = compressed sensing. +Store an N-feature vector in d < N dimensions via a random projection Phi (d x N); +recover it. This is exactly what a residual stream does: pack many features into +few dims with near-orthogonal directions, read them back by correlation. + +Claim under test: recovery is near-LOSSLESS iff the vector is SPARSE (few features +active). Past the compressed-sensing bound (k ~ d/(2 log N)) recovery COLLAPSES — +"recoverable" becomes "lost". So an LLM drops N->d recoverably ONLY to the extent +the signal is sparse/redundant; on dense/random data it cannot recover. Same +entropy wall, dressed as dimensionality reduction. + +Two decoders: + OMP - greedy orthogonal matching pursuit (pick most-correlated feature, repeat); + given the TRUE k as its budget = best case for the model. + MF - matched filter: x_hat = top-k of Phi^T y (the raw dot-product readout). +""" +import numpy as np +rng = np.random.default_rng(0) + +N, d = 1024, 128 # 1024 features packed into a 128-dim stream (8x squeeze) +TRIALS = 20 + +def make_phi(): + Phi = rng.standard_normal((d, N)) + return Phi / np.linalg.norm(Phi, axis=0, keepdims=True) # unit-norm columns + +def omp(Phi, y, k): + r = y.copy(); supp = [] + for _ in range(k): + j = int(np.argmax(np.abs(Phi.T @ r))) + if j in supp: break + supp.append(j) + Ps = Phi[:, supp] + xs, *_ = np.linalg.lstsq(Ps, y, rcond=None) + r = y - Ps @ xs + x = np.zeros(N); + if supp: x[supp] = xs + return x, supp + +def rel_err(a, b): + return float(np.linalg.norm(a - b) / (np.linalg.norm(b) + 1e-12)) + +bound = d / (2 * np.log(N)) +print(f"N={N} features -> d={d} dims ({N/d:.0f}x squeeze). " + f"CS bound k* ~ d/(2 ln N) = {bound:.1f}\n") +print(f"{'k active':>9}{'stored N->d':>12}{'OMP rel-err':>13}{'OMP supp-exact':>15}" + f"{'MF rel-err':>12} recover?") +print("-" * 82) + +for k in [1, 2, 4, 8, 16, 24, 32, 48, 64, 96, 128, 256, 512, 1024]: + oe = me = se = 0.0 + for _ in range(TRIALS): + Phi = make_phi() + x = np.zeros(N) + supp_true = rng.choice(N, size=k, replace=False) + x[supp_true] = rng.standard_normal(k) + y = Phi @ x # the compressed d-dim code + xo, supp = omp(Phi, y, k) + oe += rel_err(xo, x) + se += len(set(supp) & set(supp_true.tolist())) / k + # matched filter: top-k by |correlation|, LS on that support + corr = np.abs(Phi.T @ y) + top = np.argsort(corr)[-k:] + xm = np.zeros(N) + xm[top], *_ = np.linalg.lstsq(Phi[:, top], y, rcond=None) + me += rel_err(xm, x) + oe/=TRIALS; me/=TRIALS; se/=TRIALS + verdict = "LOSSLESS-ish" if oe < 1e-6 else ("degrading" if oe < 0.3 else "LOST") + print(f"{k:>9}{d/N:>11.2f}x{oe:>13.2e}{se*100:>14.0f}%{me:>12.2e} {verdict}") + +print(f"\nSmall k: 8x squeeze recovered to ~machine-zero (sparse -> lossless-ish).") +print(f"Past k* ~ {bound:.0f}: OMP support-recovery breaks, error -> O(1) = LOST.") +print(f"k=N (dense/random): error ~1.0 — a random vector CANNOT be dropped-and-recovered.") +print(f"=> 'recoverable' is bought entirely by sparsity; it is not free compression.")