math(fundamental): FUNDAMENTAL_CHENTSOV_RECONSTRUCTION.md

docs/fundamental_math/FUNDAMENTAL_CHENTSOV_RECONSTRUCTION.md

@@ -0,0 +1,295 @@
+# Chentsov's Theorem — Fundamental Reconstruction
+## From Verified Math to Outputs (No Code, No Lean, Just Proof)
+
+---
+
+## 1. THE ACTUAL THEOREM (Chentsov 1972, Amari 1985)
+
+**Theorem (Chentsov).** Let Δₙ = {p ∈ ℝⁿ⁺¹ : pᵢ > 0, Σpᵢ = 1} be the open
+probability simplex. Let g be a Riemannian metric on Δₙ such that for every
+Markov morphism (stochastic map) T: Δₙ → Δₘ induced by a sufficient statistic,
+the map T is a contraction:
+
+    ‖T(v)‖_{g(T(p))} ≤ ‖v‖_{g(p)}  for all p ∈ Δₙ, v ∈ T_pΔₙ
+
+Then g is unique up to scalar multiple and is the Fisher metric:
+
+    g_p(u,v) = c · Σᵢ (uᵢ vᵢ / pᵢ)    for some c > 0
+
+**What this actually says:** Any metric that respects sufficient statistic
+reduction (i.e., coarse-graining doesn't create information) MUST be the Fisher
+metric. The condition is INFORMATION MONOTONICITY.
+
+---
+
+## 2. WHAT WE CAN DERIVE (rigorously)
+
+### 2.1 The Fisher Metric on Δ₇ (our 8-state system)
+
+For p = (p₁,...,p₈) ∈ Δ₇ with pᵢ > 0, Σpᵢ = 1:
+
+    g_p(u,v) = Σᵢ₌₁⁸ (uᵢ vᵢ / pᵢ)
+
+**Proof this is a metric:**
+- Symmetric: obvious from formula ✓
+- Bilinear: obvious ✓
+- Positive definite: g_p(v,v) = Σᵢ vᵢ²/pᵢ ≥ 0, with equality iff vᵢ=0 ∀i ✓
+- Tangent compatibility: for v ∈ T_pΔ₇, Σvᵢ = 0, so g_p is well-defined ✓
+
+### 2.2 The √p Embedding into S⁷
+
+Define the map:
+
+    φ: Δ₇ → S⁷,    φ(p) = (√p₁, √p₂, ..., √p₈)
+
+**Claim:** φ is an isometric embedding (up to factor 4).
+
+**Proof:**
+
+Let c(t) be a curve in Δ₇ with c(0) = p, ċ(0) = v.
+Then γ(t) = φ(c(t)) is a curve in S⁷.
+
+    γ̇(t) = (v₁/(2√c₁), ..., v₈/(2√c₈))
+
+The Euclidean metric on S⁷ ⊂ ℝ⁸ pulls back to:
+
+    ⟨γ̇, γ̇⟩ = Σᵢ vᵢ² / (4pᵢ) = ¼ · g_p(v,v)
+
+Therefore the Fisher metric on Δ₇ is 4 times the pullback of the
+round metric on S⁷. ∎
+
+**This is the key geometric fact.** Our 8-state system lives naturally on
+S⁷ with the round metric.
+
+### 2.3 The Fisher Distance
+
+For p, q ∈ Δ₇:
+
+    d_F(p,q) = 2 · arccos(Σᵢ √(pᵢqᵢ))
+
+**Proof:** Since φ is an isometry (up to factor 4), geodesic distance on
+Δ₇ equals chordal distance on S⁷:
+
+    d_F(p,q) = 2 · d_{S⁷}(φ(p), φ(q))
+              = 2 · arccos(⟨φ(p), φ(q)⟩)
+              = 2 · arccos(Σᵢ √(pᵢqᵢ))
+
+∎
+
+This is the **Bhattacharyya arc distance** — a known, well-defined quantity.
+
+### 2.4 Geodesics are Great Circles
+
+**Claim:** Fisher geodesics on Δ₇ map to great circles on S⁷.
+
+**Proof:** φ pulls back the Levi-Civita connection of the round metric on
+S⁷. Geodesics of the round metric are great circles. By isometry, Fisher
+geodesics are their preimages — arcs of great circles projected back to
+Δ₇ via the square map. ∎
+
+---
+
+## 3. IMPLICATIONS FOR THE 8-STATE SYSTEM
+
+### 3.1 What is the State Space?
+
+The 8 Hachimoji states Λ = {A, B, C, G, P, S, T, Z} label the basis of ℝ⁸.
+A probability distribution p ∈ Δ₇ assigns to each state a frequency pᵢ.
+
+**The classification problem:** Given an equation string, produce a point
+p(equation) ∈ Δ₇ such that semantically similar equations are close in
+the Fisher metric.
+
+### 3.2 How to Map Equations to Δ₇
+
+From the byte-level co-occurrence analysis:
+
+Given equation string E, compute:
+- fᵢ = count of byte-class i in E (i = 0,...,7 from the 8 buckets)
+- pᵢ = fᵢ / Σⱼ fⱼ  (normalization to Δ₇)
+
+**This is a valid map E → Δ₇ provided:**
+1. At least one byte appears (non-empty input) ✓
+2. fᵢ ≥ 0 (obvious) ✓
+3. Σfᵢ > 0 (obvious for non-empty) ✓
+
+**Question:** Is this map injective? **No.** Different equations can have
+the same byte frequencies. This is the collision problem identified in the
+adversarial review.
+
+**What CAN we claim:** The map is well-defined and Lipschitz-continuous:
+if two equations differ by one character, their Fisher distance is bounded.
+
+**Proof of Lipschitz:**
+Let E, E' differ by one character in bucket k.
+p(E') = p(E) with bucket k count incremented by 1.
+
+    |pᵢ(E') - pᵢ(E)| ≤ 2/(n+1)  for all i
+
+The Fisher distance between nearby distributions on Δₙ is bounded by
+the ℓ² distance on S⁷, which is O(1/√n) for large n. ∎
+
+### 3.3 The Φ-Corkscrew Lives on S⁷
+
+The golden spiral map:
+
+    f: ℕ → ℝ²,  f(n) = (√n · cos(nψ), √n · sin(nψ))
+    ψ = 2π/φ²,  φ = (1+√5)/2
+
+**Claim:** f is injective because ψ/2π = 1/φ² is irrational.
+
+**Proof:** Suppose f(m) = f(n) with m > n. Then:
+1. √m = √n  (from radial coordinate) → m = n ✓
+2. Or: m ≠ n but cos(mψ) = cos(nψ) and sin(mψ) = sin(nψ)
+   This requires (m-n)ψ ∈ 2πℤ, i.e., ψ/2π = k/(m-n) ∈ ℚ.
+   But ψ/2π = 1/φ² and φ² = φ+1 is irrational, contradiction. ∎
+
+**Connecting to S⁷:** For a spectral coefficient vector c = (c₀,...,c₈),
+pack into a single integer via phinary, then use f to place on the spiral.
+The spiral index n is a natural number that can be mapped to S⁷ via:
+
+    c̃ = (√(n₀/n), ..., √(n₇/n)) where n = Σnᵢ
+
+This gives a point on S⁷, hence on Δ₇ via squaring.
+
+**What this gives us:** A deterministic, injective map from spectral
+coefficients to Δ₇. Injectivity comes from the irrational rotation.
+
+---
+
+## 4. WHAT THE ALGORITHM SHOULD COMPUTE (derived from math)
+
+### 4.1 Required Outputs
+
+Given an equation string E, the system MUST produce:
+
+**Output 1: Probability vector** p(E) ∈ Δ₇
+- Computed from byte-class frequencies
+- Normalized to sum to 1
+- Strictly positive (add small ε to avoid boundary)
+
+**Output 2: Fisher distance** d_F(p(E), p_ref) for reference points
+- Computed as 2·arccos(Σ√(pᵢ(E)·p_ref,ᵢ))
+- Reference points are known equations in the corpus
+
+**Output 3: S⁷ coordinates** φ(p(E)) = (√p₁,...,√p₈)
+- These live on the unit sphere in ℝ⁸
+- Geodesic walks are great circle arcs
+
+**Output 4: Spiral index** n(E) ∈ ℕ
+- Computed from the phinary packing of spectral features
+- Deterministic and injective (proven above)
+
+### 4.2 What "Classification" Means Geometrically
+
+The chaos game converges to a point in Δ₇. Two equations are "similar"
+if their convergence points are close in Fisher distance.
+
+**The classification boundary:** Given training points {p(E₁),...,p(Eₖ)}
+with labels, the decision boundary on Δ₇ is the Voronoi decomposition
+under the Fisher metric.
+
+**This is well-defined:** The Voronoi cells of a finite point set in a
+Riemannian manifold partition the manifold. Fisher geodesics are unique
+(locally) because the metric is positive definite.
+
+### 4.3 What "Compression" Means Geometrically
+
+The eigensolid is a fixed point of the braid crossing operator on Δ₇.
+At the fixed point, the state is stable under coarse-graining.
+
+**The compression ratio:** The number of bits needed to specify a point
+to precision ε on Δ₇ is approximately the Kolmogorov complexity of the
+spectral features that map to that point.
+
+**For the Φ-corkscrew:** The spiral index n requires ~log₂(n) bits.
+The spectral coefficients require 9 × 16 = 144 bits in Q16_16.
+The compression ratio is the ratio of raw input size to the spiral
+index bit count.
+
+---
+
+## 5. VERIFICATION CRITERIA (what a correct implementation must satisfy)
+
+### Criterion 1: Metric properties
+For any implementation computing d_F(p,q):
+- d_F(p,p) = 0                              (identity)
+- d_F(p,q) = d_F(q,p) > 0 for p ≠ q        (symmetry, positivity)
+- d_F(p,q) ≤ d_F(p,r) + d_F(r,q)           (triangle inequality)
+
+### Criterion 2: S⁷ embedding
+For p ∈ Δ₇:
+- ‖φ(p)‖₂ = 1                               (on the sphere)
+- φ(p)ᵢ = √pᵢ ≥ 0                           (nonnegative coordinates)
+- Σᵢ φ(p)ᵢ² = 1                             (unit normalization)
+
+### Criterion 3: Φ-corkscrew injectivity
+For n ≠ m:
+- f(n) ≠ f(m)                               (proven via irrationality)
+
+### Criterion 4: Probability preservation
+For any equation E:
+- p(E)ᵢ ≥ 0                                 (nonnegative)
+- Σᵢ p(E)ᵢ = 1                              (normalized)
+- p(E)ᵢ ≤ 1 for all i                       (bounded)
+
+### Criterion 5: Reference consistency
+For the Erdős-Rényi test case G(20, 1/20):
+- The Laplacian eigenvalues are real and nonnegative ✓
+- The dominant eigenvalue equals the spectral radius ✓
+- The spectral gap equals λ₁ - λ₂             ✓
+- These are standard spectral graph theory results
+
+---
+
+## 6. WHAT STILL NEEDS PROOF
+
+### Open Problem 1: Does the chaos game converge?
+The chaos game with IFS contractions on Δ₇ converges to a unique attractor
+if the IFS is contractive in the Fisher metric. **This requires proof**
+that the specific IFS used is indeed contractive.
+
+### Open Problem 2: Is the classification correct?
+Similar equations mapping to nearby points in Δ₇ is a **hypothesis**, not
+a theorem. The adversarial review found collisions — equations with
+different semantics but identical byte frequencies. This needs either:
+- (a) A larger feature set that breaks collisions, OR
+- (b) Acceptance that classification is approximate
+
+### Open Problem 3: What does the eigensolid compress TO?
+The fixed point of braid crossing on Δ₇ exists under certain conditions.
+**Proving** these conditions and characterizing the fixed point requires
+showing the crossing operator is a contraction on a suitable subset of Δ₇.
+
+---
+
+## 7. SUMMARY: THE MATH PRIMITIVES WE STAND ON
+
+| Primitive | Status | Reference |
+|-----------|--------|-----------|
+| Chentsov's theorem | **Proven (1972)** | Chentsov, N. N. (1972). Statistical Decision Rules and Optimal Inference |
+| Fisher metric formula | **Proven** | Amari, S. (1985). Differential-Geometrical Methods in Statistics |
+| S⁷ embedding | **Proven above** | √p map, explicit calculation |
+| Fisher distance | **Proven above** | Bhattacharyya arc distance |
+| Great circle geodesics | **Proven above** | Isometry pulls back S⁷ geodesics |
+| Φ-corkscrew injectivity | **Proven above** | ψ/2π irrational |
+| Byte-frequency map E→Δ₇ | **Well-defined** | Lipschitz, not injective |
+| Voronoi classification | **Well-defined** | Standard Riemannian geometry |
+| Chaos game convergence | **OPEN** | Needs IFS contraction proof |
+| Semantic collision-free | **FALSE** | Adversarial review found counterexamples |
+
+---
+
+## 8. THE DECISION
+
+Before writing ANY code, we must:
+
+1. ✅ Accept Chentsov's theorem as given (1972 proven result)
+2. ✅ Derive Fisher metric on Δ₇ explicitly (done above)
+3. ✅ Prove the S⁷ embedding (done above)
+4. ✅ Prove Φ-corkscrew injectivity (done above)
+5. ❓ Decide: do we need semantic features to break collisions?
+6. ❓ Prove: chaos game IFS is contractive on Δ₇
+7. ❓ Characterize: the eigensolid fixed point
+
+Items 5-7 are the gates. No code passes these gates until the math is done.