Archive HACHIMOJI_DNA_SYNTAX.md

archive/2026-07-02/docs/HACHIMOJI_DNA_SYNTAX.md

@@ -0,0 +1,477 @@
+# Hachimoji DNA Encoding Syntax — Formal Specification
+
+**Version:** 1.0
+**Date:** 2026-06-23
+**Status:** Active
+**Purpose:** Computational substrate for manifold/QUBO/eigenvalue work.
+
+---
+
+## 1. Alphabet
+
+### 1.1 Base Set
+
+Eight bases, ordered by ASCII value for monotone lexicographic sorting:
+
+| Index | Base | ASCII | Phase | Binary (3-bit) |
+|-------|------|-------|-------|----------------|
+| 0 | A | 0x41 | 0° | 000 |
+| 1 | B | 0x42 | 45° | 001 |
+| 2 | C | 0x43 | 90° | 010 |
+| 3 | G | 0x47 | 135° | 011 |
+| 4 | P | 0x50 | 180° | 100 |
+| 5 | S | 0x53 | 225° | 101 |
+| 6 | T | 0x54 | 270° | 110 |
+| 7 | Z | 0x5A | 315° | 111 |
+
+### 1.2 Ordering Axiom
+
+```
+A < B < C < G < P < S < T < Z
+```
+
+This ordering is **canonical** and **immutable**. It satisfies:
+
+1. **ASCII order = index order.** `ord(A) < ord(B) < ... < ord(Z)`.
+2. **Index order = lexicographic rank.** For any two sequences of equal length, `s₁ < s₂` (lexicographic) if and only if `dna_to_int(s₁) < dna_to_int(s₂)`.
+3. **Monotone encoding.** Assigning sequences by increasing integer rank produces lexicographically sorted output.
+
+**Proof:** The bases are chosen such that their ASCII codes are in ascending order: 0x41 < 0x42 < 0x43 < 0x47 < 0x50 < 0x53 < 0x54 < 0x5A. Since lexicographic comparison proceeds character-by-character using ASCII ordering, and our index ordering matches ASCII ordering, integer rank ordering implies lexicographic ordering. ∎
+
+---
+
+## 2. Integer ↔ DNA Conversion
+
+### 2.1 Encoding (integer → DNA)
+
+```
+int_to_dna(value: int, length: int) → string
+```
+
+Converts a non-negative integer to a fixed-length DNA sequence using base-8 representation, most-significant digit first.
+
+```
+Algorithm:
+  seq = []
+  for i in 1..length:
+      seq.append(BASES[value mod 8])
+      value = value ÷ 8
+  return reverse(seq)
+```
+
+**Constraints:**
+- `value ≥ 0`
+- `length ≥ 1`
+- `value < 8^length` (otherwise the sequence cannot represent the value)
+
+**Examples:**
+```
+int_to_dna(0, 3)   → "AAA"
+int_to_dna(1, 3)   → "AAB"
+int_to_dna(7, 3)   → "AAZ"
+int_to_dna(8, 3)   → "ABA"
+int_to_dna(511, 3) → "ZZZ"
+```
+
+### 2.2 Decoding (DNA → integer)
+
+```
+dna_to_int(sequence: string) → int
+```
+
+Converts a DNA sequence back to its integer value.
+
+```
+Algorithm:
+  value = 0
+  for each base b in sequence:
+      value = value × 8 + BASE_TO_INDEX[b]
+  return value
+```
+
+**Examples:**
+```
+dna_to_int("AAA") → 0
+dna_to_int("AAB") → 1
+dna_to_int("ABA") → 8
+dna_to_int("ZZZ") → 511
+```
+
+### 2.3 Roundtrip Axiom
+
+```
+∀ value ∈ [0, 8^length):
+    dna_to_int(int_to_dna(value, length)) = value
+```
+
+### 2.4 Lexicographic Ordering Axiom
+
+```
+∀ v₁, v₂ ∈ [0, 8^length):
+    v₁ < v₂ ⟺ int_to_dna(v₁, length) < int_to_dna(v₂, length)
+    (where < on strings is lexicographic comparison)
+```
+
+---
+
+## 3. Symbol Encoding
+
+### 3.1 Chunks
+
+A **chunk** is a contiguous group of bytes treated as a single symbol.
+
+| Chunk size | Range | Symbols | Bases needed |
+|---|---|---|---|
+| 1 byte | 0x00–0xFF | 256 | 3 (8³ = 512 ≥ 256) |
+| 2 bytes | 0x0000–0xFFFF | 65,536 | 6 (8⁶ = 262,144 ≥ 65,536) |
+| n unique | — | n | ⌈log₈(n)⌉ |
+
+### 3.2 Bases Per Symbol
+
+```
+bases_needed(n_symbols: int) → int
+    length = 1
+    while 8^length < n_symbols:
+        length += 1
+    return length
+```
+
+---
+
+## 4. Monotone LUT
+
+### 4.1 Definition
+
+A **monotone LUT** is a bijection:
+
+```
+L: {0, 1, ..., n-1} → DNA_sequences × Solutions × Energies
+```
+
+such that:
+
+```
+∀ i < j: L(i).energy ≤ L(j).energy
+```
+
+and:
+
+```
+∀ i < j: L(i).sequence < L(j).sequence  (lexicographic)
+```
+
+### 4.2 Construction
+
+```
+build_monotone_lut(solutions, energies) → LUT
+
+Algorithm:
+  1. Sort solutions by energy (ascending)
+  2. Assign DNA sequences in order:
+       rank 0 → int_to_dna(0, seq_len)
+       rank 1 → int_to_dna(1, seq_len)
+       ...
+       rank n-1 → int_to_dna(n-1, seq_len)
+  3. Return LUT: sequence → (solution, energy)
+```
+
+### 4.3 Properties
+
+1. **Monotonicity.** Lexicographic sort of sequences = energy sort of solutions.
+2. **Completeness.** Every solution has exactly one DNA sequence.
+3. **Injectivity.** Every DNA sequence maps to at most one solution.
+4. **Minimal encoding.** The optimal solution always maps to `AAA...A` (the lexicographically smallest sequence).
+
+### 4.4 Verification
+
+```
+verify_monotone(lut) → (bool, float)
+
+  is_monotone = (sort_by_sequence(lut) == sort_by_energy(lut))
+  rank_correlation = spearman(sequence_indices, energy_ranks)
+  return (is_monotone, rank_correlation)
+```
+
+A valid monotone LUT has `is_monotone = true` and `rank_correlation = 1.0`.
+
+---
+
+## 5. File Formats
+
+### 5.1 DNA File (`.dna`)
+
+Plain text file containing a single DNA sequence.
+
+```
+Format: [ACGTBPSZ]+
+Encoding: ASCII
+Line ending: LF (optional)
+```
+
+### 5.2 LUT File (`.lut`)
+
+JSON file mapping DNA sequences to solutions and energies.
+
+```json
+{
+  "format": "hachimoji_monotone_lut_v1",
+  "bases": "ABCGPSTZ",
+  "n_vars": 20,
+  "n_solutions": 1048576,
+  "seq_length": 7,
+  "encoding": "monotone",
+  "monotone": true,
+  "rank_correlation": 1.0,
+  "qubo_matrix": [[...]],
+  "entries": {
+    "AAAAAAA": {"x": [0,0,...,0], "energy": 0.0},
+    "AAAAAAB": {"x": [1,0,...,0], "energy": 3.074},
+    ...
+  }
+}
+```
+
+**Required fields:**
+- `format` — always `"hachimoji_monotone_lut_v1"`
+- `bases` — the base alphabet (must be `"ABCGPSTZ"`)
+- `n_vars` — number of variables in the problem
+- `n_solutions` — total number of entries
+- `seq_length` — bases per sequence
+- `encoding` — always `"monotone"`
+- `monotone` — must be `true` for a valid LUT
+- `rank_correlation` — must be `1.0` for a valid LUT
+- `entries` — the mapping: sequence → {x, energy}
+
+### 5.3 Metadata File (`.json`)
+
+Problem-level metadata (optional).
+
+```json
+{
+  "problem": "banded_qubo_20var",
+  "n_vars": 20,
+  "n_solutions": 1048576,
+  "optimal": {"x": [...], "energy": 0.0, "seq": "AAAAAAA"},
+  "worst": {"x": [...], "energy": 60.66, "seq": "GZZZZZZ"},
+  "timing": {"generate": 0.185, "energy": 0.113, "sort": 0.096, "total": 0.394}
+}
+```
+
+---
+
+## 6. Operations
+
+### 6.1 Encode
+
+```
+encode(data: bytes, chunk_size: int) → (dna: string, lut: dict)
+
+  1. Split data into chunks of chunk_size bytes
+  2. Rank chunks by frequency (most frequent → rank 0)
+  3. Assign DNA sequences by rank
+  4. Concatenate sequences
+  5. Return (dna_string, decode_lut)
+```
+
+### 6.2 Decode
+
+```
+decode(dna: string, lut: dict, bases_per_symbol: int) → bytes
+
+  1. Split dna into groups of bases_per_symbol
+  2. Look up each group in lut
+  3. Concatenate results
+  4. Return bytes
+```
+
+### 6.3 Roundtrip
+
+```
+decode(encode(data)) = data
+```
+
+This must hold for all valid inputs. Verified at encode time.
+
+---
+
+## 7. QUBO Integration
+
+### 7.1 Problem Encoding
+
+A QUBO problem `minimize x^T Q x` over `x ∈ {0,1}^n` is encoded as:
+
+1. **Matrix:** QUBO matrix Q encoded as bytes → DNA (via `encode`)
+2. **Solutions:** All (or sampled) solutions encoded as DNA sequences (via monotone LUT)
+3. **LUT:** The monotone LUT maps DNA sequences to (solution, energy) pairs
+
+### 7.2 Solving
+
+```
+solve_qubo(Q) → (optimal_x, optimal_energy, optimal_seq)
+
+  1. Enumerate all 2^n solutions (or sample)
+  2. Compute energies: E_i = x_i^T Q x_i
+  3. Build monotone LUT
+  4. Return: optimal = LUT["AAA...A"]
+```
+
+### 7.3 Sorting as Computation
+
+The act of sorting DNA sequences IS the act of solving the QUBO:
+
+```
+sorted(dna_sequences) → solutions in energy order
+first(sorted) = optimal solution
+last(sorted) = worst solution
+```
+
+This is the core insight: **sorting is solving**.
+
+---
+
+## 8. GPU Integration
+
+### 8.1 Radix Sort
+
+DNA sequences are base-8 digit arrays. Radix sort on these arrays is:
+
+- **O(n · k)** where n = number of sequences, k = sequence length
+- For constant k, this is **O(n)** — linear time
+- Each digit is 3 bits, perfectly suited for GPU parallel processing
+
+### 8.2 Zero Copy
+
+CPU writes DNA sequences to GPU-accessible unified memory. GPU sorts in-place. CPU reads result. No memcpy.
+
+```
+CPU → [unified memory] → GPU (radix sort) → [unified memory] → CPU
+```
+
+### 8.3 Braid Sort Kernel
+
+The GPU compute shader performs braid crossings:
+
+```
+braid_cross(a, b):
+    if a > b: return (b, a)  // triangle rotation
+    else:     return (a, b)  // eigensolid (converged)
+```
+
+Each workgroup processes a chunk of the array. After log₂(n) passes, the array is sorted.
+
+---
+
+## 9. Surface Rendering
+
+### 9.1 8×8 Hachimoji Surface
+
+A QUBO solution is rendered as an 8×8 pixel grid:
+
+- Each pixel = one variable
+- x[i] = 0 → dark (A-state, RGB: 13,13,13)
+- x[i] = 1 → bright (G-state, RGB: 26,204,77)
+- Variables laid out in row-major order
+
+### 9.2 Color Map
+
+| Base | Color | RGB | Meaning |
+|---|---|---|---|
+| A | Near black | (13, 13, 13) | x = 0 |
+| B | Deep purple | (51, 26, 77) | synthetic |
+| C | Ocean blue | (26, 77, 128) | synthetic |
+| G | Hachimoji green | (26, 204, 77) | x = 1 |
+| P | Plasma orange | (230, 102, 26) | synthetic |
+| S | Spectral violet | (153, 51, 204) | synthetic |
+| T | Teal | (26, 179, 179) | synthetic |
+| Z | Near white | (242, 242, 242) | synthetic |
+
+### 9.3 Eigenvalue Fingerprint
+
+The 8×8 surface is the **eigenvalue fingerprint** of the QUBO solution. Different QUBOs produce different surfaces. The surface IS the answer.
+
+---
+
+## 10. Invariants
+
+The following properties must hold for any valid Hachimoji DNA encoding:
+
+1. **Alphabet consistency.** All sequences use only bases from `{A, B, C, G, P, S, T, Z}`.
+2. **Ordering consistency.** ASCII order = index order = lexicographic rank.
+3. **Monotonicity.** In a monotone LUT, `sort(sequence) = sort(energy)`.
+4. **Roundtrip.** `decode(encode(data)) = data` for all valid inputs.
+5. **Uniqueness.** Each solution maps to exactly one DNA sequence.
+6. **Minimality.** The optimal (lowest-energy) solution maps to `AAA...A`.
+7. **Completeness.** Every entry in the LUT has a valid solution and energy.
+8. **Correlation.** Rank correlation between sequence index and energy = 1.0.
+
+---
+
+## 11. Anti-Patterns
+
+The following are **forbidden**:
+
+1. **Non-ASCII bases.** Sequences must use only the 8 canonical bases.
+2. **Variable-length symbols within a LUT.** All sequences in a LUT must have the same length.
+3. **Non-monotone assignment.** If `encoding = "monotone"`, the LUT must satisfy the monotonicity axiom.
+4. **Lossy encoding.** Roundtrip must be exact. No approximation.
+5. **Mutable base ordering.** The base ordering `A < B < C < G < P < S < T < Z` is fixed forever.
+
+---
+
+## 12. Extensions
+
+Future extensions (not yet specified):
+
+- **Multi-pass radix sort** for sequences longer than 8 bases
+- **Hierarchical LUTs** for problems with structure (banded, sparse, block-diagonal)
+- **Streaming encode/decode** for large files
+- **WebGPU compute shader** for GPU-accelerated sorting
+- **Finsler metric integration** for manifold-aware encoding
+- **Eigenvalue surface** for visual comparison of solutions
+
+---
+
+## Appendix A: Reference Implementation
+
+| Component | File | Language |
+|---|---|---|
+| LUT builder | `python/dna_lut.py` | Python |
+| File encoder | `python/dna_encode_file.py` | Python |
+| Radix sort | `python/dna_radix_gpu.py` | Python + NumPy |
+| GPU kernel | `python/dna_braid.wgsl` | WGSL |
+| WebGPU host | `python/dna_webgpu.js` | JavaScript |
+| Surface render | `python/dna_surface.html` | HTML + Canvas |
+
+## Appendix B: Proof of Monotonicity
+
+**Theorem:** The monotone encoding satisfies the lexicographic ordering axiom.
+
+**Proof:**
+
+1. Let `S = {s₀, s₁, ..., s_{n-1}}` be solutions sorted by energy: `E(s₀) ≤ E(s₁) ≤ ... ≤ E(s_{n-1})`.
+2. Assign `seq_i = int_to_dna(i, k)` where `k = ⌈log₈(n)⌉`.
+3. By construction, `i < j ⟹ seq_i < seq_j` (lexicographic), because `int_to_dna` preserves ordering (§2.4).
+4. Therefore, `seq_i < seq_j ⟹ E(s_i) ≤ E(s_j)`.
+5. The LUT is monotone. ∎
+
+## Appendix C: Worked Example
+
+**Problem:** 3-variable diagonal QUBO, Q = diag(3, 2, 1).
+
+| Rank | DNA | Solution | Energy |
+|---|---|---|---|
+| 0 | AAA | [0,0,0] | 0.0 |
+| 1 | AAB | [0,0,1] | 1.0 |
+| 2 | AAC | [0,1,0] | 2.0 |
+| 3 | AAG | [0,1,1] | 3.0 |
+| 4 | AAP | [1,0,0] | 3.0 |
+| 5 | AAS | [1,0,1] | 4.0 |
+| 6 | AAT | [1,1,0] | 5.0 |
+| 7 | AAZ | [1,1,1] | 6.0 |
+
+**Verification:**
+- Lexicographic sort: AAA < AAB < AAC < AAG < AAP < AAS < AAT < AAZ
+- Energy sort: 0.0 ≤ 1.0 ≤ 2.0 ≤ 3.0 ≤ 3.0 ≤ 4.0 ≤ 5.0 ≤ 6.0
+- Monotone: ✓
+- Optimal: AAA → E=0.0
+- Worst: AAZ → E=6.0