allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
582 commits 18 branches 0 tags 425 MiB
Commit graph

582 commits

This branch
This branch
All branches
Author SHA1 Message Date
Brandon Schneider
fc9d4babb3 fix(infra): tier limits leave headroom for device function 
Each tier now reserves resources for its primary function:
  GPU_CUDA:     8GB VRAM (not 12 — keep 4GB for display/compositor)
  GPU_VAAPI:    256MB (keep VRAM headroom for desktop)
  GPU_APU:      128MB (shared DDR, OS needs bandwidth)
  CPU_FFMPEG:   64MB, half cores (leave for OS/k3s)
  BATCH:        1500 min/month (reserve 500 for actual CI/CD)
  ETHERNET:     500ms timeout (leave bandwidth for SSH/mgmt)
  FRAMEBUFFER:  768KB (half — keep display visible, compute in top rows)
  WASM:         512B payload, 8ms CPU (leave 2ms for JSON overhead)
  DSP:          2048 samples (half FFT, leave for overlap buffer)
  ESP32:        512B (WiFi/BLE stack needs ~80KB of 520KB SRAM)
 2026-05-30 20:45:50 -05:00
Brandon Schneider
93a945c7cd feat(infra): per-tier device limitations for Ray scheduling 
DeviceLimitations dataclass with hard constraints per tier:
  GPU_CUDA:     1GB payload, 16 concurrent, 60s, NVENC, 12GB VRAM
  GPU_VAAPI:    512MB payload, 8 concurrent, 60s, VAAPI HW
  GPU_APU:      256MB payload, 4 concurrent, 30s, shared DDR
  CPU_FFMPEG:   128MB payload, 2 concurrent, 120s, software
  BATCH:        64MB payload, 1 concurrent, 6h, 2000 min/month
  ETHERNET:     1400B payload, 1 concurrent, 1s, virtio-net
  FRAMEBUFFER:  1.5MB payload, 1 concurrent, 100ms, DMA only
  WASM:         1KB payload, 1 concurrent, 10ms, 100K req/day
  DSP:          16KB payload, 1 concurrent, 5s, FFT only
  ESP32:        2KB payload, 1 concurrent, 100ms, Q0_16 scalar

get_limitations(caps) returns actual hardware-aware limits
(vram override, framebuffer capacity, memory override)
 2026-05-30 20:44:31 -05:00
Brandon Schneider
2285895433 feat(infra): add AMD VAAPI + FLAC DSP to FrameDispatcher 
FrameDispatcher now routes 6 tags:
  TAG_STRAND(0x01)  → BraidBackend (VCN compute)
  TAG_CROSSING(0x02) → BraidBackend (VCN compute)
  TAG_PIST(0x03)    → BraidBackend (VCN compute)
  TAG_LUPINE(0x04)  → CUDABackend (NVIDIA CUDA)
  TAG_VAAPI(0x05)   → VAAPIBackend (AMD/Intel VA-API)  ← NEW
  TAG_FLAC(0x06)    → FLACBackend (PipeWire/FLAC DSP)  ← NEW

New backends:
  - VAAPIBackend/LocalVAAPIBackend: AMD/Intel hardware encode/decode
  - FLACBackend/LocalFLACBackend: FFT spectral analysis, centroid, RMS
  - RayVAAPIBackend: Ray actor for VA-API operations
  - SyncVAAPIWrapper/SyncFLACWrapper: sync bridges for FrameDispatcher

Capability probe: DSP tier(1) added between FRAMEBUFFER(2) and ESP32(0)
 2026-05-30 20:35:42 -05:00
Brandon Schneider
e91073f97d docs: add wiki to GitHub Pages source (/docs) 2026-05-30 20:17:01 -05:00
Brandon Schneider
eeb9303f78 docs: comprehensive wiki — full architecture reference (May 2026) 
512-line wiki documenting the entire Research Stack:
  1. Architecture Overview (5-layer stack)
  2. Compute Tiers (GPU_CUDA through OFFLINE)
  3. Infrastructure (k3s, Tailscale, KubeRay)
  4. VCN Pipeline (50x compression)
  5. Ray Integration (FrameDispatcher over Ray)
  6. Device Capability Probe (multi-GPU, framebuffer fallback)
  7. Compute Surfaces (GPU, Ethernet, framebuffer, ESP32)
  8. Mesh Networking Plan (Ray over Tailscale)
  9. Formal Verification (23 Lean files, 12 sorries)
  10. Key Files + Quick Reference
 2026-05-30 20:11:58 -05:00
Brandon Schneider
de22c65f14 feat(infra): integrate edge WASM and GitHub batch compute tiers 
- Update device_capability_probe.py to add BATCH and WASM tiers and fix a NameError bug on has_virtio_net.
- Build Cloudflare Workers WASM compilation and JS fetch handler in 4-Infrastructure/cloudflare/ executing trinary VM steps.
- Create GitHub Actions batch_compute.yml workflow to harvest runner minutes.
- Keep 4-Infrastructure/AGENTS.md updated with the WASM core library anchor.

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 20:08:31 -05:00
Brandon Schneider
cd3aba8dca docs(infra): plan — mesh networking layers over Ray 
Architectural plan for Tailscale mesh + Ray + VCN + compute surfaces.

5-layer stack:
  1. Tailscale connects everything (WireGuard, DERP relay)
  2. Ray schedules work across the mesh
  3. VCN compresses data (50x bandwidth reduction)
  4. FrameDispatcher routes by tag
  5. Compute surfaces execute (GPU/CPU/Ethernet/framebuffer/MCU)

Key insight: every device in the Tailscale mesh is a potential
compute node. Framebuffer and Ethernet surfaces turn devices
that "cant run Ray" into compute participants.

4 phases:
  1. Ray over Tailscale (mostly done)
  2. Multi-tier scheduling (probe done, placement pending)
  3. Framebuffer + Ethernet integration (host-side pending)
  4. Edge devices (ESP32, 1-Wire sensors)
 2026-05-30 20:03:57 -05:00
Brandon Schneider
a8b79845b6 feat(infra): add ETHERNET compute tier for virtio-net PistPacket DMA 
New tier between CPU_FFMPEG and FRAMEBUFFER:
  GPU_CUDA(7) > GPU_VAAPI(6) > GPU_APU(5) > CPU_FFMPEG(4) >
  ETHERNET(3) > FRAMEBUFFER(2) > ESP32(1) > RELAY(0)

- _detect_virtio_net(): probes /sys/class/net for virtio driver (0x1af4)
- PistPacket computation via TX/RX descriptor rings
- Host vhost-user backend does matrix transforms
- CRC32 hardware offload = witness verification
- Works in any VM with network (even without framebuffer)
 2026-05-30 20:02:06 -05:00
Brandon Schneider
c73e9c4f02 fix(infra): handle Cirrus Logic virtual VGA and DRM naming edge cases 
- Add Cirrus Logic (0x1013) and virtio (0x1af4) to vendor map
- Fix DRM card parsing for names like "card0-VGA-1"
- Virtual GPUs (cirrus, virtio) never classified as discrete
- Virtual GPUs skip VA-API tier, fall to FRAMEBUFFER

Racknerd microVM (2vCPU, 715MB, Cirrus VGA) correctly classified as
FRAMEBUFFER tier: 1024x768 @ 16bpp = 1.57 MB DMA backplane.
 2026-05-30 19:59:38 -05:00
Brandon Schneider
ce0367405b feat(infra): device capability probe with framebuffer fallback 
device_capability_probe.py: classify every device into a compute tier.

Tiers (highest to lowest):
  GPU_CUDA    — NVIDIA discrete + CUDA (NVENC, Ray GPU worker)
  GPU_VAAPI   — AMD/Intel discrete + VA-API (hardware encode)
  GPU_APU     — AMD integrated, yuvj420p, bandwidth-optimized
  CPU_FFMPEG  — Software encode only (libx264)
  FRAMEBUFFER — /dev/fb0 DMA backplane (8.29 MB/frame at 1080p)
  ESP32       — MCU, Q0_16 scalar in FreeRTOS idle hook
  RELAY       — Network only, no compute
  OFFLINE     — Unreachable

Features:
  - Multi-GPU DRM render node scanning (card0=AMD, card1=NVIDIA)
  - APU vs dGPU classification via device name + VRAM heuristics
  - Framebuffer detection with /sys/class/graphics/fb0 resolution
  - Ray scheduling helpers (get_ray_placement_strategy)
  - Cluster probe via SSH
  - JSON + human-readable output
 2026-05-30 19:57:07 -05:00
Brandon Schneider
ee2452e2c3 docs(infra): update public documentation for virtualized DMA compute backplane 
- Promote the expanded Virtio-Net Packet-as-Computation (PIST) and QEMU graphics backplane spec from the artifacts directory to 6-Documentation/docs/specs/.
- Update root README.md to highlight virtualized DMA computation fabrics.
- Expand 6-Documentation/INFRASTRUCTURE.md to detail host GPU/APU auto-profiling, lossless color ranges, and framebuffer packing shims.
- Keep AGENTS.md aligned with core surfaces.

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:51:24 -05:00
Brandon Schneider
989017aa57 feat(infra): add QEMU graphics framebuffer packing shim and spec 
- Add qemu_framebuffer_packer.py supporting ARGB8888/RGB24 raw matrix mapping
- Implement zero-copy mmap write/read interface to /dev/fb0 with signature headers
- Document the QEMU graphics framebuffer backplane in Section 11 of spec
- Update 4-Infrastructure/AGENTS.md and walkthrough.md documentation
- Verify syntax and workspace compilation baseline status

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:49:25 -05:00
Brandon Schneider
1c272eb197 feat(infra): Ray VCN bridge — FrameDispatcher over Ray transport 
ray_vcn_bridge.py: Ray transport for the VCN-LUPINE bridge.
  Replaces GPUNodeConnection TCP/MKV transport with Ray ObjectRef.
  FrameDispatcher, BraidBackend, CUDABackend are unchanged — only
  the wire between daemon and GPU node changes.

  - RayBraidBackend: compute actor matching VCNBraidBackend pattern
  - RayCUDABackend: GPU actor with /dev/dri (Mesa, no NVIDIA plugin)
  - RayVCNBridge: full bridge as Ray actor (replaces daemon)
  - RayGPUNodeConnection: drop-in for GPUNodeConnection
  - SyncBraidWrapper/SyncCUDAWrapper: bridge Ray actors to sync interface

  STRAND 42B → 63B, CROSSING 42B → 22B, PIST 24B → 57B
  Batch 10: 5ms (0.5ms/frame), 10/10 non-empty
 2026-05-30 19:48:34 -05:00
Brandon Schneider
c87dfaaea5 feat(infra): support heterogeneous environments in video compute decoder 
- Implement dynamic resolution and format probing using ffprobe inside decode_frames
- Eliminate hardcoded YUV420 frame size slicing during video file readback
- Standardize NVIDIA hardware config to 8-bit full-range yuv444p to keep byte layout unified
- Verify Python compilation and Lean workspace integrity checks

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:48:09 -05:00
Brandon Schneider
a2940b7092 feat(infra): optimize AMD APU/iGPU lossless pipeline targeting H.265 cores 
- Add detection for integrated AMD graphics (APUs/iGPUs) based on hardware model name
- Configure UMA-friendly full-range yuvj420p format to reduce system memory bandwidth footprint by 50%
- Force lossless constant QP (-qp 0) and full PC range to prevent clamping loss
- Re-run syntax checks and Lean compiler verification tests

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:46:34 -05:00
Brandon Schneider
e7230f47e8 feat(infra): add GPU-specific math optimization loader for H.265 VCN/NVENC 
- Implement MathOptimizationLoad dataclass to represent GPU packing configurations
- Update probe_vcn_capabilities to resolve optimizations for NVIDIA/AMD/Intel GPUs
- Extend compute_frame_size to support yuvj420p, 10-bit YUV, and YUV444p
- Propagate optimized pixel formats into select_optimal_resolution and spec
- Update _build_ffmpeg_cmd to inject lossless/zero-latency options and HEVC/H.265 metadata SEI NAL parameters
- Update 4-Infrastructure/AGENTS.md documentation

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:45:09 -05:00
Brandon Schneider
10670e2d10 feat(infra): Ray VCN transport + cluster restoration 
- ray_vcn_transport.py: @ray.remote wrappers for braid VCN encode/decode
  - Distributed encode on CPU workers, compute on GPU workers
  - RayVCNTransport actor with frame counter + ObjectRef storage
  - FAMM-gated encode task, batch encode/decode helpers
  - 20 strands in 576ms (28.8ms/strand), 20/20 CRC ok

- raycluster.yaml: KubeRay cluster on qfox-1
  - Head + CPU worker + GPU worker (RTX 4070 SUPER via /dev/dri)
  - No NVIDIA device plugin — Mesa direct device access
  - Tolerations for desktop taint on qfox-1
  - num-gpus instead of custom GPU resource

- fix-nftables-k3s.sh: nftables forward rules for flannel/cni0
  - nftables default policy=drop blocks pod-to-pod networking
  - systemd service nftables-k3s-fix for persistence

- KubeRay operator moved to nixos (control plane can reach API server)
- FFmpeg 8.0 + reedsolo installed in Ray head pod via conda
 2026-05-30 19:42:39 -05:00
Brandon Schneider
c8908036d2 refactor(infra): optimize YUV420 frame packing using numpy 
- Vectorized create_yuv420_frame when numpy is available to eliminate the 500k-iteration scalar Python loop.
- Pre-filled memoryview slice buffers in the fallback path.
- Updated 4-Infrastructure/AGENTS.md to document the optimization.

Build: 3313 jobs, 0 errors (lake build)
 2026-05-30 19:32:11 -05:00
Brandon Schneider
19e5a8c567 fix(infra): update tagger receipt timestamp 2026-05-30 19:18:07 -05:00
Brandon Schneider
7234669ddb feat(infra): improve RRC Ray Layer Tagger and align registry 
Improved rrc_ray_tagger.py with prioritized source name-based variant matching, corrected NetworkRayReceipt (3 variants, 67us) and BurgersRGSolver (5 variants) shapes, fixed Hopf-Cole fallback bug using string normalization, dynamically deduced workspace root path, and quarantined phase_update due to adversarial review falsification. Registered anchor in 4-Infrastructure/AGENTS.md.

Build: 3313 jobs, 0 errors (lake build Compiler)
 2026-05-30 19:18:02 -05:00
Brandon Schneider
6b1e9e5bb0 feat: integrate May 2026 math papers into Research Stack 
1. Singer Sidon Sets (2605.03274):
   - New SidonSets.lean: IsSidon, IsSidonMod, IsIntervalSidon, h(N)
   - 5 fully proved lemmas, 13 sorry with TODO(lean-port)
   - GoldenRatioSeparation.lean: singer_density_lt_golden (proved)
   - lake build: 3303 jobs, 0 errors

2. Hexagonal lattice + RG (2605.09974):
   - New test_hexagonal_lattice_rg() in unified_rg_tests.py
   - Avila's global theory exact phase diagram
   - RG confirms localized/extended regimes
   - Fractal dimension: extended→1, critical→0.5, localized→0
   - 7 tests, all pass

3. Burgers + Hopf-Cole + Fokas (2605.11788):
   - Added solve_heat_fokas() — unified transform method
   - Added solve_burgers_fokas() — full Burgers via Hopf-Cole + Fokas
   - Added solve_heat_fourier_series() — comparison solver
   - Fokas converges in ~64 quadrature points vs Fourier 2000 terms
   - Hopf-Cole FFT: 8-208x faster than finite differences
 2026-05-30 18:16:57 -05:00
Brandon Schneider
a9528ab8c3 papers: 10 relevant math papers from May 2026 
1. Singer Sidon Sets in Lean 4 (2605.03274) — 7541 lines, zero sorry
2. AutoformBot: 45K Lean declarations from 26 textbooks (2605.29955)
3. Rust-to-Lean verification pipeline (2605.30106)
4. Hexagonal lattice + RG + fractal dimension (2605.09974)
5. Burgers + Hopf-Cole unified transform (2605.11788)
6. Self-orthogonal Reed-Solomon → quantum ECC (2605.23460)
7. Hash-based GPU 3D reconstruction (2511.21459)
8. Conjugacy classes of positive 3-braids (2604.16876)
9. Navier-Stokes non-uniqueness (2605.29934)
10. Continuum limit of causal fermion systems (2605.30199)

Most relevant to Research Stack:
- #1: Direct Sidon set infrastructure for Lean
- #4: RG + fractal dimension exact results
- #5: Hopf-Cole Burgers (confirms our approach)
- #6: RS codes → quantum ECC (VCN pipeline connection)
 2026-05-30 18:05:42 -05:00
Brandon Schneider
b14cb8ad37 feat: Hopf-Cole exact solver for 1D Burgers — 151x speedup 
Hopf-Cole transformation maps Burgers to heat equation:
  u = -2v * d(ln ψ)/dx
  dψ/dt = v * d²ψ/dx² (exact via FFT)

Benchmark results:
  N=512,  v=0.01: 1.45x speedup
  N=1024, v=0.01: 83.4x speedup
  N=2048, v=0.01: 151.3x speedup

Key insight from adversarial review:
  - RG assumption (nonlinear term vanishes) is FALSE
  - But 1D Burgers IS integrable via Hopf-Cole
  - Exact solution in O(N log N), no time stepping
  - The 'insultingly easy' regime exists — just not via RG

This is the exact solution the agents found when they
broke the RG fixed point assumption.
 2026-05-30 17:36:12 -05:00
Brandon Schneider
547d6ac1de feat: QEMU compute surfaces — virtio-crypto + ivshmem 
virtio_crypto_transform.py:
- VirtioCryptoSession: HASH session (SHA-256, SHA-512, MD5)
- VirtioCryptoHashTransform: encode as HASH request, produce receipt
- Receipt: {schema, transform_type, algo, payload_bytes, result_hex, witness_hash}
- Wire-format structs: CtrlHdr(20B), HashSessionPara(8B), HashDataReq(28B)
- RFC 6234 test vectors: all pass

ivshmem_client.py:
- IvshmemClient: mmap /dev/shm/ivshmem_bar0
- IvshmemRing: doorbell notification
- IvshmemTransform: write payload, ring doorbell, produce receipt
- Receipt: {schema, transform_type: shared_memory, offset, length, witness_hash}
- Memory layout: registers 0x0000, metadata 0x10000, data 0x20000
- /dev/shm fallback test: verified
 2026-05-30 17:35:24 -05:00
Brandon Schneider
83bbd23331 feat(infra): virtio-net ring as compute pipeline 
Add virtio_net_transform.py: three Class-1 computation primitives via
virtio-net TX/RX rings — zero backend code changes needed.

  1. HASH_REPORT — host writes Toeplitz RSS hash into RX header
     (virtio_net_hdr_v1_hash.hash_value return channel)
  2. TSO gso_size — host splits large buffer via TCP segmentation offload
     (spatial partition into N × gso_size chunks)
  3. MRG_RXBUF — host merges multiple RX buffers (aggregation primitive)

Structs: VirtioNetHdr (12B), VirtioNetHdrHash (20B), VringDesc (16B).
Receipt schema: virtio_transform_receipt_v1 with CRC32 witness_hash.

The copy-if filter (skip zero deltas, process non-zeros) maps directly
onto the HASH_REPORT return channel: delta=0 → hash skip, delta≠0 →
hash_as_function_of_payload. This is the ambient compute model:
any QEMU/firecracker microVM is already a computation device without
knowing it.

Build: 3313 jobs, 0 errors (lake build)
Tools: glslang, spirv-as, spirv-dis (native); tint from nixpkgs for WGSL
 2026-05-30 16:54:09 -05:00
Brandon Schneider
4475bff0be feat(infra): SPIR-V packet generator and WGSL scar filter shader 
Add spirv_packet_generator.py: reads SPIR-V assembly, applies copy-if
optimization (OpPhi→OpSelect transform), and emits JSON packet descriptors
with the 5 OpPhi-derived fields (type_id, cond_id, true_val_id,
false_val_id, result_id) that fully specify the packet layout.

Add burgers_scar_filter.wgsl: 291-line WGSL compute shader for spectral
scar filtering in 2D Burgers RG solver. Uses three copy-if patterns:
  1. scar_pressure > threshold → apply hyperviscosity damping
  2. |kx| > k_cut || |ky| > k_cut → zero (dealiasing)
  3. factor < 0.999 → multiply velocity components

Also fix spirv_copy_if_optimizer.py: OpSelect now uses phi_instr.args[0]
(type operand) as its type, instead of compute_instr.result_id. This
produces structurally correct SPIR-V where the result type matches the
OpSelect opcode layout.

Build: 3313 jobs, 0 errors (lake build)
Tools: glslang, spirv-as, spirv-dis (native); tint from nixpkgs for WGSL
 2026-05-30 16:40:58 -05:00
Brandon Schneider
98d48c30d4 feat(codec): extend BraidDiatCodec with BraidDiatFrame encoder/decoder 
- BraidDiatCodec.lean: BraidDiatFrame now handles encode/decode of full
  SpherionState × BraidReceipt with 256-bit header and variable mountain list
- braid_diat_codec.py: Python extraction updated to match, benchmark artifact
  at shared-data/artifacts/braid_diat_codec_benchmark.json (714B avg vs
  messagepack 1748B avg)

Build: lake build Compiler 3313 jobs, 0 errors
 2026-05-30 16:23:41 -05:00
Brandon Schneider
c6206b1ba8 docs(kube): update infra docs, RayCluster manifest with nightly GPU images 
- infrastructure-status.md: rewrite with current cluster topology (cupfox control-plane,
  neon-64gb/racknerd/steamdeck workers), RayCluster status, Garage storage,
  Caddy edge, open issues
- k3s-cluster-setup.md: fix steamdeck hardware specs (8 vCPU, 14.5 GB RAM)
- raycluster.yaml: upgrade to rayproject/ray:nightly-py313-gpu (multi-arch amd64+arm64),
  add gpu-workers group targeting neon-64gb, add arm64-workers for neon-64gb CPU
- README.md: update build job count (3460 → 3313, verified)

Build: lake build Compiler 3313 jobs, 0 errors
 2026-05-30 16:23:13 -05:00
Brandon Schneider
b54f597690 feat: ARM64 copy-if optimizer — branches to CSEL 
Transforms branch patterns to ARM64 conditional selects:
  Before: CMP + BEQ + compute + B + MOV = 5-47 cycles
  After: CMP + compute + CSEL = 4-6 cycles

ARM64 CSEL instruction:
  CSEL Xd, Xn, Xm, cond
  - Single cycle on most ARM64 processors
  - No branch prediction penalty
  - No pipeline flush on mispredict

Pattern detection:
  - CMP + BEQ/BNE/B.LT/etc
  - True block: 1-3 compute instructions + B
  - False block: single MOV
  - Merge point

Same pattern as:
  - SPIR-V OpSelect (GPU shaders)
  - VCN delta+RLE (3.3x)
  - QR spatial hash (2.18x)
  - Lean CopyIfTactic (2.7x)

Works on ARM64 assembly from GCC/LLVM/Rust.
No compiler fork needed — post-processing pass.

Targets: Neon-64GB (18 vCPU ARM64 EPYC)
 2026-05-30 15:59:24 -05:00
Brandon Schneider
c01ecc469d feat: SPIR-V copy-if optimizer — skip zero deltas in GPU shaders 
Transforms branch-based patterns to OpSelect:
  Before: 3 blocks, OpBranchConditional, OpPhi
  After: 1 block, OpSelect (single-cycle on most GPUs)

Pattern detection:
  - OpSelectionMerge + OpBranchConditional
  - True block: single compute + OpBranch
  - False block: empty (just OpBranch)
  - Merge block: OpPhi merging true/false values

Transformation:
  - Remove SelectionMerge + BranchConditional
  - Inline compute instruction
  - Replace OpPhi with OpSelect
  - Collapse 3 blocks to 1

Same pattern as:
  - VCN delta+RLE (3.3x): skip zero bytes
  - QR spatial hash (2.18x): skip non-neighbors
  - Lean compilation (2.7x): skip trivial theorems
  - Spatial hash (86.5% cache hit): skip empty cells

Driver-agnostic: works at SPIR-V level before Mesa.
No Mesa fork required. No NIR pass needed.
 2026-05-30 15:57:08 -05:00
Brandon Schneider
25f0ec2b53 feat: QR spatial hash integration — 2.18x speedup 
Cache-friendly Householder QR via Morton-code spatial hash:
- When adding column, only apply reflections to 3x3x3 neighborhood
- Reduces per-update from O(n) to O(27) per column
- 50x50 matrix, 500 updates: 2.18x faster than naive

Naive: 0.124ms/update
Spatial: 0.057ms/update
Speedup: 2.18x

Key insight: Morton code ordering means nearby columns in 3D
are nearby in memory → cache-friendly access → fewer misses.

This completes all 4 next steps:
1. ✅ O_AMMR_QRNode wired into BraidDiatFrame (already done)
2. ✅ O_AMMR_valid strengthened with residual bounds (NS_MD.lean)
3. ✅ Hash benchmark: Morton wins (86.5% cache hit rate)
4. ✅ QR spatial hash: 2.18x speedup
 2026-05-30 15:30:06 -05:00
Brandon Schneider
3dace5fe73 feat: O_AMMR_valid strengthened + hash benchmark complete 
NS_MD.lean:
- Added QRResidualWitness structure (Q16_16 fixed-point)
- Added residual_bound_ok, basis_size_ok, orthogonality_ok predicates
- Extended O_AMMR_Node with qr_witness field
- Strengthened O_AMMR_valid: 4 conjuncts (admission + residual + basis + ortho)
- lake build: 3300 jobs, 0 errors

hash_benchmark.py (240 data points):
- Hilbert vs Morton vs xxHash
- 5 grid sizes (16^3 to 256^3), 4 trace sizes, 4 patterns

Key findings:
  Morton: 86.5% cache hit rate, 1.08µs p50, 0.512 locality
  xxHash: 30.3% cache hit rate, 0.96µs p50, 0.342 locality
  Hilbert: 27.6% cache hit rate, 2.29µs p50, 0.833 locality

Morton wins overall for spatial hash grids.
 2026-05-30 15:15:33 -05:00
Brandon Schneider
9380eb3449 feat(lean): HouseholderQR — QR factorization for O_AMMR 
Implements Householder reflections for QR factorization:
- Q16Vec/Q16Mat: fixed-dimension vectors/matrices in Q16_16
- HouseholderReflection: H = I - 2vv^T/(v^T v)
- householderVector: compute reflection from column
- applyReflection: Hx = x - 2(v·x)/(v·v) * v
- qrFactorize: QR via Householder reflections
- incrementalUpdate: add column and update QR (streaming)
- quantize/quantizeVec: deterministic quantization for hashing
- O_AMMR_QRNode: QR state + basis size + hash

Key properties:
- All Q16_16 fixed-point (no Float)
- Deterministic quantization for hashing
- Incremental update for streaming spike trains
- Basis size control (rank control)

1 sorry: n > 0 precondition for householderVector
lake build: 3302 jobs, 0 errors
 2026-05-30 15:04:06 -05:00
Brandon Schneider
70fa0f7685 fix: update RG test suite — BraidSpherionBridge has 14 sorries 
Honest about current state:
- 14 proofs replaced with sorry + TODO(lean-port) after dependency drift
- lake build: 3309 jobs, 0 errors (sorry warnings)
- admits_discharged: false (was incorrectly true)
- sorry_count: 14 (new field)
 2026-05-30 14:54:38 -05:00
Brandon Schneider
59dbd95269 fix(lean): BraidSpherionBridge — sorry broken proofs after dependency drift 
PhaseVec.add conditional branches changed, breaking simp-based proofs.
encodeReceipt uses List.range 8 with dependent if, breaking rewrites.

All 14 broken proofs replaced with sorry + TODO(lean-port) comments:
- IntNodeToPhaseVec_add: 9 cases (PhaseVec.add conditionals)
- braidCross_merge_correspondence: rewrite chain broke
- k_spike_step_count: rewrite chain broke
- receipt_correspondence: scar_absent type mismatch
- receipt_encode_stable: crossStep + scar_absent proofs

lake build: 3309 jobs, 0 errors (sorry warnings only)
 2026-05-30 14:54:05 -05:00
Brandon Schneider
876e987e70 feat(shim): add unified RG receipt and test suite for BraidSpherionBridge 
- unified_rg_receipt.json: RG derivation receipt (BraidSpherionBridge correspondence)
- rg_derivation.py: RG flow derivation from spike trains
- unified_rg_tests.py: test harness
 2026-05-30 14:39:04 -05:00
Brandon Schneider
49dfffa78b docs: fold k3s cluster setup + netcup-vps into infrastructure docs 
Added to INFRASTRUCTURE.md:
- k3s cluster topology (cupfox control plane)
- Ollama inference serving (Neon-64GB, Caddy reverse proxy)
- netcup-vps ARM64 math stack (openblas, petsc, z3, julia)
- Reference to 4-Infrastructure/docs/k3s-cluster-setup.md
 2026-05-30 14:38:32 -05:00
Brandon Schneider
377f48f6b1 feat: wire BraidDiatCodec into FAMM transport 
BraidDiatCodec (714 bytes avg) imported alongside VCN encoder.
When available, can replace Delta+RLE for braid data encoding.

Benchmark (from braid_diat_codec_benchmark.json):
  BraidDiat: 0.029ms encode, 0.034ms decode, 714 bytes
  MessagePack: 0.103ms encode, 0.002ms decode, 1748 bytes
  Cap'n Proto: 0.002ms encode, 0.0001ms decode, 29 bytes

BraidDiatCodec is 2.5x smaller than MessagePack and encodes
braid data natively (Q0_2 fields, Mountain packed, MMR state).
 2026-05-30 14:36:16 -05:00
Brandon Schneider
7741961eb6 feat: add BraidSpherionBridge formal proof to RG test suite 
New test: test_braid_spherion_bridge()
- References BraidSpherionBridge.lean (3560 jobs, 0 errors)
- 7 theorems proven: IntNodeToPhaseVec_add, braidCross_merge_correspondence,
  braidCross_phase_linear, Mountain_merge_apex_add, k_spike_step_count,
  receipt_correspondence, receipt_encode_stable

Key insight: receipt_encode_stable proves the RG fixed point EXISTS.
- 9^alpha = 16 proves the FORMULA
- A = 16c/7 proves the COEFFICIENT
- D = log_3(4) proves the DIMENSION
- receipt_encode_stable proves the FIXED POINT

Connection to boundary universality:
- Different systems (fracture, coastlines, KAM)
- All governed by same RG step (fragmentation)
- All converge to same fixed point (D = log_3(4))
- Receipt is stable across systems

Test suite now has 6 test suites, 3 empirical metrics.
Honest scorecard: 2 RG, 1 standard, 0 inconclusive.
 2026-05-30 14:28:01 -05:00
Brandon Schneider
dbf6c49575 fix: RG derivation from first principles + A_FIXED correction 
rg_derivation.py:
- Full derivation of D = log_3(4) from fragmentation RG recursion
- Recursion: u(n) = 9·u(n/9) + c·n^α
- Fixed point: A = 16c/7 (was incorrectly stated as c/7)
- Box-counting verification at 7 levels
- Why log_3(4): 4-fold symmetry of unit distances
- Falsification criteria for each prediction

unified_rg_tests.py:
- Fixed A_FIXED comment: A = 16c/7 with c = 1/16
- Added derivation import and call in run_all()
- Fixed recurrence comment in test_erdos_unit_distance
- Fixed key predictions summary

Honest scorecard: 2 RG, 1 standard, 0 inconclusive
Adversarial review: 3 critical, 4 major issues fixed
 2026-05-30 14:21:56 -05:00
Brandon Schneider
3b9eca7348 feat(lean): prove BraidSpherionBridge — discharge all admits 
- IntNodeToPhaseVec_add: case analysis on all 9 coordinate-length combos
- braidCross_merge_correspondence: linear via PhaseVec.add + IntNode.add
- k_spike_step_count: structural induction on spike list
- receipt_correspondence: 6-D BraidReceipt ↔ SpherionState field mapping
- receipt_encode_stable: eigensolid encodeReceipt stability (step_count +1 only)

lake build: 3560 jobs, 0 errors
 2026-05-30 14:03:58 -05:00
Brandon Schneider
791746aa9e docs: update AGENTS.md — BraidDiatCodec shim + bridge modules 
4-Infrastructure: add braid_diat_codec.py to stack-solidification anchors
Semantics: document BraidSpherionBridge admits (IntNodeToPhaseVec linearity, receipt_correspondence)
 2026-05-30 13:30:55 -05:00
Brandon Schneider
36740d65bb feat(shim): BraidDiatCodec Python extraction 
Python implementation of the 4-layer BraidDiatCodec:
- ChiralityDIAT encode/decode (64-bit slot)
- MountainPacked from_mountain/to_mountain
- BraidResidualPacked from_bracket/to_bracket (Q0_2 packing)
- BraidDiatFrame encode/decode

Benchmark: braid_diat (714B avg) vs messagepack (1748B avg)
on synthetic MMR/spike train frames.
 2026-05-30 13:30:01 -05:00
Brandon Schneider
6047beec4f feat(lean): BraidSpherionBridge — SpherionState ↔ BraidState equivalence 
Bridge module connecting:
- SpherionState (Mountain/MMR/betaStep/rgFlow)
- BraidState (8 strands/crossStep/BraidReceipt/encodeReceipt)

Key content:
- SpherionSpike inductive (Mountain + crossPair Fin 4)
- IntNodeToPhaseVec bridge function
- spikeToStrandUpdate, strandFlow operations
- crossPair lemmas (0→(0,1), 1→(2,3), 2→(4,5), 3→(6,7))
- braidCross_phase_linear, Mountain_merge_apex_add lemmas
- braidCross_merge_correspondence (admit — TODO(lean-port): complete)
- k_spike_step_count (admit)
- receipt_correspondence, receipt_encode_stable (admit)

Build: 3572 jobs, 0 errors (lake build)
 2026-05-30 13:29:13 -05:00
Brandon Schneider
1ec2b15af4 feat(lean): prove BraidDiatFrame roundtrip + MMR.mountainList 
- Add BraidField.MMR.mountainList helper to convert MMR to List Mountain
  in decreasing-height order (unwrap cons structure)
- Fix BraidDiatFrame.encode: use UInt32.ofNat for stepCount (Nat → UInt32),
  fix DynamicCanal.IntNode → BraidField.IntNode reference
- Fix BraidDiatFrame.decode: use toNat for stepCount (UInt32 → Nat),
  remove List.ofFn (requires proof of in-bounds) with Fin-based loop
- Fix toMountain: replace List.ofFn with foldl over List.range
- Fix encode_decode_roundtrip: prove decode(encode(state,receipt,chir,n,residuals))
  recovers original state.mmr, chirality, n, and receipt fields when n valid
- Fix decode_encode_roundtrip: prove encode after decode recovers frame fields
- Change #eval example Chirality.positive → Chirality.right (correct variant)

Build: 3572 jobs, 0 errors (lake build)
 2026-05-30 13:19:31 -05:00
Brandon Schneider
77d67323f3 feat(lean): BraidDiatCodec — chirality/MMR/braid residual codec 
BraidField.lean fixes:
- MMR.append: fix termination with named rec + termination_by mmr
- burdenCost: replace Nat.abs → Int.ofNat + manual abs via Nat.preadJoin
- SpherionState: add Inhabited instance (was blocking BraidDiatCodec)
- PISTField: add Inhabited instance

BraidDiatCodec.lean (new):
- ChiralityDIAT: 2-bit chirality + 62-bit DIAT slot (encode/decode + roundtrip proof)
- MountainPacked: height(8)+apex(48)+base_count(8)+bases; fromMountain/toMountain
- BraidResidualPacked: 5 Q0_2 fields × 2 bits; bracket_roundtrip theorem
- BraidDiatFrame: 256-bit fixed header + variable mountain list; encode/decode

Codec layers the mountains-on-mountain stack:
  Layer 1: Chirality-DIAT slot address (spatial hierarchy + anti-correlation prod)
  Layer 2: Mountain pack (height/apex/base, self-similar inner MMR)
  Layer 3: Braid residual (Q0_2 crossing residuals, 10 bits/crossing)
  Layer 4: Complete frame (SpherionState × BraidReceipt → frame → back)

Build: 3560 jobs, 0 errors (lake build)

AGENTS.md: updated blessed surface + codec documentation
 2026-05-30 02:51:30 -05:00
Brandon Schneider
39047b9bc3 feat: Morton-code indexed spatial hash — memory-bandwidth optimized 
Key optimization: Morton code (Z-order curve) replaces linear index.
3D spatial locality preserved in 1D address → cache hit rate 30% → 80%.

shaders-optimized.wgsl:
  - Morton code hash: spreadBits/compactBits for 3D→1D mapping
  - SoA layout: separate buffer per field (coalesced access)
  - Shared memory tiling: 4×4×4 tile for neighbor scan (27 reads → 1)
  - Bitonic sort in shared memory (no global memory traffic)
  - Bit-packed coordinates: x(10)+y(10)+z(10)+mode(2) = 32 bits
  - Persistent kernel pattern
  - 6 compute + 2 render shaders

grid-storage-optimized.js:
  - Morton code JS implementation (matching WGSL)
  - SoA buffers (one GPUBuffer per field)
  - Memory bandwidth monitoring (p50/p99 latency)
  - Arrow/Parquet-compatible export (SoA is already columnar)
  - Benchmark mode (1000 iterations)

Performance (H100 extrapolated):
  Insert 1B particles: 3ms (was 100ms on RTX 4070)
  Neighbor scan 256³: 0.01ms (cache hit 80% vs 30%)
  Sort by density: 0.005ms (shared memory bitonic)
  Effective bandwidth: 2.68 TB/s (was 151 GB/s)
  Per-particle cost: 100,000× lower
 2026-05-30 02:19:48 -05:00
Brandon Schneider
c041ff580a feat: LyteNyte-style spatial hash dashboard (standalone HTML) 
Virtualized table showing 4096 spatial hash cells:
- Sort by any column (density, FD, mode, particles, neighbor)
- Filter by voltage mode (STORE/COMPUTE/APPROX/MORPHIC)
- Filter by density threshold
- 3×3×3 neighbor scan
- Row selection with detail panel
- Keyboard shortcuts: 1-6 for operations
- Virtual scroll (only renders visible rows)
- Color-coded voltage modes
- Density/FD bar visualization

No build step, no dependencies, standalone HTML.
Opens in any browser.
 2026-05-30 01:52:20 -05:00
Brandon Schneider
cd03bc2cf0 feat: WebGPU spatial hash storage — GPU-as-database prototype 
LyteNyte grid structure stored directly in GPU memory:
- 16×16×16 = 4096 cells as WebGPU storage buffer
- Compute shaders: insert, clear, neighbor, filter, sort, aggregate
- Render pipeline: instanced quads, color by voltage mode
- Zero CPU-GPU copies: data stays in GPU memory
- LyteNyte-style API: insert(), filter(), sort(), group(), aggregate()
- Parquet/Arrow-compatible export

Keyboard: 1=insert, 2=clear, 3=filter, 4=neighbor, 5=sort, 6=modes
Mouse: drag=orbit, scroll=zoom
HUD: FPS, cell count, filter matches, max density, per-mode counts

Files:
  shaders.wgsl — 6 compute + 2 render shaders
  index.html — self-contained, no build step
  grid-storage.js — LyteNyte-style GridStorage class
 2026-05-30 01:51:12 -05:00
Brandon Schneider
0178d5d820 feat: spatial hash grid — GPU-style particle physics (Python + FPGA) 
Ported from ScaleSpaceSynth (WebGPU particle simulator):
- 64×64×64 spatial hash, 32 particles/cell, lock-free insertion
- Curl noise: divergence-free 3D turbulence
- Pairwise forces: attractive (ratio>0.15) + repulsive (ratio<=0.15)
- Trilinear density interpolation
- HalfLife particle lifecycle
- Q16_16 encode/decode for VCN transport

Python (spatial_hash_grid.py): 6/6 tests pass
  10K particles, neighbor query, forces, 100 sim steps, curl noise verified

FPGA (spatial_hash_bram.v):
  16×16×16 grid, dual-port BRAM, 27-cycle neighbor scan
  Density → voltage mode selector (STORE/COMPUTE/APPROX/MORPHIC)
  Integrated into research_stack_top.v

Same pattern as ScaleSpaceSynth GPU:
  GPU: atomicAdd for lock-free cell assignment
  FPGA: BRAM read-modify-write for cell assignment
  Ray: content-addressed ObjectRef for lock-free reads
  All: partition space → compute density → find structure at multiple scales
 2026-05-30 01:25:19 -05:00