Research-Stack/6-Documentation/INFRASTRUCTURE.md

Research Stack Infrastructure

Last updated: 2026-05-29 Domain: researchstack.info Repo: ~/Research Stack (git main) Hosting: Self-hosted bare metal — no cloud instances. All compute runs on physical nodes connected via Tailscale mesh. AWS instances were shut down and migrated to this setup.

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                        Tailscale Mesh Network                       │
│                                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐           │
│  │ QFox     │  │ nixos    │  │ 361395-1 │  │ racknerd │           │
│  │ RTX 4070 │  │ control  │  │ edge     │  │ VPS      │           │
│  │ 100.88.  │  │ plane    │  │ 100.110. │  │ 100.80.  │           │
│  │ 57.96    │  │ 100.102. │  │ 163.82   │  │ 39.40    │           │
│  │          │  │ 173.61   │  │          │  │          │           │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘           │
│       │             │             │             │                   │
│       └─────────────┴─────────────┴─────────────┘                   │
│                        Tailscale Funnel                             │
│                        361395-1.tail4e7094.ts.net                   │
└─────────────────────────────────────────────────────────────────────┘

Nodes

Node	Hostname	Role	IP (Tailscale)	Hardware
nixos	nixos-laptop	k3s control plane	100.102.173.61	NixOS 26.05, 6.18.32 kernel
qfox-1	qfox-1	GPU worker	100.88.57.96	CachyOS, RTX 4070, 7.0.9 kernel
361395-1	361395-1	Edge/Funnel	100.110.163.82	Proxmox VPS, Debian 13
racknerd	racknerd-510bd9c	Edge worker	100.80.39.40	Debian 13, 6.12.43 kernel
steamdeck	nixos-steamdeck-1	Worker	100.85.244.73	NixOS 25.11

k3s Cluster

Version: v1.35.4+k3s1 (v1.34.5 on steamdeck) CNI: Flannel (tailscale0 interface) Ingress: Traefik Server: nixos (control plane + etcd)

Namespaces

Namespace	Services
services	Homer, Hermes, Actual Budget, Uptime Kuma, Homarr, Vaultwarden, Heimdall, Authentik, Credential Server, Registry API, Jobs API, Blobs API
media	Jellyfin, Navidrome, Audiobookshelf, Sonarr, Radarr, Prowlarr, SABnzbd
mail	Roundcube (Postfix + ProtonMail Bridge disabled)
monitoring	Cluster Dashboard (FastAPI + Vite, port 8787)
edge	WebRTC bridge (port 8080)
ai-models	Ollama (deepseek-coder-v2:16b, NodePort 31434)
research	AlphaProof loop

DNS & TLS

• Domain: researchstack.info
• Auth: auth.researchstack.info (Authentik OIDC)
• Registry: registry.researchstack.info
• TLS: Caddy with Porkbun DNS-01 (wildcard certs)
• Funnel: 361395-1.tail4e7094.ts.net → Traefik NodePort 30080
• LE cert: Valid until 2026-08-18

URL Routing (Traefik Ingress)

Path	Service	Auth
/	Homer	SSO-gated
/apps/chat/*	Hermes	SSO + strip-prefix
/apps/budget/*	Actual Budget	SSO + strip-prefix
/server/status/*	Uptime Kuma	SSO + strip-prefix
/server/dash/*	Homarr	SSO + strip-prefix
/server/vault/*	Vaultwarden	SSO + strip-prefix
/api/cred/*	Credential Server	token-auth
/api/registry/*	Registry API	token-auth
/api/jobs/*	Jobs API	token-auth
/api/blobs/*	Blobs API	token-auth
auth.researchstack.info	Authentik	none

FPGA — Tang Nano 9K

Board: Sipeed Tang Nano 9K FPGA: Gowin GW1NR-LV9QN88PC6/I5 Clock: 27 MHz BRAM: 26 blocks (9Kbit each, 288Kbit total) LUTs: 6,480

Unified Bitstream: research_stack_top.fs

Module	LUTs	FFs	Function
Blitter6502OISC	~2000	~500	SUBLEQ CPU, 4K memory, UART
Q16 LUT Core	266	68	8 ops, 2-stage pipeline, 74ns
Memory Map	~100	~50	8-bit ↔ 32-bit bridge
Voltage Controller	~200	~100	4 BRAM modes (STORE/COMPUTE/APPROX/MORPHIC)
Scale Space BRAM	~300	~150	4 Gaussian kernel banks (σ=0.25/0.50/0.75/1.00)
HiGHS Pivot	~150	~80	3-stage simplex pipeline

Timing: 195.92 MHz (7.2x margin over 27 MHz target) Status: Flashed, Verilator verified (5/5 sims pass), UART TX confirmed in simulation

FPGA Toolchain

Tool	Version	Path
Yosys	0.64	/usr/bin/yosys
nextpnr-himbaechel	0.10-75	/usr/bin/nextpnr-himbaechel
gowin_pack	—	/usr/bin/gowin_pack
Verilator	5.048	/usr/bin/verilator
openFPGALoader	—	/usr/bin/openFPGALoader

Pin Mapping (Tang Nano 9K)

Pin	Signal	Direction	Type
52	clk	input	LVCMOS33
4	rst_n	input	LVCMOS33 (pull-up)
3	user_btn	input	LVCMOS33 (pull-up)
10-16	led[0:5]	output	LVCMOS18
17	uart_tx	output	LVCMOS33
18	uart_rx	input	LVCMOS33 (pull-up)

GPU Cores & Video-as-Compute

In addition to deep learning inference (served via Ollama), physical GPU accelerators and integrated APUs are utilized as high-throughput fixed-point vector engines via hardware video encoders/decoders (AMD VCN, NVIDIA NVDEC/NVENC).

GPU Detection & Optimization Routing

The system dynamically profiles host hardware configurations via vcn_compute_substrate.py to select optimal encoding/decoding strategies:

Hardware Type	Profile / Format	Key Optimization	Target / Use Case
NVIDIA dGPU (e.g. H100, RTX 4070)	H.265 / HEVC (yuv444p)	Lossless CABAC, High-Bandwidth Full Range	Dense vector structures, raw matrix mappings
AMD dGPU (e.g. Radeon)	H.265 / HEVC (yuv444p)	Full-Range Lossless VCN Bypass	Dedicated PCIe lanes, unconstrained memory
AMD APU / iGPU (e.g. Steam Deck)	H.265 / HEVC (yuvj420p)	UMA UMA-aware memory bandwidth throttling	Shared CPU/GPU DDR bus (saves 50% system memory bandwidth)
CPU / Fallback	Software H.265	Vector Processing (no hardware VCN)	Standard system CPU (low throughput)

• Lossless Range Correction: Limited-range video profiles clamp and corrupt raw binary bytes. All profiles enforce full-range YUV (yuvj420p, yuvj444p, or -color_range pc in FFmpeg) to maintain bit-level losslessness.
• Heterogeneous Frame Slicing: Dynamic payload routing extracts video formatting metrics via ffprobe to slice and distribute frame segments safely across heterogeneous compute nodes.

Node Details

Card: NVIDIA RTX 4070 Driver: NVML 610.43 Ollama: deepseek-coder-v2:16b (Q4_0, 8.9GB) NodePort: 31434

Lean 4 / Semantics

Toolchain: Lean 4.30.0-rc2 + Mathlib Build: lake build — 3572 jobs, 0 errors Modules: 10,353 .lean files

Key Modules

Module	Lines	Jobs	Status
FixedPoint	900+	—	Q16_16 type, arithmetic
AdjugateMatrix	535	3300	Matrix inversion, cofactor identity
GoldenRatioSeparation	123	3301	Lemma 3.4, φ boundary
DegeneracyConversion	311	3302	Unified gate condition
LadderBraidAlgebra	312	3314	Ladder operators → braid crossings
PenguinDecayLUT	356	3318	HEP equations → OTOM framework
RiemannianResonanceCorrelator	373	3317	PDE discovery from data
PhysicsPipeline	360	3320	8-stage particle physics pipeline
RouteCost	475	2	9-dimension cost (with latency)
OptimizedRoute	42	3	2-opt proof (14.1% shorter)

Sorry Inventory

File	Count	Status
AdjugateMatrix.lean	3	Q16_16 obstruction (documented)
FourPrimitiveErdosRenyi.lean	4	Research grade
HyperbolicStateSurface.lean	1	TODO with proof sketch
HamiltonianMechanics.lean	1	Legacy Picard-Lindelöf

Python Shims

Version: Python 3.14.5 Tests: 68/68 pass (test_braid_pipeline.py)

Core Dependencies

Package	Version	Purpose
highspy	1.14.0	HiGHS MIP/LP solver
reedsolo	≥1.7.0	Reed-Solomon ECC
cryptography	≥41.0.0	ChaCha20 encryption
numpy	≥1.24.0	Numerical operations
requests	≥2.31.0	HTTP client

Key Modules

Module	Function
qubo_highs.py	QUBO→MIP via HiGHS (8.4ms vs 53.5ms SA)
braid_search.py	Dense Sidon sets (Mian-Chowla), soliton search
alphaproof_loop.py	Ollama → lake build → feedback
scale_space_solver.py	Multi-scale optimization, Tailscale detection
particle_physics_lut.py	34 particle masses, 11 cross-sections as Q16_16
reed_solomon_vcn.py	RS encode/decode for VCN frames
chacha20_braid.py	ChaCha20 encryption for braid data
polynomial_commitment.py	KZG polynomial commitments
vcn_compute_substrate.py	Video-as-compute coordinator with GPU profiling, heterogeneous format detection, and lossless range clamping avoidance
qemu_framebuffer_packer.py	QEMU graphics framebuffer /dev/fb0 zero-copy mmap packer (ARGB8888 100% density mapping: 1 pixel = 1 scalar)

Skills (111 enabled)

Research Stack Skills (6)

Skill	Triggers	Enforcement
lean-proof	.lean, lake build, sorry, Q16_16, theorem	Proof quality contract
lean-autoformalization	autoformalization, paper to Lean	Paper→Lean pipeline
gowin-fpga-synthesis	synthesize verilog, fpga bitstream	Gowin bitstream flow
vcn-compute-substrate	VCN, braid encode, MKV blitter	VCN pipeline
research-stack-contracts	Research Stack, Q16_16, PIST, FAMM	12 hard-enforced rules
python-numpy-preference	python, numpy, array	Prefer NumPy

Hub Skills (12)

fpga, systemverilog, verilog-design, math-help, physics-intuition, hardware-counters, tensorrt-llm, formal-provers, pipeworx-newton, acorn-prover, numpy, cuopt-numerical-optimization, nemo-curator, Pre-flight Check

MCP Servers

Server	Tools	Path
eda	synthesize_verilog, simulate_verilog, view_waveform, run_openlane, view_gds, read_openlane_reports	~/.hermes/mcp-eda/build/index.js
lean	verify_lean_theorem	uvx lean-mcp
sympy	100+ symbolic math tools	uvx mcp-sympy
contextstream	memory, search, session management	MCP server
aws	AWS CLI tools (legacy — AWS instances shut down)	uvx awslabs.aws-api-mcp-server

Key Constants

Constant	Value	Source
Q16_16.one	65536	FixedPoint.lean
goldenAngleStep	40503	GoldenAngleEncoding.lean
goldenRatio	106008	GoldenRatioSeparation.lean
uartBaudDivisor	233	GenerateSparklePhiS3C.lean
UART baud rate	115384 Hz	27MHz / 234
MAX_CYCLES	1,000,000	Blitter6502OISC_small.v
Kolmogorov 4/5	52429	DegeneracyConversion.lean

Build Commands

# Lean full workspace
cd 0-Core-Formalism/lean/Semantics && lake buildObj

# Python tests & pipeline validation
cd 4-Infrastructure/shim && python3 test_braid_pipeline.py

# Framebuffer packer verification
python3 4-Infrastructure/shim/qemu_framebuffer_packer.py --verify

# GPU auto-detection & profiling
python3 4-Infrastructure/shim/vcn_compute_substrate.py --detect-gpu

# FPGA synthesis
cd 4-Infrastructure/hardware && bash build_research_stack.sh

# FPGA simulation
cd /tmp/fpga_sim_full && ./obj_dir/sim_top

# FPGA flash
openFPGALoader -b tangnano9k research_stack_top.fs

# k3s status
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
kubectl get pods -A

k3s Cluster Setup

Detailed cluster setup documentation: 4-Infrastructure/docs/k3s-cluster-setup.md

Topology (cupfox control plane)

Node	Tailscale IP	Role	Arch
cupfox	100.110.163.82	control-plane	amd64
neon-64gb	100.64.19.78	worker (heavy)	arm64
steamdeck	100.85.244.73	worker (gpu)	amd64
racknerd	100.80.39.40	worker (edge)	amd64

Ollama Inference Serving

• Host-level deployment on Neon-64GB (bypasses KServe RAM limits)
• Caddy reverse proxy on racknerd (:8443 → 100.64.19.78:11434)
• Troubleshooting: autosave.json stale config, passt port interception

netcup-vps (ARM64 EPYC)

NixOS configuration: 4-Infrastructure/netcup-vps/configuration.nix

ARM64-optimized packages:

• openblas, blis, lapack — multi-threaded linear algebra
• petsc, slepc — sparse/eigenvalue solvers
• flintqs, pari, gap, singular — number theory / algebra
• symengine — fast C++ symbolic (SymPy backend)
• fftw, suitesparse — FFT and sparse direct solvers
• z3, julia_11 — SMT and JIT numerics

k3s ports: 6443, 2379, 2380 (firewall updated)