Research-Stack/.hermes/plans/2026-05-29_220000-ray-deployment.md

Plan: Ray Deployment on Research Stack k3s

Date: 2026-05-29 Status: Draft Estimated effort: 2-3 days

---

Goal

Deploy Ray on the Research Stack k3s cluster with:

1. Distributed compute across 5 nodes (2 Ready, 3 flaky)
2. H.264 transport via Ray Object Store
3. GPU acceleration on qfox (RTX 4070)
4. Thundering herd mitigations for node reconnection
5. Graceful degradation when nodes go offline

---

Current State

Node	Status	Role	Hardware
nixos	Ready	control-plane	NixOS, 6.18.32 kernel
qfox-1	Ready	worker	CachyOS, RTX 4070
361395-1	NotReady	worker	Debian, edge VPS
racknerd	NotReady	worker	Debian, VPS
steamdeck	NotReady	worker	NixOS

Namespaces: ai-models (empty), services, media, mail, monitoring, research, edge

Existing workloads:

• VCN pipeline: braid_vcn_encoder.py (Delta+RLE + RS ECC + ChaCha20 + H.264)
• Fractal dimension: fractal_dimension.py (DBC algorithm, 29x vectorized)
• GPU node: vcn_lupine_gpu_node.py (MKV decode + GPU compute)
• AlphaProof: alphaproof_loop.py (Ollama → lake build → feedback)
• Lean: 3572 jobs, 0 errors
• Python: 68/68 tests pass

---

Architecture

Ray Topology

Neon-64GB (Ray Head + Worker)
├── GCS (Global Control Store) — PVC-backed
├── Dashboard (:8265) — via Traefik ingress
├── Object Store — 32GB allocated
├── Raylet — 18 ARM64 cores
└── VCN Encoder — Delta+RLE + RS ECC + ChaCha20

qfox-1 (Ray GPU Worker)
├── Raylet — RTX 4070
├── NVDEC — H.264 hardware decode (0.1ms)
├── CUDA — GPU compute (fractal dimension, Q16 LUT)
└── Object Store — 8GB allocated

nixos (Ray CPU Worker)
├── Raylet — CPU cores
├── Software decode — H.264 fallback
├── Lean builds — lake build via Ray tasks
└── Object Store — 4GB allocated

racknerd (Ray Edge Worker — standby)
├── Raylet — 2 vCPU EPYC
├── Edge relay — forward to external clients
└── Object Store — 2GB allocated

steamdeck (Ray Idle Worker — standby)
├── Raylet — idle until needed
└── Object Store — 2GB allocated

Network Flow

Braid data (Python)
    → Delta+RLE (vectorized copy-if, 3.3x)
    → RS ECC
    → ChaCha20
    → H.264 frame (YUV420)
    → MKV container
    → ray.put() → ObjectRef
    → Ray Object Store (shared memory)
    → GPU hardware decode (NVDEC) or CPU fallback
    → GPU compute or CPU compute
    → Result → ray.put() → ObjectRef
    → Downstream consumer

Port Map

Port	Service	Access
6379	Ray GCS	Internal (Tailscale only)
8265	Ray Dashboard	Via Traefik ingress
10001	Ray Client	Tailscale only
8000	Ray Serve	Via Traefik ingress

---

Implementation Steps

Phase 1: Ray Head Deployment (Day 1)

Step 1.1: Create Ray namespace and RBAC

• File: 4-Infrastructure/k3s/ray/namespace.yaml
• Create ai namespace (already exists, use it)
• ServiceAccount, ClusterRole, ClusterRoleBinding for Ray

Step 1.2: Deploy Ray Head on Neon-64GB

• File: 4-Infrastructure/k3s/ray/head-deployment.yaml
• Image: rayproject/ray:2.43.0-py311-cu124
• Command: ray start --head --dashboard-host=0.0.0.0 --object-store-memory=34359738368
• PVC: 10Gi for GCS persistence
• Resources: 4 CPU request, 16 CPU limit, 8Gi RAM request, 32Gi RAM limit
• NodeSelector: kubernetes.io/hostname: neon-64gb
• Liveness probe: ray health-check
• Readiness probe: ray status

Step 1.3: Deploy Ray Dashboard Service

• File: 4-Infrastructure/k3s/ray/dashboard-service.yaml
• ClusterIP service on port 8265
• Traefik IngressRoute at /ray/dashboard

Step 1.4: Verify Head is running

• kubectl get pods -n ai
• curl http://localhost:8265 (port-forward)
• ray status inside head pod

Phase 2: Worker Deployment (Day 1)

Step 2.1: Deploy Ray GPU Worker on qfox-1

• File: 4-Infrastructure/k3s/ray/gpu-worker-deployment.yaml
• Image: rayproject/ray:2.43.0-py311-cu124
• Command: ray start --address=ray-head.ai.svc.cluster.local:6379
• Resources: 2 CPU, 8Gi RAM, 1 nvidia.com/gpu
• NodeSelector: kubernetes.io/hostname: qfox-1
• Toleration: nvidia.com/gpu:NoSchedule
• Environment: NVIDIA_VISIBLE_DEVICES=all

Step 2.2: Deploy Ray CPU Worker on nixos

• File: 4-Infrastructure/k3s/ray/cpu-worker-deployment.yaml
• Image: rayproject/ray:2.43.0-py311-cu124
• Command: ray start --address=ray-head.ai.svc.cluster.local:6379
• Resources: 4 CPU, 8Gi RAM
• No GPU resources

Step 2.3: Deploy Standby Workers (racknerd, steamdeck)

• File: 4-Infrastructure/k3s/ray/standby-worker-deployment.yaml
• Same as CPU worker but with lower resources
• Toleration: node.kubernetes.io/unreachable:NoSchedule
• Priority: low (evicted first when nodes are under pressure)

Step 2.4: Verify all workers connected

• ray status inside head pod
• Check: 4 workers connected (Neon, qfox, nixos, racknerd/steamdeck if online)

Phase 3: Thundering Herd Mitigations (Day 2)

Step 3.1: Staggered reconnection

• File: 4-Infrastructure/k3s/ray/worker-entrypoint.sh
• Script: compute jitter from Tailscale IP hash, sleep before connecting
• delay = hash(tailscale_ip) % 30
• Mount as ConfigMap, use as worker command

Step 3.2: Token bucket on Head

• File: 4-Infrastructure/k3s/ray/head-entrypoint.sh
• Configure Ray with --max-worker-startup-concurrency=1
• One worker connects per second, burst of 3
• Workers get 429 + Retry-After on rejection

Step 3.3: Object Store watermarks

• File: 4-Infrastructure/k3s/ray/object-store-config.yaml
• Configure Ray with memory thresholds:
  • 60%: throttle incoming syncs to 1/s
  • 80%: reject new connections
  • 95%: emergency GC
• Spill cold objects to PVC at 60%

Step 3.4: Circuit breaker per worker

• File: 4-Infrastructure/shim/ray_circuit_breaker.py
• Python class wrapping Ray client calls
• States: CLOSED (normal), OPEN (head overwhelmed), HALF-OPEN (testing)
• Failure threshold: 5 failures in 30s → OPEN
• Recovery: 60s wait → HALF-OPEN → probe → CLOSED

Step 3.5: Graduated task migration

• File: 4-Infrastructure/shim/ray_task_migrator.py
• When GPU node returns, migrate tasks in batches:
  • t=0s: 10% of tasks
  • t=30s: 20% more
  • t=60s: remaining 70%
• Abort if any batch fails

Step 3.6: Witness deduplication

• Ray ObjectRefs are already content-addressed
• Verify: ray.put(data) returns same ObjectRef for same data
• No custom code needed — document the behavior

Phase 4: VCN Pipeline Integration (Day 2)

Step 4.1: Wire VCN encoder to Ray Object Store

• File: 4-Infrastructure/shim/vcn_ray_transport.py
• encode_and_store(braid_data) → ObjectRef
• decode_on_gpu(ObjectRef) → np.ndarray
• decode_on_cpu(ObjectRef) → np.ndarray (fallback)

Step 4.2: Wire fractal dimension to Ray

• File: 4-Infrastructure/shim/fractal_ray_task.py
• @ray.remote(num_gpus=1) def compute_fd_gpu(data)
• @ray.remote(num_cpus=4) def compute_fd_cpu(data)
• Auto-select based on GPU availability

Step 4.3: Wire AlphaProof to Ray

• File: 4-Infrastructure/shim/alphaproof_ray.py
• @ray.remote def prove_theorem(problem)
• Parallel proof search across workers
• Pre-filter: skip trivial theorems (copy-if pattern)

Step 4.4: Test end-to-end

• Encode 100 braid strands → Object Store → GPU decode → compute → verify
• Measure: encode time, transfer time, decode time, compute time
• Verify: 3.3x speedup on Delta+RLE, 100x on H.264 decode

Phase 5: k3s Migration (Day 3)

Step 5.1: Migrate control plane

• Stop k3s-server on Neon-64GB
• Install k3s-server on cupfox
• Update kubeconfig to point to cupfox
• Update Tailscale Funnel target

Step 5.2: Re-point workers

• Update k3s-agent on all workers to point to cupfox
• Verify: all nodes join new control plane

Step 5.3: Update DNS and ingress

• Update researchstack.info A record → cupfox IP
• Update Caddy upstream on racknerd
• Verify: Traefik ingress works

Step 5.4: Verify Ray survives migration

• Ray Head should reschedule on Neon (worker node now)
• GCS persists on PVC (survives restart)
• Workers reconnect to new Head

Phase 6: Monitoring and Observability (Day 3)

Step 6.1: Ray Dashboard ingress

• File: 4-Infrastructure/k3s/ray/dashboard-ingress.yaml
• Traefik IngressRoute at researchstack.info/ray/
• SSO-gated via Authentik

Step 6.2: Prometheus metrics

• Ray exports Prometheus metrics at :8080/metrics
• Scrape config in monitoring namespace
• Grafana dashboard for Ray cluster health

Step 6.3: Alerting

• Alert on: worker disconnect, Object Store pressure, task failures
• Route to: Telegram/Discord via Alertmanager

---

Files to Create

File	Purpose
4-Infrastructure/k3s/ray/namespace.yaml	Namespace + RBAC
4-Infrastructure/k3s/ray/head-deployment.yaml	Ray Head on Neon
4-Infrastructure/k3s/ray/head-service.yaml	ClusterIP + IngressRoute
4-Infrastructure/k3s/ray/gpu-worker-deployment.yaml	GPU worker on qfox
4-Infrastructure/k3s/ray/cpu-worker-deployment.yaml	CPU worker on nixos
4-Infrastructure/k3s/ray/standby-worker-deployment.yaml	Standby on racknerd/steamdeck
4-Infrastructure/k3s/ray/worker-entrypoint.sh	Staggered reconnection script
4-Infrastructure/k3s/ray/head-entrypoint.sh	Token bucket config
4-Infrastructure/k3s/ray/object-store-config.yaml	Watermark thresholds
4-Infrastructure/k3s/ray/dashboard-ingress.yaml	Dashboard Traefik route
4-Infrastructure/shim/ray_circuit_breaker.py	Circuit breaker per node
4-Infrastructure/shim/ray_task_migrator.py	Graduated migration
4-Infrastructure/shim/vcn_ray_transport.py	VCN → Ray Object Store
4-Infrastructure/shim/fractal_ray_task.py	Fractal dimension on Ray
4-Infrastructure/shim/alphaproof_ray.py	AlphaProof on Ray

---

Files to Modify

File	Change
4-Infrastructure/k3s-flake/k3s-edge.nix	Update Traefik for Ray ingress
6-Documentation/INFRASTRUCTURE.md	Add Ray section
6-Documentation/RUNBOOK.md	Add Ray troubleshooting
AGENTS.md	Add Ray deployment rules

---

Validation

Phase 1 Validation

• Ray Head pod Running on Neon
• Dashboard accessible at localhost:8265 (port-forward)
• ray status shows 1 node (Head)

Phase 2 Validation

• GPU worker connected (qfox)
• CPU worker connected (nixos)
• Standby workers connected (racknerd, steamdeck if online)
• ray status shows 4+ nodes

Phase 3 Validation

• Simulate node disconnect: kill qfox pod
• Verify: GPU tasks queued, not failed
• Restart qfox pod
• Verify: staggered reconnection (watch logs)
• Verify: graduated migration (5→15→35→50 tasks)
• Verify: Object Store watermark triggers at 60%

Phase 4 Validation

• ray.put(braid_data) returns ObjectRef
• ray.get(ObjectRef) returns original data
• GPU decode works on qfox
• CPU decode fallback works on nixos
• 100 braid strands: end-to-end < 10s

Phase 5 Validation

• Control plane migrated to cupfox
• All workers join new control plane
• Ray Head reschedules on Neon
• GCS persists across restart

Phase 6 Validation

• Dashboard accessible via Traefik
• Prometheus scraping Ray metrics
• Grafana dashboard shows cluster health

---

Risks and Mitigations

Risk	Impact	Mitigation
Neon ARM64 image compatibility	Some services won't run	Test images before deploying
Ray Head on 4GB cupfox	OOM	Keep Head on Neon (64GB), cupfox is control-plane only
Tailscale DERP latency	Slow Ray RPC	Use direct connections when possible
Object Store memory pressure	OOM on workers	Watermarks + PVC spill
GPU driver mismatch	NVDEC fails	Fall back to CPU software decode
etcd on cupfox 128GB disk	Disk full	Monitor disk usage, set alerts

---

Open Questions

1. ARM64 Ray image: Does rayproject/ray:2.43.0-py311-cu124 have ARM64 builds? If not, need to build custom image.
2. NVDEC on qfox: Is the NVIDIA driver new enough for NVDEC? Need to verify with nvidia-smi.
3. PVC access from workers: Can Ray workers on different nodes mount the same PVC? Need ReadWriteMany or NFS.
4. Ray version: 2.43.0 is latest stable. Check compatibility with k3s 1.35.4.
5. Cost: Ray Head on Neon uses 8-32GB RAM. Is that acceptable alongside other workloads?

---

Success Criteria

• Ray cluster operational with 4+ nodes
• VCN pipeline runs on Ray (encode → transfer → decode → compute)
• GPU acceleration working (NVDEC + CUDA)
• Thundering herd mitigations tested (node disconnect/reconnect)
• Dashboard accessible via Traefik
• AlphaProof runs on Ray (parallel proof search)
• All existing workloads still functional (k3s services, Lean, Python)