Research-Stack/4-Infrastructure/docs/mesh-networking-over-ray-plan.md

Plan: Mesh Networking Layers Over Ray

The Stack (top to bottom)

┌─────────────────────────────────────────────────────────────┐
│  Application Layer                                            │
│  Braid search, AlphaProof, RG tests, PIST classification     │
├─────────────────────────────────────────────────────────────┤
│  Ray Scheduler                                                │
│  Distributed task scheduling, ObjectRef transport             │
│  @ray.remote, ray.put(), ray.get()                            │
├─────────────────────────────────────────────────────────────┤
│  VCN Pipeline (compression)                                   │
│  Delta+RLE → RS ECC → ChaCha20 → H.264 → MKV               │
│  50x bandwidth reduction over raw data                        │
├─────────────────────────────────────────────────────────────┤
│  FrameDispatcher (routing)                                    │
│  TAG_STRAND → VCNBraidBackend                                 │
│  TAG_CROSSING → VCNBraidBackend                               │
│  TAG_PIST → VCNBraidBackend                                   │
│  TAG_LUPINE → CUDABackend                                     │
├─────────────────────────────────────────────────────────────┤
│  Compute Surfaces (execution)                                 │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │
│  │ GPU_CUDA │ │ GPU_VAAPI│ │ ETHERNET │ │FRAMEBUFF │       │
│  │ NVENC    │ │ VAAPI    │ │ virtio   │ │ /dev/fb0 │       │
│  │ 12GB VRAM│ │ shared   │ │ TX/RX    │ │ DMA      │       │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘       │
├─────────────────────────────────────────────────────────────┤
│  Tailscale Mesh (connectivity)                                │
│  WireGuard encryption, DERP relay fallback                    │
│  Direct P2P when possible, relay when NAT'd                  │
│  MagicDNS, subnet routing, exit nodes                         │
├─────────────────────────────────────────────────────────────┤
│  Physical Network                                             │
│  Ethernet, WiFi, 4G/5G, satellite, mesh radio                │
└─────────────────────────────────────────────────────────────┘

What Changes When Tailscale Is the Transport

Current State (without mesh)

qfox-1 ──── direct IP ──── nixos (control plane)
   │
   └──── direct IP ──── racknerd (if reachable)

Ray nodes communicate via direct IPs. If a node is behind NAT or unreachable, it's offline. The cluster is limited to nodes with direct connectivity.

With Tailscale Mesh

qfox-1 ─── WireGuard ─── nixos ─── WireGuard ─── racknerd
   │            │            │            │            │
   └── WireGuard ── neon ──── WireGuard ── steamdeck ──┘
                  │
              DERP relay
              (fallback)

Every node in the Tailscale mesh becomes a potential Ray worker. The mesh handles:

• NAT traversal (DERP relay when direct fails)
• Encryption (WireGuard, post-quantum Kyber768)
• Discovery (MagicDNS, coordination server)
• Routing (direct P2P when possible, relay when not)

Architecture: Ray Over Tailscale

Layer 1: Control Plane (Ray GCS on nixos)

# Ray head on nixos (control-plane node)
# Binds to Tailscale IP 100.102.173.61
ray start --head \
  --node-ip=100.102.173.61 \
  --dashboard-host=0.0.0.0 \
  --port=6379

• Ray GCS (Global Control Store) on nixos
• Dashboard accessible via Tailscale funnel
• All workers connect via Tailscale IPs

Layer 2: GPU Workers (qfox-1)

qfox-1 (100.88.57.96)
├── Ray GPU worker (RTX 4070 SUPER)
│   └── @ray.remote(num_gpus=1)
├── Ray CPU worker (AMD iGPU)
│   └── @ray.remote(resources={"APU": 1})
├── VCN encode/decode (FFmpeg NVENC/VAAPI)
└── FrameDispatcher (VCNBraidBackend + CUDABackend)

• Highest capability tier (GPU_CUDA + GPU_APU)
• Handles all FFmpeg-dependent tasks
• NVENC for H.264 encode, VAAPI for AMD encode
• ObjectRef transfers to/from nixos over Tailscale

Layer 3: CPU Workers (neon-64gb)

neon-64gb (100.64.19.78)
├── Ray ARM64 worker (18 cores)
│   └── @ray.remote(num_cpus=16)
├── VCN encode (software libx264)
└── k3s agent (needs control plane fix)

• ARM64 compute for parallel CPU tasks
• Software H.264 encode (no GPU)
• Tailscale provides connectivity despite IP change

Layer 4: Ethernet Workers (racknerd)

racknerd (100.80.39.40)
├── Ray lightweight worker (2 vCPU)
├── PistPacket computation (virtio-net TX/RX rings)
│   └── Host vhost-user does matrix transforms
├── Framebuffer DMA (/dev/fb0, 1024x768, 1.57 MB)
└── Tier: ETHERNET + FRAMEBUFFER

• Can't run FFmpeg, can't do GPU work
• CAN send PistPackets via virtio-net
• CAN write Q16_16 to /dev/fb0
• Tailscale encrypts all traffic to/from this node
• DERP relay fallback if NAT'd

Layer 5: Edge Workers (steamdeck, ESP32, sensors)

steamdeck (100.85.244.73)
├── Ray worker (RDNA 2 iGPU, when online)
└── Tier: GPU_APU

ESP32 (hypothetical)
├── Q0_16 scalar in FreeRTOS idle hook
├── 520 KB SRAM, 240 MHz Xtensa
└── Tier: ESP32

DS18B20 (hypothetical)
├── 1-Wire trits during 750ms temperature conversion
└── Tier: RELAY (data collection only)

Data Flow: Task Submission to Result

User submits task
    │
    ▼
Ray scheduler (on nixos)
    │
    ├── Is it GPU work? → qfox-1 (GPU_CUDA)
    │   └── NVENC encode → ObjectRef → Tailscale → result
    │
    ├── Is it CPU-parallel? → neon-64gb (GPU_APU/CPU_FFMPEG)
    │   └── ARM64 parallel → ObjectRef → Tailscale → result
    │
    ├── Is it lightweight? → racknerd (ETHERNET)
    │   └── PistPacket → virtio-net TX ring → host transform → RX ring
    │   └── OR framebuffer → /dev/fb0 DMA → host reads → result
    │
    └── Is it data collection? → ESP32/sensors (RELAY)
        └── 1-Wire/I2C/SPI → data → Tailscale → Ray scheduler

VCN Compression Over Tailscale

The VCN pipeline compresses data before sending over the Tailscale mesh:

Raw braid data: 1 MB per strand
    ↓ Delta+RLE: 200 KB (5x)
    ↓ RS ECC: 250 KB (with parity)
    ↓ ChaCha20: 250 KB (encrypted)
    ↓ H.264 frame: 20 KB (50x)
    ↓ MKV container: 22 KB

Tailscale WireGuard overhead: ~60 bytes per packet
Total per strand: ~22 KB over mesh

1000 strands:
  Raw: 1000 MB over mesh → 80 seconds at 100 Mbps
  VCN: 22 MB over mesh → 1.8 seconds at 100 Mbps

DERP relay adds ~129ms per hop, but the 50x compression means the data transfer is negligible. The latency becomes the bottleneck, not bandwidth.

Mesh-Aware Scheduling

Ray should be aware of the Tailscale mesh topology:

@ray.remote(
    num_gpus=1,
    scheduling_strategy="NODE_AFFINITY",
    affinity_node="100.88.57.96",  # qfox-1 (GPU)
)
def gpu_task(data):
    ...

Latency Classes (from RouteCost.lean)

Class	RTT	Tailscale Path	FPGA Voltage
LOCAL	<1ms	Same node	1.2V
NEAR	<10ms	Direct P2P	1.0V
FAR	<100ms	Multi-hop	0.8V
DERP	<1s	Relay	0.6V
OFFLINE	≥1s	Unreachable	0V

Placement Strategy

def get_placement(caps: DeviceCapabilities, latency_class: str) -> dict:
    """Mesh-aware Ray placement strategy."""
    base = get_ray_placement_strategy(caps)

    # Penalize DERP-relayed nodes (129ms extra latency)
    if latency_class == "DERP":
        base["resources"]["derp_penalty"] = 1

    # Prefer direct P2P nodes
    if latency_class in ("LOCAL", "NEAR"):
        base["resources"]["direct_link"] = 1

    return base

Implementation Steps

Phase 1: Ray over Tailscale (current)

• Ray cluster on qfox-1 (head + workers)
• KubeRay operator on nixos
• nftables fix for pod-to-pod networking
• Fix neon-64gb k3s agent → nixos control plane
• Ray head binds to Tailscale IP
• Workers connect via Tailscale IPs

Phase 2: Multi-tier scheduling

• Device capability probe (GPU_CUDA → ESP32)
• Framebuffer fallback (qemu_framebuffer_packer.py)
• Ethernet tier (virtio-net PistPacket)
• Mesh-aware placement strategy
• VCN compression for cross-node ObjectRef transfers

Phase 3: Framebuffer + Ethernet integration

• Host-side framebuffer reader (QEMU display buffer)
• Host-side PistPacket transform engine (vhost-user)
• Ray actor for framebuffer devices
• Ray actor for Ethernet devices
• Round-trip test: write → host transform → readback

Phase 4: Edge devices

• ESP32 firmware (Q0_16 scalar, FreeRTOS idle hook)
• 1-Wire sensor integration (DS18B20 trit stream)
• Ray relay for data collection devices
• Tailscale exit node for internet access

Key Insight

The mesh networking layer doesn't just provide connectivity — it provides a capability-adaptive compute fabric. Every device in the Tailscale mesh is a potential compute node. The capability probe determines what each device can do, and Ray schedules work accordingly.

The hierarchy:

1. Tailscale connects everything (mesh networking)
2. Ray schedules work across the mesh (distributed computing)
3. VCN compresses data for the mesh (bandwidth optimization)
4. FrameDispatcher routes to the right backend (tag-based routing)
5. Compute surfaces execute the work (GPU, CPU, framebuffer, Ethernet, MCU)

The framebuffer and Ethernet surfaces are the breakthrough — they turn devices that "can't run Ray" into compute nodes. A microVM with no GPU and no FFmpeg can still participate via /dev/fb0 or virtio-net. An ESP32 with 520 KB SRAM can still contribute Q0_16 scalars.