Research-Stack/4-Infrastructure/shim/vcn_lupine_bridge_spec.md

VCN-LUPINE Bridge — Unified Compute Transport over Video

Schema: vcn_lupine_bridge_spec_v1 Date: 2026-05-29 Status: Research artifact — not yet promoted

---

1. Motivation

VCN (Video Core Next) and LUPINE are both GPU-over-IP bridges at different layers:

Layer	VCN	LUPINE
Transport	H.264/MKV video frames	HTTP/2
Hardware	AMD VCN (video encode)	NVIDIA CUDA (compute)
Data model	Braid operations (strand, crossing, PIST)	CUDA driver API calls
Use case	Eigensolid compressor, braid crossing loop	CUDA-compute workloads

Both encode structured computation as byte sequences that must survive network transmission with integrity guarantees. Both need encryption and error correction.

Thesis: These are the same abstraction at different levels of the stack. We can unify them by:

1. Encoding both VCN braid-ops and LUPINE CUDA calls in the same binary frame format
2. Transporting those frames via H.264 video (the VCN pipeline)
3. Decoding on the GPU node and dispatching to the appropriate compute backend

This eliminates the HTTP/2 transport layer for LUPINE — it rides on VCN instead.

---

2. Architecture

2.1 Network Topology

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Tailscale Mesh                                     │
│                                                                             │
│   ┌──────────────┐                              ┌──────────────────────┐  │
│   │   netcup VPS │  ◄──── MKV stream ────────► │   qfox-1             │  │
│   │  rs-vps      │  (H.264, VCN-encoded)        │  (AMD VCN + NVIDIA) │  │
│   │  (ARM64)     │                              │  100.88.57.96       │  │
│   │  152.53.81.  │                              │                      │  │
│   │  164         │                              │  AMD VCN ← decode   │  │
│   │              │                              │  NVIDIA ← CUDA      │  │
│   │  LUPINE      │                              │                      │  │
│   │  preload     │                              │                      │  │
│   │  libcuda.so.1│                              │                      │  │
│   └──────┬───────┘                              └──────────────────────┘  │
│          │                                                                │
│          │  LD_PRELOAD intercepts CUDA API calls                           │
│          │  Serializes to JSON-braid → unified frame format               │
│          │                                                                │
└──────────┼────────────────────────────────────────────────────────────────┘
           │
           ▼
    IPC socket (abstract)
    /run/vcn-lupine/daemon.sock

2.2 Compute Flow

Path A — CUDA compute (LUPINE):

1. Python/C binary calls cuBLAS/cuDNN/cuSOLVER API
2. LD_PRELOAD=libcuda.so.1 intercepts call
3. Call serialized to JSON → JSON-braid frame
4. Frame sent over IPC to vcn-lupine-daemon
5. Daemon: JSON → unified frame bytes (TAG_LUPINE)
6. FFmpeg: frame bytes → H.264 YUV420 frame
7. MKV mux → sent over Tailscale UDP to GPU node
8. GPU node: MKV demux → H.264 decode
9. GPU node dispatch: TAG_LUPINE → LUPINE server → NVIDIA GPU
10. Result serialized → JSON → unified frame (reply)
11. Reverse path back to VPS
12. libcuda.so.1 returns result to calling program

Path B — Braid compute (VCN):

1. braid_vcn_encoder.py builds BraidStrand/BraidBracket dict
2. Calls encode_braid_strand() / encode_braid_crossing()
3. Frame bytes (TAG_STRAND / TAG_CROSSING) → IPC → daemon
4. Daemon → FFmpeg → H.264 → MKV → Tailscale
5. GPU node decode → dispatch: TAG_STRAND → VCN compute path
6. Result (phase_acc, crossing_residual) → reply frame

2.3 Unified Frame Format

Frames are the same binary format as vcn_compute_substrate.py with one new tag:

TAG_STRAND   = 0x01   # braid strand state
TAG_CROSSING = 0x02   # braid crossing operation
TAG_PIST     = 0x03   # PIST spectral data
TAG_LUPINE   = 0x04   # LUPINE CUDA operation (JSON-braid wrapper)

LUPINE frame payload (when TAG_LUPINE):

[4 bytes: request_id (UInt32LE)]
[4 bytes: opcode (UInt32LE: 1=cudaMalloc, 2=cuBLASgemm, 3=cuDNN conv, etc.)]
[4 bytes: payload_length (UInt32LE)]
[payload_length bytes: JSON-encoded argument struct]

JSON argument structure (matches LUPINE CUDA API calls):

{
  "api": "cudaMalloc",
  "args": {
    "ptr": 0,
    "size": 4096
  }
}

Full LUPINE opcode map:

Opcode	API	JSON args
1	cudaMalloc	{"ptr": 0, "size": N}
2	cudaFree	{"ptr": HANDLE}
3	cudaMemcpy	{"dst": HANDLE, "src": HANDLE, "bytes": N, "kind": 1}
4	cuBLASgemm	{"handle": H, "transA": 0, "transB": 0, "m": M, "n": N, "k": K, "alpha": FP32, "A": HANDLE, "lda": INT, "B": HANDLE, "ldb": INT, "beta": FP32, "C": HANDLE, "ldc": INT}
5	cuDNNConvolutionForward	{"handle": H, "x_desc": HANDLE, "x": HANDLE, "w_desc": HANDLE, "w": HANDLE, "conv_desc": HANDLE, "algo": INT, "workspace": HANDLE, "workSize": N}
6	cuSOLVERDnorgqr	{"handle": H, "A": HANDLE, "lda": INT, "n": INT, "tau": HANDLE}
7	nvmlDeviceGetCount	{}
8	nvmlDeviceGetName	{"index": 0}

Reply frame (same TAG, different flag):

[1 byte: TAG_LUPINE]
[1 byte: flags (0x80 = reply)]
[4 bytes: request_id (echo)]
[4 bytes: status (0=OK, -1=error)]
[4 bytes: result_length]
[result_length bytes: JSON result]

---

3. Components

3.1 libcuda.so.1 Preload Shim (VPS side)

Intercepts CUDA driver API calls. Lives at /opt/vcn-lupine/lib/libcuda.so.1 on VPS.

Intercepted symbols (from client.exports):

cudaMalloc, cudaFree, cudaMemcpy, cudaMemcpyAsync,
cuBLAScreate, cuBLASdestroy, cuBLASgemm,
cuDNNcreate, cuDNNdestroy, cuDNNconvolutionForward,
cuSOLVERcreate, cuSOLVERdestroy, cuSOLVERdnorgqr,
nvmlInit, nvmlDeviceGetCount, nvmlDeviceGetName, nvmlDeviceGetHandleByIndex

Flow:

1. dlsym(RTLD_NEXT, "cudaMalloc") gets real libcuda (if any — on CPU-only there is none)
2. Serialize call to JSON
3. Write JSON to /run/vcn-lupine/daemon.sock (abstract Unix socket)
4. Read reply JSON from socket
5. Return result to caller

Socket protocol:

Send:    JSON bytes (length-prefixed with 4-byte UInt32LE)
Recv:    JSON bytes (length-prefixed)

3.2 vcn-lupine-daemon (VPS side)

Daemon running on the VPS. Receives IPC from libcuda shim and braid encoders.

Endpoints:

• bind:unix:/run/vcn-lupine/daemon.sock — IPC from shim + encoders
• connect:tcp:100.88.57.96:14834 — MKV stream to GPU node

One socket pair per request:

IPC client → daemon (Unix socket)
  └─► GPU node (TCP/MKV stream)
        └─► LUPINE server / VCN decoder
        ◄── reply
      ◄─ daemon
    ◄─ IPC client

Commands:

FRAME_SEND  <tag> <seq> <payload_bytes>  → sends frame to GPU node
FRAME_RECV  <tag> <seq>                 → receives frame from GPU node
STATUS                                → returns connection health

3.3 GPU Node Receiver (qfox-1 side)

Receives MKV stream from VPS. Lives on qfox-1.

Components:

1. MKV demux listener — TCP:14834, receives VCN-encoded MKV from VPS
2. H.264 decode — AMD VCN hardware decode via ffmpeg -c:v h264_qsv or h264_vaapi
3. Frame dispatch — reads TAG byte, routes to:
   • TAG_LUPINE → LUPINE CUDA shim (libcuda.so.1) → NVIDIA GPU
   • TAG_STRAND/TAG_CROSSING/TAG_PIST → existing vcn_compute_substrate decode path

Reply path: result encoded as TAG_LUPINE frame → H.264 encode → MKV mux → TCP back to VPS

3.4 Unified Frame Byte Layout

All frames share the same outer structure (matches vcn_compute_substrate.py):

Byte 0:      tag (0x01-0x04)
Byte 1:      flags (0x00=request, 0x80=reply)
Bytes 2-5:   sequence number (UInt32LE, big-endian)
Bytes 6-9:   payload_length (UInt32LE, little-endian)
Bytes 10+:   payload (payload_length bytes)

Payload for TAG_LUPINE:
  [4]  request_id
  [4]  opcode
  [4]  args_length
  [N]  JSON args

---

4. Implementation Plan

4.1 Files to Create

File	Purpose
4-Infrastructure/shim/vcn_lupine_bridge.py	Core: unified frame format, dispatch, socket protocol
4-Infrastructure/shim/vcn_lupine_daemon.py	VPS-side daemon (IPC socket server, MKV client)
4-Infrastructure/shim/vcn_lupine_gpu_node.py	GPU-node-side receiver + dispatch
4-Infrastructure/shim/libcuda_preload.c	CUDA preload shim (C, compiled to .so)
4-Infrastructure/shim/vcn_lupine_opcodes.py	LUPINE opcode constants + JSON arg schemas

4.2 Existing Files to Modify

File	Change
4-Infrastructure/shim/vcn_compute_substrate.py	Add TAG_LUPINE = 0x04, TAG_REPLY = 0x80, OP_LUPINE_CUDA frame handler
4-Infrastructure/netcup-vps/configuration.nix	Add vcn-lupine-daemon service, libcuda_preload package
4-Infrastructure/netcup-vps/flake.nix	Add lupine flake input

4.3 NixOS Service Config (VPS)

# VPS side: daemon + preload shim
systemd.services.vcn-lupine-daemon = {
  wantedBy = [ "multi-user.target" ];
  after = [ "network.target" ];
  serviceConfig = {
    ExecStart = "${vcn-lupine}/bin/vcn-lupine-daemon";
    RuntimeDirectory = "vcn-lupine";
    RestrictNamespaces = true;
    PrivateTmp = true;
  };
};
environment.etc."ld.so.preload" = {
  text = "/opt/vcn-lupine/lib/libcuda.so.1";
};

4.4 GPU Node Service Config (qfox-1)

# qfox-1 side: receiver + dispatch
systemd.services.vcn-lupine-gpu-node = {
  wantedBy = [ "multi-user.target" ];
  after = [ "network.target" ];
  serviceConfig = {
    ExecStart = "${vcn-lupine}/bin/vcn-lupine-gpu-node --port 14834";
    RuntimeDirectory = "vcn-lupine";
  };
};

---

5. VCN Compute vs LUPINE: Same Abstraction

The key insight is that both systems are doing structured computation serialization:

Aspect	VCN braid compute	LUPINE CUDA compute
Input	BraidStrand dict	CUDA API call (JSON)
Serialization	Binary (21/42 bytes)	JSON text
Integrity	RS(255,223) + ChaCha20	HTTP/2 + TLS
Transport	H.264/MKV frames	HTTP/2/TCP
Hardware	AMD VCN encode	NVIDIA GPU
Computation	Phase accumulation, crossing residual	GEMM, convolution, QR

The unification is encoding both as video frames:

• VCN native: binary → RS → ChaCha20 → VCN frame
• LUPINE: JSON → unified frame → RS → ChaCha20 → VCN frame

Both arrive at the same H.264/MKV output. The GPU node dispatches by TAG.

---

6. Seal (Receipt)

No formal proof yet — this is a design document. The formal claim:

Unified Transport Thesis: Every GPU-compute operation (CUDA driver API or braid crossing) can be encoded as an H.264 video frame carrying a binary payload, transmitted over an unreliable UDP-based network, and correctly reconstructed at the receiver with bounded error (RS纠错码 < 16 symbol errors, ChaCha20 authenticated encryption).

This requires two Lean proofs:

1. Frame encoding soundness — the binary serialization is bijective for all valid inputs
2. Dispatch correctness — TAG routing delivers LUPINE frames to CUDA backend and braid frames to VCN compute path

---

7. Open Questions

1. H.264 encode latency — VCN encode on AMD takes ~33ms/frame at 1080p30. LUPINE's HTTP/2 is much lower latency. Does the video codec overhead dominate?
2. MKV container vs raw UDP — VCN currently uses MKV for frame packaging (container, streaming). LUPINE uses raw HTTP/2. Should we keep MKV or drop to raw H.264 PES?
3. Multi-GPU dispatch — LUPINE supports multiple servers (comma-separated). Does the unified bridge need to track which GPU node owns which CUDA device?
4. LUPINE server on VPS side — Currently LUPINE server runs on GPU node. Does the unified bridge need a reverse path where GPU node initiates work to VPS?
5. FFmpeg dependency — The daemon relies on FFmpeg for H.264 encode/decode. Can we use the VCN hardware encoder directly via libva/AMF, or is FFmpeg sufficient?