# Infrastructure Status — Research Stack

> **Last verified:** 2026-05-30
> **Scope:** All live nodes, services, and storage in the Tailscale mesh.

---

## Cluster Overview

All nodes run k3s v1.35.5+k3s1 (except steamdeck: v1.34.5+k3s1) joined into a single cluster,
networked over Tailscale. cupfox is the control-plane; all other nodes are workers.

**kubeconfig:** retrieve from cupfox at `/etc/rancher/k3s/k3s.yaml` (replace `0.0.0.0` → `100.110.163.82` for Tailscale access).

```bash
# Access cluster from anywhere via Tailscale
ssh root@100.110.163.82 'cat /etc/rancher/k3s/k3s.yaml' | sed 's|0.0.0.0|100.110.163.82|g' > ~/.kube/config
kubectl get nodes
```

| Node | k3s Role | CPU | RAM | Arch | Public IP | Tailscale | Notes |
|------|----------|-----|-----|------|-----------|-----------|-------|
| cupfox | control-plane | 2 | 3.8 GiB | amd64 | 46.232.249.226 | 100.110.163.82 | k3s + KubeRay operator |
| neon-64gb | worker | 18 | 62.7 GiB | **arm64** | 152.53.81.164 | 100.100.75.113 | Heavy stateful + inference |
| racknerd | worker | 1 | 715 MiB | amd64 | 172.245.19.182 | 100.80.39.40 | TLS via Caddy, too small for Ray |
| steamdeck | worker | 8 | 14.5 GiB | amd64 | — | 100.85.244.73 | NixOS 25.11, Ray head + worker running |

> **Note:** neon-64gb is ARM64 — `rayproject/ray` images are amd64 only. Build or pull multi-arch
> images to use its 18 cores. Racknerd has only 715 MiB RAM and cannot run Ray pods.

---

## k3s Cluster — cupfox

k3s v1.35.5+k3s1 runs as control-plane on cupfox. Flannel.1 (mtu 1280) handles pod networking.

### System pods

| Pod | Ready | Node | Status |
|-----|-------|------|--------|
| coredns-6b6544b569-n86sp | 1/1 | steamdeck | Running |
| local-path-provisioner-5d9d9885bc-5hbt2 | 1/1 | cupfox | Running |
| metrics-server-56dd944747-g59vn | 1/1 | steamdeck | Running |

### Ray on k3s — KubeRay Operator

| Component | Version | Namespace | Status |
|-----------|---------|-----------|--------|
| kuberay-operator | 1.6.1 | ray-system | Running |
| raycluster (head) | 2.40.0 | ray-system | Running on steamdeck |
| raycluster (worker) | 2.40.0 | ray-system | Running on steamdeck |

```bash
# Check Ray status
kubectl exec -it raycluster-head-xxx -n ray-system -- ray status

# Ray Python test
kubectl exec -it raycluster-head-xxx -n ray-system -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
```

**RayCluster manifest:** `4-Infrastructure/kube/raycluster.yaml`

---

## Storage: Garage S3

Garage v2.3.0 — self-hosted S3-compatible object store, Tailscale-only access.

### Cluster Topology

| Node | Tailscale IP | Address | Capacity | DataAvail |
|------|-------------|---------|----------|-----------|
| cupfox | 100.110.163.82 | — | 68.9 GiB | 65.2 GiB (52.3%) |
| qfox-1 (local) | 100.88.57.96 | — | 780.4 GiB | 1.5 TiB (83.5%) |
| nixos-laptop | 100.102.173.61 | — | 346.5 GiB | 393.5 GiB (85.8%) |

- **Replication factor:** 3
- **Layout version:** 1

### Ports

| Port | Purpose | Binding |
|------|---------|---------|
| 3900 | S3 API | qfox-1 localhost + Tailscale |
| 3901 | RPC | all nodes, Tailscale-only |
| 3903 | Admin API | loopback only |

### Buckets

| Bucket | Purpose |
|--------|---------|
| `research-stack` | Primary project objects |
| `db-scratch` | Active SQLite scratch DBs |
| `rds-overflow` | pg_dump / COPY TO exports |
| `snap-zone` | ZFS send/receive snapshots |
| `gdrive-mirror` | Mirror of gdrive:research-stack |

### Scripts

| Script | Purpose |
|--------|---------|
| `backup.sh snap [tag]` | Snapshot repo tree → Garage |
| `backup.sh snap-db [dir]` | Snapshot SQLite scratch DBs |
| `backup.sh cold-copy` | rclone copy Garage → gdrive:restic-mirror |
| `backup.sh forget` | Retention prune (7d/4w/6m) |
| `backup.sh verify` | restic check --read-data-subset=5% |
| `backup.sh full` | snap + cold-copy + sync-gdrive + forget |

Daily timer fires at 03:00 ±30 min. Post-commit hook runs `db-consolidate.sh offload + consolidate` async.

---

## Public Edge: Caddy on racknerd

Caddy handles TLS termination for `*.researchstack.info` domains.

### Domains & Routing

| Domain | Upstream | Notes |
|--------|----------|-------|
| `researchstack.info` | 100.102.173.61:30803 | Homer dashboard |
| `dash.researchstack.info` | 100.102.173.61:30802 | Heimdall |
| `status.researchstack.info` | 100.102.173.61:30801 | Uptime Kuma |
| `auth.researchstack.info` | 100.102.173.61:9000 | Authentik SSO (direct) |

### TLS

- **Certificate:** `*.researchstack.info` + `researchstack.info`
- **Issuer:** Let's Encrypt
- **Challenge:** DNS-01 via Porkbun API
- **Valid:** 2026-05-21 → 2026-08-19

---

## Services by Node

### cupfox (100.110.163.82)
- k3s control-plane
- KubeRay operator (ray-system namespace)
- Garage storage node (fra zone)

### neon-64gb (100.100.75.113)
- k3s worker (ARM64, 18 cores)
- Docker: ollama, cert-manager, knative (no Ray images available for ARM64)
- **Note:** Ray images must be built/pulled for linux/arm64 to use GPU workers here

### racknerd (100.80.39.40)
- k3s worker (715 MiB RAM — too small for Ray)
- Docker: docker-mailserver
- Caddy TLS termination

### steamdeck (100.85.244.73)
- k3s worker (NixOS 25.11, 8 cores, 14 GiB RAM)
- Docker: nginx, postgres, ollama, authentik, homarr, audiobookshelf, roundcube
- Ray head + worker running on this node

### nixos-laptop (100.102.173.61)
- Authentik SSO (standalone Podman, port 9000)
- Uptime Kuma, Heimdall, Homer (k3s NodePorts)
- Garage storage node (ord zone)
- SSH access via Tailscale key

### qfox-1 (this machine, 100.88.57.96)
- Primary Garage S3 endpoint
- Build host (CUDA 13.2, RTX, 30 GiB RAM)
- Local Nix environment

---

## Post-Quantum Cryptography

| Layer | Status |
|-------|--------|
| **Tailscale** | X25519Kyber768 hybrid key exchange active (v1.98+) |
| **SSH (all nodes)** | `mlkem768x25519-sha256` preferred |
| **Garage RPC** | Tailscale transport only |

---

## Access Cheat Sheet

```bash
# Cluster admin (via Tailscale)
ssh root@100.110.163.82 'cat /etc/rancher/k3s/k3s.yaml' | sed 's|0.0.0.0|100.110.163.82|g' > /tmp/kubeconfig
KUBECONFIG=/tmp/kubeconfig kubectl get pods -n ray-system

# Ray status
KUBECONFIG=/tmp/kubeconfig kubectl exec -it raycluster-head-xxx -n ray-system -- ray status

# SSH shortcuts (from ~/.ssh/config)
ssh root@100.110.163.82   # cupfox
ssh root@152.53.81.164    # neon-64gb
ssh root@172.245.19.182   # racknerd
ssh root@100.85.244.73    # steamdeck

# Garage status (local)
garage status

# Decrypt secrets
sops --decrypt 4-Infrastructure/infra/secrets/credentials.json
```

---

## Open Issues

| # | Issue | Node | Priority |
|---|-------|------|----------|
| 1 | neon-64gb ARM64 — no Ray images available | neon-64gb | High |
| 2 | Dependabot alert #75 (`@ai-sdk/provider-utils`, transitive) | — | Low |
| 3 | racknerd RAM too small for Ray (715 MiB) | racknerd | Won't fix |
| 4 | CUDA driver mismatch (610.x vs 13.2 toolkit) — needs reboot | qfox-1 | Medium |

---

## Changelog

| Date | Change |
|------|--------|
| 2026-05-30 | k3s cluster migrated to cupfox (was nixos-laptop); kubeconfig accessible via Tailscale |
| 2026-05-30 | KubeRay operator 1.6.1 deployed in ray-system namespace |
| 2026-05-30 | RayCluster deployed: head + worker on steamdeck (2 CPUs total) |
| 2026-05-30 | Node naming standardized: cupfox / neon-64gb / racknerd / steamdeck |