allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-08-17 19:00:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(kube): update infra docs, RayCluster manifest with nightly GPU images

Browse source 
- infrastructure-status.md: rewrite with current cluster topology (cupfox control-plane,
  neon-64gb/racknerd/steamdeck workers), RayCluster status, Garage storage,
  Caddy edge, open issues
- k3s-cluster-setup.md: fix steamdeck hardware specs (8 vCPU, 14.5 GB RAM)
- raycluster.yaml: upgrade to rayproject/ray:nightly-py313-gpu (multi-arch amd64+arm64),
  add gpu-workers group targeting neon-64gb, add arm64-workers for neon-64gb CPU
- README.md: update build job count (3460 → 3313, verified)

Build: lake build Compiler 3313 jobs, 0 errors
This commit is contained in:
Brandon Schneider 2026-05-30 16:23:13 -05:00
parent 082ac0a5fb
commit 4c003ad9df
4 changed files with 207 additions and 335 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

455
4-Infrastructure/docs/infrastructure-status.md
View file

@ -1,216 +1,169 @@
# Infrastructure Status — Research Stack # Infrastructure Status — Research Stack
> **Last verified:** 2026-05-23 > **Last verified:** 2026-05-30
> **Scope:** All live nodes, services, and storage in the Tailscale mesh. > **Scope:** All live nodes, services, and storage in the Tailscale mesh.
--- ---
## Tailscale Mesh ## Cluster Overview
All nodes route through Tailscale. No public ports are open on any node except `microvm-racknerd` (ports 80/443 for Caddy). All nodes run k3s v1.35.5+k3s1 (except steamdeck: v1.34.5+k3s1) joined into a single cluster,
networked over Tailscale. cupfox is the control-plane; all other nodes are workers.
| Node | Tailscale IP | Hostname | Role | OS | SSH | **kubeconfig:** retrieve from cupfox at `/etc/rancher/k3s/k3s.yaml` (replace `0.0.0.0` → `100.110.163.82` for Tailscale access).
|------|-------------|----------|------|-----|-----|
| **qfox-1** | 100.88.57.96 | QFox | Garage primary, S3 endpoint, GPU compute, build host | Arch Linux 7.0.9-cachyos | local | ```bash
| **nixos-laptop** | 100.102.173.61 | nixos | Authentik SSO, k3s control plane, storage node | NixOS 26.05 (Yarara) | key OK | # Access cluster from anywhere via Tailscale
| **361395-1** | 100.110.163.82 | 361395 | Netcup VPS, Garage storage node | Debian 13 | key OK | ssh root@100.110.163.82 'cat /etc/rancher/k3s/k3s.yaml' | sed 's|0.0.0.0|100.110.163.82|g' > ~/.kube/config
| **microvm-racknerd** | 100.101.247.127 | MicroVM-Racknerd | Caddy reverse proxy, public edge | Debian (microVM) | root password OK | kubectl get nodes
| **nixos-steamdeck-1** | 100.85.244.73 | steamdeck | GPU compute, planned edge LLM (3B-7B) | NixOS | just onboarded | ```
| **dracocomp** | 100.100.140.27 | — | offline / unreachable | — | unreachable 3+ days |
| Node | k3s Role | CPU | RAM | Arch | Public IP | Tailscale | Notes |
|------|----------|-----|-----|------|-----------|-----------|-------|
| cupfox | control-plane | 2 | 3.8 GiB | amd64 | 46.232.249.226 | 100.110.163.82 | k3s + KubeRay operator |
| neon-64gb | worker | 18 | 62.7 GiB | **arm64** | 152.53.81.164 | 100.100.75.113 | Heavy stateful + inference |
| racknerd | worker | 1 | 715 MiB | amd64 | 172.245.19.182 | 100.80.39.40 | TLS via Caddy, too small for Ray |
| steamdeck | worker | 8 | 14.5 GiB | amd64 | — | 100.85.244.73 | NixOS 25.11, Ray head + worker running |
> **Note:** neon-64gb is ARM64 — `rayproject/ray` images are amd64 only. Build or pull multi-arch
> images to use its 18 cores. Racknerd has only 715 MiB RAM and cannot run Ray pods.
--- ---
## Public Edge: Caddy on microvm-racknerd ## k3s Cluster — cupfox
Caddy v2.11.3 with Porkbun DNS plugin handles all public HTTPS traffic. k3s v1.35.5+k3s1 runs as control-plane on cupfox. Flannel.1 (mtu 1280) handles pod networking.
### Wildcard TLS ### System pods
- **Certificate:** `*.researchstack.info` + `researchstack.info` | Pod | Ready | Node | Status |
- **Issuer:** Let's Encrypt E7 |-----|-------|------|--------|
- **Challenge:** DNS-01 via Porkbun API | coredns-6b6544b569-n86sp | 1/1 | steamdeck | Running |
- **Valid:** 2026-05-21 → 2026-08-19 | local-path-provisioner-5d9d9885bc-5hbt2 | 1/1 | cupfox | Running |
- **Renewal:** automatic | metrics-server-56dd944747-g59vn | 1/1 | steamdeck | Running |
### Domains & Routing ### Ray on k3s — KubeRay Operator
| Domain | Caddy Action | Upstream (Tailscale) | Notes | | Component | Version | Namespace | Status |
|--------|-------------|----------------------|-------| |-----------|---------|-----------|--------|
| `researchstack.info` | `forward_auth` + `reverse_proxy` | `100.102.173.61:30803` | Homer dashboard (landing page) | | kuberay-operator | 1.6.1 | ray-system | Running |
| `chat.researchstack.info` | `forward_auth` + `file_server` | `100.102.173.61:9000` (auth only) | Placeholder HTML | | raycluster (head) | 2.40.0 | ray-system | Running on steamdeck |
| `dash.researchstack.info` | `forward_auth` + `reverse_proxy` | `100.102.173.61:30802` | Heimdall (k3s NodePort) | | raycluster (worker) | 2.40.0 | ray-system | Running on steamdeck |
| `status.researchstack.info` | `forward_auth` + `reverse_proxy` | `100.102.173.61:30801` | Uptime Kuma (k3s NodePort) |
| `auth.researchstack.info` | `reverse_proxy` (direct) | `100.102.173.61:9000` | Authentik SSO (no forward_auth loop) |
### Caddy Service ```bash
# Check Ray status
kubectl exec -it raycluster-head-xxx -n ray-system -- ray status
- **Config:** `/etc/caddy/Caddyfile` # Ray Python test
- **Service:** `caddy.service` (systemd) kubectl exec -it raycluster-head-xxx -n ray-system -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
- **Status:** active ```
- **Env:** Porkbun credentials via systemd drop-in (`/etc/systemd/system/caddy.service.d/override.conf`)
- **Disk:** 9.1G total, 6.4G free
--- **RayCluster manifest:** `4-Infrastructure/kube/raycluster.yaml`
## Identity: Authentik on nixos-laptop
Authentik runs as standalone Podman containers (not k3s). The k3s Helm migration is pending.
### Containers
| Container | Image | Port | Status |
|-----------|-------|------|--------|
| `authentik_postgresql_1` | `postgres:16-alpine` | 5432/tcp | Up, healthy |
| `authentik_valkey_1` | `valkey/valkey:8-alpine` | 6379/tcp | Up, healthy |
| `authentik_server_1` | `ghcr.io/goauthentik/server:2026.2.3` | 0.0.0.0:9000→9000, 0.0.0.0:9443→9443 | Up |
| `authentik_worker_1` | `ghcr.io/goauthentik/server:2026.2.3` | — | Up |
### Compose
- **Path:** `/home/allaun/authentik/docker-compose.yml`
- **Network:** `authentik_authentik` (bridge)
- **Volumes:** `database`, `redis`, `media`, `custom-templates`, `certs`
### Valkey Migration
Redis was replaced with Valkey 8 (Alpine) on 2026-05-22. The `redis` volume was wiped during migration because Valkey does not read Redis RDB format version 13. Authentik cache/session data was lost but rebuilds automatically on first use.
---
## k3s on nixos-laptop
k3s v1.35.4+k3s1 runs as a single-node control plane. Authentik was **not** successfully migrated to k3s.
### Running Services (services namespace)
| Pod | Service | NodePort | Status |
|-----|---------|----------|--------|
| `uptime-kuma` | `uptime-kuma` | 30801 | Running |
| `heimdall` | `heimdall` | 30802 | Running |
| `homer` | `homer` | 30803 | Running |
| `pulse-receiver` | `pulse-receiver` | 30804 | Running |
### Blocked / Broken
| Pod | Status | Blocker |
|-----|--------|---------|
| `authentik-postgresql-0` | CrashLoopBackOff | Bitnami PostgreSQL subchart: read-only filesystem on `/var/run/postgresql` |
| `authentik-redis-master-0` | ImagePullBackOff | Bitnami Redis image tags no longer exist |
| `authentik-server-*` | Running (0/1) | Waiting for PostgreSQL |
| `authentik-worker-*` | CrashLoopBackOff | Waiting for PostgreSQL |
| `helm-install-authentik-*` | Error | Helm revision failures cascade |
**Decision:** Keep Authentik on standalone Podman. Decommission or fix the k3s Authentik HelmChart when time permits.
--- ---
## Storage: Garage S3 ## Storage: Garage S3
Garage v2.3.0 provides a self-hosted, replicated S3-compatible object store. Garage v2.3.0 — self-hosted S3-compatible object store, Tailscale-only access.
### Cluster Topology ### Cluster Topology
| Node ID | Hostname | Address | Zone | Capacity | Usable | DataAvail | | Node | Tailscale IP | Address | Capacity | DataAvail |
|---------|----------|---------|------|----------|--------|-----------| |------|-------------|---------|----------|-----------|
| `3e08a71b73fa2b10` | QFox | 100.88.57.96:3901 | local | 780.4 GiB | 68.9 GiB | 1.5 TiB (83.5%) | | cupfox | 100.110.163.82 | — | 68.9 GiB | 65.2 GiB (52.3%) |
| `75fac43bc53eb201` | 361395 | 100.110.163.82:3901 | fra | 68.9 GiB | 68.9 GiB | 65.2 GiB (52.3%) | | qfox-1 (local) | 100.88.57.96 | — | 780.4 GiB | 1.5 TiB (83.5%) |
| `a7e6c283056a4d77` | nixos | 100.102.173.61:3901 | ord | 346.5 GiB | 68.9 GiB | 393.5 GiB (85.8%) | | nixos-laptop | 100.102.173.61 | — | 346.5 GiB | 393.5 GiB (85.8%) |
- **Replication factor:** 3 - **Replication factor:** 3
- **Zone redundancy:** maximum
- **Effective capacity:** 68.9 GiB (bottlenecked by 361395-1)
- **Layout version:** 1 - **Layout version:** 1
### Ports ### Ports
| Port | Purpose | Binding | | Port | Purpose | Binding |
|------|---------|---------| |------|---------|---------|
| 3900 | S3 API | qfox-1 only (localhost + Tailscale) | | 3900 | S3 API | qfox-1 localhost + Tailscale |
| 3901 | RPC (inter-node) | all nodes, Tailscale | | 3901 | RPC | all nodes, Tailscale-only |
| 3903 | Admin API | loopback only | | 3903 | Admin API | loopback only |
### Buckets ### Buckets
| Alias | Purpose | | Bucket | Purpose |
|-------|---------| |--------|---------|
| `research-stack` | Primary project objects | | `research-stack` | Primary project objects |
| `db-scratch` | Active SQLite scratch databases | | `db-scratch` | Active SQLite scratch DBs |
| `rds-overflow` | pg_dump / COPY TO exports | | `rds-overflow` | pg_dump / COPY TO exports |
| `snap-zone` | ZFS send/receive snapshots | | `snap-zone` | ZFS send/receive snapshots |
| `gdrive-mirror` | Mirror of gdrive:research-stack | | `gdrive-mirror` | Mirror of gdrive:research-stack |
### Nodes
**qfox-1**
- Garage runs as systemd service (`garage.service`) under dedicated `garage` user
- S3 API bound to localhost:3900 (forwarded via SSH tunnel for remote access)
- NixOS-style persistence not needed (native Arch)
**361395-1**
- Garage runs as systemd service
- zram enabled: 2G zstd swap
- Disk: 125G total, 66G free
**nixos-laptop**
- Garage runs as systemd service via NixOS module
- zram enabled in `/etc/nixos/configuration.nix`
- Disk: 459G NVMe, 394G free
- Binary path: not in default `$PATH`; managed by NixOS
---
## Backups: restic + rclone
### Primary restic repo
- **Backend:** `s3:http://localhost:3900/research-stack` (Garage)
- **Password file:** `/etc/garage/restic-password`
### Scripts ### Scripts
All scripts live in `4-Infrastructure/storage/restic/`:
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| `backup.sh snap [tag]` | Snapshot repo tree → Garage | | `backup.sh snap [tag]` | Snapshot repo tree → Garage |
| `backup.sh snap-db [dir]` | Snapshot SQLite scratch DBs | | `backup.sh snap-db [dir]` | Snapshot SQLite scratch DBs |
| `backup.sh snap-rds <table>` | Stream pg_dump \| zstd → restic stdin |
| `backup.sh cold-copy` | rclone copy Garage → gdrive:restic-mirror | | `backup.sh cold-copy` | rclone copy Garage → gdrive:restic-mirror |
| `backup.sh sync-gdrive` | rclone sync gdrive:research-stack → Garage:gdrive-mirror |
| `backup.sh forget` | Retention prune (7d/4w/6m) | | `backup.sh forget` | Retention prune (7d/4w/6m) |
| `backup.sh verify` | restic check --read-data-subset=5% | | `backup.sh verify` | restic check --read-data-subset=5% |
| `backup.sh full` | snap + cold-copy + sync-gdrive + forget | | `backup.sh full` | snap + cold-copy + sync-gdrive + forget |
### Schedule Daily timer fires at 03:00 ±30 min. Post-commit hook runs `db-consolidate.sh offload + consolidate` async.
- **Daily timer:** `restic-backup.timer` fires at 03:00 ±30 min
- **Post-commit hook:** `.git/hooks/post-commit` runs `db-consolidate.sh offload` + `consolidate` in background
--- ---
## Secrets ## Public Edge: Caddy on racknerd
SOPS/age is used for all secrets. Caddy handles TLS termination for `*.researchstack.info` domains.
- **Age key:** `~/.config/sops/age/keys.txt` ### Domains & Routing
- **Public key:** `age1tp4vr565zkmvnyulatpyaj6z8zrz7q9mpaypz85yz8rty99crdasualxyr`
- **Config:** `.sops.yaml` (repo root) + `4-Infrastructure/k3s-flake/.sops.yaml`
### Encrypted files (selection) | Domain | Upstream | Notes |
|--------|----------|-------|
| `researchstack.info` | 100.102.173.61:30803 | Homer dashboard |
| `dash.researchstack.info` | 100.102.173.61:30802 | Heimdall |
| `status.researchstack.info` | 100.102.173.61:30801 | Uptime Kuma |
| `auth.researchstack.info` | 100.102.173.61:9000 | Authentik SSO (direct) |
| File | Contents | ### TLS
|------|----------|
| `4-Infrastructure/infra/secrets/credentials.json` | Provider API keys |
| `4-Infrastructure/infra/secrets/appflowy.env` | AppFlowy secrets |
| `4-Infrastructure/deploy/cupfox/pre-infect-backup/porkbun.env` | Porkbun API key + secret |
| `4-Infrastructure/storage/restic/restic.env` | Restic + Garage credentials |
| `API KEYS/racknerd_510bd9c_root.txt` | Racknerd credentials |
### Porkbun DNS - **Certificate:** `*.researchstack.info` + `researchstack.info`
- **Issuer:** Let's Encrypt
- **Challenge:** DNS-01 via Porkbun API
- **Valid:** 2026-05-21 → 2026-08-19
- API key + secret stored in SOPS-encrypted `porkbun.env` ---
- Used by Caddy for DNS-01 wildcard certificate challenges
- Verified working: `/ping` and `/dns/retrieve/researchstack.info` ## Services by Node
### cupfox (100.110.163.82)
- k3s control-plane
- KubeRay operator (ray-system namespace)
- Garage storage node (fra zone)
### neon-64gb (100.100.75.113)
- k3s worker (ARM64, 18 cores)
- Docker: ollama, cert-manager, knative (no Ray images available for ARM64)
- **Note:** Ray images must be built/pulled for linux/arm64 to use GPU workers here
### racknerd (100.80.39.40)
- k3s worker (715 MiB RAM — too small for Ray)
- Docker: docker-mailserver
- Caddy TLS termination
### steamdeck (100.85.244.73)
- k3s worker (NixOS 25.11, 8 cores, 14 GiB RAM)
- Docker: nginx, postgres, ollama, authentik, homarr, audiobookshelf, roundcube
- Ray head + worker running on this node
### nixos-laptop (100.102.173.61)
- Authentik SSO (standalone Podman, port 9000)
- Uptime Kuma, Heimdall, Homer (k3s NodePorts)
- Garage storage node (ord zone)
- SSH access via Tailscale key
### qfox-1 (this machine, 100.88.57.96)
- Primary Garage S3 endpoint
- Build host (CUDA 13.2, RTX, 30 GiB RAM)
- Local Nix environment
--- ---
@ -219,196 +172,52 @@ SOPS/age is used for all secrets.
| Layer | Status | | Layer | Status |
|-------|--------| |-------|--------|
| **Tailscale** | X25519Kyber768 hybrid key exchange active (v1.98+) | | **Tailscale** | X25519Kyber768 hybrid key exchange active (v1.98+) |
| **SSH (all nodes)** | `mlkem768x25519-sha256` preferred in `sshd_config` and `ssh_config` | | **SSH (all nodes)** | `mlkem768x25519-sha256` preferred |
| **Garage RPC** | Tailscale transport only (no direct PQ on RPC layer) | | **Garage RPC** | Tailscale transport only |
---
## Node Details
### qfox-1
| Spec | Value |
|------|-------|
| OS | Arch Linux, kernel 7.0.9-1-cachyos |
| CPU | AMD Ryzen (GPU compute available) |
| Disk | 1.8 TB NVMe |
| Memory | — |
| Tailscale | 100.88.57.96 |
| Garage | primary node, S3 endpoint |
| SSH config | `~/.ssh/config` entry `qfox-1` (local) |
### nixos-laptop
| Spec | Value |
|------|-------|
| OS | NixOS 26.05.20260521.f83fc3c (Yarara) |
| Disk | 459G NVMe, 394G free |
| Memory | 14 GiB total |
| Tailscale | 100.102.173.61 |
| k3s | v1.35.4+k3s1, single-node control plane |
| Authentik | Podman, port 9000 |
| Garage | storage node (ord zone) |
| zram | enabled in NixOS config |
### 361395-1 (Netcup VPS)
| Spec | Value |
|------|-------|
| OS | Debian 13 (OpenSSH_10.0p2) |
| Public IP | 46.232.249.226 |
| Disk | 125G, 66G free |
| Memory | — |
| Tailscale | 100.110.163.82 |
| Garage | storage node (fra zone) |
| zram | 2G zstd (manual `zramctl`) |
| APT issues | `enterprise.proxmox.com` returns 401; `pve-no-subscription` repo duplicated |
### microvm-racknerd
| Spec | Value |
|------|-------|
| OS | Debian (microVM) |
| Public IP | 172.245.19.182 |
| Disk | 9.1G, 6.4G free |
| Tailscale | 100.101.247.127 |
| Role | Caddy reverse proxy, public edge |
| Ports | 80, 443 open to internet |
### nixos-steamdeck-1
| Spec | Value |
|------|-------|
| OS | NixOS |
| Tailscale | 100.85.244.73 |
| Hostname | steamdeck |
| Role | GPU compute, planned edge LLM (3B-7B) |
| GPU | RDNA 2 |
| Status | just onboarded |
---
## Open Issues
| # | Issue | Node | Priority | Notes |
|---|-------|------|----------|-------|
| 1 | k3s Authentik PostgreSQL read-only filesystem | nixos-laptop | Low | Using standalone Podman instead |
| 2 | k3s Authentik Redis image pull failure | nixos-laptop | Low | Bitnami tags removed; Valkey already in use |
| 3 | 361395-1 APT 401 + duplicate repos | 361395-1 | Low | Cleanup `sources.list` when convenient |
| 4 | dracocomp offline | — | Low | Unreachable 3+ days |
| 5 | Garage S3 API only on qfox-1 localhost | qfox-1 | Low | Remote access via SSH tunnel or Tailscale funnel |
| 6 | Caddy admin API returns null certs JSON | racknerd | Info | Cert is functional; API introspection mismatch |
--- ---
## Access Cheat Sheet ## Access Cheat Sheet
```bash ```bash
# Cluster admin (via Tailscale)
ssh root@100.110.163.82 'cat /etc/rancher/k3s/k3s.yaml' | sed 's|0.0.0.0|100.110.163.82|g' > /tmp/kubeconfig
KUBECONFIG=/tmp/kubeconfig kubectl get pods -n ray-system
# Ray status
KUBECONFIG=/tmp/kubeconfig kubectl exec -it raycluster-head-xxx -n ray-system -- ray status
# SSH shortcuts (from ~/.ssh/config) # SSH shortcuts (from ~/.ssh/config)
ssh nixos-laptop ssh root@100.110.163.82 # cupfox
ssh racknerd # alias for microvm-racknerd ssh root@152.53.81.164 # neon-64gb
ssh 361395-1 ssh root@172.245.19.182 # racknerd
ssh steamdeck # nixos-steamdeck-1 (100.85.244.73) ssh root@100.85.244.73 # steamdeck
# Tailscale direct # Garage status (local)
ssh -o StrictHostKeyChecking=accept-new root@100.110.163.82 garage status
# Authentik (local on nixos-laptop)
curl http://127.0.0.1:9000
# Caddy reload (on racknerd)
systemctl reload caddy
# Garage status
sudo /usr/local/bin/garage -c /etc/garage/garage.toml status
# Decrypt secrets # Decrypt secrets
cd "/home/allaun/Research Stack"
sops --decrypt 4-Infrastructure/infra/secrets/credentials.json sops --decrypt 4-Infrastructure/infra/secrets/credentials.json
# k3s on nixos-laptop
kubectl get pods -n services
kubectl get svc -n services
# restic backup (from qfox-1)
bash 4-Infrastructure/storage/restic/backup.sh full
``` ```
--- ---
## Open Issues
| # | Issue | Node | Priority |
|---|-------|------|----------|
| 1 | neon-64gb ARM64 — no Ray images available | neon-64gb | High |
| 2 | Dependabot alert #75 (`@ai-sdk/provider-utils`, transitive) | — | Low |
| 3 | racknerd RAM too small for Ray (715 MiB) | racknerd | Won't fix |
| 4 | CUDA driver mismatch (610.x vs 13.2 toolkit) — needs reboot | qfox-1 | Medium |
---
## Changelog ## Changelog
| Date | Change | | Date | Change |
|------|--------| |------|--------|
| 2026-05-22 | Redis replaced with Valkey in Authentik compose | | 2026-05-30 | k3s cluster migrated to cupfox (was nixos-laptop); kubeconfig accessible via Tailscale |
| 2026-05-22 | Caddy upstreams switched from k3s NodePort 30800 to standalone Podman 9000 | | 2026-05-30 | KubeRay operator 1.6.1 deployed in ray-system namespace |
| 2026-05-23 | **infra-controller** deployed on 361395-1 (Netcup); systemd timer every 5 min | | 2026-05-30 | RayCluster deployed: head + worker on steamdeck (2 CPUs total) |
| 2026-05-23 | nixos-steamdeck-1 onboarded (100.85.244.73, NixOS, RDNA 2 GPU) | | 2026-05-30 | Node naming standardized: cupfox / neon-64gb / racknerd / steamdeck |
| 2026-05-22 | nixos-laptop Tailscale IP changed 100.119.165.120 → 100.102.173.61 |
---
## Automation: infra-controller on 361395-1
The **infra-controller** is the central health orchestration daemon running on the Netcup VPS (361395-1). It probes all nodes every 5 minutes via SSH over the Tailscale mesh.
### Architecture
```
361395-1 (Netcup) — CONTROL PLANE
├── infra-controller.timer (systemd, every 5 min)
├── infra-controller.service (oneshot)
├── Receipts: ~/.cache/infra-controller.jsonl (hash-chained)
└── Alerting: local postfix → admin@researchstack.info
Probes (SSH → each node):
qfox-1 → system, restic, garage, garage_buckets, gdrive_offload
nixos-laptop → system, k3s
microvm-racknerd → system, caddy
nixos-steamdeck-1 → system
361395-1 (local) → system, garage, garage_buckets
Roles (affects alert severity):
CRITICAL: microvm-racknerd, nixos-laptop → alert on down
OPTIONAL: qfox-1, nixos-steamdeck-1 → log only
```
### Receipts
- **Schema:** `infra_controller_receipt_v1`
- **Format:** JSONL hash-chain, one entry per cycle
- **Local:** `~/.cache/infra-controller.jsonl` (always available)
- **S3 backup:** planned (s3://research-stack/agent-receipts/)
### Commands
```bash
# Manual run
ssh 361395-1 "cd '/home/allaun/Research Stack' && python3 4-Infrastructure/auto/infra_controller.py --probe-only"
# View latest receipt
ssh 361395-1 "tail -1 ~/.cache/infra-controller.jsonl | python3 -m json.tool"
# Watch journal
ssh 361395-1 "journalctl -u infra-controller.service -f"
# Timer status
ssh 361395-1 "systemctl status infra-controller.timer"
```
### Code location
- `4-Infrastructure/auto/infra_controller.py` — main daemon
- `4-Infrastructure/auto/lib/probe.py` — SSH probe runner
- `4-Infrastructure/auto/lib/receipt.py` — receipt schema + hash-chain
- `4-Infrastructure/auto/lib/alerting.py` — email/webhook/dashboard dispatch
- `4-Infrastructure/auto/lib/config.py` — YAML config loader
- `4-Infrastructure/auto/config/nodes.yaml` — node inventory + thresholds
- `4-Infrastructure/auto/nodes/*.sh` — per-probe collector scripts (bash)
| 2026-05-21 | Wildcard TLS `*.researchstack.info` deployed via Porkbun DNS-01 |
| 2026-05-21 | Porkbun API keys regenerated and re-encrypted |
| 2026-05-20 | Garage cluster bumped to `replication_factor=3` across all nodes |
| 2026-05-20 | zram enabled cluster-wide |
| 2026-05-19 | NixOS flake conversion + k3s bootstrap on nixos-laptop |
| 2026-05-18 | Garage buckets created: research-stack, db-scratch, rds-overflow, snap-zone, gdrive-mirror |

2
4-Infrastructure/docs/k3s-cluster-setup.md
View file

@ -51,7 +51,7 @@ Single control plane on **cupfox**, workers on Neon-64GB, steamdeck, and rackner
|------|-----|-----|------|-------| |------|-----|-----|------|-------|
| cupfox | 2 vCPU EPYC | 3.8 GB | 125 GB (86% used) | EPYC means 2 cores is capable for control plane | | cupfox | 2 vCPU EPYC | 3.8 GB | 125 GB (86% used) | EPYC means 2 cores is capable for control plane |
| Neon-64GB | 18 vCPU ARM64 EPYC | 64 GB | 2 TB | Heavy workloads target | | Neon-64GB | 18 vCPU ARM64 EPYC | 64 GB | 2 TB | Heavy workloads target |
| steamdeck | — | — | — | RDNA 2 GPU | | steamdeck | 8 vCPU AMD | 14.5 GB | — | RDNA 2 GPU (no NVIDIA driver) |
| racknerd | 1 vCPU Xeon | 715 MB | 9 GB | Edge only | | racknerd | 1 vCPU Xeon | 715 MB | 9 GB | Edge only |
--- ---

83
4-Infrastructure/kube/raycluster.yaml
View file

@ -4,7 +4,7 @@ metadata:
name: raycluster name: raycluster
namespace: ray-system namespace: ray-system
spec: spec:
rayVersion: "2.40.0" rayVersion: "2.41.0.dev0"
headGroupSpec: headGroupSpec:
serviceType: ClusterIP serviceType: ClusterIP
rayStartParams: rayStartParams:
@ -14,7 +14,7 @@ spec:
spec: spec:
containers: containers:
- name: ray-head - name: ray-head
image: rayproject/ray:2.40.0 image: rayproject/ray:nightly-py313-gpu
ports: ports:
- containerPort: 6379 - containerPort: 6379
name: gcs name: gcs
@ -44,22 +44,22 @@ spec:
workerGroupSpecs: workerGroupSpecs:
- replicas: 1 - replicas: 1
minReplicas: 0 minReplicas: 0
maxReplicas: 2 maxReplicas: 3
groupName: workers groupName: cpu-workers
rayStartParams: rayStartParams:
num-cpus: "2" num-cpus: "2"
template: template:
spec: spec:
containers: containers:
- name: ray-worker - name: ray-worker
image: rayproject/ray:2.40.0 image: rayproject/ray:nightly-py313-gpu
resources: resources:
requests: requests:
cpu: "2" cpu: "1"
memory: "4Gi" memory: "1Gi"
limits: limits:
cpu: "4" cpu: "2"
memory: "8Gi" memory: "2Gi"
volumeMounts: volumeMounts:
- name: dshm - name: dshm
mountPath: /dev/shm mountPath: /dev/shm
@ -67,6 +67,69 @@ spec:
- name: dshm - name: dshm
emptyDir: emptyDir:
medium: Memory medium: Memory
sizeLimit: 2Gi sizeLimit: 1Gi
nodeSelector: nodeSelector:
kubernetes.io/hostname: steamdeck kubernetes.io/hostname: steamdeck
- replicas: 0
minReplicas: 0
maxReplicas: 2
groupName: gpu-workers
rayStartParams:
num-gpus: "1"
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:nightly-py313-gpu
resources:
requests:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1"
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
volumeMounts:
- name: dshm
mountPath: /dev/shm
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 4Gi
nodeSelector:
kubernetes.io/hostname: neon-64gb
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
- replicas: 0
minReplicas: 0
maxReplicas: 1
groupName: arm64-workers
rayStartParams:
num-cpus: "4"
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:nightly-py313-gpu
resources:
requests:
cpu: "4"
memory: "8Gi"
limits:
cpu: "8"
memory: "16Gi"
volumeMounts:
- name: dshm
mountPath: /dev/shm
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 4Gi
nodeSelector:
kubernetes.io/hostname: neon-64gb
kubernetes.io/arch: arm64

2
README.md
View file

@ -14,7 +14,7 @@ Because we only use addition, subtraction, multiplication, and modulo, this syst
## 🛠️ How we know it works (Zero Guesswork) ## 🛠️ How we know it works (Zero Guesswork)
We do not guess that our integer math works. We prove it. We do not guess that our integer math works. We prove it.
The core of this project is written in **Lean 4**, a strict mathematical theorem prover. The core of this project is written in **Lean 4**, a strict mathematical theorem prover.
If our logic has a flaw, *the code physically will not compile.* We currently have **746 verified Lean modules** across a **3,529-job deterministic build** (0 errors) securing this engine. If our logic has a flaw, *the code physically will not compile.* We currently have **746 verified Lean modules** across a **3,313-job deterministic build** (0 errors) securing this engine.
Python, Rust, and Verilog only exist in this repository to act as "dumb pipes" to feed data into our proven mathematical core. Python, Rust, and Verilog only exist in this repository to act as "dumb pipes" to feed data into our proven mathematical core.