Research-Stack/4-Infrastructure/k3s-flake/k3s-server.nix
Brandon Schneider 6d9c4e23de fix(infra): standardize k3s cluster — Traefik, ingress, architecture docs, Ray GPU workers
Architecture alignment:
- Rewrite k3s-server.nix from aspirational role=server to actual role=agent
  (cupfox is the real control-plane at 100.110.163.82:6443)
- Update flake.nix: correct serverAddr for all nodes, annotate dead nodes,
  fix hostname from nixos-laptop to nixos
- Update join-agent.sh default SERVER to cupfox
- Document actual vs intended architecture in comments

Ingress:
- Add rs-apps-books Ingress for audiobookshelf (/apps/books → media ns)
- Add strip-apps-books middleware for prefix stripping
- Create ray-ingress.yaml for Ray dashboard at /server/ray

Ray:
- Fix raycluster.yaml: enable dashboard, mount /dev/dri on gpu-workers
- Point gpu-workers to qfox-1 (was neon-64gb)
- Remove nvidia.com/gpu resource dependency (use /dev/dri via Mesa)

Security:
- Move OPENID_CLIENT_SECRET from plaintext to K8s Secret ref
- Update manifest to use valueFrom.secretKeyRef

Cleanup:
- Remove 53 committed test PNG screenshots from git tracking
- Remove auth-state.json from git tracking
- Add *.png, *.json to tests/.gitignore

Build: no build needed
2026-05-31 23:13:04 -05:00

51 lines
1.7 KiB
Nix

{ config, pkgs, lib, hostName, serverAddr, domain, ... }:
{
##########################################################################
# k3s-server.nix — NixOS storage + k3s agent node
#
# THIS NODE IS NOT THE CONTROL PLANE.
# The actual control plane is on cupfox (100.110.163.82).
# This node was originally the standalone server but was migrated to
# agent mode when cupfox was promoted to control-plane.
#
# Architecture:
# cupfox (Debian, control-plane) ← migrated from nixos
# nixos (NixOS, storage agent)
# qfox-1 (CachyOS, foxtop agent)
# steamdeck (NixOS, GPU agent)
# racknerd (Debian, edge agent + Caddy TLS)
# neon-64gb (Debian, ARM64 agent)
#
# Servicerouter (to be deployed):
# Edge Caddy (racknerd:443) → Tailscale → host Caddy → Traefik → Services
##########################################################################
sops.secrets.k3s-token = {
sopsFile = ./secrets/k3s-token.age;
format = "yaml";
};
systemd.services.k3s.serviceConfig.EnvironmentFile = [
config.sops.secrets.k3s-token.path
];
services.k3s = {
enable = true;
role = "agent";
serverAddr = "https://100.110.163.82:6443"; # cupfox control-plane
tokenFile = config.sops.secrets.k3s-token.path;
extraFlags = [
"--node-ip=100.102.173.61"
"--node-external-ip=100.102.173.61"
"--flannel-iface=tailscale0"
"--node-label=topology.researchstack.io/role=storage"
"--node-label=topology.researchstack.io/gpu=false"
"--node-label=topology.researchstack.io/storage-tier=nvme-ssd"
];
};
networking.firewall.allowedTCPPorts = [ 80 ];
services.tailscale.enable = true;
}