diff --git a/4-Infrastructure/k3s-flake/README.md b/4-Infrastructure/k3s-flake/README.md index c3c53c6b..d9ea601f 100644 --- a/4-Infrastructure/k3s-flake/README.md +++ b/4-Infrastructure/k3s-flake/README.md @@ -8,10 +8,68 @@ no external dependencies embedded in the flake. > A node goes online → it joins → it goes offline → the cluster adjusts. -The flake spans the full topology spectrum: -- **Server-class x86** (core, judge, mirror, foxtop) — full k8s workloads -- **Thin client / Pi** (edge) — lightweight k3s agent, pulse heartbeat only -- **microvm-nerdrack** — zero work, just pulse. Tainted `pulse-only:NoSchedule`. +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Internet │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ Public Edge (microvm-racknerd) │ │ +│ │ Caddy: TLS termination only (Porkbun DNS-01) │ │ +│ │ Ports 80/443 → forwards ALL traffic over Tailscale │ │ +│ └───────────────────────────┬─────────────────────────────────┘ │ +│ │ Tailscale mesh │ +│ ┌───────────────────────────▼─────────────────────────────────┐ │ +│ │ Traefik Ingress (nixos-laptop / k3s-server :80) │ │ +│ │ Path routing + forward_auth middleware (Authentik) │ │ +│ │ Defined in manifests/ingress/ (Ingress + Middleware CRDs) │ │ +│ └───────────────────────────┬─────────────────────────────────┘ │ +│ │ │ +│ ┌───────────────────────────▼─────────────────────────────────┐ │ +│ │ k3s Service Layer (ClusterIP) │ │ +│ │ Hermes, Authentik, Uptime Kuma, Homarr, control-plane APIs │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ Worker Pool (qfox-1, steamdeck, 361395-1, ...) │ │ +│ │ GPU compute, storage, downloaders, codecs │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +## URL Contract (canonical paths on `https://researchstack.info`) + +| Path | Service | Type | +|------|---------|------| +| `/` | Homer directory | Dashboard | +| `/gettingstarted` | Static page | Info | +| `/apps/chat/*` | Hermes (chat/orchestrator) | App | +| `/apps/jellyfin/*` | Jellyfin | App | +| `/apps/books/*` | Audiobookshelf | App | +| `/apps/music/*` | Navidrome | App | +| `/apps/budget/*` | Actual Budget | App | +| `/server/status/*` | Uptime Kuma | Ops UI | +| `/server/dash/*` | Homarr | Ops UI | +| `/server/vault/*` | Vaultwarden | Ops UI | +| `/api/cred/*` | Credential Server | Control-plane | +| `/api/registry/*` | Registry API (join/heartbeat) | Control-plane | +| `/api/jobs/*` | Job Router | Control-plane | +| `/api/blobs/*` | Blob Plane | Control-plane | + +**Stable subdomains** (not path-routed): +- `auth.researchstack.info` → Authentik (OIDC issuer — must not change) +- `mail.researchstack.info` / `webmail.researchstack.info` → Mail services + +**Legacy subdomain redirects** (301 → canonical path): +- `status.*` → `/server/status/` +- `dash.*` / `home.*` → `/` +- `media.*` → `/apps/jellyfin/` +- `books.*` → `/apps/books/` +- `music.*` → `/apps/music/` +- `vault.*` → `/server/vault/` +- `apps.*` → `/apps/` +- `pulse.*` → `/api/registry/` ## File Layout @@ -19,7 +77,8 @@ The flake spans the full topology spectrum: 4-Infrastructure/k3s-flake/ ├── flake.nix — 6 topology configurations ├── k3s-configuration.nix — base module (Tailscale, SSH, Nix, firewall, sops) -├── k3s-server.nix — control plane + Caddy/Porkbun + deploy oneshot +├── k3s-server.nix — control plane + Traefik Ingress + deploy oneshot +├── k3s-edge.nix — public TLS edge (Caddy) + mail services ├── .sops.yaml — age key rules ├── secrets/ — encrypted at rest, decrypted at activation │ ├── k3s-token.age — K3S_TOKEN= @@ -32,13 +91,21 @@ The flake spans the full topology spectrum: │ ├── edge.nix — label: role=edge, taint: pulse-only:NoSchedule │ └── foxtop.nix — label: role=foxtop ├── manifests/ — Kubernetes resources, auto-deployed by systemd -│ ├── kustomization.yaml +│ ├── kustomization.yaml — master resource list │ ├── namespace.yaml — namespace: services -│ ├── authentik/ — HelmChart CRD (official chart + in-cluster PG/Redis) -│ ├── uptime-kuma/ — Deployment + NodePort 30801 + PVC -│ ├── heimdall/ — Deployment + NodePort 30802 + PVC -│ ├── homer/ — Deployment + NodePort 30803 + ConfigMap -│ └── pulse-receiver/ — Deployment + NodePort 30804 (inline Python receiver) +│ ├── ingress/ — Traefik Ingress + Middleware CRDs (path routing) +│ ├── authentik/ — HelmChart CRD (OIDC @ auth.researchstack.info) +│ ├── hermes/ — Deployment + ClusterIP (placeholder → /apps/chat/) +│ ├── credential-server/ — Deployment + ClusterIP (/api/cred/*) +│ ├── control-plane/ — Registry, Jobs, Blobs APIs (/api/*) +│ ├── uptime-kuma/ — Deployment + Service (/server/status/) +│ ├── homer/ — Deployment + Service (/) +│ ├── homarr/ — Deployment + Service (/server/dash/) +│ ├── actual-budget/ — Deployment + Service (/apps/budget/) +│ ├── vaultwarden/ — Deployment + Service (/server/vault/) +│ ├── media/ — Jellyfin, Navidrome, Audiobookshelf, *arr stack +│ ├── heimdall/ — [LEGACY] being replaced by path routing +│ └── pulse-receiver/ — [LEGACY] being replaced by /api/registry/ └── scripts/ └── deploy-services.sh — idempotent kubectl apply, called by systemd oneshot ``` @@ -49,35 +116,51 @@ The flake spans the full topology spectrum: | Role | k3s Label | Taints | Workload | |---------|-------------------------------------------------|---------------------------|----------| -| server | — (control plane) | — | Caddy ingress + deploy-manifests | -| core | `topology.researchstack.io/role=core` | — | General compute (PG, Redis, etc.) | +| server | — (control plane) | — | Internal Caddy router + deploy-manifests | +| core | `topology.researchstack.io/role=core` | — | General compute (PG, Redis, Authentik) | | judge | `topology.researchstack.io/role=judge` | — | Validation / audit | | mirror | `topology.researchstack.io/role=mirror` | — | Storage / replication | -| edge | `topology.researchstack.io/role=edge` | `pulse-only:NoSchedule` | Pulse heartbeat only | -| foxtop | `topology.researchstack.io/role=foxtop` | — | Top-level orchestrator | +| edge | `topology.researchstack.io/role=edge` | `pulse-only:NoSchedule` | Public TLS edge + mail | +| foxtop | `topology.researchstack.io/role=foxtop` | — | Primary compute / orchestrator | ### Service Placement -| Service | NodePort | Prefers Role | Stateful? | -|--------------------|----------|--------------|-----------| -| Authentik | 30800 | core, server | Yes (PG + Redis PVCs) | -| Uptime Kuma | 30801 | any | PVC (1Gi) — node-bound | -| Heimdall | 30802 | any | PVC (1Gi) — node-bound | -| Homer | 30803 | any | No (ConfigMap) | -| Pulse Receiver | 30804 | any | No (stateless) | +| Service | Port | Path | Prefers Role | Stateful? | +|--------------------|-------|-------------------|--------------|-----------| +| Authentik | 80 | auth.* subdomain | core | Yes (PG + Redis PVCs) | +| Hermes | 80 | /apps/chat/ | any | No (placeholder) | +| Uptime Kuma | 3001 | /server/status/ | any | PVC (1Gi) | +| Homer | 8080 | / | any | No (ConfigMap) | +| Homarr | 7575 | /server/dash/ | any | PVC | +| Actual Budget | 5006 | /apps/budget/ | any | PVC | +| Vaultwarden | 80 | /server/vault/ | any | PVC | +| Credential Server | 8444 | /api/cred/ | any | Secret vol | +| Registry API | 8080 | /api/registry/ | any | No (stub) | +| Jobs API | 8080 | /api/jobs/ | any | No (stub) | +| Blobs API | 8080 | /api/blobs/ | any | No (stub) | -### Domain Mapping (Caddy) +All services use ClusterIP and are routed via Traefik Ingress. Legacy +NodePort assignments are preserved for backward compat but are not the +primary routing path. -All services are served under `*.YOUR_DOMAIN` (e.g. `researchstack.info`): +### Routing Model (Caddy edge + Traefik Ingress) -- `auth.YOUR_DOMAIN` → NodePort 30800 → Authentik -- `status.YOUR_DOMAIN` → NodePort 30801 → Uptime Kuma -- `apps.YOUR_DOMAIN` → NodePort 30802 → Heimdall -- `home.YOUR_DOMAIN` → NodePort 30803 → Homer -- `pulse.YOUR_DOMAIN` → NodePort 30804 → Pulse Receiver -- `YOUR_DOMAIN` → static response "k3s unified topology — Research Stack" +**Public edge** (`k3s-edge.nix` / microvm-racknerd): +- Caddy terminates TLS via Porkbun DNS-01 wildcard +- Forwards all traffic to Traefik (`100.102.173.61:80`) over Tailscale +- Handles legacy subdomain 301 redirects at the edge +- `auth.*` and `mail.*`/`webmail.*` forwarded with Host header preserved -TLS via Porkbun DNS challenge (caddy-dns/porkbun plugin). +**Traefik Ingress** (k3s built-in, nixos-laptop): +- Listens on `:80` (node port, HTTP — TLS handled by edge) +- Path-based routing defined in `manifests/ingress/ingress.yaml` +- Traefik Middleware CRDs for: + - `authentik-forward-auth` — SSO gate for `/apps/*`, `/server/*`, `/` + - `strip-*` — prefix stripping per route +- `/api/*` routes skip `forward_auth` (token-authenticated) +- `auth.researchstack.info` matched by separate Ingress (no middleware) + +TLS via Porkbun DNS challenge (caddy-dns/porkbun plugin) at the edge. ## Node Lifecycle diff --git a/4-Infrastructure/k3s-flake/k3s-edge.nix b/4-Infrastructure/k3s-flake/k3s-edge.nix index ac191072..d1bbb948 100644 --- a/4-Infrastructure/k3s-flake/k3s-edge.nix +++ b/4-Infrastructure/k3s-flake/k3s-edge.nix @@ -2,6 +2,8 @@ let mailDomain = domain; + # Internal router target (k3s-server / nixos-laptop over Tailscale) + internalRouter = "100.102.173.61:80"; sendAlert = pkgs.writeShellScriptBin "send-alert" '' set -euo pipefail subject="''${1:-Alert}" @@ -35,7 +37,7 @@ in networking.hostName = lib.mkDefault hostName; networking.domain = lib.mkDefault mailDomain; - # ── Postfix ─────────────────────────────────────────────────────────────── + # ── Postfix (mail stays on edge — non-HTTP protocol) ───────────────────── services.postfix = { enable = true; hostname = "mail.${mailDomain}"; @@ -62,7 +64,6 @@ in ssl_min_protocol = TLSv1.2 mail_location = maildir:/var/mail/%n auth_mechanisms = plain login - ssl = yes passdb { driver = passwd-file args = /etc/dovecot/users @@ -74,7 +75,6 @@ in ''; }; - # Create mailbox user and directory users.users.postmaster = { isSystemUser = true; uid = 5000; @@ -87,37 +87,7 @@ in members = [ "postmaster" ]; }; - # ── ProtonMail Bridge ───────────────────────────────────────────────────── - systemd.services.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) { - description = "ProtonMail Bridge SMTP relay"; - after = [ "network.target" ]; - wants = [ "network.target" ]; - wantedBy = [ "multi-user.target" ]; - serviceConfig = { - Type = "simple"; - User = "protonmail-bridge"; - StateDirectory = "protonmail/bridge"; - ExecStart = "${pkgs.writeShellScript "bridge-run" '' - ${pkgs.socat}/bin/socat TCP-LISTEN:1025,fork TCP:127.0.0.1:1025 & - ${pkgs.socat}/bin/socat TCP-LISTEN:1143,fork TCP:127.0.0.1:1143 & - rm -f /tmp/bridge-faketty - mkfifo /tmp/bridge-faketty - cat /tmp/bridge-faketty | ${pkgs.protonmail-bridge}/bin/protonmail-bridge --cli - ''}"; - Restart = "on-failure"; - RestartSec = 10; - }; - }; - - users.users.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) { - isSystemUser = true; - group = "protonmail-bridge"; - home = "/var/lib/protonmail"; - createHome = true; - }; - users.groups.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) {}; - - # ── ProtonMail alert relay ──────────────────────────────────────────────── + # ── msmtp alert relay ──────────────────────────────────────────────────── environment.etc."msmtprc".text = '' defaults auth login @@ -149,45 +119,129 @@ in msmtp sendAlert socat - ] ++ lib.optional (pkgs ? protonmail-bridge) pkgs.protonmail-bridge; + ]; - # ── rs-surface ──────────────────────────────────────────────────────────── - systemd.services.rs-surface = { - description = "Research Stack Surface Daemon"; - after = [ "network.target" "tailscaled.service" ]; - wants = [ "network.target" "tailscaled.service" ]; - wantedBy = [ "multi-user.target" ]; - environment = { - RS_CREDENTIAL_CONFIG = "/etc/rs-surface/credentials.json"; - RS_SURFACE_PORT = "8444"; - RS_SURFACE_HOST = "0.0.0.0"; - RUST_LOG = "info"; - }; - serviceConfig = { - Type = "simple"; - WorkingDirectory = "/opt/rs-surface"; - ExecStart = "${pkgs.bash}/bin/bash /opt/rs-surface/run.sh"; - Restart = "always"; - RestartSec = 5; - }; + # ── Caddy — Public TLS edge ────────────────────────────────────────────── + # This is a "dumb TLS edge": terminates TLS for researchstack.info + + # *.researchstack.info via Porkbun DNS-01 challenge, then forwards ALL + # HTTP traffic to the internal Caddy router over Tailscale. + # + # Exceptions: + # - auth.researchstack.info → forwarded with Host preserved (OIDC issuer) + # - mail/webmail subdomains → forwarded to internal router + # + # No path routing logic lives here. Traefik (k3s built-in) handles all + # path-based routing inside the cluster via Ingress resources. + systemd.services.caddy.serviceConfig.EnvironmentFile = [ "/etc/caddy/porkbun.env" ]; + + sops.secrets.porkbun-env = { + sopsFile = ./secrets/porkbun-env.age; + format = "yaml"; + path = "/etc/caddy/porkbun.env"; }; - # ── Caddy ───────────────────────────────────────────────────────────────── services.caddy = { enable = true; + package = pkgs.caddy; extraConfig = '' - mail.${domain} { reverse_proxy 127.0.0.1:30808 } - webmail.${domain} { reverse_proxy 127.0.0.1:30808 } - cred.${domain} { reverse_proxy 127.0.0.1:8444 } + (porkbun_tls) { + tls { + dns porkbun { + api_key {$PORKBUN_API_KEY} + api_secret_key {$PORKBUN_SECRET_KEY} + } + } + } + + # Primary domain — forward everything to Traefik on k3s-server + researchstack.info { + import porkbun_tls + reverse_proxy ${internalRouter} { + header_up Host {host} + header_up X-Real-IP {remote} + header_up X-Forwarded-For {remote} + header_up X-Forwarded-Proto {scheme} + header_up X-Forwarded-Host {host} + } + } + + # Auth subdomain — stable OIDC issuer (preserved, not redirected) + auth.${domain} { + import porkbun_tls + reverse_proxy ${internalRouter} { + header_up Host auth.${domain} + header_up X-Real-IP {remote} + header_up X-Forwarded-For {remote} + header_up X-Forwarded-Proto {scheme} + header_up X-Forwarded-Host auth.${domain} + } + } + + # Mail subdomains — forward to Traefik (mail Ingress pending) + mail.${domain}, webmail.${domain} { + import porkbun_tls + reverse_proxy ${internalRouter} { + header_up Host {host} + header_up X-Real-IP {remote} + header_up X-Forwarded-For {remote} + header_up X-Forwarded-Proto {scheme} + header_up X-Forwarded-Host {host} + } + } + + # Legacy subdomain catch-all — 301 to canonical paths + # These exist for bookmark/client compat during migration. + status.${domain} { + import porkbun_tls + redir https://${domain}/server/status/{uri} 301 + } + + dash.${domain}, home.${domain} { + import porkbun_tls + redir https://${domain}/ 301 + } + + media.${domain} { + import porkbun_tls + redir https://${domain}/apps/jellyfin/{uri} 301 + } + + books.${domain} { + import porkbun_tls + redir https://${domain}/apps/books/{uri} 301 + } + + music.${domain} { + import porkbun_tls + redir https://${domain}/apps/music/{uri} 301 + } + + vault.${domain} { + import porkbun_tls + redir https://${domain}/server/vault/{uri} 301 + } + + pulse.${domain} { + import porkbun_tls + redir https://${domain}/api/registry/{uri} 301 + } + + apps.${domain} { + import porkbun_tls + redir https://${domain}/apps/{uri} 301 + } + + # Wildcard fallback — anything else gets a redirect to root + *.${domain} { + import porkbun_tls + redir https://${domain}{uri} 301 + } ''; }; # ── Firewall ────────────────────────────────────────────────────────────── networking.firewall.allowedTCPPorts = [ - 25 143 465 587 993 - 1025 1143 - 8444 - 30808 + 25 143 465 587 993 # Mail protocols (Postfix + Dovecot) ]; } diff --git a/4-Infrastructure/k3s-flake/k3s-server.nix b/4-Infrastructure/k3s-flake/k3s-server.nix index e64e31b0..76846e9d 100644 --- a/4-Infrastructure/k3s-flake/k3s-server.nix +++ b/4-Infrastructure/k3s-flake/k3s-server.nix @@ -79,12 +79,11 @@ default: true exposedPort: 30080 protocol: TCP + # websecure NodePort not exposed — TLS terminates at the edge Caddy, + # never reaches Traefik. Disabling reduces surface area. websecure: - port: 8443 - nodePort: 30443 expose: default: false - protocol: TCP service: type: NodePort ingressRoute: @@ -101,11 +100,13 @@ package = pkgs.caddy; extraConfig = '' :80 { - reverse_proxy 127.0.0.1:30080 { + reverse_proxy 100.102.173.61:30080 { header_up Host {host} header_up X-Real-IP {remote} header_up X-Forwarded-For {remote} - header_up X-Forwarded-Proto https + # Preserve the proto set by the edge Caddy (https) rather than + # overwriting it. The edge is authoritative; we just pass it through. + header_up X-Forwarded-Proto {http.request.header.X-Forwarded-Proto} header_up X-Forwarded-Host {host} } }