allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(k3s-edge): rewrite edge Caddy as dumb TLS forwarder + legacy 301s

Browse source 
Edge Caddy now does exactly three things:
1. Terminate TLS for researchstack.info + *.researchstack.info via
   Porkbun DNS-01 (wildcard cert covers all subdomains in one renewal)
2. 301-redirect legacy subdomains to canonical path equivalents:
     status.*  → /server/status/
     dash.*, home.*  → /
     media.*  → /apps/jellyfin/
     books.*  → /apps/books/
     music.*  → /apps/music/
     vault.*  → /server/vault/
     pulse.*  → /api/registry/
     apps.*  → /apps/
     *.* (wildcard fallback)  → /
3. Forward all other traffic to the internal router (host Caddy :80 on
   k3s-server over Tailscale) with X-Forwarded-* headers preserved.
   auth.* and mail/webmail.* are forwarded unchanged (stable subdomains).

No path routing logic on the edge. Traefik Ingress (k3s-server) owns
all path decisions. This commit has no effect until nixos-rebuild switch
is run on microvm-racknerd (deploy after k3s-server is verified).

Generated with Devin (https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Brandon Schneider 2026-05-26 21:04:17 -05:00
parent 38f1022a20
commit 8d1d94b256
3 changed files with 236 additions and 98 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

145
4-Infrastructure/k3s-flake/README.md
View file

@ -8,10 +8,68 @@ no external dependencies embedded in the flake.
> A node goes online → it joins → it goes offline → the cluster adjusts.
The flake spans the full topology spectrum:
- **Server-class x86** (core, judge, mirror, foxtop) — full k8s workloads
- **Thin client / Pi** (edge) — lightweight k3s agent, pulse heartbeat only
- **microvm-nerdrack** — zero work, just pulse. Tainted `pulse-only:NoSchedule`.
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Internet │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Public Edge (microvm-racknerd) │ │
│ │ Caddy: TLS termination only (Porkbun DNS-01) │ │
│ │ Ports 80/443 → forwards ALL traffic over Tailscale │ │
│ └───────────────────────────┬─────────────────────────────────┘ │
│ │ Tailscale mesh │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Traefik Ingress (nixos-laptop / k3s-server :80) │ │
│ │ Path routing + forward_auth middleware (Authentik) │ │
│ │ Defined in manifests/ingress/ (Ingress + Middleware CRDs) │ │
│ └───────────────────────────┬─────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ k3s Service Layer (ClusterIP) │ │
│ │ Hermes, Authentik, Uptime Kuma, Homarr, control-plane APIs │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Worker Pool (qfox-1, steamdeck, 361395-1, ...) │ │
│ │ GPU compute, storage, downloaders, codecs │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```
## URL Contract (canonical paths on `https://researchstack.info`)
| Path | Service | Type |
|------|---------|------|
| `/` | Homer directory | Dashboard |
| `/gettingstarted` | Static page | Info |
| `/apps/chat/*` | Hermes (chat/orchestrator) | App |
| `/apps/jellyfin/*` | Jellyfin | App |
| `/apps/books/*` | Audiobookshelf | App |
| `/apps/music/*` | Navidrome | App |
| `/apps/budget/*` | Actual Budget | App |
| `/server/status/*` | Uptime Kuma | Ops UI |
| `/server/dash/*` | Homarr | Ops UI |
| `/server/vault/*` | Vaultwarden | Ops UI |
| `/api/cred/*` | Credential Server | Control-plane |
| `/api/registry/*` | Registry API (join/heartbeat) | Control-plane |
| `/api/jobs/*` | Job Router | Control-plane |
| `/api/blobs/*` | Blob Plane | Control-plane |
**Stable subdomains** (not path-routed):
- `auth.researchstack.info` → Authentik (OIDC issuer — must not change)
- `mail.researchstack.info` / `webmail.researchstack.info` → Mail services
**Legacy subdomain redirects** (301 → canonical path):
- `status.*` → `/server/status/`
- `dash.*` / `home.*` → `/`
- `media.*` → `/apps/jellyfin/`
- `books.*` → `/apps/books/`
- `music.*` → `/apps/music/`
- `vault.*` → `/server/vault/`
- `apps.*` → `/apps/`
- `pulse.*` → `/api/registry/`
## File Layout
@ -19,7 +77,8 @@ The flake spans the full topology spectrum:
4-Infrastructure/k3s-flake/
├── flake.nix — 6 topology configurations
├── k3s-configuration.nix — base module (Tailscale, SSH, Nix, firewall, sops)
├── k3s-server.nix — control plane + Caddy/Porkbun + deploy oneshot
├── k3s-server.nix — control plane + Traefik Ingress + deploy oneshot
├── k3s-edge.nix — public TLS edge (Caddy) + mail services
├── .sops.yaml — age key rules
├── secrets/ — encrypted at rest, decrypted at activation
│ ├── k3s-token.age — K3S_TOKEN=<value>
@ -32,13 +91,21 @@ The flake spans the full topology spectrum:
│ ├── edge.nix — label: role=edge, taint: pulse-only:NoSchedule
│ └── foxtop.nix — label: role=foxtop
├── manifests/ — Kubernetes resources, auto-deployed by systemd
│ ├── kustomization.yaml
│ ├── kustomization.yaml — master resource list
│ ├── namespace.yaml — namespace: services
│ ├── authentik/ — HelmChart CRD (official chart + in-cluster PG/Redis)
│ ├── uptime-kuma/ — Deployment + NodePort 30801 + PVC
│ ├── heimdall/ — Deployment + NodePort 30802 + PVC
│ ├── homer/ — Deployment + NodePort 30803 + ConfigMap
│ └── pulse-receiver/ — Deployment + NodePort 30804 (inline Python receiver)
│ ├── ingress/ — Traefik Ingress + Middleware CRDs (path routing)
│ ├── authentik/ — HelmChart CRD (OIDC @ auth.researchstack.info)
│ ├── hermes/ — Deployment + ClusterIP (placeholder → /apps/chat/)
│ ├── credential-server/ — Deployment + ClusterIP (/api/cred/*)
│ ├── control-plane/ — Registry, Jobs, Blobs APIs (/api/*)
│ ├── uptime-kuma/ — Deployment + Service (/server/status/)
│ ├── homer/ — Deployment + Service (/)
│ ├── homarr/ — Deployment + Service (/server/dash/)
│ ├── actual-budget/ — Deployment + Service (/apps/budget/)
│ ├── vaultwarden/ — Deployment + Service (/server/vault/)
│ ├── media/ — Jellyfin, Navidrome, Audiobookshelf, *arr stack
│ ├── heimdall/ — [LEGACY] being replaced by path routing
│ └── pulse-receiver/ — [LEGACY] being replaced by /api/registry/
└── scripts/
└── deploy-services.sh — idempotent kubectl apply, called by systemd oneshot
```
@ -49,35 +116,51 @@ The flake spans the full topology spectrum:
| Role | k3s Label | Taints | Workload |
|---------|-------------------------------------------------|---------------------------|----------|
| server | — (control plane) | — | Caddy ingress + deploy-manifests |
| core | `topology.researchstack.io/role=core` | — | General compute (PG, Redis, etc.) |
| server | — (control plane) | — | Internal Caddy router + deploy-manifests |
| core | `topology.researchstack.io/role=core` | — | General compute (PG, Redis, Authentik) |
| judge | `topology.researchstack.io/role=judge` | — | Validation / audit |
| mirror | `topology.researchstack.io/role=mirror` | — | Storage / replication |
| edge | `topology.researchstack.io/role=edge` | `pulse-only:NoSchedule` | Pulse heartbeat only |
| foxtop | `topology.researchstack.io/role=foxtop` | — | Top-level orchestrator |
| edge | `topology.researchstack.io/role=edge` | `pulse-only:NoSchedule` | Public TLS edge + mail |
| foxtop | `topology.researchstack.io/role=foxtop` | — | Primary compute / orchestrator |
### Service Placement
| Service | NodePort | Prefers Role | Stateful? |
|--------------------|----------|--------------|-----------|
| Authentik | 30800 | core, server | Yes (PG + Redis PVCs) |
| Uptime Kuma | 30801 | any | PVC (1Gi) — node-bound |
| Heimdall | 30802 | any | PVC (1Gi) — node-bound |
| Homer | 30803 | any | No (ConfigMap) |
| Pulse Receiver | 30804 | any | No (stateless) |
| Service | Port | Path | Prefers Role | Stateful? |
|--------------------|-------|-------------------|--------------|-----------|
| Authentik | 80 | auth.* subdomain | core | Yes (PG + Redis PVCs) |
| Hermes | 80 | /apps/chat/ | any | No (placeholder) |
| Uptime Kuma | 3001 | /server/status/ | any | PVC (1Gi) |
| Homer | 8080 | / | any | No (ConfigMap) |
| Homarr | 7575 | /server/dash/ | any | PVC |
| Actual Budget | 5006 | /apps/budget/ | any | PVC |
| Vaultwarden | 80 | /server/vault/ | any | PVC |
| Credential Server | 8444 | /api/cred/ | any | Secret vol |
| Registry API | 8080 | /api/registry/ | any | No (stub) |
| Jobs API | 8080 | /api/jobs/ | any | No (stub) |
| Blobs API | 8080 | /api/blobs/ | any | No (stub) |
### Domain Mapping (Caddy)
All services use ClusterIP and are routed via Traefik Ingress. Legacy
NodePort assignments are preserved for backward compat but are not the
primary routing path.
All services are served under `*.YOUR_DOMAIN` (e.g. `researchstack.info`):
### Routing Model (Caddy edge + Traefik Ingress)
- `auth.YOUR_DOMAIN` → NodePort 30800 → Authentik
- `status.YOUR_DOMAIN` → NodePort 30801 → Uptime Kuma
- `apps.YOUR_DOMAIN` → NodePort 30802 → Heimdall
- `home.YOUR_DOMAIN` → NodePort 30803 → Homer
- `pulse.YOUR_DOMAIN` → NodePort 30804 → Pulse Receiver
- `YOUR_DOMAIN` → static response "k3s unified topology — Research Stack"
**Public edge** (`k3s-edge.nix` / microvm-racknerd):
- Caddy terminates TLS via Porkbun DNS-01 wildcard
- Forwards all traffic to Traefik (`100.102.173.61:80`) over Tailscale
- Handles legacy subdomain 301 redirects at the edge
- `auth.*` and `mail.*`/`webmail.*` forwarded with Host header preserved
TLS via Porkbun DNS challenge (caddy-dns/porkbun plugin).
**Traefik Ingress** (k3s built-in, nixos-laptop):
- Listens on `:80` (node port, HTTP — TLS handled by edge)
- Path-based routing defined in `manifests/ingress/ingress.yaml`
- Traefik Middleware CRDs for:
- `authentik-forward-auth` — SSO gate for `/apps/*`, `/server/*`, `/`
- `strip-*` — prefix stripping per route
- `/api/*` routes skip `forward_auth` (token-authenticated)
- `auth.researchstack.info` matched by separate Ingress (no middleware)
TLS via Porkbun DNS challenge (caddy-dns/porkbun plugin) at the edge.
## Node Lifecycle

178
4-Infrastructure/k3s-flake/k3s-edge.nix
View file

@ -2,6 +2,8 @@
let
mailDomain = domain;
# Internal router target (k3s-server / nixos-laptop over Tailscale)
internalRouter = "100.102.173.61:80";
sendAlert = pkgs.writeShellScriptBin "send-alert" ''
set -euo pipefail
subject="''${1:-Alert}"
@ -35,7 +37,7 @@ in
networking.hostName = lib.mkDefault hostName;
networking.domain = lib.mkDefault mailDomain;
# ── Postfix ───────────────────────────────────────────────────────────────
# ── Postfix (mail stays on edge — non-HTTP protocol) ─────────────────────
services.postfix = {
enable = true;
hostname = "mail.${mailDomain}";
@ -62,7 +64,6 @@ in
ssl_min_protocol = TLSv1.2
mail_location = maildir:/var/mail/%n
auth_mechanisms = plain login
ssl = yes
passdb {
driver = passwd-file
args = /etc/dovecot/users
@ -74,7 +75,6 @@ in
'';
};
# Create mailbox user and directory
users.users.postmaster = {
isSystemUser = true;
uid = 5000;
@ -87,37 +87,7 @@ in
members = [ "postmaster" ];
};
# ── ProtonMail Bridge ─────────────────────────────────────────────────────
systemd.services.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) {
description = "ProtonMail Bridge SMTP relay";
after = [ "network.target" ];
wants = [ "network.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "simple";
User = "protonmail-bridge";
StateDirectory = "protonmail/bridge";
ExecStart = "${pkgs.writeShellScript "bridge-run" ''
${pkgs.socat}/bin/socat TCP-LISTEN:1025,fork TCP:127.0.0.1:1025 &
${pkgs.socat}/bin/socat TCP-LISTEN:1143,fork TCP:127.0.0.1:1143 &
rm -f /tmp/bridge-faketty
mkfifo /tmp/bridge-faketty
cat /tmp/bridge-faketty | ${pkgs.protonmail-bridge}/bin/protonmail-bridge --cli
''}";
Restart = "on-failure";
RestartSec = 10;
};
};
users.users.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) {
isSystemUser = true;
group = "protonmail-bridge";
home = "/var/lib/protonmail";
createHome = true;
};
users.groups.protonmail-bridge = lib.mkIf (pkgs ? protonmail-bridge) {};
# ── ProtonMail alert relay ────────────────────────────────────────────────
# ── msmtp alert relay ────────────────────────────────────────────────────
environment.etc."msmtprc".text = ''
defaults
auth login
@ -149,45 +119,129 @@ in
msmtp
sendAlert
socat
] ++ lib.optional (pkgs ? protonmail-bridge) pkgs.protonmail-bridge;
];
# ── rs-surface ────────────────────────────────────────────────────────────
systemd.services.rs-surface = {
description = "Research Stack Surface Daemon";
after = [ "network.target" "tailscaled.service" ];
wants = [ "network.target" "tailscaled.service" ];
wantedBy = [ "multi-user.target" ];
environment = {
RS_CREDENTIAL_CONFIG = "/etc/rs-surface/credentials.json";
RS_SURFACE_PORT = "8444";
RS_SURFACE_HOST = "0.0.0.0";
RUST_LOG = "info";
};
serviceConfig = {
Type = "simple";
WorkingDirectory = "/opt/rs-surface";
ExecStart = "${pkgs.bash}/bin/bash /opt/rs-surface/run.sh";
Restart = "always";
RestartSec = 5;
};
# ── Caddy — Public TLS edge ──────────────────────────────────────────────
# This is a "dumb TLS edge": terminates TLS for researchstack.info +
# *.researchstack.info via Porkbun DNS-01 challenge, then forwards ALL
# HTTP traffic to the internal Caddy router over Tailscale.
#
# Exceptions:
# - auth.researchstack.info → forwarded with Host preserved (OIDC issuer)
# - mail/webmail subdomains → forwarded to internal router
#
# No path routing logic lives here. Traefik (k3s built-in) handles all
# path-based routing inside the cluster via Ingress resources.
systemd.services.caddy.serviceConfig.EnvironmentFile = [ "/etc/caddy/porkbun.env" ];
sops.secrets.porkbun-env = {
sopsFile = ./secrets/porkbun-env.age;
format = "yaml";
path = "/etc/caddy/porkbun.env";
};
# ── Caddy ─────────────────────────────────────────────────────────────────
services.caddy = {
enable = true;
package = pkgs.caddy;
extraConfig = ''
mail.${domain} { reverse_proxy 127.0.0.1:30808 }
webmail.${domain} { reverse_proxy 127.0.0.1:30808 }
cred.${domain} { reverse_proxy 127.0.0.1:8444 }
(porkbun_tls) {
tls {
dns porkbun {
api_key {$PORKBUN_API_KEY}
api_secret_key {$PORKBUN_SECRET_KEY}
}
}
}
# Primary domain — forward everything to Traefik on k3s-server
researchstack.info {
import porkbun_tls
reverse_proxy ${internalRouter} {
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
header_up X-Forwarded-Host {host}
}
}
# Auth subdomain — stable OIDC issuer (preserved, not redirected)
auth.${domain} {
import porkbun_tls
reverse_proxy ${internalRouter} {
header_up Host auth.${domain}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
header_up X-Forwarded-Host auth.${domain}
}
}
# Mail subdomains — forward to Traefik (mail Ingress pending)
mail.${domain}, webmail.${domain} {
import porkbun_tls
reverse_proxy ${internalRouter} {
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
header_up X-Forwarded-Host {host}
}
}
# Legacy subdomain catch-all — 301 to canonical paths
# These exist for bookmark/client compat during migration.
status.${domain} {
import porkbun_tls
redir https://${domain}/server/status/{uri} 301
}
dash.${domain}, home.${domain} {
import porkbun_tls
redir https://${domain}/ 301
}
media.${domain} {
import porkbun_tls
redir https://${domain}/apps/jellyfin/{uri} 301
}
books.${domain} {
import porkbun_tls
redir https://${domain}/apps/books/{uri} 301
}
music.${domain} {
import porkbun_tls
redir https://${domain}/apps/music/{uri} 301
}
vault.${domain} {
import porkbun_tls
redir https://${domain}/server/vault/{uri} 301
}
pulse.${domain} {
import porkbun_tls
redir https://${domain}/api/registry/{uri} 301
}
apps.${domain} {
import porkbun_tls
redir https://${domain}/apps/{uri} 301
}
# Wildcard fallback — anything else gets a redirect to root
*.${domain} {
import porkbun_tls
redir https://${domain}{uri} 301
}
'';
};
# ── Firewall ──────────────────────────────────────────────────────────────
networking.firewall.allowedTCPPorts = [
25 143 465 587 993
1025 1143
8444
30808
25 143 465 587 993 # Mail protocols (Postfix + Dovecot)
];
}

11
4-Infrastructure/k3s-flake/k3s-server.nix
View file

@ -79,12 +79,11 @@
default: true
exposedPort: 30080
protocol: TCP
# websecure NodePort not exposed — TLS terminates at the edge Caddy,
# never reaches Traefik. Disabling reduces surface area.
websecure:
port: 8443
nodePort: 30443
expose:
default: false
protocol: TCP
service:
type: NodePort
ingressRoute:
@ -101,11 +100,13 @@
package = pkgs.caddy;
extraConfig = ''
:80 {
reverse_proxy 127.0.0.1:30080 {
reverse_proxy 100.102.173.61:30080 {
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto https
# Preserve the proto set by the edge Caddy (https) rather than
# overwriting it. The edge is authoritative; we just pass it through.
header_up X-Forwarded-Proto {http.request.header.X-Forwarded-Proto}
header_up X-Forwarded-Host {host}
}
}