mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-31 03:05:21 +00:00
allaun
475f6319ea
This squashes all local history (768 commits) onto the scrubbed PR #90 baseline. Individual commits were lost during filter-repo corruption; the working tree content is preserved intact. Build: N/A (working tree state only)
399 lines
8.9 KiB
Markdown
399 lines
8.9 KiB
Markdown
# Ops Runbook — Research Stack
|
|
|
|
**Last updated:** 2026-05-29
|
|
|
|
Quick-reference operational procedures for k3s, FPGA, Tailscale, GPU, and DNS.
|
|
|
|
---
|
|
|
|
## k3s Cluster
|
|
|
|
### Start / Stop
|
|
|
|
```bash
|
|
# Set kubeconfig
|
|
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
|
|
|
|
# Check cluster status
|
|
kubectl get nodes
|
|
kubectl get pods -A
|
|
```
|
|
|
|
**Control plane (nixos):**
|
|
```bash
|
|
# Restart k3s server
|
|
sudo systemctl restart k3s
|
|
|
|
# Check k3s server health
|
|
sudo systemctl status k3s
|
|
sudo journalctl -u k3s -f --lines=50
|
|
```
|
|
|
|
### Check Health
|
|
|
|
```bash
|
|
# All nodes ready
|
|
kubectl get nodes -o wide
|
|
|
|
# All pods running (any namespace)
|
|
kubectl get pods -A --field-selector='status.phase!=Running,status.phase!=Succeeded'
|
|
|
|
# Specific namespace
|
|
kubectl get pods -n services
|
|
kubectl get pods -n media
|
|
kubectl get pods -n monitoring
|
|
kubectl get pods -n ai-models
|
|
```
|
|
|
|
### Restart Pods
|
|
|
|
```bash
|
|
# Restart a deployment (rolling restart)
|
|
kubectl rollout restart deployment/<name> -n <namespace>
|
|
|
|
# Examples
|
|
kubectl rollout restart deployment/homer -n services
|
|
kubectl rollout restart deployment/ollama -n ai-models
|
|
kubectl rollout restart deployment/cluster-dashboard -n monitoring
|
|
|
|
# Force delete stuck pod
|
|
kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force
|
|
```
|
|
|
|
### View Logs
|
|
|
|
```bash
|
|
# Pod logs
|
|
kubectl logs <pod-name> -n <namespace> --tail=100 -f
|
|
|
|
# Previous container (if crashed)
|
|
kubectl logs <pod-name> -n <namespace> --previous
|
|
|
|
# All pods matching label
|
|
kubectl logs -l app=<label> -n <namespace> --tail=50
|
|
```
|
|
|
|
### Common Pods
|
|
|
|
| Namespace | Service | Deployment Name |
|
|
|-----------|---------|-----------------|
|
|
| `services` | Homer | `homer` |
|
|
| `services` | Hermes | `hermes` |
|
|
| `services` | Actual Budget | `actual-budget` |
|
|
| `services` | Uptime Kuma | `uptime-kuma` |
|
|
| `services` | Vaultwarden | `vaultwarden` |
|
|
| `services` | Authentik | `authentik` |
|
|
| `services` | Credential Server | `credential-server` |
|
|
| `services` | Registry API | `registry-api` |
|
|
| `services` | Jobs API | `jobs-api` |
|
|
| `services` | Blobs API | `blobs-api` |
|
|
| `ai-models` | Ollama | `ollama` |
|
|
| `monitoring` | Cluster Dashboard | `cluster-dashboard` |
|
|
| `media` | Jellyfin | `jellyfin` |
|
|
| `research` | AlphaProof | `alphaproof` |
|
|
|
|
---
|
|
|
|
## FPGA — Tang Nano 9K
|
|
|
|
### Flash Bitstream
|
|
|
|
```bash
|
|
# Build
|
|
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
|
|
|
# Flash via USB-JTAG
|
|
openFPGALoader -b tangnano9k research_stack_top.fs
|
|
|
|
# Verify flash
|
|
openFPGALoader -b tangnano9k --verify research_stack_top.fs
|
|
```
|
|
|
|
### Verify LEDs
|
|
|
|
After power-on, `led[0:5]` (pins 10-16) should show SUBLEQ state.
|
|
- **All off:** CPU halted or not clocked
|
|
- **Blinking:** CPU running (heartbeat)
|
|
- **LED 0 solid on:** CPU halted (trap)
|
|
|
|
### Debug UART
|
|
|
|
```bash
|
|
# Connect to UART TX (pin 17) via USB-serial adapter
|
|
# Baud rate: 115384 (27MHz / 234)
|
|
picocom -b 115384 /dev/ttyUSB0
|
|
|
|
# Or with screen
|
|
screen /dev/ttyUSB0 115384
|
|
```
|
|
|
|
### Run Simulation
|
|
|
|
```bash
|
|
cd /tmp/fpga_sim_full && ./obj_dir/sim_top
|
|
```
|
|
|
|
### Pin Reference
|
|
|
|
| Pin | Signal | Direction | Notes |
|
|
|-----|--------|-----------|-------|
|
|
| 52 | clk | input | 27 MHz oscillator |
|
|
| 4 | rst_n | input | Active-low reset (pull-up) |
|
|
| 3 | user_btn | input | Active-low (pull-up) |
|
|
| 10-16 | led[0:5] | output | LVCMOS18 |
|
|
| 17 | uart_tx | output | 115384 baud |
|
|
| 18 | uart_rx | input | Pull-up |
|
|
|
|
### FPGA Not Responding
|
|
|
|
1. Check USB connection to Tang Nano 9K
|
|
2. Verify `openFPGALoader` sees the device: `openFPGALoader --detect`
|
|
3. Re-flash bitstream: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
|
4. Check clock: pin 52 should have 27 MHz (oscilloscope)
|
|
5. Assert reset: pull pin 4 low momentarily, then release
|
|
6. Check power: board should draw ~100mA from USB
|
|
|
|
---
|
|
|
|
## Tailscale
|
|
|
|
### Check Status
|
|
|
|
```bash
|
|
# Show tailnet status
|
|
tailscale status
|
|
|
|
# Show current node IP
|
|
tailscale ip
|
|
|
|
# Check connectivity to other nodes
|
|
tailscale ping qfox-1
|
|
tailscale ping 361395-1
|
|
tailscale ping racknerd-510bd9c
|
|
```
|
|
|
|
### Restart Funnel
|
|
|
|
The Funnel runs on `361395-1` and routes to Traefik NodePort 30080:
|
|
|
|
```bash
|
|
# On 361395-1 (edge node):
|
|
tailscale funnel 8080 off
|
|
tailscale funnel 8080
|
|
|
|
# Verify funnel URL
|
|
tailscale funnel status
|
|
```
|
|
|
|
### Debug Relay
|
|
|
|
```bash
|
|
# Check if relay is in use
|
|
tailscale status | grep relay
|
|
|
|
# Force direct connections (disable relay)
|
|
tailscale set --direct-peers-only=true
|
|
|
|
# Reset to default
|
|
tailscale set --direct-peers-only=false
|
|
```
|
|
|
|
### Re-authenticate
|
|
|
|
```bash
|
|
# On any node:
|
|
tailscale up --authkey=<TS_AUTH_KEY>
|
|
|
|
# Or interactive login
|
|
tailscale up
|
|
```
|
|
|
|
### Node IPs
|
|
|
|
| Node | Tailscale IP | Role |
|
|
|------|-------------|------|
|
|
| nixos | 100.102.173.61 | Control plane |
|
|
| qfox-1 | 100.88.57.96 | GPU worker |
|
|
| 361395-1 | 100.72.130.76 | Edge/Funnel |
|
|
| racknerd | 100.80.39.40 | Edge worker |
|
|
| steamdeck | 100.85.244.73 | Worker |
|
|
|
|
---
|
|
|
|
## GPU — QFox (RTX 4070)
|
|
|
|
### Check nvidia-smi
|
|
|
|
```bash
|
|
# GPU status
|
|
nvidia-smi
|
|
|
|
# Watch GPU utilization
|
|
nvidia-smi -l 1
|
|
|
|
# Check driver version
|
|
nvidia-smi --query-gpu=driver_version --format=csv,noheader
|
|
# Expected: 610.43
|
|
```
|
|
|
|
### Restart Ollama
|
|
|
|
```bash
|
|
# If running as k3s pod
|
|
kubectl rollout restart deployment/ollama -n ai-models
|
|
kubectl logs -l app=ollama -n ai-models --tail=20 -f
|
|
|
|
# If running as systemd service on qfox-1
|
|
sudo systemctl restart ollama
|
|
sudo journalctl -u ollama -f --tail=20
|
|
```
|
|
|
|
### Pull / Manage Models
|
|
|
|
```bash
|
|
# List loaded models
|
|
curl http://100.88.57.96:11434/api/tags
|
|
|
|
# Pull a model
|
|
curl http://100.88.57.96:11434/api/pull -d '{"name": "deepseek-coder-v2:16b"}'
|
|
|
|
# Test inference
|
|
curl http://100.88.57.96:11434/api/generate \
|
|
-d '{"model": "deepseek-coder-v2:16b", "prompt": "Hello", "stream": false}'
|
|
```
|
|
|
|
### Ollama via NodePort
|
|
|
|
```bash
|
|
# Access from cluster
|
|
curl http://<any-node-ip>:31434/api/tags
|
|
```
|
|
|
|
---
|
|
|
|
## DNS & TLS
|
|
|
|
### Check LE Certificates
|
|
|
|
```bash
|
|
# Check cert expiry
|
|
echo | openssl s_client -connect researchstack.info:443 -servername researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
|
|
|
# Wildcard cert
|
|
echo | openssl s_client -connect registry.researchstack.info:443 -servername registry.researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
|
```
|
|
|
|
**Current cert valid until:** 2026-08-18
|
|
|
|
### Renew Certificates
|
|
|
|
Caddy auto-renews via Porkbun DNS-01. If manual renewal is needed:
|
|
|
|
```bash
|
|
# Restart Caddy to trigger renewal check
|
|
kubectl rollout restart deployment/caddy -n services
|
|
|
|
# Check Caddy logs for renewal
|
|
kubectl logs -l app=caddy -n services --tail=50 | grep -i renew
|
|
```
|
|
|
|
### Check Porkbun DNS
|
|
|
|
```bash
|
|
# Verify A record
|
|
dig researchstack.info +short
|
|
dig auth.researchstack.info +short
|
|
dig registry.researchstack.info +short
|
|
|
|
# Check API key (set PORKBUN_API_KEY env var)
|
|
curl -X POST https://api.porkbun.com/api/json/v3/ping \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"apikey": "'$PORKBUN_API_KEY'"}'
|
|
```
|
|
|
|
---
|
|
|
|
## Common Failures
|
|
|
|
### Node Down
|
|
|
|
```bash
|
|
# Identify down node
|
|
kubectl get nodes
|
|
|
|
# Check node conditions
|
|
kubectl describe node <node-name>
|
|
|
|
# On the node itself:
|
|
sudo systemctl status k3s-agent # worker nodes
|
|
sudo systemctl status k3s # control plane
|
|
|
|
# Rejoin cluster if needed (worker):
|
|
sudo k3s agent --server https://100.102.173.61:6443 --token <TOKEN>
|
|
```
|
|
|
|
### Pod CrashLoopBackOff
|
|
|
|
```bash
|
|
# Check events
|
|
kubectl describe pod <pod-name> -n <namespace>
|
|
|
|
# Check logs (including previous crash)
|
|
kubectl logs <pod-name> -n <namespace> --previous --tail=100
|
|
|
|
# Common fixes:
|
|
# - ConfigMap/Secret missing: kubectl get configmap -n <ns>; kubectl get secret -n <ns>
|
|
# - Image pull error: check image tag and registry access
|
|
# - Resource limits: kubectl top pod -n <namespace>
|
|
# - OOMKill: increase memory limit in deployment spec
|
|
```
|
|
|
|
### FPGA Not Responding
|
|
|
|
1. Unplug and replug USB cable
|
|
2. Check `ls /dev/ttyUSB*` — device should appear
|
|
3. Re-flash: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
|
4. If JTAG fails: try holding reset (pin 4 low) while plugging in
|
|
5. Check power LED on Tang Nano 9K board
|
|
6. Try different USB port / cable
|
|
|
|
### Tailscale Tunnel Down
|
|
|
|
```bash
|
|
# Check daemon
|
|
sudo systemctl status tailscaled
|
|
|
|
# Restart
|
|
sudo systemctl restart tailscaled
|
|
tailscale up
|
|
|
|
# Check if key expired
|
|
tailscale status | grep -i expir
|
|
```
|
|
|
|
### DNS Resolution Failing
|
|
|
|
```bash
|
|
# Check Caddy pod
|
|
kubectl logs -l app=caddy -n services --tail=50
|
|
|
|
# Test DNS from inside cluster
|
|
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup researchstack.info
|
|
|
|
# Check Porkbun API key
|
|
kubectl get secret porkbun-credentials -n services -o jsonpath='{.data.api-key}' | base64 -d
|
|
```
|
|
|
|
---
|
|
|
|
## Escalation Matrix
|
|
|
|
| Issue | First Response | Escalation |
|
|
|-------|---------------|------------|
|
|
| Pod down | `kubectl rollout restart` | Check node, check resource limits |
|
|
| Node down | SSH to node, check `systemctl` | Reboot, check hardware |
|
|
| FPGA unresponsive | Re-flash bitstream | Check USB, try different board |
|
|
| Tailscale tunnel | `tailscale up` | Check auth key, restart daemon |
|
|
| DNS/cert | Restart Caddy | Check Porkbun API, check ingress |
|
|
| GPU errors | Check `nvidia-smi` | Restart driver, check PCIe seating |
|
|
| OOM on Ollama | Restart pod | Reduce context length, switch model |
|