mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-30 18:56:16 +00:00
docs: runbook, FPGA programming guide, disaster recovery, API docs
- RUNBOOK.md: k3s/FPGA/Tailscale/GPU/DNS ops procedures - FPGA_PROGRAMMING_GUIDE.md: SUBLEQ format, memory map, 3 examples - DISASTER_RECOVERY.md: backup/restore for all components - API_DOCS.md: dashboard, credential, registry, jobs, blobs APIs
This commit is contained in:
Brandon Schneider
parent
251ea9d9bf
commit
7a6df586b2
4 changed files with 1570 additions and 0 deletions
465
6-Documentation/API_DOCS.md
Normal file
465
6-Documentation/API_DOCS.md
Normal file
|
|
@ -0,0 +1,465 @@
|
|||
# API Documentation — Research Stack Services
|
||||
|
||||
**Last updated:** 2026-05-29
|
||||
**Base URL:** `https://researchstack.info`
|
||||
**Auth:** Authentik OIDC (SSO) or Bearer token (API endpoints)
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
### OIDC (SSO) — Web Endpoints
|
||||
|
||||
Web-facing endpoints use Authentik OIDC. Access via browser redirects to `auth.researchstack.info`.
|
||||
|
||||
### Token Auth — API Endpoints
|
||||
|
||||
API endpoints (`/api/*`) use Bearer token authentication:
|
||||
|
||||
```
|
||||
Authorization: Bearer <token>
|
||||
```
|
||||
|
||||
Tokens are managed by the Credential Server.
|
||||
|
||||
---
|
||||
|
||||
## Cluster Dashboard
|
||||
|
||||
**Namespace:** `monitoring`
|
||||
**Internal URL:** `http://cluster-dashboard:8787`
|
||||
**External URL:** `https://researchstack.info/server/dash/` (via Homarr)
|
||||
**Tech:** FastAPI + Vite
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### `GET /`
|
||||
|
||||
Dashboard UI (Vite SPA).
|
||||
|
||||
#### `GET /api/status`
|
||||
|
||||
Cluster health summary.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"name": "nixos-laptop",
|
||||
"status": "Ready",
|
||||
"ip": "100.102.173.61",
|
||||
"roles": "control-plane",
|
||||
"version": "v1.35.4+k3s1"
|
||||
}
|
||||
],
|
||||
"pods": {
|
||||
"total": 24,
|
||||
"running": 23,
|
||||
"pending": 0,
|
||||
"failed": 1
|
||||
},
|
||||
"namespaces": ["services", "media", "monitoring", "ai-models", "edge", "research", "mail"]
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /api/metrics`
|
||||
|
||||
Prometheus-compatible metrics.
|
||||
|
||||
#### `WebSocket /ws/live`
|
||||
|
||||
Real-time cluster events stream.
|
||||
|
||||
**Message format:**
|
||||
```json
|
||||
{
|
||||
"type": "pod_event",
|
||||
"namespace": "services",
|
||||
"pod": "homer-abc123",
|
||||
"status": "Running",
|
||||
"timestamp": "2026-05-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Credential Server
|
||||
|
||||
**Namespace:** `services`
|
||||
**Internal URL:** `http://credential-server:8080`
|
||||
**External URL:** `https://researchstack.info/api/cred/`
|
||||
**Auth:** Bearer token
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### `GET /api/cred/health`
|
||||
|
||||
Health check.
|
||||
|
||||
**Response:** `200 OK`
|
||||
```json
|
||||
{"status": "ok"}
|
||||
```
|
||||
|
||||
#### `GET /api/cred/tokens`
|
||||
|
||||
List available tokens (requires admin token).
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"tokens": [
|
||||
{"name": "registry", "scope": "read,write", "expires": "2026-12-31"},
|
||||
{"name": "jobs", "scope": "read,write,execute", "expires": "2026-12-31"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/cred/issue`
|
||||
|
||||
Issue a new token.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "service-name",
|
||||
"scope": "read,write",
|
||||
"ttl": "30d"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"token": "rs_tk_...",
|
||||
"expires": "2026-06-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/cred/validate`
|
||||
|
||||
Validate a token.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{"token": "rs_tk_..."}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"valid": true,
|
||||
"name": "registry",
|
||||
"scope": "read,write"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Registry API
|
||||
|
||||
**Namespace:** `services`
|
||||
**Internal URL:** `http://registry-api:8080`
|
||||
**External URL:** `https://researchstack.info/api/registry/`
|
||||
**Auth:** Bearer token
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### `GET /api/registry/health`
|
||||
|
||||
Health check.
|
||||
|
||||
#### `GET /api/registry/artifacts`
|
||||
|
||||
List all registered artifacts.
|
||||
|
||||
**Query Parameters:**
|
||||
- `type` — Filter by artifact type (e.g., `bitstream`, `lean-build`, `receipt`)
|
||||
- `since` — ISO 8601 timestamp
|
||||
- `limit` — Max results (default: 50)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"id": "art_abc123",
|
||||
"name": "research_stack_top.fs",
|
||||
"type": "bitstream",
|
||||
"size": 184320,
|
||||
"sha256": "...",
|
||||
"created": "2026-05-29T10:00:00Z",
|
||||
"tags": ["fpga", "production"]
|
||||
}
|
||||
],
|
||||
"total": 42
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/registry/artifacts`
|
||||
|
||||
Register a new artifact.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "research_stack_top.fs",
|
||||
"type": "bitstream",
|
||||
"sha256": "...",
|
||||
"blob_ref": "blob_xyz",
|
||||
"tags": ["fpga"]
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /api/registry/artifacts/{id}`
|
||||
|
||||
Get artifact by ID.
|
||||
|
||||
#### `PUT /api/registry/artifacts/{id}`
|
||||
|
||||
Update artifact metadata.
|
||||
|
||||
#### `DELETE /api/registry/artifacts/{id}`
|
||||
|
||||
Remove artifact registration (does not delete blob).
|
||||
|
||||
---
|
||||
|
||||
## Jobs API
|
||||
|
||||
**Namespace:** `services`
|
||||
**Internal URL:** `http://jobs-api:8080`
|
||||
**External URL:** `https://researchstack.info/api/jobs/`
|
||||
**Auth:** Bearer token
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### `GET /api/jobs/health`
|
||||
|
||||
Health check.
|
||||
|
||||
#### `GET /api/jobs`
|
||||
|
||||
List jobs.
|
||||
|
||||
**Query Parameters:**
|
||||
- `status` — Filter: `pending`, `running`, `completed`, `failed`
|
||||
- `type` — Filter by job type
|
||||
- `limit` — Max results (default: 20)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": "job_abc123",
|
||||
"type": "lean-build",
|
||||
"status": "completed",
|
||||
"created": "2026-05-29T10:00:00Z",
|
||||
"started": "2026-05-29T10:00:05Z",
|
||||
"finished": "2026-05-29T10:05:30Z",
|
||||
"result": {
|
||||
"jobs": 3572,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"total": 156
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/jobs`
|
||||
|
||||
Create a new job.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"type": "lean-build",
|
||||
"params": {
|
||||
"target": "Semantics",
|
||||
"branch": "main"
|
||||
},
|
||||
"priority": "normal"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "job_def456",
|
||||
"status": "pending",
|
||||
"created": "2026-05-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /api/jobs/{id}`
|
||||
|
||||
Get job status and result.
|
||||
|
||||
#### `POST /api/jobs/{id}/cancel`
|
||||
|
||||
Cancel a pending or running job.
|
||||
|
||||
#### `GET /api/jobs/{id}/logs`
|
||||
|
||||
Stream job logs (Server-Sent Events).
|
||||
|
||||
**Response:** `text/event-stream`
|
||||
```
|
||||
data: {"line": "Building Semantics...", "ts": "2026-05-29T12:00:01Z"}
|
||||
data: {"line": "3572 jobs, 0 errors", "ts": "2026-05-29T12:05:30Z"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Blobs API
|
||||
|
||||
**Namespace:** `services`
|
||||
**Internal URL:** `http://blobs-api:8080`
|
||||
**External URL:** `https://researchstack.info/api/blobs/`
|
||||
**Auth:** Bearer token
|
||||
|
||||
### Endpoints
|
||||
|
||||
#### `GET /api/blobs/health`
|
||||
|
||||
Health check.
|
||||
|
||||
#### `POST /api/blobs`
|
||||
|
||||
Upload a blob.
|
||||
|
||||
**Request:** `multipart/form-data`
|
||||
- `file` — Binary file content
|
||||
- `sha256` — Expected SHA-256 hash (verification)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"id": "blob_xyz789",
|
||||
"size": 184320,
|
||||
"sha256": "...",
|
||||
"created": "2026-05-29T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /api/blobs/{id}`
|
||||
|
||||
Download a blob.
|
||||
|
||||
**Response:** `application/octet-stream` with blob content.
|
||||
|
||||
#### `HEAD /api/blobs/{id}`
|
||||
|
||||
Check blob existence.
|
||||
|
||||
**Response:** `200 OK` (exists) or `404 Not Found`
|
||||
|
||||
#### `DELETE /api/blobs/{id}`
|
||||
|
||||
Delete a blob.
|
||||
|
||||
#### `GET /api/blobs`
|
||||
|
||||
List blobs.
|
||||
|
||||
**Query Parameters:**
|
||||
- `limit` — Max results
|
||||
- `offset` — Pagination offset
|
||||
|
||||
---
|
||||
|
||||
## Authentik — OIDC Configuration
|
||||
|
||||
**URL:** `https://auth.researchstack.info`
|
||||
**Protocol:** OpenID Connect
|
||||
|
||||
### Provider Configuration
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Authorization URL | `https://auth.researchstack.info/application/o/authorize/` |
|
||||
| Token URL | `https://auth.researchstack.info/application/o/token/` |
|
||||
| UserInfo URL | `https://auth.researchstack.info/application/o/userinfo/` |
|
||||
| JWKS URL | `https://auth.researchstack.info/application/o/research-stack/jwks/` |
|
||||
| Issuer | `https://auth.researchstack.info/application/o/research-stack/` |
|
||||
|
||||
### Scopes
|
||||
|
||||
| Scope | Description |
|
||||
|-------|-------------|
|
||||
| `openid` | Standard OIDC |
|
||||
| `profile` | User profile (name, email) |
|
||||
| `email` | Email address |
|
||||
| `groups` | Group membership |
|
||||
|
||||
### Redirect URIs
|
||||
|
||||
```
|
||||
https://researchstack.info/oidc/callback
|
||||
https://researchstack.info/apps/chat/oidc/callback
|
||||
https://researchstack.info/apps/budget/oidc/callback
|
||||
```
|
||||
|
||||
### Application Setup
|
||||
|
||||
In Authentik admin:
|
||||
|
||||
1. **Applications** → Create → Name: `Research Stack`
|
||||
2. **Providers** → Create → OAuth2/OpenID Provider
|
||||
3. Set client type: `Confidential`
|
||||
4. Set redirect URIs above
|
||||
5. Assign to application
|
||||
|
||||
### Service Account Tokens
|
||||
|
||||
For API-to-API auth (no user interaction):
|
||||
|
||||
```bash
|
||||
# Create service account in Authentik
|
||||
# Admin → Directory → Users → Create
|
||||
# Type: Service Account
|
||||
|
||||
# Get token
|
||||
curl -X POST https://auth.researchstack.info/application/o/token/ \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=<service-account-id>" \
|
||||
-d "client_secret=<service-account-secret>" \
|
||||
-d "scope=openid"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Response Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| 200 | Success |
|
||||
| 201 | Created |
|
||||
| 400 | Bad request (invalid parameters) |
|
||||
| 401 | Unauthorized (missing or invalid token) |
|
||||
| 403 | Forbidden (insufficient scope) |
|
||||
| 404 | Not found |
|
||||
| 409 | Conflict (duplicate resource) |
|
||||
| 422 | Unprocessable entity (validation error) |
|
||||
| 500 | Internal server error |
|
||||
|
||||
---
|
||||
|
||||
## Rate Limits
|
||||
|
||||
| Endpoint | Limit | Window |
|
||||
|----------|-------|--------|
|
||||
| `/api/cred/*` | 100 req | 1 minute |
|
||||
| `/api/registry/*` | 200 req | 1 minute |
|
||||
| `/api/jobs/*` | 50 req | 1 minute |
|
||||
| `/api/blobs/*` | 100 req | 1 minute |
|
||||
|
||||
Rate limit headers:
|
||||
```
|
||||
X-RateLimit-Limit: 100
|
||||
X-RateLimit-Remaining: 95
|
||||
X-RateLimit-Reset: 1685366400
|
||||
```
|
||||
343
6-Documentation/DISASTER_RECOVERY.md
Normal file
343
6-Documentation/DISASTER_RECOVERY.md
Normal file
|
|
@ -0,0 +1,343 @@
|
|||
# Disaster Recovery — Research Stack
|
||||
|
||||
**Last updated:** 2026-05-29
|
||||
|
||||
Procedures for backing up and restoring all Research Stack components.
|
||||
|
||||
---
|
||||
|
||||
## k3s Cluster
|
||||
|
||||
### Backup etcd
|
||||
|
||||
```bash
|
||||
# Snapshot etcd (on control plane node — nixos)
|
||||
sudo k3s etcd-snapshot save \
|
||||
--name researchstack-backup \
|
||||
--dir /var/lib/rancher/k3s/server/db/snapshots
|
||||
|
||||
# List snapshots
|
||||
sudo ls -la /var/lib/rancher/k3s/server/db/snapshots/
|
||||
|
||||
# Copy snapshot off-node
|
||||
scp /var/lib/rancher/k3s/server/db/snapshots/researchstack-backup-* \
|
||||
user@backup-host:/backups/k3s/
|
||||
```
|
||||
|
||||
### Restore from Snapshot
|
||||
|
||||
```bash
|
||||
# On control plane (nixos):
|
||||
sudo systemctl stop k3s
|
||||
|
||||
# Restore
|
||||
sudo k3s server \
|
||||
--cluster-reset \
|
||||
--cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/researchstack-backup-<timestamp>
|
||||
|
||||
# Restart
|
||||
sudo systemctl start k3s
|
||||
|
||||
# Verify
|
||||
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
|
||||
kubectl get nodes
|
||||
kubectl get pods -A
|
||||
```
|
||||
|
||||
### Backup Manifests
|
||||
|
||||
All Kubernetes manifests are in the git repo. The source of truth is:
|
||||
|
||||
```bash
|
||||
cd ~/Research\ Stack
|
||||
git push origin main
|
||||
```
|
||||
|
||||
### Automated Backup Schedule
|
||||
|
||||
Create a cron job on nixos:
|
||||
|
||||
```bash
|
||||
# /etc/cron.d/k3s-backup
|
||||
0 3 * * * root k3s etcd-snapshot save --name researchstack-backup --dir /var/lib/rancher/k3s/server/db/snapshots && \
|
||||
find /var/lib/rancher/k3s/server/db/snapshots/ -mtime +7 -delete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FPGA — Tang Nano 9K
|
||||
|
||||
### Backup Bitstream
|
||||
|
||||
The bitstream source and compiled output are in the git repo:
|
||||
|
||||
```bash
|
||||
cd ~/Research\ Stack
|
||||
|
||||
# Source
|
||||
ls 4-Infrastructure/hardware/*.v
|
||||
|
||||
# Compiled bitstream
|
||||
ls 4-Infrastructure/hardware/research_stack_top.fs
|
||||
```
|
||||
|
||||
### Re-flash from Git
|
||||
|
||||
```bash
|
||||
cd ~/Research\ Stack
|
||||
|
||||
# Rebuild if needed
|
||||
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||
|
||||
# Flash
|
||||
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||
|
||||
# Verify
|
||||
openFPGALoader -b tangnano9k --verify research_stack_top.fs
|
||||
```
|
||||
|
||||
### Backup SRAM Contents
|
||||
|
||||
SRAM contents are volatile (lost on power cycle). To preserve runtime state:
|
||||
|
||||
```python
|
||||
# Dump memory via UART before power-off
|
||||
import serial, struct
|
||||
|
||||
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=10)
|
||||
|
||||
# Trigger memory dump (send HALT with dump flag)
|
||||
# ... protocol-specific ...
|
||||
|
||||
# Read and save
|
||||
with open('fpga_memory_dump.bin', 'wb') as f:
|
||||
data = ser.read(8192) # 4K words = 8K bytes
|
||||
f.write(data)
|
||||
|
||||
ser.close()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tailscale
|
||||
|
||||
### Re-authenticate Node
|
||||
|
||||
```bash
|
||||
# On the node that lost auth:
|
||||
sudo systemctl restart tailscaled
|
||||
tailscale up --authkey=<TS_AUTH_KEY>
|
||||
|
||||
# Or interactive
|
||||
tailscale up
|
||||
```
|
||||
|
||||
### Re-join Tailnet
|
||||
|
||||
```bash
|
||||
# Check current status
|
||||
tailscale status
|
||||
|
||||
# If node is missing from tailnet:
|
||||
# 1. Generate new auth key at https://login.tailscale.com/admin/settings/keys
|
||||
# 2. On the node:
|
||||
sudo tailscale up --authkey=tskey-auth-<key>
|
||||
|
||||
# Verify connectivity
|
||||
tailscale ping nixos-laptop
|
||||
tailscale ping qfox-1
|
||||
```
|
||||
|
||||
### Backup Tailscale State
|
||||
|
||||
Tailscale state is stored at `/var/lib/tailscale/`:
|
||||
|
||||
```bash
|
||||
# Backup state directory
|
||||
sudo tar czf /backups/tailscale-state-$(date +%Y%m%d).tar.gz \
|
||||
/var/lib/tailscale/
|
||||
|
||||
# Restore
|
||||
sudo systemctl stop tailscaled
|
||||
sudo tar xzf /backups/tailscale-state-*.tar.gz -C /
|
||||
sudo systemctl start tailscaled
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Git Repository
|
||||
|
||||
### Backup
|
||||
|
||||
The canonical backup is the remote origin. Ensure all changes are pushed:
|
||||
|
||||
```bash
|
||||
cd ~/Research\ Stack
|
||||
|
||||
# Check for unpushed commits
|
||||
git log --oneline origin/main..HEAD
|
||||
|
||||
# Push everything
|
||||
git push origin main --tags
|
||||
|
||||
# Verify clean state
|
||||
git status --branch --short --untracked-files=all
|
||||
```
|
||||
|
||||
### Restore from Remote
|
||||
|
||||
```bash
|
||||
# Clone fresh
|
||||
cd ~
|
||||
git clone <remote-url> "Research Stack"
|
||||
|
||||
# Or reset to remote state
|
||||
cd ~/Research\ Stack
|
||||
git fetch origin
|
||||
git reset --hard origin/main
|
||||
```
|
||||
|
||||
### Backup Remotes
|
||||
|
||||
```bash
|
||||
# List remotes
|
||||
cd ~/Research\ Stack
|
||||
git remote -v
|
||||
|
||||
# Add backup remote
|
||||
git remote add backup <backup-url>
|
||||
git push backup main --tags
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Secrets (sops-nix / age)
|
||||
|
||||
### Backup age Keys
|
||||
|
||||
```bash
|
||||
# age key location (NixOS)
|
||||
cat /etc/age/keys.txt
|
||||
|
||||
# Backup
|
||||
cp /etc/age/keys.txt /backups/age-keys-$(date +%Y%m%d).txt
|
||||
chmod 600 /backups/age-keys-*.txt
|
||||
|
||||
# Or from home directory
|
||||
cat ~/.config/sops/age/keys.txt
|
||||
```
|
||||
|
||||
### Restore age Keys
|
||||
|
||||
```bash
|
||||
# Restore key file
|
||||
cp /backups/age-keys-*.txt /etc/age/keys.txt
|
||||
chmod 600 /etc/age/keys.txt
|
||||
|
||||
# Verify sops can decrypt
|
||||
sops -d secrets.enc.yaml
|
||||
```
|
||||
|
||||
### Backup sops Configuration
|
||||
|
||||
```bash
|
||||
# The .sops.yaml config is in the repo
|
||||
cat ~/Research\ Stack/.sops.yaml
|
||||
|
||||
# Encrypted secrets are also in the repo
|
||||
find ~/Research\ Stack -name '*.enc.yaml' -o -name '*.enc.json'
|
||||
```
|
||||
|
||||
### Regenerate age Key (Last Resort)
|
||||
|
||||
```bash
|
||||
# Generate new key pair
|
||||
age-keygen -o /etc/age/keys.txt
|
||||
|
||||
# Get public key
|
||||
age-keygen -y /etc/age/keys.txt
|
||||
|
||||
# Re-encrypt all secrets with new key
|
||||
# (requires old key or plaintext backup)
|
||||
sops updatekeys secrets.enc.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## DNS — Porkbun
|
||||
|
||||
### Backup API Key
|
||||
|
||||
```bash
|
||||
# From k8s secret
|
||||
kubectl get secret porkbun-credentials -n services \
|
||||
-o jsonpath='{.data.api-key}' | base64 -d
|
||||
|
||||
# Back up securely
|
||||
echo "<api-key>" | age -r <backup-age-pubkey> > /backups/porkbun-api.age
|
||||
|
||||
# Or store in password manager
|
||||
```
|
||||
|
||||
### Recover API Key
|
||||
|
||||
1. Log in to [Porkbun](https://porkbun.com)
|
||||
2. Navigate to **Account → API Access**
|
||||
3. Regenerate or copy existing API key
|
||||
4. Update k8s secret:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic porkbun-credentials \
|
||||
-n services \
|
||||
--from-literal=api-key=<new-key> \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Restart Caddy to pick up new key
|
||||
kubectl rollout restart deployment/caddy -n services
|
||||
```
|
||||
|
||||
### Verify DNS Records
|
||||
|
||||
```bash
|
||||
dig researchstack.info +short
|
||||
dig auth.researchstack.info +short
|
||||
dig registry.researchstack.info +short
|
||||
|
||||
# Test Porkbun API
|
||||
curl -X POST https://api.porkbun.com/api/json/v3/ping \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"apikey": "<key>", "secretapikey": "<secret>"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recovery Priority Order
|
||||
|
||||
When restoring from a total failure, follow this order:
|
||||
|
||||
| Step | Component | Depends On |
|
||||
|------|-----------|------------|
|
||||
| 1 | **Tailscale** | Auth key (from password manager or backup) |
|
||||
| 2 | **Git repo** | Network connectivity (Tailscale or direct) |
|
||||
| 3 | **age keys** | Backup location |
|
||||
| 4 | **k3s cluster** | etcd snapshot + age keys for secrets |
|
||||
| 5 | **DNS/Porkbun** | API key (from backup or Porkbun account) |
|
||||
| 6 | **FPGA** | Git repo (source + bitstream) |
|
||||
| 7 | **Services** | k3s cluster + secrets + DNS |
|
||||
|
||||
---
|
||||
|
||||
## Full System Checklist
|
||||
|
||||
After recovery, verify each component:
|
||||
|
||||
- [ ] Tailscale: `tailscale status` shows all 5 nodes
|
||||
- [ ] k3s: `kubectl get nodes` shows all nodes Ready
|
||||
- [ ] Pods: `kubectl get pods -A` — all Running/Succeeded
|
||||
- [ ] DNS: `dig researchstack.info` resolves correctly
|
||||
- [ ] TLS: cert valid (`openssl s_client`)
|
||||
- [ ] Auth: `auth.researchstack.info` loads Authentik
|
||||
- [ ] Funnel: `361395-1.tail4e7094.ts.net` responds
|
||||
- [ ] Ollama: `curl http://100.88.57.96:31434/api/tags` returns models
|
||||
- [ ] FPGA: `openFPGALoader --detect` sees Tang Nano 9K
|
||||
- [ ] Dashboard: `https://researchstack.info` shows Homer
|
||||
363
6-Documentation/FPGA_PROGRAMMING_GUIDE.md
Normal file
363
6-Documentation/FPGA_PROGRAMMING_GUIDE.md
Normal file
|
|
@ -0,0 +1,363 @@
|
|||
# FPGA Programming Guide — SUBLEQ on the Blitter
|
||||
|
||||
**Last updated:** 2026-05-29
|
||||
**Board:** Sipeed Tang Nano 9K (Gowin GW1NR-LV9)
|
||||
**CPU:** Blitter6502OISC (One Instruction Set Computer — SUBLEQ)
|
||||
|
||||
---
|
||||
|
||||
## SUBLEQ Instruction Format
|
||||
|
||||
The Blitter implements a **SUBLEQ** (Subtract and Branch if Less-than-or-Equal to Zero) CPU.
|
||||
|
||||
Each instruction is **3 words** (3 × 16-bit = 6 bytes):
|
||||
|
||||
```
|
||||
[src] [dst] [next]
|
||||
```
|
||||
|
||||
**Semantics:**
|
||||
```
|
||||
mem[dst] = mem[dst] - mem[src]
|
||||
if mem[dst] <= 0:
|
||||
PC = next
|
||||
else:
|
||||
PC = PC + 3
|
||||
```
|
||||
|
||||
**Special addresses:**
|
||||
- If `next == 0` (or `next == PC`): **HALT**
|
||||
- If `src == 0`: reads zero (constant source)
|
||||
- If `dst == IO_ADDR`: writes to I/O
|
||||
|
||||
---
|
||||
|
||||
## Memory Map ($0000–$FFFF)
|
||||
|
||||
The SUBLEQ address space is 16-bit (64K words). The memory map is divided into regions:
|
||||
|
||||
| Address Range | Size | Function |
|
||||
|---------------|------|----------|
|
||||
| `$0000–$0FFF` | 4K words | Program + data (BRAM) |
|
||||
| `$1000–$7FFF` | 28K words | Extended data (if available) |
|
||||
| `$8000–$800F` | 16 words | Q16 LUT result registers |
|
||||
| `$8010` | 1 word | Voltage controller mode |
|
||||
| `$8011` | 1 word | Scale space parameter |
|
||||
| `$8020–$8025` | 6 words | HiGHS pivot registers |
|
||||
| `$FF00` | 1 word | UART TX data register |
|
||||
| `$FF01` | 1 word | UART TX status (bit 0 = busy) |
|
||||
| `$FF02` | 1 word | UART RX data register |
|
||||
| `$FF03` | 1 word | UART RX status (bit 0 = data available) |
|
||||
| `$FFF0` | 1 word | LED output (bits 0-5 = led[0:5]) |
|
||||
| `$FFF1` | 1 word | Button input (bit 0 = user_btn) |
|
||||
|
||||
---
|
||||
|
||||
## Q16 LUT ($8000–$8025)
|
||||
|
||||
The Q16 LUT is a hardware-accelerated fixed-point arithmetic unit. It operates on Q16.16 values (16-bit integer, 16-bit fraction; total 1.0 = 65536).
|
||||
|
||||
### Q16 Operations
|
||||
|
||||
Write operands to the LUT registers, then read the result:
|
||||
|
||||
| Address | Register | Function |
|
||||
|---------|----------|----------|
|
||||
| `$8000` | OP_A (lo) | Operand A, low word |
|
||||
| `$8001` | OP_A (hi) | Operand A, high word |
|
||||
| `$8002` | OP_B (lo) | Operand B, low word |
|
||||
| `$8003` | OP_B (hi) | Operand B, high word |
|
||||
| `$8004` | OPCODE | Operation selector (0-7) |
|
||||
| `$8008` | RESULT (lo) | Result, low word |
|
||||
| `$8009` | RESULT (hi) | Result, high word |
|
||||
|
||||
### Opcodes
|
||||
|
||||
| Code | Operation | Latency |
|
||||
|------|-----------|---------|
|
||||
| 0 | A + B | 2 cycles (74ns @ 27MHz) |
|
||||
| 1 | A - B | 2 cycles |
|
||||
| 2 | A × B | 2 cycles |
|
||||
| 3 | A ÷ B | 2 cycles |
|
||||
| 4 | √A | 2 cycles |
|
||||
| 5 | \|A\| | 2 cycles |
|
||||
| 6 | min(A, B) | 2 cycles |
|
||||
| 7 | max(A, B) | 2 cycles |
|
||||
|
||||
### Q16.16 Encoding
|
||||
|
||||
```
|
||||
value = integer_part × 65536 + fraction_part
|
||||
|
||||
Examples:
|
||||
1.0 = 65536 (0x00010000)
|
||||
0.5 = 32768 (0x00008000)
|
||||
3.14 = 205887 (0x000323D7)
|
||||
-1.0 = -65536 (0xFFFF0000)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Voltage Controller ($8010)
|
||||
|
||||
The voltage controller manages BRAM access modes:
|
||||
|
||||
| Mode | Value | Description |
|
||||
|------|-------|-------------|
|
||||
| STORE | 0 | Direct memory read/write |
|
||||
| COMPUTE | 1 | Q16 LUT computation mode |
|
||||
| APPROX | 2 | Approximate computation (fast) |
|
||||
| MORPHIC | 3 | Morphic field mode |
|
||||
|
||||
```subleq
|
||||
; Set voltage controller to COMPUTE mode
|
||||
; Write 1 to address $8010
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scale Space ($8011)
|
||||
|
||||
The scale space parameter controls Gaussian kernel selection:
|
||||
|
||||
| Value | σ (sigma) | Kernel Bank |
|
||||
|-------|-----------|-------------|
|
||||
| 0 | 0.25 | Bank 0 |
|
||||
| 1 | 0.50 | Bank 1 |
|
||||
| 2 | 0.75 | Bank 2 |
|
||||
| 3 | 1.00 | Bank 3 |
|
||||
|
||||
---
|
||||
|
||||
## HiGHS Pivot Registers ($8020–$8025)
|
||||
|
||||
3-stage simplex pipeline interface for hardware-accelerated LP solving:
|
||||
|
||||
| Address | Register | Function |
|
||||
|---------|----------|----------|
|
||||
| `$8020` | PIVOT_ROW | Row index |
|
||||
| `$8021` | PIVOT_COL | Column index |
|
||||
| `$8022` | PIVOT_VAL (lo) | Pivot value, low |
|
||||
| `$8023` | PIVOT_VAL (hi) | Pivot value, high |
|
||||
| `$8024` | PIVOT_CTRL | Control/status |
|
||||
| `$8025` | PIVOT_RESULT | Result/iteration count |
|
||||
|
||||
---
|
||||
|
||||
## Example Programs
|
||||
|
||||
### 1. Blink LED
|
||||
|
||||
Blink LED 0 in a loop:
|
||||
|
||||
```subleq
|
||||
; Program at address 0
|
||||
; Toggle LED 0 by XOR with 1
|
||||
|
||||
; mem[100] = 1 (constant)
|
||||
; mem[101] = LED address ($FFF0)
|
||||
; mem[102] = current LED state
|
||||
; mem[103] = 0 (zero constant)
|
||||
|
||||
; Instruction 0: sub 103 from 102, store in 102 (clear 102)
|
||||
addr 0: 103 102 3 ; mem[102] = mem[102] - mem[103] = 0
|
||||
|
||||
; Instruction 3: sub 103 from LED, store in LED (clear LED)
|
||||
addr 3: 103 2545 6 ; mem[$FFF0] = mem[$FFF0] - 0
|
||||
|
||||
; Instruction 6: sub 100 from LED, store in LED (set bit 0)
|
||||
addr 6: 100 2545 9 ; mem[$FFF0] = mem[$FFF0] - 1
|
||||
|
||||
; Instruction 9: delay loop
|
||||
addr 9: 104 104 12 ; mem[104] = mem[104] - 1
|
||||
addr 12: 104 104 0 ; if mem[104] <= 0, jump to 0 (restart)
|
||||
|
||||
; Data
|
||||
addr 100: 1 ; toggle mask
|
||||
addr 101: 0 ; unused
|
||||
addr 102: 0 ; LED state
|
||||
addr 103: 0 ; zero
|
||||
addr 104: 50000 ; delay counter
|
||||
```
|
||||
|
||||
**Assembled binary:**
|
||||
```
|
||||
0064 0066 0003
|
||||
0067 09F1 0006
|
||||
0064 09F1 0009
|
||||
0068 0068 000C
|
||||
0068 0068 0000
|
||||
0001 0000 0000 0000 C350
|
||||
```
|
||||
|
||||
### 2. Q16 Addition
|
||||
|
||||
Add two Q16.16 values using the hardware LUT:
|
||||
|
||||
```subleq
|
||||
; Write operands to Q16 LUT, read result
|
||||
|
||||
; mem[200] = operand A = 3.14 (Q16: 205887 = 0x000323D7)
|
||||
; mem[201] = operand B = 2.72 (Q16: 178258 = 0x0002B8F2)
|
||||
|
||||
; Write A low word to $8000
|
||||
addr 0: 200 32768 3 ; mem[$8000] = mem[200] (A low)
|
||||
|
||||
; Write A high word to $8001
|
||||
addr 3: 201 32769 6 ; mem[$8001] = 0 (A high)
|
||||
|
||||
; Write B low word to $8002
|
||||
addr 6: 202 32770 9 ; mem[$8002] = mem[202] (B low)
|
||||
|
||||
; Write B high word to $8003
|
||||
addr 9: 203 32771 12 ; mem[$8003] = 0 (B high)
|
||||
|
||||
; Set opcode to 0 (add)
|
||||
addr 12: 204 32772 15 ; mem[$8004] = 0
|
||||
|
||||
; Read result low from $8008
|
||||
addr 15: 204 32776 18 ; mem[$8008] -> read
|
||||
|
||||
; Store result to mem[210]
|
||||
addr 18: 32776 210 21 ; mem[210] = result low
|
||||
|
||||
; HALT
|
||||
addr 21: 0 0 0
|
||||
|
||||
; Data
|
||||
addr 200: 23D7 ; A low (3.14)
|
||||
addr 201: 0003 ; A high
|
||||
addr 202: B8F2 ; B low (2.72)
|
||||
addr 203: 0002 ; B high
|
||||
addr 204: 0000 ; opcode 0 (add)
|
||||
```
|
||||
|
||||
### 3. UART Send Character
|
||||
|
||||
Send 'A' (0x41) over UART:
|
||||
|
||||
```subleq
|
||||
; Wait for TX to be not busy, then send character
|
||||
|
||||
; mem[300] = 0x41 ('A')
|
||||
; mem[301] = 0 (zero)
|
||||
; mem[302] = UART TX status address ($FF01)
|
||||
; mem[303] = UART TX data address ($FF00)
|
||||
|
||||
; Check TX status (poll loop)
|
||||
addr 0: 302 304 3 ; mem[304] = mem[$FF01]
|
||||
addr 3: 304 304 6 ; mem[304] -= mem[304] (test if zero)
|
||||
addr 6: 304 304 9 ; if <= 0 (not busy), continue
|
||||
addr 9: 301 304 0 ; else, reset and retry
|
||||
|
||||
; Send character
|
||||
addr 12: 300 303 15 ; mem[$FF00] = 0x41
|
||||
|
||||
; HALT
|
||||
addr 15: 0 0 0
|
||||
|
||||
; Data
|
||||
addr 300: 0041 ; 'A'
|
||||
addr 301: 0000 ; zero
|
||||
addr 302: FF01 ; UART TX status
|
||||
addr 303: FF00 ; UART TX data
|
||||
addr 304: 0000 ; temp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Loading Programs via UART
|
||||
|
||||
### Using Python
|
||||
|
||||
```python
|
||||
import serial
|
||||
import struct
|
||||
|
||||
# Load assembled program (array of 16-bit words)
|
||||
program = [0x0064, 0x0066, 0x0003, ...] # assembled instructions
|
||||
|
||||
# Connect to FPGA UART
|
||||
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=1)
|
||||
|
||||
# Send program: each word as 2 bytes, big-endian
|
||||
for word in program:
|
||||
ser.write(struct.pack('>H', word))
|
||||
|
||||
ser.close()
|
||||
```
|
||||
|
||||
### Using openFPGALoader (SRAM load)
|
||||
|
||||
For quick iteration (non-persistent):
|
||||
|
||||
```bash
|
||||
# Load bitstream to SRAM (lost on power cycle)
|
||||
openFPGALoader -b tangnano9k --sram research_stack_top.fs
|
||||
|
||||
# Load to flash (persistent)
|
||||
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Reading Results
|
||||
|
||||
### Via UART
|
||||
|
||||
```python
|
||||
import serial
|
||||
|
||||
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=5)
|
||||
|
||||
# Read result bytes
|
||||
data = ser.read(2) # 1 word = 2 bytes
|
||||
result = struct.unpack('>H', data)[0]
|
||||
print(f"Result: {result} (0x{result:04X})")
|
||||
|
||||
ser.close()
|
||||
```
|
||||
|
||||
### Via LEDs
|
||||
|
||||
Read the 6 LEDs (pins 10-16) as a 6-bit value from `led[0:5]`.
|
||||
- LED 0 = bit 0 (rightmost)
|
||||
|
||||
### Via Memory Dump
|
||||
|
||||
After HALT, the UART TX beacon outputs the full memory contents.
|
||||
Connect a serial terminal and observe the dump.
|
||||
|
||||
---
|
||||
|
||||
## Build Toolchain
|
||||
|
||||
```bash
|
||||
# Synthesis
|
||||
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||
|
||||
# Simulation
|
||||
cd /tmp/fpga_sim_full && ./obj_dir/sim_top
|
||||
|
||||
# Flash
|
||||
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||
```
|
||||
|
||||
### Tool Versions
|
||||
|
||||
| Tool | Version |
|
||||
|------|---------|
|
||||
| Yosys | 0.64 |
|
||||
| nextpnr-himbaechel | 0.10-75 |
|
||||
| gowin_pack | latest |
|
||||
| Verilator | 5.048 |
|
||||
| openFPGALoader | latest |
|
||||
|
||||
---
|
||||
|
||||
## Timing
|
||||
|
||||
- **Clock:** 27 MHz (37.04 ns period)
|
||||
- **Achieved Fmax:** 195.92 MHz (7.2× margin)
|
||||
- **Q16 LUT latency:** 2 cycles (74 ns)
|
||||
- **MAX_CYCLES:** 1,000,000
|
||||
399
6-Documentation/RUNBOOK.md
Normal file
399
6-Documentation/RUNBOOK.md
Normal file
|
|
@ -0,0 +1,399 @@
|
|||
# Ops Runbook — Research Stack
|
||||
|
||||
**Last updated:** 2026-05-29
|
||||
|
||||
Quick-reference operational procedures for k3s, FPGA, Tailscale, GPU, and DNS.
|
||||
|
||||
---
|
||||
|
||||
## k3s Cluster
|
||||
|
||||
### Start / Stop
|
||||
|
||||
```bash
|
||||
# Set kubeconfig
|
||||
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
|
||||
|
||||
# Check cluster status
|
||||
kubectl get nodes
|
||||
kubectl get pods -A
|
||||
```
|
||||
|
||||
**Control plane (nixos):**
|
||||
```bash
|
||||
# Restart k3s server
|
||||
sudo systemctl restart k3s
|
||||
|
||||
# Check k3s server health
|
||||
sudo systemctl status k3s
|
||||
sudo journalctl -u k3s -f --lines=50
|
||||
```
|
||||
|
||||
### Check Health
|
||||
|
||||
```bash
|
||||
# All nodes ready
|
||||
kubectl get nodes -o wide
|
||||
|
||||
# All pods running (any namespace)
|
||||
kubectl get pods -A --field-selector='status.phase!=Running,status.phase!=Succeeded'
|
||||
|
||||
# Specific namespace
|
||||
kubectl get pods -n services
|
||||
kubectl get pods -n media
|
||||
kubectl get pods -n monitoring
|
||||
kubectl get pods -n ai-models
|
||||
```
|
||||
|
||||
### Restart Pods
|
||||
|
||||
```bash
|
||||
# Restart a deployment (rolling restart)
|
||||
kubectl rollout restart deployment/<name> -n <namespace>
|
||||
|
||||
# Examples
|
||||
kubectl rollout restart deployment/homer -n services
|
||||
kubectl rollout restart deployment/ollama -n ai-models
|
||||
kubectl rollout restart deployment/cluster-dashboard -n monitoring
|
||||
|
||||
# Force delete stuck pod
|
||||
kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# Pod logs
|
||||
kubectl logs <pod-name> -n <namespace> --tail=100 -f
|
||||
|
||||
# Previous container (if crashed)
|
||||
kubectl logs <pod-name> -n <namespace> --previous
|
||||
|
||||
# All pods matching label
|
||||
kubectl logs -l app=<label> -n <namespace> --tail=50
|
||||
```
|
||||
|
||||
### Common Pods
|
||||
|
||||
| Namespace | Service | Deployment Name |
|
||||
|-----------|---------|-----------------|
|
||||
| `services` | Homer | `homer` |
|
||||
| `services` | Hermes | `hermes` |
|
||||
| `services` | Actual Budget | `actual-budget` |
|
||||
| `services` | Uptime Kuma | `uptime-kuma` |
|
||||
| `services` | Vaultwarden | `vaultwarden` |
|
||||
| `services` | Authentik | `authentik` |
|
||||
| `services` | Credential Server | `credential-server` |
|
||||
| `services` | Registry API | `registry-api` |
|
||||
| `services` | Jobs API | `jobs-api` |
|
||||
| `services` | Blobs API | `blobs-api` |
|
||||
| `ai-models` | Ollama | `ollama` |
|
||||
| `monitoring` | Cluster Dashboard | `cluster-dashboard` |
|
||||
| `media` | Jellyfin | `jellyfin` |
|
||||
| `research` | AlphaProof | `alphaproof` |
|
||||
|
||||
---
|
||||
|
||||
## FPGA — Tang Nano 9K
|
||||
|
||||
### Flash Bitstream
|
||||
|
||||
```bash
|
||||
# Build
|
||||
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||
|
||||
# Flash via USB-JTAG
|
||||
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||
|
||||
# Verify flash
|
||||
openFPGALoader -b tangnano9k --verify research_stack_top.fs
|
||||
```
|
||||
|
||||
### Verify LEDs
|
||||
|
||||
After power-on, `led[0:5]` (pins 10-16) should show SUBLEQ state.
|
||||
- **All off:** CPU halted or not clocked
|
||||
- **Blinking:** CPU running (heartbeat)
|
||||
- **LED 0 solid on:** CPU halted (trap)
|
||||
|
||||
### Debug UART
|
||||
|
||||
```bash
|
||||
# Connect to UART TX (pin 17) via USB-serial adapter
|
||||
# Baud rate: 115384 (27MHz / 234)
|
||||
picocom -b 115384 /dev/ttyUSB0
|
||||
|
||||
# Or with screen
|
||||
screen /dev/ttyUSB0 115384
|
||||
```
|
||||
|
||||
### Run Simulation
|
||||
|
||||
```bash
|
||||
cd /tmp/fpga_sim_full && ./obj_dir/sim_top
|
||||
```
|
||||
|
||||
### Pin Reference
|
||||
|
||||
| Pin | Signal | Direction | Notes |
|
||||
|-----|--------|-----------|-------|
|
||||
| 52 | clk | input | 27 MHz oscillator |
|
||||
| 4 | rst_n | input | Active-low reset (pull-up) |
|
||||
| 3 | user_btn | input | Active-low (pull-up) |
|
||||
| 10-16 | led[0:5] | output | LVCMOS18 |
|
||||
| 17 | uart_tx | output | 115384 baud |
|
||||
| 18 | uart_rx | input | Pull-up |
|
||||
|
||||
### FPGA Not Responding
|
||||
|
||||
1. Check USB connection to Tang Nano 9K
|
||||
2. Verify `openFPGALoader` sees the device: `openFPGALoader --detect`
|
||||
3. Re-flash bitstream: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
||||
4. Check clock: pin 52 should have 27 MHz (oscilloscope)
|
||||
5. Assert reset: pull pin 4 low momentarily, then release
|
||||
6. Check power: board should draw ~100mA from USB
|
||||
|
||||
---
|
||||
|
||||
## Tailscale
|
||||
|
||||
### Check Status
|
||||
|
||||
```bash
|
||||
# Show tailnet status
|
||||
tailscale status
|
||||
|
||||
# Show current node IP
|
||||
tailscale ip
|
||||
|
||||
# Check connectivity to other nodes
|
||||
tailscale ping qfox-1
|
||||
tailscale ping 361395-1
|
||||
tailscale ping racknerd-510bd9c
|
||||
```
|
||||
|
||||
### Restart Funnel
|
||||
|
||||
The Funnel runs on `361395-1` and routes to Traefik NodePort 30080:
|
||||
|
||||
```bash
|
||||
# On 361395-1 (edge node):
|
||||
tailscale funnel 8080 off
|
||||
tailscale funnel 8080
|
||||
|
||||
# Verify funnel URL
|
||||
tailscale funnel status
|
||||
```
|
||||
|
||||
### Debug Relay
|
||||
|
||||
```bash
|
||||
# Check if relay is in use
|
||||
tailscale status | grep relay
|
||||
|
||||
# Force direct connections (disable relay)
|
||||
tailscale set --direct-peers-only=true
|
||||
|
||||
# Reset to default
|
||||
tailscale set --direct-peers-only=false
|
||||
```
|
||||
|
||||
### Re-authenticate
|
||||
|
||||
```bash
|
||||
# On any node:
|
||||
tailscale up --authkey=<TS_AUTH_KEY>
|
||||
|
||||
# Or interactive login
|
||||
tailscale up
|
||||
```
|
||||
|
||||
### Node IPs
|
||||
|
||||
| Node | Tailscale IP | Role |
|
||||
|------|-------------|------|
|
||||
| nixos | 100.102.173.61 | Control plane |
|
||||
| qfox-1 | 100.88.57.96 | GPU worker |
|
||||
| 361395-1 | 100.110.163.82 | Edge/Funnel |
|
||||
| racknerd | 100.80.39.40 | Edge worker |
|
||||
| steamdeck | 100.85.244.73 | Worker |
|
||||
|
||||
---
|
||||
|
||||
## GPU — QFox (RTX 4070)
|
||||
|
||||
### Check nvidia-smi
|
||||
|
||||
```bash
|
||||
# GPU status
|
||||
nvidia-smi
|
||||
|
||||
# Watch GPU utilization
|
||||
nvidia-smi -l 1
|
||||
|
||||
# Check driver version
|
||||
nvidia-smi --query-gpu=driver_version --format=csv,noheader
|
||||
# Expected: 610.43
|
||||
```
|
||||
|
||||
### Restart Ollama
|
||||
|
||||
```bash
|
||||
# If running as k3s pod
|
||||
kubectl rollout restart deployment/ollama -n ai-models
|
||||
kubectl logs -l app=ollama -n ai-models --tail=20 -f
|
||||
|
||||
# If running as systemd service on qfox-1
|
||||
sudo systemctl restart ollama
|
||||
sudo journalctl -u ollama -f --tail=20
|
||||
```
|
||||
|
||||
### Pull / Manage Models
|
||||
|
||||
```bash
|
||||
# List loaded models
|
||||
curl http://100.88.57.96:11434/api/tags
|
||||
|
||||
# Pull a model
|
||||
curl http://100.88.57.96:11434/api/pull -d '{"name": "deepseek-coder-v2:16b"}'
|
||||
|
||||
# Test inference
|
||||
curl http://100.88.57.96:11434/api/generate \
|
||||
-d '{"model": "deepseek-coder-v2:16b", "prompt": "Hello", "stream": false}'
|
||||
```
|
||||
|
||||
### Ollama via NodePort
|
||||
|
||||
```bash
|
||||
# Access from cluster
|
||||
curl http://<any-node-ip>:31434/api/tags
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## DNS & TLS
|
||||
|
||||
### Check LE Certificates
|
||||
|
||||
```bash
|
||||
# Check cert expiry
|
||||
echo | openssl s_client -connect researchstack.info:443 -servername researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
||||
|
||||
# Wildcard cert
|
||||
echo | openssl s_client -connect registry.researchstack.info:443 -servername registry.researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
||||
```
|
||||
|
||||
**Current cert valid until:** 2026-08-18
|
||||
|
||||
### Renew Certificates
|
||||
|
||||
Caddy auto-renews via Porkbun DNS-01. If manual renewal is needed:
|
||||
|
||||
```bash
|
||||
# Restart Caddy to trigger renewal check
|
||||
kubectl rollout restart deployment/caddy -n services
|
||||
|
||||
# Check Caddy logs for renewal
|
||||
kubectl logs -l app=caddy -n services --tail=50 | grep -i renew
|
||||
```
|
||||
|
||||
### Check Porkbun DNS
|
||||
|
||||
```bash
|
||||
# Verify A record
|
||||
dig researchstack.info +short
|
||||
dig auth.researchstack.info +short
|
||||
dig registry.researchstack.info +short
|
||||
|
||||
# Check API key (set PORKBUN_API_KEY env var)
|
||||
curl -X POST https://api.porkbun.com/api/json/v3/ping \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"apikey": "'$PORKBUN_API_KEY'"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Failures
|
||||
|
||||
### Node Down
|
||||
|
||||
```bash
|
||||
# Identify down node
|
||||
kubectl get nodes
|
||||
|
||||
# Check node conditions
|
||||
kubectl describe node <node-name>
|
||||
|
||||
# On the node itself:
|
||||
sudo systemctl status k3s-agent # worker nodes
|
||||
sudo systemctl status k3s # control plane
|
||||
|
||||
# Rejoin cluster if needed (worker):
|
||||
sudo k3s agent --server https://100.102.173.61:6443 --token <TOKEN>
|
||||
```
|
||||
|
||||
### Pod CrashLoopBackOff
|
||||
|
||||
```bash
|
||||
# Check events
|
||||
kubectl describe pod <pod-name> -n <namespace>
|
||||
|
||||
# Check logs (including previous crash)
|
||||
kubectl logs <pod-name> -n <namespace> --previous --tail=100
|
||||
|
||||
# Common fixes:
|
||||
# - ConfigMap/Secret missing: kubectl get configmap -n <ns>; kubectl get secret -n <ns>
|
||||
# - Image pull error: check image tag and registry access
|
||||
# - Resource limits: kubectl top pod -n <namespace>
|
||||
# - OOMKill: increase memory limit in deployment spec
|
||||
```
|
||||
|
||||
### FPGA Not Responding
|
||||
|
||||
1. Unplug and replug USB cable
|
||||
2. Check `ls /dev/ttyUSB*` — device should appear
|
||||
3. Re-flash: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
||||
4. If JTAG fails: try holding reset (pin 4 low) while plugging in
|
||||
5. Check power LED on Tang Nano 9K board
|
||||
6. Try different USB port / cable
|
||||
|
||||
### Tailscale Tunnel Down
|
||||
|
||||
```bash
|
||||
# Check daemon
|
||||
sudo systemctl status tailscaled
|
||||
|
||||
# Restart
|
||||
sudo systemctl restart tailscaled
|
||||
tailscale up
|
||||
|
||||
# Check if key expired
|
||||
tailscale status | grep -i expir
|
||||
```
|
||||
|
||||
### DNS Resolution Failing
|
||||
|
||||
```bash
|
||||
# Check Caddy pod
|
||||
kubectl logs -l app=caddy -n services --tail=50
|
||||
|
||||
# Test DNS from inside cluster
|
||||
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup researchstack.info
|
||||
|
||||
# Check Porkbun API key
|
||||
kubectl get secret porkbun-credentials -n services -o jsonpath='{.data.api-key}' | base64 -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Escalation Matrix
|
||||
|
||||
| Issue | First Response | Escalation |
|
||||
|-------|---------------|------------|
|
||||
| Pod down | `kubectl rollout restart` | Check node, check resource limits |
|
||||
| Node down | SSH to node, check `systemctl` | Reboot, check hardware |
|
||||
| FPGA unresponsive | Re-flash bitstream | Check USB, try different board |
|
||||
| Tailscale tunnel | `tailscale up` | Check auth key, restart daemon |
|
||||
| DNS/cert | Restart Caddy | Check Porkbun API, check ingress |
|
||||
| GPU errors | Check `nvidia-smi` | Restart driver, check PCIe seating |
|
||||
| OOM on Ollama | Restart pod | Reduce context length, switch model |
|
||||
Loading…
Add table
Reference in a new issue