mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-31 03:05:21 +00:00
docs: runbook, FPGA programming guide, disaster recovery, API docs
- RUNBOOK.md: k3s/FPGA/Tailscale/GPU/DNS ops procedures - FPGA_PROGRAMMING_GUIDE.md: SUBLEQ format, memory map, 3 examples - DISASTER_RECOVERY.md: backup/restore for all components - API_DOCS.md: dashboard, credential, registry, jobs, blobs APIs
This commit is contained in:
Brandon Schneider
parent
251ea9d9bf
commit
7a6df586b2
4 changed files with 1570 additions and 0 deletions
465
6-Documentation/API_DOCS.md
Normal file
465
6-Documentation/API_DOCS.md
Normal file
|
|
@ -0,0 +1,465 @@
|
||||||
|
# API Documentation — Research Stack Services
|
||||||
|
|
||||||
|
**Last updated:** 2026-05-29
|
||||||
|
**Base URL:** `https://researchstack.info`
|
||||||
|
**Auth:** Authentik OIDC (SSO) or Bearer token (API endpoints)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
### OIDC (SSO) — Web Endpoints
|
||||||
|
|
||||||
|
Web-facing endpoints use Authentik OIDC. Access via browser redirects to `auth.researchstack.info`.
|
||||||
|
|
||||||
|
### Token Auth — API Endpoints
|
||||||
|
|
||||||
|
API endpoints (`/api/*`) use Bearer token authentication:
|
||||||
|
|
||||||
|
```
|
||||||
|
Authorization: Bearer <token>
|
||||||
|
```
|
||||||
|
|
||||||
|
Tokens are managed by the Credential Server.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cluster Dashboard
|
||||||
|
|
||||||
|
**Namespace:** `monitoring`
|
||||||
|
**Internal URL:** `http://cluster-dashboard:8787`
|
||||||
|
**External URL:** `https://researchstack.info/server/dash/` (via Homarr)
|
||||||
|
**Tech:** FastAPI + Vite
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
#### `GET /`
|
||||||
|
|
||||||
|
Dashboard UI (Vite SPA).
|
||||||
|
|
||||||
|
#### `GET /api/status`
|
||||||
|
|
||||||
|
Cluster health summary.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"nodes": [
|
||||||
|
{
|
||||||
|
"name": "nixos-laptop",
|
||||||
|
"status": "Ready",
|
||||||
|
"ip": "100.102.173.61",
|
||||||
|
"roles": "control-plane",
|
||||||
|
"version": "v1.35.4+k3s1"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"pods": {
|
||||||
|
"total": 24,
|
||||||
|
"running": 23,
|
||||||
|
"pending": 0,
|
||||||
|
"failed": 1
|
||||||
|
},
|
||||||
|
"namespaces": ["services", "media", "monitoring", "ai-models", "edge", "research", "mail"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `GET /api/metrics`
|
||||||
|
|
||||||
|
Prometheus-compatible metrics.
|
||||||
|
|
||||||
|
#### `WebSocket /ws/live`
|
||||||
|
|
||||||
|
Real-time cluster events stream.
|
||||||
|
|
||||||
|
**Message format:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "pod_event",
|
||||||
|
"namespace": "services",
|
||||||
|
"pod": "homer-abc123",
|
||||||
|
"status": "Running",
|
||||||
|
"timestamp": "2026-05-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Credential Server
|
||||||
|
|
||||||
|
**Namespace:** `services`
|
||||||
|
**Internal URL:** `http://credential-server:8080`
|
||||||
|
**External URL:** `https://researchstack.info/api/cred/`
|
||||||
|
**Auth:** Bearer token
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
#### `GET /api/cred/health`
|
||||||
|
|
||||||
|
Health check.
|
||||||
|
|
||||||
|
**Response:** `200 OK`
|
||||||
|
```json
|
||||||
|
{"status": "ok"}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `GET /api/cred/tokens`
|
||||||
|
|
||||||
|
List available tokens (requires admin token).
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"tokens": [
|
||||||
|
{"name": "registry", "scope": "read,write", "expires": "2026-12-31"},
|
||||||
|
{"name": "jobs", "scope": "read,write,execute", "expires": "2026-12-31"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `POST /api/cred/issue`
|
||||||
|
|
||||||
|
Issue a new token.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "service-name",
|
||||||
|
"scope": "read,write",
|
||||||
|
"ttl": "30d"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"token": "rs_tk_...",
|
||||||
|
"expires": "2026-06-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `POST /api/cred/validate`
|
||||||
|
|
||||||
|
Validate a token.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{"token": "rs_tk_..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"valid": true,
|
||||||
|
"name": "registry",
|
||||||
|
"scope": "read,write"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Registry API
|
||||||
|
|
||||||
|
**Namespace:** `services`
|
||||||
|
**Internal URL:** `http://registry-api:8080`
|
||||||
|
**External URL:** `https://researchstack.info/api/registry/`
|
||||||
|
**Auth:** Bearer token
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
#### `GET /api/registry/health`
|
||||||
|
|
||||||
|
Health check.
|
||||||
|
|
||||||
|
#### `GET /api/registry/artifacts`
|
||||||
|
|
||||||
|
List all registered artifacts.
|
||||||
|
|
||||||
|
**Query Parameters:**
|
||||||
|
- `type` — Filter by artifact type (e.g., `bitstream`, `lean-build`, `receipt`)
|
||||||
|
- `since` — ISO 8601 timestamp
|
||||||
|
- `limit` — Max results (default: 50)
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"artifacts": [
|
||||||
|
{
|
||||||
|
"id": "art_abc123",
|
||||||
|
"name": "research_stack_top.fs",
|
||||||
|
"type": "bitstream",
|
||||||
|
"size": 184320,
|
||||||
|
"sha256": "...",
|
||||||
|
"created": "2026-05-29T10:00:00Z",
|
||||||
|
"tags": ["fpga", "production"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total": 42
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `POST /api/registry/artifacts`
|
||||||
|
|
||||||
|
Register a new artifact.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "research_stack_top.fs",
|
||||||
|
"type": "bitstream",
|
||||||
|
"sha256": "...",
|
||||||
|
"blob_ref": "blob_xyz",
|
||||||
|
"tags": ["fpga"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `GET /api/registry/artifacts/{id}`
|
||||||
|
|
||||||
|
Get artifact by ID.
|
||||||
|
|
||||||
|
#### `PUT /api/registry/artifacts/{id}`
|
||||||
|
|
||||||
|
Update artifact metadata.
|
||||||
|
|
||||||
|
#### `DELETE /api/registry/artifacts/{id}`
|
||||||
|
|
||||||
|
Remove artifact registration (does not delete blob).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Jobs API
|
||||||
|
|
||||||
|
**Namespace:** `services`
|
||||||
|
**Internal URL:** `http://jobs-api:8080`
|
||||||
|
**External URL:** `https://researchstack.info/api/jobs/`
|
||||||
|
**Auth:** Bearer token
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
#### `GET /api/jobs/health`
|
||||||
|
|
||||||
|
Health check.
|
||||||
|
|
||||||
|
#### `GET /api/jobs`
|
||||||
|
|
||||||
|
List jobs.
|
||||||
|
|
||||||
|
**Query Parameters:**
|
||||||
|
- `status` — Filter: `pending`, `running`, `completed`, `failed`
|
||||||
|
- `type` — Filter by job type
|
||||||
|
- `limit` — Max results (default: 20)
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"jobs": [
|
||||||
|
{
|
||||||
|
"id": "job_abc123",
|
||||||
|
"type": "lean-build",
|
||||||
|
"status": "completed",
|
||||||
|
"created": "2026-05-29T10:00:00Z",
|
||||||
|
"started": "2026-05-29T10:00:05Z",
|
||||||
|
"finished": "2026-05-29T10:05:30Z",
|
||||||
|
"result": {
|
||||||
|
"jobs": 3572,
|
||||||
|
"errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total": 156
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `POST /api/jobs`
|
||||||
|
|
||||||
|
Create a new job.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "lean-build",
|
||||||
|
"params": {
|
||||||
|
"target": "Semantics",
|
||||||
|
"branch": "main"
|
||||||
|
},
|
||||||
|
"priority": "normal"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "job_def456",
|
||||||
|
"status": "pending",
|
||||||
|
"created": "2026-05-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `GET /api/jobs/{id}`
|
||||||
|
|
||||||
|
Get job status and result.
|
||||||
|
|
||||||
|
#### `POST /api/jobs/{id}/cancel`
|
||||||
|
|
||||||
|
Cancel a pending or running job.
|
||||||
|
|
||||||
|
#### `GET /api/jobs/{id}/logs`
|
||||||
|
|
||||||
|
Stream job logs (Server-Sent Events).
|
||||||
|
|
||||||
|
**Response:** `text/event-stream`
|
||||||
|
```
|
||||||
|
data: {"line": "Building Semantics...", "ts": "2026-05-29T12:00:01Z"}
|
||||||
|
data: {"line": "3572 jobs, 0 errors", "ts": "2026-05-29T12:05:30Z"}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Blobs API
|
||||||
|
|
||||||
|
**Namespace:** `services`
|
||||||
|
**Internal URL:** `http://blobs-api:8080`
|
||||||
|
**External URL:** `https://researchstack.info/api/blobs/`
|
||||||
|
**Auth:** Bearer token
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
#### `GET /api/blobs/health`
|
||||||
|
|
||||||
|
Health check.
|
||||||
|
|
||||||
|
#### `POST /api/blobs`
|
||||||
|
|
||||||
|
Upload a blob.
|
||||||
|
|
||||||
|
**Request:** `multipart/form-data`
|
||||||
|
- `file` — Binary file content
|
||||||
|
- `sha256` — Expected SHA-256 hash (verification)
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "blob_xyz789",
|
||||||
|
"size": 184320,
|
||||||
|
"sha256": "...",
|
||||||
|
"created": "2026-05-29T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `GET /api/blobs/{id}`
|
||||||
|
|
||||||
|
Download a blob.
|
||||||
|
|
||||||
|
**Response:** `application/octet-stream` with blob content.
|
||||||
|
|
||||||
|
#### `HEAD /api/blobs/{id}`
|
||||||
|
|
||||||
|
Check blob existence.
|
||||||
|
|
||||||
|
**Response:** `200 OK` (exists) or `404 Not Found`
|
||||||
|
|
||||||
|
#### `DELETE /api/blobs/{id}`
|
||||||
|
|
||||||
|
Delete a blob.
|
||||||
|
|
||||||
|
#### `GET /api/blobs`
|
||||||
|
|
||||||
|
List blobs.
|
||||||
|
|
||||||
|
**Query Parameters:**
|
||||||
|
- `limit` — Max results
|
||||||
|
- `offset` — Pagination offset
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Authentik — OIDC Configuration
|
||||||
|
|
||||||
|
**URL:** `https://auth.researchstack.info`
|
||||||
|
**Protocol:** OpenID Connect
|
||||||
|
|
||||||
|
### Provider Configuration
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Authorization URL | `https://auth.researchstack.info/application/o/authorize/` |
|
||||||
|
| Token URL | `https://auth.researchstack.info/application/o/token/` |
|
||||||
|
| UserInfo URL | `https://auth.researchstack.info/application/o/userinfo/` |
|
||||||
|
| JWKS URL | `https://auth.researchstack.info/application/o/research-stack/jwks/` |
|
||||||
|
| Issuer | `https://auth.researchstack.info/application/o/research-stack/` |
|
||||||
|
|
||||||
|
### Scopes
|
||||||
|
|
||||||
|
| Scope | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| `openid` | Standard OIDC |
|
||||||
|
| `profile` | User profile (name, email) |
|
||||||
|
| `email` | Email address |
|
||||||
|
| `groups` | Group membership |
|
||||||
|
|
||||||
|
### Redirect URIs
|
||||||
|
|
||||||
|
```
|
||||||
|
https://researchstack.info/oidc/callback
|
||||||
|
https://researchstack.info/apps/chat/oidc/callback
|
||||||
|
https://researchstack.info/apps/budget/oidc/callback
|
||||||
|
```
|
||||||
|
|
||||||
|
### Application Setup
|
||||||
|
|
||||||
|
In Authentik admin:
|
||||||
|
|
||||||
|
1. **Applications** → Create → Name: `Research Stack`
|
||||||
|
2. **Providers** → Create → OAuth2/OpenID Provider
|
||||||
|
3. Set client type: `Confidential`
|
||||||
|
4. Set redirect URIs above
|
||||||
|
5. Assign to application
|
||||||
|
|
||||||
|
### Service Account Tokens
|
||||||
|
|
||||||
|
For API-to-API auth (no user interaction):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create service account in Authentik
|
||||||
|
# Admin → Directory → Users → Create
|
||||||
|
# Type: Service Account
|
||||||
|
|
||||||
|
# Get token
|
||||||
|
curl -X POST https://auth.researchstack.info/application/o/token/ \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=<service-account-id>" \
|
||||||
|
-d "client_secret=<service-account-secret>" \
|
||||||
|
-d "scope=openid"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Response Codes
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
|------|---------|
|
||||||
|
| 200 | Success |
|
||||||
|
| 201 | Created |
|
||||||
|
| 400 | Bad request (invalid parameters) |
|
||||||
|
| 401 | Unauthorized (missing or invalid token) |
|
||||||
|
| 403 | Forbidden (insufficient scope) |
|
||||||
|
| 404 | Not found |
|
||||||
|
| 409 | Conflict (duplicate resource) |
|
||||||
|
| 422 | Unprocessable entity (validation error) |
|
||||||
|
| 500 | Internal server error |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rate Limits
|
||||||
|
|
||||||
|
| Endpoint | Limit | Window |
|
||||||
|
|----------|-------|--------|
|
||||||
|
| `/api/cred/*` | 100 req | 1 minute |
|
||||||
|
| `/api/registry/*` | 200 req | 1 minute |
|
||||||
|
| `/api/jobs/*` | 50 req | 1 minute |
|
||||||
|
| `/api/blobs/*` | 100 req | 1 minute |
|
||||||
|
|
||||||
|
Rate limit headers:
|
||||||
|
```
|
||||||
|
X-RateLimit-Limit: 100
|
||||||
|
X-RateLimit-Remaining: 95
|
||||||
|
X-RateLimit-Reset: 1685366400
|
||||||
|
```
|
||||||
343
6-Documentation/DISASTER_RECOVERY.md
Normal file
343
6-Documentation/DISASTER_RECOVERY.md
Normal file
|
|
@ -0,0 +1,343 @@
|
||||||
|
# Disaster Recovery — Research Stack
|
||||||
|
|
||||||
|
**Last updated:** 2026-05-29
|
||||||
|
|
||||||
|
Procedures for backing up and restoring all Research Stack components.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## k3s Cluster
|
||||||
|
|
||||||
|
### Backup etcd
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Snapshot etcd (on control plane node — nixos)
|
||||||
|
sudo k3s etcd-snapshot save \
|
||||||
|
--name researchstack-backup \
|
||||||
|
--dir /var/lib/rancher/k3s/server/db/snapshots
|
||||||
|
|
||||||
|
# List snapshots
|
||||||
|
sudo ls -la /var/lib/rancher/k3s/server/db/snapshots/
|
||||||
|
|
||||||
|
# Copy snapshot off-node
|
||||||
|
scp /var/lib/rancher/k3s/server/db/snapshots/researchstack-backup-* \
|
||||||
|
user@backup-host:/backups/k3s/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore from Snapshot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On control plane (nixos):
|
||||||
|
sudo systemctl stop k3s
|
||||||
|
|
||||||
|
# Restore
|
||||||
|
sudo k3s server \
|
||||||
|
--cluster-reset \
|
||||||
|
--cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/researchstack-backup-<timestamp>
|
||||||
|
|
||||||
|
# Restart
|
||||||
|
sudo systemctl start k3s
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl get pods -A
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Manifests
|
||||||
|
|
||||||
|
All Kubernetes manifests are in the git repo. The source of truth is:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Automated Backup Schedule
|
||||||
|
|
||||||
|
Create a cron job on nixos:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# /etc/cron.d/k3s-backup
|
||||||
|
0 3 * * * root k3s etcd-snapshot save --name researchstack-backup --dir /var/lib/rancher/k3s/server/db/snapshots && \
|
||||||
|
find /var/lib/rancher/k3s/server/db/snapshots/ -mtime +7 -delete
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FPGA — Tang Nano 9K
|
||||||
|
|
||||||
|
### Backup Bitstream
|
||||||
|
|
||||||
|
The bitstream source and compiled output are in the git repo:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
|
||||||
|
# Source
|
||||||
|
ls 4-Infrastructure/hardware/*.v
|
||||||
|
|
||||||
|
# Compiled bitstream
|
||||||
|
ls 4-Infrastructure/hardware/research_stack_top.fs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Re-flash from Git
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
|
||||||
|
# Rebuild if needed
|
||||||
|
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||||
|
|
||||||
|
# Flash
|
||||||
|
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
openFPGALoader -b tangnano9k --verify research_stack_top.fs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup SRAM Contents
|
||||||
|
|
||||||
|
SRAM contents are volatile (lost on power cycle). To preserve runtime state:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Dump memory via UART before power-off
|
||||||
|
import serial, struct
|
||||||
|
|
||||||
|
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=10)
|
||||||
|
|
||||||
|
# Trigger memory dump (send HALT with dump flag)
|
||||||
|
# ... protocol-specific ...
|
||||||
|
|
||||||
|
# Read and save
|
||||||
|
with open('fpga_memory_dump.bin', 'wb') as f:
|
||||||
|
data = ser.read(8192) # 4K words = 8K bytes
|
||||||
|
f.write(data)
|
||||||
|
|
||||||
|
ser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tailscale
|
||||||
|
|
||||||
|
### Re-authenticate Node
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On the node that lost auth:
|
||||||
|
sudo systemctl restart tailscaled
|
||||||
|
tailscale up --authkey=<TS_AUTH_KEY>
|
||||||
|
|
||||||
|
# Or interactive
|
||||||
|
tailscale up
|
||||||
|
```
|
||||||
|
|
||||||
|
### Re-join Tailnet
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check current status
|
||||||
|
tailscale status
|
||||||
|
|
||||||
|
# If node is missing from tailnet:
|
||||||
|
# 1. Generate new auth key at https://login.tailscale.com/admin/settings/keys
|
||||||
|
# 2. On the node:
|
||||||
|
sudo tailscale up --authkey=tskey-auth-<key>
|
||||||
|
|
||||||
|
# Verify connectivity
|
||||||
|
tailscale ping nixos-laptop
|
||||||
|
tailscale ping qfox-1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Tailscale State
|
||||||
|
|
||||||
|
Tailscale state is stored at `/var/lib/tailscale/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup state directory
|
||||||
|
sudo tar czf /backups/tailscale-state-$(date +%Y%m%d).tar.gz \
|
||||||
|
/var/lib/tailscale/
|
||||||
|
|
||||||
|
# Restore
|
||||||
|
sudo systemctl stop tailscaled
|
||||||
|
sudo tar xzf /backups/tailscale-state-*.tar.gz -C /
|
||||||
|
sudo systemctl start tailscaled
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Git Repository
|
||||||
|
|
||||||
|
### Backup
|
||||||
|
|
||||||
|
The canonical backup is the remote origin. Ensure all changes are pushed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
|
||||||
|
# Check for unpushed commits
|
||||||
|
git log --oneline origin/main..HEAD
|
||||||
|
|
||||||
|
# Push everything
|
||||||
|
git push origin main --tags
|
||||||
|
|
||||||
|
# Verify clean state
|
||||||
|
git status --branch --short --untracked-files=all
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore from Remote
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone fresh
|
||||||
|
cd ~
|
||||||
|
git clone <remote-url> "Research Stack"
|
||||||
|
|
||||||
|
# Or reset to remote state
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
git fetch origin
|
||||||
|
git reset --hard origin/main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Remotes
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List remotes
|
||||||
|
cd ~/Research\ Stack
|
||||||
|
git remote -v
|
||||||
|
|
||||||
|
# Add backup remote
|
||||||
|
git remote add backup <backup-url>
|
||||||
|
git push backup main --tags
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Secrets (sops-nix / age)
|
||||||
|
|
||||||
|
### Backup age Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# age key location (NixOS)
|
||||||
|
cat /etc/age/keys.txt
|
||||||
|
|
||||||
|
# Backup
|
||||||
|
cp /etc/age/keys.txt /backups/age-keys-$(date +%Y%m%d).txt
|
||||||
|
chmod 600 /backups/age-keys-*.txt
|
||||||
|
|
||||||
|
# Or from home directory
|
||||||
|
cat ~/.config/sops/age/keys.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore age Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restore key file
|
||||||
|
cp /backups/age-keys-*.txt /etc/age/keys.txt
|
||||||
|
chmod 600 /etc/age/keys.txt
|
||||||
|
|
||||||
|
# Verify sops can decrypt
|
||||||
|
sops -d secrets.enc.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup sops Configuration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# The .sops.yaml config is in the repo
|
||||||
|
cat ~/Research\ Stack/.sops.yaml
|
||||||
|
|
||||||
|
# Encrypted secrets are also in the repo
|
||||||
|
find ~/Research\ Stack -name '*.enc.yaml' -o -name '*.enc.json'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Regenerate age Key (Last Resort)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate new key pair
|
||||||
|
age-keygen -o /etc/age/keys.txt
|
||||||
|
|
||||||
|
# Get public key
|
||||||
|
age-keygen -y /etc/age/keys.txt
|
||||||
|
|
||||||
|
# Re-encrypt all secrets with new key
|
||||||
|
# (requires old key or plaintext backup)
|
||||||
|
sops updatekeys secrets.enc.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## DNS — Porkbun
|
||||||
|
|
||||||
|
### Backup API Key
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From k8s secret
|
||||||
|
kubectl get secret porkbun-credentials -n services \
|
||||||
|
-o jsonpath='{.data.api-key}' | base64 -d
|
||||||
|
|
||||||
|
# Back up securely
|
||||||
|
echo "<api-key>" | age -r <backup-age-pubkey> > /backups/porkbun-api.age
|
||||||
|
|
||||||
|
# Or store in password manager
|
||||||
|
```
|
||||||
|
|
||||||
|
### Recover API Key
|
||||||
|
|
||||||
|
1. Log in to [Porkbun](https://porkbun.com)
|
||||||
|
2. Navigate to **Account → API Access**
|
||||||
|
3. Regenerate or copy existing API key
|
||||||
|
4. Update k8s secret:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl create secret generic porkbun-credentials \
|
||||||
|
-n services \
|
||||||
|
--from-literal=api-key=<new-key> \
|
||||||
|
--dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
|
||||||
|
# Restart Caddy to pick up new key
|
||||||
|
kubectl rollout restart deployment/caddy -n services
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify DNS Records
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig researchstack.info +short
|
||||||
|
dig auth.researchstack.info +short
|
||||||
|
dig registry.researchstack.info +short
|
||||||
|
|
||||||
|
# Test Porkbun API
|
||||||
|
curl -X POST https://api.porkbun.com/api/json/v3/ping \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"apikey": "<key>", "secretapikey": "<secret>"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recovery Priority Order
|
||||||
|
|
||||||
|
When restoring from a total failure, follow this order:
|
||||||
|
|
||||||
|
| Step | Component | Depends On |
|
||||||
|
|------|-----------|------------|
|
||||||
|
| 1 | **Tailscale** | Auth key (from password manager or backup) |
|
||||||
|
| 2 | **Git repo** | Network connectivity (Tailscale or direct) |
|
||||||
|
| 3 | **age keys** | Backup location |
|
||||||
|
| 4 | **k3s cluster** | etcd snapshot + age keys for secrets |
|
||||||
|
| 5 | **DNS/Porkbun** | API key (from backup or Porkbun account) |
|
||||||
|
| 6 | **FPGA** | Git repo (source + bitstream) |
|
||||||
|
| 7 | **Services** | k3s cluster + secrets + DNS |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Full System Checklist
|
||||||
|
|
||||||
|
After recovery, verify each component:
|
||||||
|
|
||||||
|
- [ ] Tailscale: `tailscale status` shows all 5 nodes
|
||||||
|
- [ ] k3s: `kubectl get nodes` shows all nodes Ready
|
||||||
|
- [ ] Pods: `kubectl get pods -A` — all Running/Succeeded
|
||||||
|
- [ ] DNS: `dig researchstack.info` resolves correctly
|
||||||
|
- [ ] TLS: cert valid (`openssl s_client`)
|
||||||
|
- [ ] Auth: `auth.researchstack.info` loads Authentik
|
||||||
|
- [ ] Funnel: `361395-1.tail4e7094.ts.net` responds
|
||||||
|
- [ ] Ollama: `curl http://100.88.57.96:31434/api/tags` returns models
|
||||||
|
- [ ] FPGA: `openFPGALoader --detect` sees Tang Nano 9K
|
||||||
|
- [ ] Dashboard: `https://researchstack.info` shows Homer
|
||||||
363
6-Documentation/FPGA_PROGRAMMING_GUIDE.md
Normal file
363
6-Documentation/FPGA_PROGRAMMING_GUIDE.md
Normal file
|
|
@ -0,0 +1,363 @@
|
||||||
|
# FPGA Programming Guide — SUBLEQ on the Blitter
|
||||||
|
|
||||||
|
**Last updated:** 2026-05-29
|
||||||
|
**Board:** Sipeed Tang Nano 9K (Gowin GW1NR-LV9)
|
||||||
|
**CPU:** Blitter6502OISC (One Instruction Set Computer — SUBLEQ)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SUBLEQ Instruction Format
|
||||||
|
|
||||||
|
The Blitter implements a **SUBLEQ** (Subtract and Branch if Less-than-or-Equal to Zero) CPU.
|
||||||
|
|
||||||
|
Each instruction is **3 words** (3 × 16-bit = 6 bytes):
|
||||||
|
|
||||||
|
```
|
||||||
|
[src] [dst] [next]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Semantics:**
|
||||||
|
```
|
||||||
|
mem[dst] = mem[dst] - mem[src]
|
||||||
|
if mem[dst] <= 0:
|
||||||
|
PC = next
|
||||||
|
else:
|
||||||
|
PC = PC + 3
|
||||||
|
```
|
||||||
|
|
||||||
|
**Special addresses:**
|
||||||
|
- If `next == 0` (or `next == PC`): **HALT**
|
||||||
|
- If `src == 0`: reads zero (constant source)
|
||||||
|
- If `dst == IO_ADDR`: writes to I/O
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Memory Map ($0000–$FFFF)
|
||||||
|
|
||||||
|
The SUBLEQ address space is 16-bit (64K words). The memory map is divided into regions:
|
||||||
|
|
||||||
|
| Address Range | Size | Function |
|
||||||
|
|---------------|------|----------|
|
||||||
|
| `$0000–$0FFF` | 4K words | Program + data (BRAM) |
|
||||||
|
| `$1000–$7FFF` | 28K words | Extended data (if available) |
|
||||||
|
| `$8000–$800F` | 16 words | Q16 LUT result registers |
|
||||||
|
| `$8010` | 1 word | Voltage controller mode |
|
||||||
|
| `$8011` | 1 word | Scale space parameter |
|
||||||
|
| `$8020–$8025` | 6 words | HiGHS pivot registers |
|
||||||
|
| `$FF00` | 1 word | UART TX data register |
|
||||||
|
| `$FF01` | 1 word | UART TX status (bit 0 = busy) |
|
||||||
|
| `$FF02` | 1 word | UART RX data register |
|
||||||
|
| `$FF03` | 1 word | UART RX status (bit 0 = data available) |
|
||||||
|
| `$FFF0` | 1 word | LED output (bits 0-5 = led[0:5]) |
|
||||||
|
| `$FFF1` | 1 word | Button input (bit 0 = user_btn) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q16 LUT ($8000–$8025)
|
||||||
|
|
||||||
|
The Q16 LUT is a hardware-accelerated fixed-point arithmetic unit. It operates on Q16.16 values (16-bit integer, 16-bit fraction; total 1.0 = 65536).
|
||||||
|
|
||||||
|
### Q16 Operations
|
||||||
|
|
||||||
|
Write operands to the LUT registers, then read the result:
|
||||||
|
|
||||||
|
| Address | Register | Function |
|
||||||
|
|---------|----------|----------|
|
||||||
|
| `$8000` | OP_A (lo) | Operand A, low word |
|
||||||
|
| `$8001` | OP_A (hi) | Operand A, high word |
|
||||||
|
| `$8002` | OP_B (lo) | Operand B, low word |
|
||||||
|
| `$8003` | OP_B (hi) | Operand B, high word |
|
||||||
|
| `$8004` | OPCODE | Operation selector (0-7) |
|
||||||
|
| `$8008` | RESULT (lo) | Result, low word |
|
||||||
|
| `$8009` | RESULT (hi) | Result, high word |
|
||||||
|
|
||||||
|
### Opcodes
|
||||||
|
|
||||||
|
| Code | Operation | Latency |
|
||||||
|
|------|-----------|---------|
|
||||||
|
| 0 | A + B | 2 cycles (74ns @ 27MHz) |
|
||||||
|
| 1 | A - B | 2 cycles |
|
||||||
|
| 2 | A × B | 2 cycles |
|
||||||
|
| 3 | A ÷ B | 2 cycles |
|
||||||
|
| 4 | √A | 2 cycles |
|
||||||
|
| 5 | \|A\| | 2 cycles |
|
||||||
|
| 6 | min(A, B) | 2 cycles |
|
||||||
|
| 7 | max(A, B) | 2 cycles |
|
||||||
|
|
||||||
|
### Q16.16 Encoding
|
||||||
|
|
||||||
|
```
|
||||||
|
value = integer_part × 65536 + fraction_part
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
1.0 = 65536 (0x00010000)
|
||||||
|
0.5 = 32768 (0x00008000)
|
||||||
|
3.14 = 205887 (0x000323D7)
|
||||||
|
-1.0 = -65536 (0xFFFF0000)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Voltage Controller ($8010)
|
||||||
|
|
||||||
|
The voltage controller manages BRAM access modes:
|
||||||
|
|
||||||
|
| Mode | Value | Description |
|
||||||
|
|------|-------|-------------|
|
||||||
|
| STORE | 0 | Direct memory read/write |
|
||||||
|
| COMPUTE | 1 | Q16 LUT computation mode |
|
||||||
|
| APPROX | 2 | Approximate computation (fast) |
|
||||||
|
| MORPHIC | 3 | Morphic field mode |
|
||||||
|
|
||||||
|
```subleq
|
||||||
|
; Set voltage controller to COMPUTE mode
|
||||||
|
; Write 1 to address $8010
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scale Space ($8011)
|
||||||
|
|
||||||
|
The scale space parameter controls Gaussian kernel selection:
|
||||||
|
|
||||||
|
| Value | σ (sigma) | Kernel Bank |
|
||||||
|
|-------|-----------|-------------|
|
||||||
|
| 0 | 0.25 | Bank 0 |
|
||||||
|
| 1 | 0.50 | Bank 1 |
|
||||||
|
| 2 | 0.75 | Bank 2 |
|
||||||
|
| 3 | 1.00 | Bank 3 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## HiGHS Pivot Registers ($8020–$8025)
|
||||||
|
|
||||||
|
3-stage simplex pipeline interface for hardware-accelerated LP solving:
|
||||||
|
|
||||||
|
| Address | Register | Function |
|
||||||
|
|---------|----------|----------|
|
||||||
|
| `$8020` | PIVOT_ROW | Row index |
|
||||||
|
| `$8021` | PIVOT_COL | Column index |
|
||||||
|
| `$8022` | PIVOT_VAL (lo) | Pivot value, low |
|
||||||
|
| `$8023` | PIVOT_VAL (hi) | Pivot value, high |
|
||||||
|
| `$8024` | PIVOT_CTRL | Control/status |
|
||||||
|
| `$8025` | PIVOT_RESULT | Result/iteration count |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Example Programs
|
||||||
|
|
||||||
|
### 1. Blink LED
|
||||||
|
|
||||||
|
Blink LED 0 in a loop:
|
||||||
|
|
||||||
|
```subleq
|
||||||
|
; Program at address 0
|
||||||
|
; Toggle LED 0 by XOR with 1
|
||||||
|
|
||||||
|
; mem[100] = 1 (constant)
|
||||||
|
; mem[101] = LED address ($FFF0)
|
||||||
|
; mem[102] = current LED state
|
||||||
|
; mem[103] = 0 (zero constant)
|
||||||
|
|
||||||
|
; Instruction 0: sub 103 from 102, store in 102 (clear 102)
|
||||||
|
addr 0: 103 102 3 ; mem[102] = mem[102] - mem[103] = 0
|
||||||
|
|
||||||
|
; Instruction 3: sub 103 from LED, store in LED (clear LED)
|
||||||
|
addr 3: 103 2545 6 ; mem[$FFF0] = mem[$FFF0] - 0
|
||||||
|
|
||||||
|
; Instruction 6: sub 100 from LED, store in LED (set bit 0)
|
||||||
|
addr 6: 100 2545 9 ; mem[$FFF0] = mem[$FFF0] - 1
|
||||||
|
|
||||||
|
; Instruction 9: delay loop
|
||||||
|
addr 9: 104 104 12 ; mem[104] = mem[104] - 1
|
||||||
|
addr 12: 104 104 0 ; if mem[104] <= 0, jump to 0 (restart)
|
||||||
|
|
||||||
|
; Data
|
||||||
|
addr 100: 1 ; toggle mask
|
||||||
|
addr 101: 0 ; unused
|
||||||
|
addr 102: 0 ; LED state
|
||||||
|
addr 103: 0 ; zero
|
||||||
|
addr 104: 50000 ; delay counter
|
||||||
|
```
|
||||||
|
|
||||||
|
**Assembled binary:**
|
||||||
|
```
|
||||||
|
0064 0066 0003
|
||||||
|
0067 09F1 0006
|
||||||
|
0064 09F1 0009
|
||||||
|
0068 0068 000C
|
||||||
|
0068 0068 0000
|
||||||
|
0001 0000 0000 0000 C350
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Q16 Addition
|
||||||
|
|
||||||
|
Add two Q16.16 values using the hardware LUT:
|
||||||
|
|
||||||
|
```subleq
|
||||||
|
; Write operands to Q16 LUT, read result
|
||||||
|
|
||||||
|
; mem[200] = operand A = 3.14 (Q16: 205887 = 0x000323D7)
|
||||||
|
; mem[201] = operand B = 2.72 (Q16: 178258 = 0x0002B8F2)
|
||||||
|
|
||||||
|
; Write A low word to $8000
|
||||||
|
addr 0: 200 32768 3 ; mem[$8000] = mem[200] (A low)
|
||||||
|
|
||||||
|
; Write A high word to $8001
|
||||||
|
addr 3: 201 32769 6 ; mem[$8001] = 0 (A high)
|
||||||
|
|
||||||
|
; Write B low word to $8002
|
||||||
|
addr 6: 202 32770 9 ; mem[$8002] = mem[202] (B low)
|
||||||
|
|
||||||
|
; Write B high word to $8003
|
||||||
|
addr 9: 203 32771 12 ; mem[$8003] = 0 (B high)
|
||||||
|
|
||||||
|
; Set opcode to 0 (add)
|
||||||
|
addr 12: 204 32772 15 ; mem[$8004] = 0
|
||||||
|
|
||||||
|
; Read result low from $8008
|
||||||
|
addr 15: 204 32776 18 ; mem[$8008] -> read
|
||||||
|
|
||||||
|
; Store result to mem[210]
|
||||||
|
addr 18: 32776 210 21 ; mem[210] = result low
|
||||||
|
|
||||||
|
; HALT
|
||||||
|
addr 21: 0 0 0
|
||||||
|
|
||||||
|
; Data
|
||||||
|
addr 200: 23D7 ; A low (3.14)
|
||||||
|
addr 201: 0003 ; A high
|
||||||
|
addr 202: B8F2 ; B low (2.72)
|
||||||
|
addr 203: 0002 ; B high
|
||||||
|
addr 204: 0000 ; opcode 0 (add)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. UART Send Character
|
||||||
|
|
||||||
|
Send 'A' (0x41) over UART:
|
||||||
|
|
||||||
|
```subleq
|
||||||
|
; Wait for TX to be not busy, then send character
|
||||||
|
|
||||||
|
; mem[300] = 0x41 ('A')
|
||||||
|
; mem[301] = 0 (zero)
|
||||||
|
; mem[302] = UART TX status address ($FF01)
|
||||||
|
; mem[303] = UART TX data address ($FF00)
|
||||||
|
|
||||||
|
; Check TX status (poll loop)
|
||||||
|
addr 0: 302 304 3 ; mem[304] = mem[$FF01]
|
||||||
|
addr 3: 304 304 6 ; mem[304] -= mem[304] (test if zero)
|
||||||
|
addr 6: 304 304 9 ; if <= 0 (not busy), continue
|
||||||
|
addr 9: 301 304 0 ; else, reset and retry
|
||||||
|
|
||||||
|
; Send character
|
||||||
|
addr 12: 300 303 15 ; mem[$FF00] = 0x41
|
||||||
|
|
||||||
|
; HALT
|
||||||
|
addr 15: 0 0 0
|
||||||
|
|
||||||
|
; Data
|
||||||
|
addr 300: 0041 ; 'A'
|
||||||
|
addr 301: 0000 ; zero
|
||||||
|
addr 302: FF01 ; UART TX status
|
||||||
|
addr 303: FF00 ; UART TX data
|
||||||
|
addr 304: 0000 ; temp
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Loading Programs via UART
|
||||||
|
|
||||||
|
### Using Python
|
||||||
|
|
||||||
|
```python
|
||||||
|
import serial
|
||||||
|
import struct
|
||||||
|
|
||||||
|
# Load assembled program (array of 16-bit words)
|
||||||
|
program = [0x0064, 0x0066, 0x0003, ...] # assembled instructions
|
||||||
|
|
||||||
|
# Connect to FPGA UART
|
||||||
|
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=1)
|
||||||
|
|
||||||
|
# Send program: each word as 2 bytes, big-endian
|
||||||
|
for word in program:
|
||||||
|
ser.write(struct.pack('>H', word))
|
||||||
|
|
||||||
|
ser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using openFPGALoader (SRAM load)
|
||||||
|
|
||||||
|
For quick iteration (non-persistent):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Load bitstream to SRAM (lost on power cycle)
|
||||||
|
openFPGALoader -b tangnano9k --sram research_stack_top.fs
|
||||||
|
|
||||||
|
# Load to flash (persistent)
|
||||||
|
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reading Results
|
||||||
|
|
||||||
|
### Via UART
|
||||||
|
|
||||||
|
```python
|
||||||
|
import serial
|
||||||
|
|
||||||
|
ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=5)
|
||||||
|
|
||||||
|
# Read result bytes
|
||||||
|
data = ser.read(2) # 1 word = 2 bytes
|
||||||
|
result = struct.unpack('>H', data)[0]
|
||||||
|
print(f"Result: {result} (0x{result:04X})")
|
||||||
|
|
||||||
|
ser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Via LEDs
|
||||||
|
|
||||||
|
Read the 6 LEDs (pins 10-16) as a 6-bit value from `led[0:5]`.
|
||||||
|
- LED 0 = bit 0 (rightmost)
|
||||||
|
|
||||||
|
### Via Memory Dump
|
||||||
|
|
||||||
|
After HALT, the UART TX beacon outputs the full memory contents.
|
||||||
|
Connect a serial terminal and observe the dump.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Build Toolchain
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Synthesis
|
||||||
|
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||||
|
|
||||||
|
# Simulation
|
||||||
|
cd /tmp/fpga_sim_full && ./obj_dir/sim_top
|
||||||
|
|
||||||
|
# Flash
|
||||||
|
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tool Versions
|
||||||
|
|
||||||
|
| Tool | Version |
|
||||||
|
|------|---------|
|
||||||
|
| Yosys | 0.64 |
|
||||||
|
| nextpnr-himbaechel | 0.10-75 |
|
||||||
|
| gowin_pack | latest |
|
||||||
|
| Verilator | 5.048 |
|
||||||
|
| openFPGALoader | latest |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Timing
|
||||||
|
|
||||||
|
- **Clock:** 27 MHz (37.04 ns period)
|
||||||
|
- **Achieved Fmax:** 195.92 MHz (7.2× margin)
|
||||||
|
- **Q16 LUT latency:** 2 cycles (74 ns)
|
||||||
|
- **MAX_CYCLES:** 1,000,000
|
||||||
399
6-Documentation/RUNBOOK.md
Normal file
399
6-Documentation/RUNBOOK.md
Normal file
|
|
@ -0,0 +1,399 @@
|
||||||
|
# Ops Runbook — Research Stack
|
||||||
|
|
||||||
|
**Last updated:** 2026-05-29
|
||||||
|
|
||||||
|
Quick-reference operational procedures for k3s, FPGA, Tailscale, GPU, and DNS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## k3s Cluster
|
||||||
|
|
||||||
|
### Start / Stop
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set kubeconfig
|
||||||
|
export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml
|
||||||
|
|
||||||
|
# Check cluster status
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl get pods -A
|
||||||
|
```
|
||||||
|
|
||||||
|
**Control plane (nixos):**
|
||||||
|
```bash
|
||||||
|
# Restart k3s server
|
||||||
|
sudo systemctl restart k3s
|
||||||
|
|
||||||
|
# Check k3s server health
|
||||||
|
sudo systemctl status k3s
|
||||||
|
sudo journalctl -u k3s -f --lines=50
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All nodes ready
|
||||||
|
kubectl get nodes -o wide
|
||||||
|
|
||||||
|
# All pods running (any namespace)
|
||||||
|
kubectl get pods -A --field-selector='status.phase!=Running,status.phase!=Succeeded'
|
||||||
|
|
||||||
|
# Specific namespace
|
||||||
|
kubectl get pods -n services
|
||||||
|
kubectl get pods -n media
|
||||||
|
kubectl get pods -n monitoring
|
||||||
|
kubectl get pods -n ai-models
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Pods
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restart a deployment (rolling restart)
|
||||||
|
kubectl rollout restart deployment/<name> -n <namespace>
|
||||||
|
|
||||||
|
# Examples
|
||||||
|
kubectl rollout restart deployment/homer -n services
|
||||||
|
kubectl rollout restart deployment/ollama -n ai-models
|
||||||
|
kubectl rollout restart deployment/cluster-dashboard -n monitoring
|
||||||
|
|
||||||
|
# Force delete stuck pod
|
||||||
|
kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pod logs
|
||||||
|
kubectl logs <pod-name> -n <namespace> --tail=100 -f
|
||||||
|
|
||||||
|
# Previous container (if crashed)
|
||||||
|
kubectl logs <pod-name> -n <namespace> --previous
|
||||||
|
|
||||||
|
# All pods matching label
|
||||||
|
kubectl logs -l app=<label> -n <namespace> --tail=50
|
||||||
|
```
|
||||||
|
|
||||||
|
### Common Pods
|
||||||
|
|
||||||
|
| Namespace | Service | Deployment Name |
|
||||||
|
|-----------|---------|-----------------|
|
||||||
|
| `services` | Homer | `homer` |
|
||||||
|
| `services` | Hermes | `hermes` |
|
||||||
|
| `services` | Actual Budget | `actual-budget` |
|
||||||
|
| `services` | Uptime Kuma | `uptime-kuma` |
|
||||||
|
| `services` | Vaultwarden | `vaultwarden` |
|
||||||
|
| `services` | Authentik | `authentik` |
|
||||||
|
| `services` | Credential Server | `credential-server` |
|
||||||
|
| `services` | Registry API | `registry-api` |
|
||||||
|
| `services` | Jobs API | `jobs-api` |
|
||||||
|
| `services` | Blobs API | `blobs-api` |
|
||||||
|
| `ai-models` | Ollama | `ollama` |
|
||||||
|
| `monitoring` | Cluster Dashboard | `cluster-dashboard` |
|
||||||
|
| `media` | Jellyfin | `jellyfin` |
|
||||||
|
| `research` | AlphaProof | `alphaproof` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FPGA — Tang Nano 9K
|
||||||
|
|
||||||
|
### Flash Bitstream
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build
|
||||||
|
cd 4-Infrastructure/hardware && bash build_research_stack.sh
|
||||||
|
|
||||||
|
# Flash via USB-JTAG
|
||||||
|
openFPGALoader -b tangnano9k research_stack_top.fs
|
||||||
|
|
||||||
|
# Verify flash
|
||||||
|
openFPGALoader -b tangnano9k --verify research_stack_top.fs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify LEDs
|
||||||
|
|
||||||
|
After power-on, `led[0:5]` (pins 10-16) should show SUBLEQ state.
|
||||||
|
- **All off:** CPU halted or not clocked
|
||||||
|
- **Blinking:** CPU running (heartbeat)
|
||||||
|
- **LED 0 solid on:** CPU halted (trap)
|
||||||
|
|
||||||
|
### Debug UART
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Connect to UART TX (pin 17) via USB-serial adapter
|
||||||
|
# Baud rate: 115384 (27MHz / 234)
|
||||||
|
picocom -b 115384 /dev/ttyUSB0
|
||||||
|
|
||||||
|
# Or with screen
|
||||||
|
screen /dev/ttyUSB0 115384
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Simulation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /tmp/fpga_sim_full && ./obj_dir/sim_top
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pin Reference
|
||||||
|
|
||||||
|
| Pin | Signal | Direction | Notes |
|
||||||
|
|-----|--------|-----------|-------|
|
||||||
|
| 52 | clk | input | 27 MHz oscillator |
|
||||||
|
| 4 | rst_n | input | Active-low reset (pull-up) |
|
||||||
|
| 3 | user_btn | input | Active-low (pull-up) |
|
||||||
|
| 10-16 | led[0:5] | output | LVCMOS18 |
|
||||||
|
| 17 | uart_tx | output | 115384 baud |
|
||||||
|
| 18 | uart_rx | input | Pull-up |
|
||||||
|
|
||||||
|
### FPGA Not Responding
|
||||||
|
|
||||||
|
1. Check USB connection to Tang Nano 9K
|
||||||
|
2. Verify `openFPGALoader` sees the device: `openFPGALoader --detect`
|
||||||
|
3. Re-flash bitstream: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
||||||
|
4. Check clock: pin 52 should have 27 MHz (oscilloscope)
|
||||||
|
5. Assert reset: pull pin 4 low momentarily, then release
|
||||||
|
6. Check power: board should draw ~100mA from USB
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tailscale
|
||||||
|
|
||||||
|
### Check Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Show tailnet status
|
||||||
|
tailscale status
|
||||||
|
|
||||||
|
# Show current node IP
|
||||||
|
tailscale ip
|
||||||
|
|
||||||
|
# Check connectivity to other nodes
|
||||||
|
tailscale ping qfox-1
|
||||||
|
tailscale ping 361395-1
|
||||||
|
tailscale ping racknerd-510bd9c
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Funnel
|
||||||
|
|
||||||
|
The Funnel runs on `361395-1` and routes to Traefik NodePort 30080:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On 361395-1 (edge node):
|
||||||
|
tailscale funnel 8080 off
|
||||||
|
tailscale funnel 8080
|
||||||
|
|
||||||
|
# Verify funnel URL
|
||||||
|
tailscale funnel status
|
||||||
|
```
|
||||||
|
|
||||||
|
### Debug Relay
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if relay is in use
|
||||||
|
tailscale status | grep relay
|
||||||
|
|
||||||
|
# Force direct connections (disable relay)
|
||||||
|
tailscale set --direct-peers-only=true
|
||||||
|
|
||||||
|
# Reset to default
|
||||||
|
tailscale set --direct-peers-only=false
|
||||||
|
```
|
||||||
|
|
||||||
|
### Re-authenticate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On any node:
|
||||||
|
tailscale up --authkey=<TS_AUTH_KEY>
|
||||||
|
|
||||||
|
# Or interactive login
|
||||||
|
tailscale up
|
||||||
|
```
|
||||||
|
|
||||||
|
### Node IPs
|
||||||
|
|
||||||
|
| Node | Tailscale IP | Role |
|
||||||
|
|------|-------------|------|
|
||||||
|
| nixos | 100.102.173.61 | Control plane |
|
||||||
|
| qfox-1 | 100.88.57.96 | GPU worker |
|
||||||
|
| 361395-1 | 100.110.163.82 | Edge/Funnel |
|
||||||
|
| racknerd | 100.80.39.40 | Edge worker |
|
||||||
|
| steamdeck | 100.85.244.73 | Worker |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GPU — QFox (RTX 4070)
|
||||||
|
|
||||||
|
### Check nvidia-smi
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# GPU status
|
||||||
|
nvidia-smi
|
||||||
|
|
||||||
|
# Watch GPU utilization
|
||||||
|
nvidia-smi -l 1
|
||||||
|
|
||||||
|
# Check driver version
|
||||||
|
nvidia-smi --query-gpu=driver_version --format=csv,noheader
|
||||||
|
# Expected: 610.43
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Ollama
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# If running as k3s pod
|
||||||
|
kubectl rollout restart deployment/ollama -n ai-models
|
||||||
|
kubectl logs -l app=ollama -n ai-models --tail=20 -f
|
||||||
|
|
||||||
|
# If running as systemd service on qfox-1
|
||||||
|
sudo systemctl restart ollama
|
||||||
|
sudo journalctl -u ollama -f --tail=20
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pull / Manage Models
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List loaded models
|
||||||
|
curl http://100.88.57.96:11434/api/tags
|
||||||
|
|
||||||
|
# Pull a model
|
||||||
|
curl http://100.88.57.96:11434/api/pull -d '{"name": "deepseek-coder-v2:16b"}'
|
||||||
|
|
||||||
|
# Test inference
|
||||||
|
curl http://100.88.57.96:11434/api/generate \
|
||||||
|
-d '{"model": "deepseek-coder-v2:16b", "prompt": "Hello", "stream": false}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ollama via NodePort
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Access from cluster
|
||||||
|
curl http://<any-node-ip>:31434/api/tags
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## DNS & TLS
|
||||||
|
|
||||||
|
### Check LE Certificates
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check cert expiry
|
||||||
|
echo | openssl s_client -connect researchstack.info:443 -servername researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
||||||
|
|
||||||
|
# Wildcard cert
|
||||||
|
echo | openssl s_client -connect registry.researchstack.info:443 -servername registry.researchstack.info 2>/dev/null | openssl x509 -noout -dates
|
||||||
|
```
|
||||||
|
|
||||||
|
**Current cert valid until:** 2026-08-18
|
||||||
|
|
||||||
|
### Renew Certificates
|
||||||
|
|
||||||
|
Caddy auto-renews via Porkbun DNS-01. If manual renewal is needed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restart Caddy to trigger renewal check
|
||||||
|
kubectl rollout restart deployment/caddy -n services
|
||||||
|
|
||||||
|
# Check Caddy logs for renewal
|
||||||
|
kubectl logs -l app=caddy -n services --tail=50 | grep -i renew
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Porkbun DNS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Verify A record
|
||||||
|
dig researchstack.info +short
|
||||||
|
dig auth.researchstack.info +short
|
||||||
|
dig registry.researchstack.info +short
|
||||||
|
|
||||||
|
# Check API key (set PORKBUN_API_KEY env var)
|
||||||
|
curl -X POST https://api.porkbun.com/api/json/v3/ping \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"apikey": "'$PORKBUN_API_KEY'"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Failures
|
||||||
|
|
||||||
|
### Node Down
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Identify down node
|
||||||
|
kubectl get nodes
|
||||||
|
|
||||||
|
# Check node conditions
|
||||||
|
kubectl describe node <node-name>
|
||||||
|
|
||||||
|
# On the node itself:
|
||||||
|
sudo systemctl status k3s-agent # worker nodes
|
||||||
|
sudo systemctl status k3s # control plane
|
||||||
|
|
||||||
|
# Rejoin cluster if needed (worker):
|
||||||
|
sudo k3s agent --server https://100.102.173.61:6443 --token <TOKEN>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pod CrashLoopBackOff
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check events
|
||||||
|
kubectl describe pod <pod-name> -n <namespace>
|
||||||
|
|
||||||
|
# Check logs (including previous crash)
|
||||||
|
kubectl logs <pod-name> -n <namespace> --previous --tail=100
|
||||||
|
|
||||||
|
# Common fixes:
|
||||||
|
# - ConfigMap/Secret missing: kubectl get configmap -n <ns>; kubectl get secret -n <ns>
|
||||||
|
# - Image pull error: check image tag and registry access
|
||||||
|
# - Resource limits: kubectl top pod -n <namespace>
|
||||||
|
# - OOMKill: increase memory limit in deployment spec
|
||||||
|
```
|
||||||
|
|
||||||
|
### FPGA Not Responding
|
||||||
|
|
||||||
|
1. Unplug and replug USB cable
|
||||||
|
2. Check `ls /dev/ttyUSB*` — device should appear
|
||||||
|
3. Re-flash: `openFPGALoader -b tangnano9k research_stack_top.fs`
|
||||||
|
4. If JTAG fails: try holding reset (pin 4 low) while plugging in
|
||||||
|
5. Check power LED on Tang Nano 9K board
|
||||||
|
6. Try different USB port / cable
|
||||||
|
|
||||||
|
### Tailscale Tunnel Down
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check daemon
|
||||||
|
sudo systemctl status tailscaled
|
||||||
|
|
||||||
|
# Restart
|
||||||
|
sudo systemctl restart tailscaled
|
||||||
|
tailscale up
|
||||||
|
|
||||||
|
# Check if key expired
|
||||||
|
tailscale status | grep -i expir
|
||||||
|
```
|
||||||
|
|
||||||
|
### DNS Resolution Failing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check Caddy pod
|
||||||
|
kubectl logs -l app=caddy -n services --tail=50
|
||||||
|
|
||||||
|
# Test DNS from inside cluster
|
||||||
|
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup researchstack.info
|
||||||
|
|
||||||
|
# Check Porkbun API key
|
||||||
|
kubectl get secret porkbun-credentials -n services -o jsonpath='{.data.api-key}' | base64 -d
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Escalation Matrix
|
||||||
|
|
||||||
|
| Issue | First Response | Escalation |
|
||||||
|
|-------|---------------|------------|
|
||||||
|
| Pod down | `kubectl rollout restart` | Check node, check resource limits |
|
||||||
|
| Node down | SSH to node, check `systemctl` | Reboot, check hardware |
|
||||||
|
| FPGA unresponsive | Re-flash bitstream | Check USB, try different board |
|
||||||
|
| Tailscale tunnel | `tailscale up` | Check auth key, restart daemon |
|
||||||
|
| DNS/cert | Restart Caddy | Check Porkbun API, check ingress |
|
||||||
|
| GPU errors | Check `nvidia-smi` | Restart driver, check PCIe seating |
|
||||||
|
| OOM on Ollama | Restart pod | Reduce context length, switch model |
|
||||||
Loading…
Add table
Reference in a new issue