diff --git a/6-Documentation/API_DOCS.md b/6-Documentation/API_DOCS.md new file mode 100644 index 00000000..d3d79e75 --- /dev/null +++ b/6-Documentation/API_DOCS.md @@ -0,0 +1,465 @@ +# API Documentation — Research Stack Services + +**Last updated:** 2026-05-29 +**Base URL:** `https://researchstack.info` +**Auth:** Authentik OIDC (SSO) or Bearer token (API endpoints) + +--- + +## Authentication + +### OIDC (SSO) — Web Endpoints + +Web-facing endpoints use Authentik OIDC. Access via browser redirects to `auth.researchstack.info`. + +### Token Auth — API Endpoints + +API endpoints (`/api/*`) use Bearer token authentication: + +``` +Authorization: Bearer +``` + +Tokens are managed by the Credential Server. + +--- + +## Cluster Dashboard + +**Namespace:** `monitoring` +**Internal URL:** `http://cluster-dashboard:8787` +**External URL:** `https://researchstack.info/server/dash/` (via Homarr) +**Tech:** FastAPI + Vite + +### Endpoints + +#### `GET /` + +Dashboard UI (Vite SPA). + +#### `GET /api/status` + +Cluster health summary. + +**Response:** +```json +{ + "nodes": [ + { + "name": "nixos-laptop", + "status": "Ready", + "ip": "100.102.173.61", + "roles": "control-plane", + "version": "v1.35.4+k3s1" + } + ], + "pods": { + "total": 24, + "running": 23, + "pending": 0, + "failed": 1 + }, + "namespaces": ["services", "media", "monitoring", "ai-models", "edge", "research", "mail"] +} +``` + +#### `GET /api/metrics` + +Prometheus-compatible metrics. + +#### `WebSocket /ws/live` + +Real-time cluster events stream. + +**Message format:** +```json +{ + "type": "pod_event", + "namespace": "services", + "pod": "homer-abc123", + "status": "Running", + "timestamp": "2026-05-29T12:00:00Z" +} +``` + +--- + +## Credential Server + +**Namespace:** `services` +**Internal URL:** `http://credential-server:8080` +**External URL:** `https://researchstack.info/api/cred/` +**Auth:** Bearer token + +### Endpoints + +#### `GET /api/cred/health` + +Health check. + +**Response:** `200 OK` +```json +{"status": "ok"} +``` + +#### `GET /api/cred/tokens` + +List available tokens (requires admin token). + +**Response:** +```json +{ + "tokens": [ + {"name": "registry", "scope": "read,write", "expires": "2026-12-31"}, + {"name": "jobs", "scope": "read,write,execute", "expires": "2026-12-31"} + ] +} +``` + +#### `POST /api/cred/issue` + +Issue a new token. + +**Request:** +```json +{ + "name": "service-name", + "scope": "read,write", + "ttl": "30d" +} +``` + +**Response:** +```json +{ + "token": "rs_tk_...", + "expires": "2026-06-29T12:00:00Z" +} +``` + +#### `POST /api/cred/validate` + +Validate a token. + +**Request:** +```json +{"token": "rs_tk_..."} +``` + +**Response:** +```json +{ + "valid": true, + "name": "registry", + "scope": "read,write" +} +``` + +--- + +## Registry API + +**Namespace:** `services` +**Internal URL:** `http://registry-api:8080` +**External URL:** `https://researchstack.info/api/registry/` +**Auth:** Bearer token + +### Endpoints + +#### `GET /api/registry/health` + +Health check. + +#### `GET /api/registry/artifacts` + +List all registered artifacts. + +**Query Parameters:** +- `type` — Filter by artifact type (e.g., `bitstream`, `lean-build`, `receipt`) +- `since` — ISO 8601 timestamp +- `limit` — Max results (default: 50) + +**Response:** +```json +{ + "artifacts": [ + { + "id": "art_abc123", + "name": "research_stack_top.fs", + "type": "bitstream", + "size": 184320, + "sha256": "...", + "created": "2026-05-29T10:00:00Z", + "tags": ["fpga", "production"] + } + ], + "total": 42 +} +``` + +#### `POST /api/registry/artifacts` + +Register a new artifact. + +**Request:** +```json +{ + "name": "research_stack_top.fs", + "type": "bitstream", + "sha256": "...", + "blob_ref": "blob_xyz", + "tags": ["fpga"] +} +``` + +#### `GET /api/registry/artifacts/{id}` + +Get artifact by ID. + +#### `PUT /api/registry/artifacts/{id}` + +Update artifact metadata. + +#### `DELETE /api/registry/artifacts/{id}` + +Remove artifact registration (does not delete blob). + +--- + +## Jobs API + +**Namespace:** `services` +**Internal URL:** `http://jobs-api:8080` +**External URL:** `https://researchstack.info/api/jobs/` +**Auth:** Bearer token + +### Endpoints + +#### `GET /api/jobs/health` + +Health check. + +#### `GET /api/jobs` + +List jobs. + +**Query Parameters:** +- `status` — Filter: `pending`, `running`, `completed`, `failed` +- `type` — Filter by job type +- `limit` — Max results (default: 20) + +**Response:** +```json +{ + "jobs": [ + { + "id": "job_abc123", + "type": "lean-build", + "status": "completed", + "created": "2026-05-29T10:00:00Z", + "started": "2026-05-29T10:00:05Z", + "finished": "2026-05-29T10:05:30Z", + "result": { + "jobs": 3572, + "errors": 0 + } + } + ], + "total": 156 +} +``` + +#### `POST /api/jobs` + +Create a new job. + +**Request:** +```json +{ + "type": "lean-build", + "params": { + "target": "Semantics", + "branch": "main" + }, + "priority": "normal" +} +``` + +**Response:** +```json +{ + "id": "job_def456", + "status": "pending", + "created": "2026-05-29T12:00:00Z" +} +``` + +#### `GET /api/jobs/{id}` + +Get job status and result. + +#### `POST /api/jobs/{id}/cancel` + +Cancel a pending or running job. + +#### `GET /api/jobs/{id}/logs` + +Stream job logs (Server-Sent Events). + +**Response:** `text/event-stream` +``` +data: {"line": "Building Semantics...", "ts": "2026-05-29T12:00:01Z"} +data: {"line": "3572 jobs, 0 errors", "ts": "2026-05-29T12:05:30Z"} +``` + +--- + +## Blobs API + +**Namespace:** `services` +**Internal URL:** `http://blobs-api:8080` +**External URL:** `https://researchstack.info/api/blobs/` +**Auth:** Bearer token + +### Endpoints + +#### `GET /api/blobs/health` + +Health check. + +#### `POST /api/blobs` + +Upload a blob. + +**Request:** `multipart/form-data` +- `file` — Binary file content +- `sha256` — Expected SHA-256 hash (verification) + +**Response:** +```json +{ + "id": "blob_xyz789", + "size": 184320, + "sha256": "...", + "created": "2026-05-29T12:00:00Z" +} +``` + +#### `GET /api/blobs/{id}` + +Download a blob. + +**Response:** `application/octet-stream` with blob content. + +#### `HEAD /api/blobs/{id}` + +Check blob existence. + +**Response:** `200 OK` (exists) or `404 Not Found` + +#### `DELETE /api/blobs/{id}` + +Delete a blob. + +#### `GET /api/blobs` + +List blobs. + +**Query Parameters:** +- `limit` — Max results +- `offset` — Pagination offset + +--- + +## Authentik — OIDC Configuration + +**URL:** `https://auth.researchstack.info` +**Protocol:** OpenID Connect + +### Provider Configuration + +| Field | Value | +|-------|-------| +| Authorization URL | `https://auth.researchstack.info/application/o/authorize/` | +| Token URL | `https://auth.researchstack.info/application/o/token/` | +| UserInfo URL | `https://auth.researchstack.info/application/o/userinfo/` | +| JWKS URL | `https://auth.researchstack.info/application/o/research-stack/jwks/` | +| Issuer | `https://auth.researchstack.info/application/o/research-stack/` | + +### Scopes + +| Scope | Description | +|-------|-------------| +| `openid` | Standard OIDC | +| `profile` | User profile (name, email) | +| `email` | Email address | +| `groups` | Group membership | + +### Redirect URIs + +``` +https://researchstack.info/oidc/callback +https://researchstack.info/apps/chat/oidc/callback +https://researchstack.info/apps/budget/oidc/callback +``` + +### Application Setup + +In Authentik admin: + +1. **Applications** → Create → Name: `Research Stack` +2. **Providers** → Create → OAuth2/OpenID Provider +3. Set client type: `Confidential` +4. Set redirect URIs above +5. Assign to application + +### Service Account Tokens + +For API-to-API auth (no user interaction): + +```bash +# Create service account in Authentik +# Admin → Directory → Users → Create +# Type: Service Account + +# Get token +curl -X POST https://auth.researchstack.info/application/o/token/ \ + -d "grant_type=client_credentials" \ + -d "client_id=" \ + -d "client_secret=" \ + -d "scope=openid" +``` + +--- + +## Common Response Codes + +| Code | Meaning | +|------|---------| +| 200 | Success | +| 201 | Created | +| 400 | Bad request (invalid parameters) | +| 401 | Unauthorized (missing or invalid token) | +| 403 | Forbidden (insufficient scope) | +| 404 | Not found | +| 409 | Conflict (duplicate resource) | +| 422 | Unprocessable entity (validation error) | +| 500 | Internal server error | + +--- + +## Rate Limits + +| Endpoint | Limit | Window | +|----------|-------|--------| +| `/api/cred/*` | 100 req | 1 minute | +| `/api/registry/*` | 200 req | 1 minute | +| `/api/jobs/*` | 50 req | 1 minute | +| `/api/blobs/*` | 100 req | 1 minute | + +Rate limit headers: +``` +X-RateLimit-Limit: 100 +X-RateLimit-Remaining: 95 +X-RateLimit-Reset: 1685366400 +``` diff --git a/6-Documentation/DISASTER_RECOVERY.md b/6-Documentation/DISASTER_RECOVERY.md new file mode 100644 index 00000000..9029a32f --- /dev/null +++ b/6-Documentation/DISASTER_RECOVERY.md @@ -0,0 +1,343 @@ +# Disaster Recovery — Research Stack + +**Last updated:** 2026-05-29 + +Procedures for backing up and restoring all Research Stack components. + +--- + +## k3s Cluster + +### Backup etcd + +```bash +# Snapshot etcd (on control plane node — nixos) +sudo k3s etcd-snapshot save \ + --name researchstack-backup \ + --dir /var/lib/rancher/k3s/server/db/snapshots + +# List snapshots +sudo ls -la /var/lib/rancher/k3s/server/db/snapshots/ + +# Copy snapshot off-node +scp /var/lib/rancher/k3s/server/db/snapshots/researchstack-backup-* \ + user@backup-host:/backups/k3s/ +``` + +### Restore from Snapshot + +```bash +# On control plane (nixos): +sudo systemctl stop k3s + +# Restore +sudo k3s server \ + --cluster-reset \ + --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/researchstack-backup- + +# Restart +sudo systemctl start k3s + +# Verify +export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml +kubectl get nodes +kubectl get pods -A +``` + +### Backup Manifests + +All Kubernetes manifests are in the git repo. The source of truth is: + +```bash +cd ~/Research\ Stack +git push origin main +``` + +### Automated Backup Schedule + +Create a cron job on nixos: + +```bash +# /etc/cron.d/k3s-backup +0 3 * * * root k3s etcd-snapshot save --name researchstack-backup --dir /var/lib/rancher/k3s/server/db/snapshots && \ + find /var/lib/rancher/k3s/server/db/snapshots/ -mtime +7 -delete +``` + +--- + +## FPGA — Tang Nano 9K + +### Backup Bitstream + +The bitstream source and compiled output are in the git repo: + +```bash +cd ~/Research\ Stack + +# Source +ls 4-Infrastructure/hardware/*.v + +# Compiled bitstream +ls 4-Infrastructure/hardware/research_stack_top.fs +``` + +### Re-flash from Git + +```bash +cd ~/Research\ Stack + +# Rebuild if needed +cd 4-Infrastructure/hardware && bash build_research_stack.sh + +# Flash +openFPGALoader -b tangnano9k research_stack_top.fs + +# Verify +openFPGALoader -b tangnano9k --verify research_stack_top.fs +``` + +### Backup SRAM Contents + +SRAM contents are volatile (lost on power cycle). To preserve runtime state: + +```python +# Dump memory via UART before power-off +import serial, struct + +ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=10) + +# Trigger memory dump (send HALT with dump flag) +# ... protocol-specific ... + +# Read and save +with open('fpga_memory_dump.bin', 'wb') as f: + data = ser.read(8192) # 4K words = 8K bytes + f.write(data) + +ser.close() +``` + +--- + +## Tailscale + +### Re-authenticate Node + +```bash +# On the node that lost auth: +sudo systemctl restart tailscaled +tailscale up --authkey= + +# Or interactive +tailscale up +``` + +### Re-join Tailnet + +```bash +# Check current status +tailscale status + +# If node is missing from tailnet: +# 1. Generate new auth key at https://login.tailscale.com/admin/settings/keys +# 2. On the node: +sudo tailscale up --authkey=tskey-auth- + +# Verify connectivity +tailscale ping nixos-laptop +tailscale ping qfox-1 +``` + +### Backup Tailscale State + +Tailscale state is stored at `/var/lib/tailscale/`: + +```bash +# Backup state directory +sudo tar czf /backups/tailscale-state-$(date +%Y%m%d).tar.gz \ + /var/lib/tailscale/ + +# Restore +sudo systemctl stop tailscaled +sudo tar xzf /backups/tailscale-state-*.tar.gz -C / +sudo systemctl start tailscaled +``` + +--- + +## Git Repository + +### Backup + +The canonical backup is the remote origin. Ensure all changes are pushed: + +```bash +cd ~/Research\ Stack + +# Check for unpushed commits +git log --oneline origin/main..HEAD + +# Push everything +git push origin main --tags + +# Verify clean state +git status --branch --short --untracked-files=all +``` + +### Restore from Remote + +```bash +# Clone fresh +cd ~ +git clone "Research Stack" + +# Or reset to remote state +cd ~/Research\ Stack +git fetch origin +git reset --hard origin/main +``` + +### Backup Remotes + +```bash +# List remotes +cd ~/Research\ Stack +git remote -v + +# Add backup remote +git remote add backup +git push backup main --tags +``` + +--- + +## Secrets (sops-nix / age) + +### Backup age Keys + +```bash +# age key location (NixOS) +cat /etc/age/keys.txt + +# Backup +cp /etc/age/keys.txt /backups/age-keys-$(date +%Y%m%d).txt +chmod 600 /backups/age-keys-*.txt + +# Or from home directory +cat ~/.config/sops/age/keys.txt +``` + +### Restore age Keys + +```bash +# Restore key file +cp /backups/age-keys-*.txt /etc/age/keys.txt +chmod 600 /etc/age/keys.txt + +# Verify sops can decrypt +sops -d secrets.enc.yaml +``` + +### Backup sops Configuration + +```bash +# The .sops.yaml config is in the repo +cat ~/Research\ Stack/.sops.yaml + +# Encrypted secrets are also in the repo +find ~/Research\ Stack -name '*.enc.yaml' -o -name '*.enc.json' +``` + +### Regenerate age Key (Last Resort) + +```bash +# Generate new key pair +age-keygen -o /etc/age/keys.txt + +# Get public key +age-keygen -y /etc/age/keys.txt + +# Re-encrypt all secrets with new key +# (requires old key or plaintext backup) +sops updatekeys secrets.enc.yaml +``` + +--- + +## DNS — Porkbun + +### Backup API Key + +```bash +# From k8s secret +kubectl get secret porkbun-credentials -n services \ + -o jsonpath='{.data.api-key}' | base64 -d + +# Back up securely +echo "" | age -r > /backups/porkbun-api.age + +# Or store in password manager +``` + +### Recover API Key + +1. Log in to [Porkbun](https://porkbun.com) +2. Navigate to **Account → API Access** +3. Regenerate or copy existing API key +4. Update k8s secret: + +```bash +kubectl create secret generic porkbun-credentials \ + -n services \ + --from-literal=api-key= \ + --dry-run=client -o yaml | kubectl apply -f - + +# Restart Caddy to pick up new key +kubectl rollout restart deployment/caddy -n services +``` + +### Verify DNS Records + +```bash +dig researchstack.info +short +dig auth.researchstack.info +short +dig registry.researchstack.info +short + +# Test Porkbun API +curl -X POST https://api.porkbun.com/api/json/v3/ping \ + -H "Content-Type: application/json" \ + -d '{"apikey": "", "secretapikey": ""}' +``` + +--- + +## Recovery Priority Order + +When restoring from a total failure, follow this order: + +| Step | Component | Depends On | +|------|-----------|------------| +| 1 | **Tailscale** | Auth key (from password manager or backup) | +| 2 | **Git repo** | Network connectivity (Tailscale or direct) | +| 3 | **age keys** | Backup location | +| 4 | **k3s cluster** | etcd snapshot + age keys for secrets | +| 5 | **DNS/Porkbun** | API key (from backup or Porkbun account) | +| 6 | **FPGA** | Git repo (source + bitstream) | +| 7 | **Services** | k3s cluster + secrets + DNS | + +--- + +## Full System Checklist + +After recovery, verify each component: + +- [ ] Tailscale: `tailscale status` shows all 5 nodes +- [ ] k3s: `kubectl get nodes` shows all nodes Ready +- [ ] Pods: `kubectl get pods -A` — all Running/Succeeded +- [ ] DNS: `dig researchstack.info` resolves correctly +- [ ] TLS: cert valid (`openssl s_client`) +- [ ] Auth: `auth.researchstack.info` loads Authentik +- [ ] Funnel: `361395-1.tail4e7094.ts.net` responds +- [ ] Ollama: `curl http://100.88.57.96:31434/api/tags` returns models +- [ ] FPGA: `openFPGALoader --detect` sees Tang Nano 9K +- [ ] Dashboard: `https://researchstack.info` shows Homer diff --git a/6-Documentation/FPGA_PROGRAMMING_GUIDE.md b/6-Documentation/FPGA_PROGRAMMING_GUIDE.md new file mode 100644 index 00000000..4e57a16b --- /dev/null +++ b/6-Documentation/FPGA_PROGRAMMING_GUIDE.md @@ -0,0 +1,363 @@ +# FPGA Programming Guide — SUBLEQ on the Blitter + +**Last updated:** 2026-05-29 +**Board:** Sipeed Tang Nano 9K (Gowin GW1NR-LV9) +**CPU:** Blitter6502OISC (One Instruction Set Computer — SUBLEQ) + +--- + +## SUBLEQ Instruction Format + +The Blitter implements a **SUBLEQ** (Subtract and Branch if Less-than-or-Equal to Zero) CPU. + +Each instruction is **3 words** (3 × 16-bit = 6 bytes): + +``` +[src] [dst] [next] +``` + +**Semantics:** +``` +mem[dst] = mem[dst] - mem[src] +if mem[dst] <= 0: + PC = next +else: + PC = PC + 3 +``` + +**Special addresses:** +- If `next == 0` (or `next == PC`): **HALT** +- If `src == 0`: reads zero (constant source) +- If `dst == IO_ADDR`: writes to I/O + +--- + +## Memory Map ($0000–$FFFF) + +The SUBLEQ address space is 16-bit (64K words). The memory map is divided into regions: + +| Address Range | Size | Function | +|---------------|------|----------| +| `$0000–$0FFF` | 4K words | Program + data (BRAM) | +| `$1000–$7FFF` | 28K words | Extended data (if available) | +| `$8000–$800F` | 16 words | Q16 LUT result registers | +| `$8010` | 1 word | Voltage controller mode | +| `$8011` | 1 word | Scale space parameter | +| `$8020–$8025` | 6 words | HiGHS pivot registers | +| `$FF00` | 1 word | UART TX data register | +| `$FF01` | 1 word | UART TX status (bit 0 = busy) | +| `$FF02` | 1 word | UART RX data register | +| `$FF03` | 1 word | UART RX status (bit 0 = data available) | +| `$FFF0` | 1 word | LED output (bits 0-5 = led[0:5]) | +| `$FFF1` | 1 word | Button input (bit 0 = user_btn) | + +--- + +## Q16 LUT ($8000–$8025) + +The Q16 LUT is a hardware-accelerated fixed-point arithmetic unit. It operates on Q16.16 values (16-bit integer, 16-bit fraction; total 1.0 = 65536). + +### Q16 Operations + +Write operands to the LUT registers, then read the result: + +| Address | Register | Function | +|---------|----------|----------| +| `$8000` | OP_A (lo) | Operand A, low word | +| `$8001` | OP_A (hi) | Operand A, high word | +| `$8002` | OP_B (lo) | Operand B, low word | +| `$8003` | OP_B (hi) | Operand B, high word | +| `$8004` | OPCODE | Operation selector (0-7) | +| `$8008` | RESULT (lo) | Result, low word | +| `$8009` | RESULT (hi) | Result, high word | + +### Opcodes + +| Code | Operation | Latency | +|------|-----------|---------| +| 0 | A + B | 2 cycles (74ns @ 27MHz) | +| 1 | A - B | 2 cycles | +| 2 | A × B | 2 cycles | +| 3 | A ÷ B | 2 cycles | +| 4 | √A | 2 cycles | +| 5 | \|A\| | 2 cycles | +| 6 | min(A, B) | 2 cycles | +| 7 | max(A, B) | 2 cycles | + +### Q16.16 Encoding + +``` +value = integer_part × 65536 + fraction_part + +Examples: + 1.0 = 65536 (0x00010000) + 0.5 = 32768 (0x00008000) + 3.14 = 205887 (0x000323D7) + -1.0 = -65536 (0xFFFF0000) +``` + +--- + +## Voltage Controller ($8010) + +The voltage controller manages BRAM access modes: + +| Mode | Value | Description | +|------|-------|-------------| +| STORE | 0 | Direct memory read/write | +| COMPUTE | 1 | Q16 LUT computation mode | +| APPROX | 2 | Approximate computation (fast) | +| MORPHIC | 3 | Morphic field mode | + +```subleq +; Set voltage controller to COMPUTE mode +; Write 1 to address $8010 +``` + +--- + +## Scale Space ($8011) + +The scale space parameter controls Gaussian kernel selection: + +| Value | σ (sigma) | Kernel Bank | +|-------|-----------|-------------| +| 0 | 0.25 | Bank 0 | +| 1 | 0.50 | Bank 1 | +| 2 | 0.75 | Bank 2 | +| 3 | 1.00 | Bank 3 | + +--- + +## HiGHS Pivot Registers ($8020–$8025) + +3-stage simplex pipeline interface for hardware-accelerated LP solving: + +| Address | Register | Function | +|---------|----------|----------| +| `$8020` | PIVOT_ROW | Row index | +| `$8021` | PIVOT_COL | Column index | +| `$8022` | PIVOT_VAL (lo) | Pivot value, low | +| `$8023` | PIVOT_VAL (hi) | Pivot value, high | +| `$8024` | PIVOT_CTRL | Control/status | +| `$8025` | PIVOT_RESULT | Result/iteration count | + +--- + +## Example Programs + +### 1. Blink LED + +Blink LED 0 in a loop: + +```subleq +; Program at address 0 +; Toggle LED 0 by XOR with 1 + +; mem[100] = 1 (constant) +; mem[101] = LED address ($FFF0) +; mem[102] = current LED state +; mem[103] = 0 (zero constant) + +; Instruction 0: sub 103 from 102, store in 102 (clear 102) +addr 0: 103 102 3 ; mem[102] = mem[102] - mem[103] = 0 + +; Instruction 3: sub 103 from LED, store in LED (clear LED) +addr 3: 103 2545 6 ; mem[$FFF0] = mem[$FFF0] - 0 + +; Instruction 6: sub 100 from LED, store in LED (set bit 0) +addr 6: 100 2545 9 ; mem[$FFF0] = mem[$FFF0] - 1 + +; Instruction 9: delay loop +addr 9: 104 104 12 ; mem[104] = mem[104] - 1 +addr 12: 104 104 0 ; if mem[104] <= 0, jump to 0 (restart) + +; Data +addr 100: 1 ; toggle mask +addr 101: 0 ; unused +addr 102: 0 ; LED state +addr 103: 0 ; zero +addr 104: 50000 ; delay counter +``` + +**Assembled binary:** +``` +0064 0066 0003 +0067 09F1 0006 +0064 09F1 0009 +0068 0068 000C +0068 0068 0000 +0001 0000 0000 0000 C350 +``` + +### 2. Q16 Addition + +Add two Q16.16 values using the hardware LUT: + +```subleq +; Write operands to Q16 LUT, read result + +; mem[200] = operand A = 3.14 (Q16: 205887 = 0x000323D7) +; mem[201] = operand B = 2.72 (Q16: 178258 = 0x0002B8F2) + +; Write A low word to $8000 +addr 0: 200 32768 3 ; mem[$8000] = mem[200] (A low) + +; Write A high word to $8001 +addr 3: 201 32769 6 ; mem[$8001] = 0 (A high) + +; Write B low word to $8002 +addr 6: 202 32770 9 ; mem[$8002] = mem[202] (B low) + +; Write B high word to $8003 +addr 9: 203 32771 12 ; mem[$8003] = 0 (B high) + +; Set opcode to 0 (add) +addr 12: 204 32772 15 ; mem[$8004] = 0 + +; Read result low from $8008 +addr 15: 204 32776 18 ; mem[$8008] -> read + +; Store result to mem[210] +addr 18: 32776 210 21 ; mem[210] = result low + +; HALT +addr 21: 0 0 0 + +; Data +addr 200: 23D7 ; A low (3.14) +addr 201: 0003 ; A high +addr 202: B8F2 ; B low (2.72) +addr 203: 0002 ; B high +addr 204: 0000 ; opcode 0 (add) +``` + +### 3. UART Send Character + +Send 'A' (0x41) over UART: + +```subleq +; Wait for TX to be not busy, then send character + +; mem[300] = 0x41 ('A') +; mem[301] = 0 (zero) +; mem[302] = UART TX status address ($FF01) +; mem[303] = UART TX data address ($FF00) + +; Check TX status (poll loop) +addr 0: 302 304 3 ; mem[304] = mem[$FF01] +addr 3: 304 304 6 ; mem[304] -= mem[304] (test if zero) +addr 6: 304 304 9 ; if <= 0 (not busy), continue +addr 9: 301 304 0 ; else, reset and retry + +; Send character +addr 12: 300 303 15 ; mem[$FF00] = 0x41 + +; HALT +addr 15: 0 0 0 + +; Data +addr 300: 0041 ; 'A' +addr 301: 0000 ; zero +addr 302: FF01 ; UART TX status +addr 303: FF00 ; UART TX data +addr 304: 0000 ; temp +``` + +--- + +## Loading Programs via UART + +### Using Python + +```python +import serial +import struct + +# Load assembled program (array of 16-bit words) +program = [0x0064, 0x0066, 0x0003, ...] # assembled instructions + +# Connect to FPGA UART +ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=1) + +# Send program: each word as 2 bytes, big-endian +for word in program: + ser.write(struct.pack('>H', word)) + +ser.close() +``` + +### Using openFPGALoader (SRAM load) + +For quick iteration (non-persistent): + +```bash +# Load bitstream to SRAM (lost on power cycle) +openFPGALoader -b tangnano9k --sram research_stack_top.fs + +# Load to flash (persistent) +openFPGALoader -b tangnano9k research_stack_top.fs +``` + +--- + +## Reading Results + +### Via UART + +```python +import serial + +ser = serial.Serial('/dev/ttyUSB0', 115384, timeout=5) + +# Read result bytes +data = ser.read(2) # 1 word = 2 bytes +result = struct.unpack('>H', data)[0] +print(f"Result: {result} (0x{result:04X})") + +ser.close() +``` + +### Via LEDs + +Read the 6 LEDs (pins 10-16) as a 6-bit value from `led[0:5]`. +- LED 0 = bit 0 (rightmost) + +### Via Memory Dump + +After HALT, the UART TX beacon outputs the full memory contents. +Connect a serial terminal and observe the dump. + +--- + +## Build Toolchain + +```bash +# Synthesis +cd 4-Infrastructure/hardware && bash build_research_stack.sh + +# Simulation +cd /tmp/fpga_sim_full && ./obj_dir/sim_top + +# Flash +openFPGALoader -b tangnano9k research_stack_top.fs +``` + +### Tool Versions + +| Tool | Version | +|------|---------| +| Yosys | 0.64 | +| nextpnr-himbaechel | 0.10-75 | +| gowin_pack | latest | +| Verilator | 5.048 | +| openFPGALoader | latest | + +--- + +## Timing + +- **Clock:** 27 MHz (37.04 ns period) +- **Achieved Fmax:** 195.92 MHz (7.2× margin) +- **Q16 LUT latency:** 2 cycles (74 ns) +- **MAX_CYCLES:** 1,000,000 diff --git a/6-Documentation/RUNBOOK.md b/6-Documentation/RUNBOOK.md new file mode 100644 index 00000000..5c796203 --- /dev/null +++ b/6-Documentation/RUNBOOK.md @@ -0,0 +1,399 @@ +# Ops Runbook — Research Stack + +**Last updated:** 2026-05-29 + +Quick-reference operational procedures for k3s, FPGA, Tailscale, GPU, and DNS. + +--- + +## k3s Cluster + +### Start / Stop + +```bash +# Set kubeconfig +export KUBECONFIG=/tmp/researchstack-kubeconfig.yaml + +# Check cluster status +kubectl get nodes +kubectl get pods -A +``` + +**Control plane (nixos):** +```bash +# Restart k3s server +sudo systemctl restart k3s + +# Check k3s server health +sudo systemctl status k3s +sudo journalctl -u k3s -f --lines=50 +``` + +### Check Health + +```bash +# All nodes ready +kubectl get nodes -o wide + +# All pods running (any namespace) +kubectl get pods -A --field-selector='status.phase!=Running,status.phase!=Succeeded' + +# Specific namespace +kubectl get pods -n services +kubectl get pods -n media +kubectl get pods -n monitoring +kubectl get pods -n ai-models +``` + +### Restart Pods + +```bash +# Restart a deployment (rolling restart) +kubectl rollout restart deployment/ -n + +# Examples +kubectl rollout restart deployment/homer -n services +kubectl rollout restart deployment/ollama -n ai-models +kubectl rollout restart deployment/cluster-dashboard -n monitoring + +# Force delete stuck pod +kubectl delete pod -n --grace-period=0 --force +``` + +### View Logs + +```bash +# Pod logs +kubectl logs -n --tail=100 -f + +# Previous container (if crashed) +kubectl logs -n --previous + +# All pods matching label +kubectl logs -l app=