Research-Stack/6-Documentation/wiki/Credential-System.md

# Research Stack Credential System

> **Canonical source**: `4-Infrastructure/infra/ene-session-sync/src/credential.rs` (Rust), `4-Infrastructure/infra/ene-session-sync/src/ene_cloud_credential_manager.rs` (Rust), `4-Infrastructure/infra/recover_credential_server.sh` (deployment script)
> **Runtime host**: `microvm-racknerd` (100.101.247.127) — Debian 13 VM on RackNerd
> **Primary backend**: PostgreSQL (configured via `RDS_*` environment variables)
> **Fallback chain**: RDS → remote server → local JSON → environment variables
> **SOPS key**: `age1tp4vr565zkmvnyulatpyaj6z8zrz7q9mpaypz85yz8rty99crdasualxyr`

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────┐
│                        CONSUMERS (any node)                         │
│  qfox-1    nixos-laptop    361395-1    microvm-racknerd          │
│     │           │              │                │                    │
│     └───────────┴──────────────┴────────────────┘                    │
│                      HTTP GET /credentials/:provider                │
│                              │                                      │
│                    ┌─────────▼─────────┐                            │
│                    │  microvm-racknerd │  ← Credential Server     │
│                    │   (port 8444)      │   (Python, systemd)      │
│                    └─────────┬─────────┘                            │
│                              │                                      │
│              ┌───────────────┼───────────────┐                    │
│              │               │               │                      │
│     ┌────────▼────────┐ ┌───▼────────┐ ┌────▼───────┐             │
│     │  PostgreSQL     │ │  Remote    │ │  Local     │             │
│     │  (primary)      │ │  Server    │ │  JSON      │             │
│     │                 │ │  (chain)   │ │  (backup)  │             │
│     └────────┬────────┘ └────────────┘ └────────────┘             │
│              │                                                      │
│     ┌────────▼────────┐                                            │
│     │ credential_store│                                            │
│     │ .credentials    │  12 active records                        │
│     │ .access_log     │  0 entries (not yet enabled)              │
│     └─────────────────┘                                            │
└─────────────────────────────────────────────────────────────────────┘
```

## Four-Tier Fallback Chain

The `credential_provider.py` (`v0.4`) implements a strict priority order. The first non-empty result wins; all lower tiers are skipped.

| Tier | Function | Trigger | Data Source |
|------|----------|---------|-------------|
| **1** | `_load_from_rds()` | Always runs first | PostgreSQL `credential_store.credentials` |
| **2** | `_load_from_remote()` | Skipped if RDS empty AND `RS_CREDENTIAL_SERVER` points to self | Another credential server URL |
| **3** | `_load_from_config()` | Skipped if tiers 1–2 empty | `/etc/rs-surface/credentials.json` |
| **4** | `_load_from_env()` | Last resort | Environment variables per `PROVIDER_ENV_MAP` |

### Why PostgreSQL is the primary source

PostgreSQL credentials are **encrypted at rest** (AES-256-GCM payload) and **auditable** (`access_log` table). The local JSON file (`/etc/rs-surface/credentials.json`) is a plaintext fallback for disaster recovery only.

**Current status** (2026-05-21): Tier 1 (RDS) is active and serving all 11 credentials. Tier 3 (local JSON) exists as a warm standby but is not consulted because RDS succeeds.

```bash
# Verify from any Tailnet node
curl -s http://100.101.247.127:8444/status | jq .
# Expected: {"backend": "rds", "count": 11, "ok": true}
```

## RDS Schema

### `credential_store.credentials`

| Column | Type | Purpose |
|--------|------|---------|
| `id` | uuid | Primary key |
| `pkg` | text | Hierarchical key name, e.g. `credentials/deepseek` |
| `provider` | text | Human-readable provider name, e.g. `deepseek` |
| `encrypted_payload` | bytea | AES-256-GCM ciphertext |
| `nonce` | bytea | GCM nonce |
| `classification` | smallint | `2` = internal, `3` = secret |
| `integrity_hash` | text | SHA-256 of decrypted payload |
| `node_assignments` | text[] | Which nodes may request this credential |
| `created_at` | timestamptz | Insert timestamp |
| `rotated_at` | timestamptz | Last rotation timestamp |
| `access_count` | bigint | Number of successful fetches |
| `is_active` | boolean | Soft-delete flag |

### `credential_store.access_log`

| Column | Type | Purpose |
|--------|------|---------|
| (table empty as of 2026-05-21 — not yet wired to the fetch path) |

### Active credentials (12)

| Provider | Classification | pkg | Notes |
|----------|---------------|-----|-------|
| bedrock | 3 | credentials/bedrock | |
| deepseek | 3 | credentials/deepseek | |
| linear | 3 | credentials/linear | |
| notion | 3 | credentials/notion | |
| ollama | 2 (internal) | credentials/ollama | |
| porkbun | 3 | credentials/porkbun | DNS provider |
| quandela | 3 | credentials/quandela | Quantum cloud |
| **racknerd_ssh** | 3 | credentials/racknerd_ssh | VM root password |
| venice | 2 | credentials/venice | |
| wolfram_alpha | 2 | credentials/wolfram_alpha | |
| **authentik** | 3 | credentials/authentik | LLM controller token for Authentik API |

## Authentication to PostgreSQL

Use a standard libpq connection string built from `RDS_*` environment variables:

```python
import os
import psycopg2

host = os.environ.get('RDS_HOST', 'localhost')
port = os.environ.get('RDS_PORT', '5432')
user = os.environ.get('RDS_USER', 'postgres')
password = os.environ.get('RDS_PASSWORD', '')
dbname = os.environ.get('RDS_DB', 'postgres')

conn = psycopg2.connect(
    host=host, port=port, user=user,
    password=password, dbname=dbname,
    sslmode='require', connect_timeout=10
)
```

**Prerequisites on the client node:**
- `psycopg2` installed
- `libpq5` (Debian) or equivalent PostgreSQL C client library
- Network reachability to the configured `RDS_HOST`

### Fallback: `RDS_PASSWORD` environment variable

If no password is configured, the provider falls back to `RDS_PASSWORD` (plain text). This is **not recommended** for production.

## MicroVM-Racknerd Deployment

### File layout

```
/opt/rs-surface/
├── credential_server.py          # HTTP server (v0.4)
├── credential_provider.py        # 4-tier fallback logic (v0.4)
└── __pycache__/

/etc/rs-surface/
├── credentials.json               # Local fallback (plaintext, 9 providers)
└── node.json                      # Node identity metadata

/opt/credential-provider/          # STALE — v0.2 code, renamed
├── credential_provider.py.v0.2.stale
└── server.py.v0.2.stale
```

### Systemd service

```ini
# /etc/systemd/system/rs-credential-server.service
[Unit]
Description=Research Stack Credential Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/rs-surface
EnvironmentFile=/opt/credential-provider/.env
Environment=RS_CREDENTIAL_SERVER=        ; empty = disable self-query
ExecStart=/usr/bin/python3 /opt/rs-surface/credential_server.py --port 8444 --bind 0.0.0.0
Restart=always
RestartSec=5
Environment=RS_CREDENTIAL_CONFIG=/etc/rs-surface/credentials.json

[Install]
WantedBy=multi-user.target
```

### Environment file (`/opt/credential-provider/.env`)

This file contains raw API keys for the **env-var fallback tier only**. Since RDS is working, these env vars are not consulted by the running service. They exist for disaster recovery and for scripts that bypass the credential server.

```
RS_SURFACE_NODE_ID=MicroVM-Racknerd
RS_SURFACE_PROFILE=/etc/rs-surface/node.json
DEEPSEEK_API_KEY=sk-...
QUANDELA_API_KEY=_T_eyJ...
WOLFRAM_ALPHA_APPID=HYJE3R3R63
# ... etc
```

**Warning**: The `.env` file is **plaintext** and should be rotated out of existence once the RDS tier is proven stable on all consumers.

## Health Checks

```bash
# From any Tailnet node
curl -s --max-time 5 http://100.101.247.127:8444/health
curl -s --max-time 5 http://100.101.247.127:8444/status | jq .
curl -s --max-time 5 http://100.101.247.127:8444/credentials | jq '.providers[].name'
curl -s --max-time 5 http://100.101.247.127:8444/credentials/deepseek | jq .

# From microvm itself
systemctl status rs-credential-server --no-pager
journalctl -u rs-credential-server --since today --no-pager
```

## Adding a New Credential

1. **Encrypt with SOPS** (on qfox-1):
   ```bash
   sops --encrypt --in-place new_secret.json
   # or edit an existing encrypted file:
   sops 4-Infrastructure/infra/secrets/credentials.json
   ```

2. **Insert into PostgreSQL** (from a node with network access):
   ```python
   import psycopg2
   # ... connect using RDS_* env vars ...
   cur.execute("""
       INSERT INTO credential_store.credentials
       (pkg, provider, encrypted_payload, nonce, classification, integrity_hash, is_active)
       VALUES (%s, %s, %s, %s, %s, %s, TRUE)
   """, (pkg, provider, ciphertext, nonce, 3, integrity_hash))
   ```

3. **Restart the credential server** (to pick up the change):
   ```bash
   systemctl restart rs-credential-server
   ```

4. **Verify**:
   ```bash
   curl -s http://100.101.247.127:8444/credentials/<provider> | jq .
   ```

## Disaster Recovery

If RDS is unreachable AND the credential server is down, the local JSON fallback on microvm contains the same credentials (minus the new `racknerd_ssh` entry, which only exists in RDS). The JSON file is:

```bash
# On microvm
cat /etc/rs-surface/credentials.json
```

If microvm itself is destroyed, the credentials are also in SOPS-encrypted files on qfox-1:

```bash
# On qfox-1
sops -d 4-Infrastructure/infra/secrets/credentials.json
```

## Historical Note: v0.2 → v0.4 Migration

The original credential system (`/opt/credential-provider/`, v0.2) had a **hardcoded database token** in `credential_provider.py` with an expiration date (2026-05-17). When that token expired, `_load_from_rds()` silently returned `[]`, causing the service to fall through to environment variables. This was the "bug" that appeared to be the database not working, but was actually an expired token in stale code.

The fix was:
1. Deploy `credential_provider.py` v0.4 to `/opt/rs-surface/` (reads `RDS_PASSWORD` or `RDS_IAM_TOKEN` from environment)
2. Point systemd `ExecStart` at `/opt/rs-surface/credential_server.py`
3. Rename old v0.2 files with `.v0.2.stale` suffix

## TODO

- [ ] Wire `access_log` insert on every successful credential fetch
- [ ] Add `node_assignments` enforcement (only allowlisted Tailscale nodes to request secrets)
- [ ] Rotate `.env` file out of existence (env-var tier is a liability)
- [ ] Encrypt local JSON fallback with SOPS instead of plaintext
- [ ] Add credential server to nixos-laptop as a hot standby