Research-Stack/4-Infrastructure/storage/ZFS_PREFLIGHT.md

ZFS Pre-Flight Checklist

Machine: qfox-1 (100.88.57.96)
Target kernel: 7.0.9-1-cachyos (currently running 7.0.8-1-cachyos — one patch behind)
Pool name: stackcache
Vdev: sparse file at /var/lib/stackcache/pool.img (500 G) on /dev/nvme0n1p2 (btrfs)
Script: 4-Infrastructure/storage/zfs-pool-setup.sh
Status: ZFS module NOT loaded; reboot required first.

---

1. Block Device Inventory

NAME         SIZE  TYPE  MOUNTPOINT  FSTYPE
zram0        30.4G disk  [SWAP]      swap
nvme0n1       1.8T disk
├─nvme0n1p1    4G  part  /boot       vfat
└─nvme0n1p2  1.8T  part  /           btrfs

Available space on btrfs root (/dev/nvme0n1p2):

Size: 1.9 TB   Used: 174 GB   Avail: 1.7 TB   Use%: 10%

There is only one NVMe. No raw partitions or spare disks are available.
The correct approach is a ZFS sparse-file vdev carved from the existing btrfs filesystem — exactly what zfs-pool-setup.sh does. Btrfs + ZFS file vdevs coexist safely; ZFS handles its own checksumming and compression independently of btrfs zstd:1.

---

2. ZFS Module Status

modinfo zfs → ERROR: Module zfs not found.
zpool list  → The ZFS modules cannot be auto-loaded.

Action required: reboot into the 7.0.9-1-cachyos kernel once it is installed.
Verify after reboot with:

uname -r                          # expect 7.0.9-1-cachyos
lsmod | grep "^zfs"               # must print a zfs line
modinfo zfs | head -3             # shows version

---

3. Exact Commands to Run After Reboot

Run these in order as root after confirming lsmod | grep "^zfs" is non-empty.

# 1. Confirm kernel and module
uname -r                          # 7.0.9-1-cachyos
lsmod | grep "^zfs"

# 2. Confirm free space (need ≥ 500 G available on btrfs)
df -h /var/lib                    # ≥ 500 G avail

# 3. Run the setup script (idempotent — safe to run again if pool exists)
sudo bash 4-Infrastructure/storage/zfs-pool-setup.sh

# 4. Verify pool came up
zpool status stackcache
zfs list -r stackcache

# 5. Confirm systemd services are enabled
systemctl is-enabled zfs-import-cache.service zfs-import-scan.service \
    zfs-mount.service zfs.target

# 6. Smoke-test thermal zone mounts
ls /mnt/stackcache/hot/db
ls /mnt/stackcache/warm/pgdump
ls /mnt/stackcache/warm/rclone
ls /mnt/stackcache/cold/snap

# 7. Confirm quotas
zfs get quota stackcache/hot/db
zfs get quota stackcache/warm/pgdump
zfs get quota stackcache/warm/rclone
zfs get quota stackcache/cold/snap

---

4. Pool Layout Recommendation

Single NVMe — no redundancy is possible with one disk. The sparse-file approach is the right call: it avoids repartitioning and gives ZFS its own namespace on top of btrfs.

stackcache          (500 G sparse file vdev, ashift=12, lz4 global)
├── hot/            (logbias=latency, primarycache=all, sync=standard)
│   └── hot/db      quota=150G, recordsize=16K   ← SQLite scratch
├── warm/           (logbias=throughput, primarycache=metadata, sync=disabled)
│   ├── warm/pgdump quota=150G, recordsize=128K, zstd-3  ← pg_dump / RDS
│   └── warm/rclone quota=50G,  recordsize=32K            ← rclone VFS cache
└── cold/           (logbias=throughput, primarycache=none, zstd-3, sync=disabled)
    └── cold/snap   quota=100G, recordsize=128K            ← ZFS send/recv

Total quotas: 450 G on a 500 G pool → 50 G headroom for pool metadata, snapshot churn, and ZFS internal bookkeeping.

Boot re-import path: zfs-import-cache.service reads /etc/zfs/zpool.cache (written by zpool create -o cachefile=). The scan-based fallback (zfs-import-scan.service) is also enabled for resilience.

---

5. Issues Found in zfs-pool-setup.sh and Fixes Applied

All issues were fixed in-place. bash -n syntax check passes.

Issue 1 — No thermal-zone hierarchy (structural, fixed)

Before: Pool had flat datasets stackcache/db, stackcache/pgdump, stackcache/rclone, stackcache/snap.
After: Datasets are nested under thermal-zone parents hot/, warm/, cold/ matching the UCM substrate schema:

stackcache/hot/db
stackcache/warm/pgdump
stackcache/warm/rclone
stackcache/cold/snap

Thermal-zone parents carry the tuning properties (logbias, primarycache, sync) so child datasets inherit them, reducing per-dataset boilerplate and making the routing intent explicit.

Issue 2 — Quota arithmetic left zero pool headroom (correctness, fixed)

Before: db=200G + pgdump=200G + rclone=50G + snap=50G = 500G — exactly the pool size. This left no free space for pool metadata, snapshot bookkeeping, or ZFS internal structures, which would cause ENOSPC during normal operation.
After: hot/db=150G + warm/pgdump=150G + warm/rclone=50G + cold/snap=100G = 450G, leaving 50 G (10%) free.

Issue 3 — cachefile not set on pool creation (reliability, fixed)

Before: zpool create had no -o cachefile= argument. File-vdev pools are not automatically found by zfs-import-scan unless the vdev path is known at scan time, which is fragile.
After: -o cachefile=/etc/zfs/zpool.cache is passed to zpool create, and zfs-import-cache.service is now enabled alongside zfs-import-scan.service. The cache file is the canonical, reliable import path for file-vdev pools.

Issue 4 — zfs-import-cache.service not enabled (reliability, fixed)

Before: Only zfs-import-scan.service was enabled.
After: Both zfs-import-cache.service and zfs-import-scan.service are enabled. Cache-based import runs first; scan is the fallback.

Non-issues confirmed

• Idempotency: zpool list "${POOL_NAME}" &>/dev/null guard at top — already idempotent (prints status and exits 0 if pool exists). ✓
• Device paths: No hard-coded /dev/sdX paths — uses a sparse file under /var/lib/stackcache/. The btrfs root has 1.7 TB free, well above the 500 G vdev. ✓
• set -euo pipefail: Present. ✓
• ZFS module check: lsmod | grep "^zfs" guard present. ✓
• || true on systemctl enable: Correct — prevents set -e abort if a service unit name differs slightly across kernel/distro versions. ✓

---

6. Thermal Routing Score → ZFS Dataset Placement

The UniversalThermalRouter.compute_thermal_signature() function in .devcontainer/Universal Computational Modeling.md produces a score in [0, 1]:

score += 0.30  if urgency=True          (real-time required)
score += 0.02  per active_region        (spatial locality, up to 0.20)
score += 0.20  if char_time < 1e-3      (fast dynamics, fs/ps timescale)
score += 0.15  if scar_density > 0.5    (historically failure-prone → demote)
score += 0.15  if parallel_efficiency < 0.5  (poor scaling → batch)

Score range	Zone	ZFS dataset path	Garage / restic role
score < 0.2	hot	stackcache/hot/db	— (local scratch only)
0.2 ≤ score < 0.7	warm	stackcache/warm/pgdump or warm/rclone	Garage S3 batch (db-scratch, rds-overflow)
score ≥ 0.7	cold	stackcache/cold/snap → then offload	restic snap → Garage snap-zone → gdrive cold copy

Concrete routing examples

Workload	Score drivers	Score	Zone → Dataset
Active SQLite DB write	urgency=T, char_time<1ms	0.50	warm → warm/pgdump or hot/db*
Interactive Lean proof search	urgency=T, active_regions>5	0.40–0.60	warm → warm/rclone (VFS)
pg_dump / RDS COPY TO	urgency=F, large sequential	0.15–0.30	warm → warm/pgdump
ZFS send/recv from node	urgency=F, rare, archival	0.70+	cold → cold/snap
restic snapshot chunks	urgency=F, background	0.70+	cold → Garage research-stack

*SQLite scratch sits at the hot/warm boundary. Route to hot/db when urgency=True AND char_time < 1ms; otherwise warm/pgdump is acceptable.

---

7. Recommended storage_agent.py Changes for ZFS Thermal Zones

These are recommendations only — not applied here, as storage_agent.py already passes py_compile and is in active use.

7.1 Add ZFS observation probe

storage_agent.py currently observes Garage and restic but has no ZFS awareness. After the pool is live, add a _probe_zfs(obs) method that calls zpool list -H -o name,health,allocated,free stackcache and adds:

obs.zfs_pool_health: str        # "ONLINE" / "DEGRADED" / absent
obs.zfs_hot_used_pct_q16: int   # Q16_16 fraction of hot/db quota used
obs.zfs_warm_used_pct_q16: int
obs.zfs_cold_used_pct_q16: int

7.2 Add ZFS routing in the decide() function

When a ZFS zone exceeds ~80% of quota (0xCCCC in Q16_16 ≈ 0.8), trigger the appropriate offload:

# trigger_zfs_hot_evict: when hot/db > 80% quota
#   → mv stackcache/hot/db/... → warm, or flush to Garage db-scratch
# trigger_zfs_cold_offload: when cold/snap > 80% quota
#   → db-consolidate.sh offload to Garage snap-zone

7.3 Route storage_agent.py offloads through thermal paths

Currently trigger_offload always calls db-consolidate.sh offload which pushes to Garage db-scratch. After ZFS is live, the agent could:

1. Write hot SQLite scratchwork to stackcache/hot/db
2. On trigger_offload, first snapshot: zfs snapshot stackcache/warm/pgdump@agent-<ts>
3. Then send to Garage snap-zone via zfs send | aws s3 cp rather than plain rclone.

This keeps the thermal routing provenance in the ZFS snapshot lineage.

7.4 Add snap-zone bucket observation

obs.garage_buckets already lists buckets. Add a check that snap-zone exists once ZFS is live (it is already defined in AGENTS.md), and alert if missing:

if "snap-zone" not in obs.garage_buckets:
    d.alerts.append("WARN: Garage snap-zone bucket missing — ZFS send/recv will fail")

---

8. Location Note

AGENTS.md (Infrastructure) lists zfs-pool-setup.sh under 4-Infrastructure/storage/garage/ in the Garage scripts table. The file actually lives at 4-Infrastructure/storage/zfs-pool-setup.sh (one level up). This is a documentation inconsistency — the file location is correct; the table entry in AGENTS.md should be updated to point to storage/zfs-pool-setup.sh.