From 7fd228a130f177b59b12d3dfec7c07ead9f9f4b3 Mon Sep 17 00:00:00 2001 From: allaun Date: Sat, 20 Jun 2026 17:03:22 -0500 Subject: [PATCH] docs(agents): update qfox-1 local inference to Ollama Hermes 3 Document the switch from the unreachable k3s llama.cpp NodePort to a local Ollama service (ollama-hermes3.service) serving hermes3:latest. Hermes config now points at http://127.0.0.1:11434/v1. Note CPU fallback until a CUDA-enabled llama-server is available. --- AGENTS.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9fe58fba..4546a7a3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,22 +13,22 @@ fully-local Devstral model on qfox-1's RTX 4070. OpenClaw is decommissioned. - Run with `--insecure` (auth disabled) since the dashboard is on tailnet only. - Public access: `https://chat.researchstack.info` via Caddy on racknerd (`reverse_proxy http://100.85.244.73:9119`). -- **Inference**: llama.cpp server on qfox-1 reachable at - `http://100.88.57.96:30516/v1` (k3s NodePort, ClusterIP unreachable from host) - - Model: `Devstral-Small-2505-UD-IQ2_M.gguf` (Unsloth, ~8.2 GB) - - Context: 131072 tokens (auto-detected as 16384, manually bumped in config) -- **Hermes config**: `~/.hermes/config.yaml` uses the canonical schema: +- **Local inference on qfox-1**: Ollama serving `hermes3:latest` built from + `/var/lib/gemma-models/Hermes-3-Llama-3.1-8B-Q4_K_M.gguf`. + - Endpoint: `http://127.0.0.1:11434/v1` + - Systemd unit: `ollama-hermes3.service` + - Note: `ollama-cuda` is installed but the packaged `llama-server` binary + currently falls back to CPU inference. GPU acceleration can be enabled later + by reinstalling Ollama from the official installer or building a CUDA-enabled + `llama-server`. +- **Hermes config**: `~/.hermes/config.yaml` points at the local Ollama: ```yaml model: - default: "Devstral-Small-2505-UD-IQ2_M.gguf" - provider: "llamacpp" - base_url: "http://100.88.57.96:30516/v1" + default: "hermes3:latest" + provider: "openai" + base_url: "http://127.0.0.1:11434/v1" api_key: "sk-local" - context_length: 131072 ``` -- **Ollama** (port 11434 on qfox-1) also has the same GGUF imported as - `devstral:latest` for ad-hoc `ollama run` use, but is **not** wired into - Hermes. Keep it as a debugging escape hatch. ### Common failure modes