diff --git a/AGENTS.md b/AGENTS.md index 9fe58fba..4546a7a3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,22 +13,22 @@ fully-local Devstral model on qfox-1's RTX 4070. OpenClaw is decommissioned. - Run with `--insecure` (auth disabled) since the dashboard is on tailnet only. - Public access: `https://chat.researchstack.info` via Caddy on racknerd (`reverse_proxy http://100.85.244.73:9119`). -- **Inference**: llama.cpp server on qfox-1 reachable at - `http://100.88.57.96:30516/v1` (k3s NodePort, ClusterIP unreachable from host) - - Model: `Devstral-Small-2505-UD-IQ2_M.gguf` (Unsloth, ~8.2 GB) - - Context: 131072 tokens (auto-detected as 16384, manually bumped in config) -- **Hermes config**: `~/.hermes/config.yaml` uses the canonical schema: +- **Local inference on qfox-1**: Ollama serving `hermes3:latest` built from + `/var/lib/gemma-models/Hermes-3-Llama-3.1-8B-Q4_K_M.gguf`. + - Endpoint: `http://127.0.0.1:11434/v1` + - Systemd unit: `ollama-hermes3.service` + - Note: `ollama-cuda` is installed but the packaged `llama-server` binary + currently falls back to CPU inference. GPU acceleration can be enabled later + by reinstalling Ollama from the official installer or building a CUDA-enabled + `llama-server`. +- **Hermes config**: `~/.hermes/config.yaml` points at the local Ollama: ```yaml model: - default: "Devstral-Small-2505-UD-IQ2_M.gguf" - provider: "llamacpp" - base_url: "http://100.88.57.96:30516/v1" + default: "hermes3:latest" + provider: "openai" + base_url: "http://127.0.0.1:11434/v1" api_key: "sk-local" - context_length: 131072 ``` -- **Ollama** (port 11434 on qfox-1) also has the same GGUF imported as - `devstral:latest` for ad-hoc `ollama run` use, but is **not** wired into - Hermes. Keep it as a debugging escape hatch. ### Common failure modes