allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(agents): update qfox-1 local inference to Ollama Hermes 3

Browse source 
Document the switch from the unreachable k3s llama.cpp NodePort to a
local Ollama service (ollama-hermes3.service) serving hermes3:latest.
Hermes config now points at http://127.0.0.1:11434/v1. Note CPU fallback
until a CUDA-enabled llama-server is available.
This commit is contained in:
allaun 2026-06-20 17:03:22 -05:00
parent 119bdb22c5
commit d2e8db21c3
1 changed files with 12 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
AGENTS.md
View file

@ -13,22 +13,22 @@ fully-local Devstral model on qfox-1's RTX 4070. OpenClaw is decommissioned.
- Run with `--insecure` (auth disabled) since the dashboard is on tailnet only.
- Public access: `https://chat.researchstack.info` via Caddy on racknerd
(`reverse_proxy http://100.85.244.73:9119`).
- **Inference**: llama.cpp server on qfox-1 reachable at
`http://100.88.57.96:30516/v1` (k3s NodePort, ClusterIP unreachable from host)
- Model: `Devstral-Small-2505-UD-IQ2_M.gguf` (Unsloth, ~8.2 GB)
- Context: 131072 tokens (auto-detected as 16384, manually bumped in config)
- **Hermes config**: `~/.hermes/config.yaml` uses the canonical schema:
- **Local inference on qfox-1**: Ollama serving `hermes3:latest` built from
`/var/lib/gemma-models/Hermes-3-Llama-3.1-8B-Q4_K_M.gguf`.
- Endpoint: `http://127.0.0.1:11434/v1`
- Systemd unit: `ollama-hermes3.service`
- Note: `ollama-cuda` is installed but the packaged `llama-server` binary
currently falls back to CPU inference. GPU acceleration can be enabled later
by reinstalling Ollama from the official installer or building a CUDA-enabled
`llama-server`.
- **Hermes config**: `~/.hermes/config.yaml` points at the local Ollama:
```yaml
model:
default: "Devstral-Small-2505-UD-IQ2_M.gguf"
provider: "llamacpp"
base_url: "http://100.88.57.96:30516/v1"
default: "hermes3:latest"
provider: "openai"
base_url: "http://127.0.0.1:11434/v1"
api_key: "sk-local"
context_length: 131072
```
- **Ollama** (port 11434 on qfox-1) also has the same GGUF imported as
`devstral:latest` for ad-hoc `ollama run` use, but is **not** wired into
Hermes. Keep it as a debugging escape hatch.
### Common failure modes