Everything that was done to build Jarvis from nothing on this lab, in the order it was done, with the commands, the files and the reasons, so that someone who reads only this page can rebuild it or extend it. Companion pages: the plan and progress log Jarvis: goal, plan and progress log, the technical reference Jarvis, how to use it User Guide, the traps we fell into Lessons Learned. The same text lives with the code in the Jarvis container:
/opt/jarvis/docs/BUILD-GUIDE.md. Written 2026-10-11 for bridge v0.9.
This is not a beginner tutorial: it assumes you can run Proxmox VE, write a systemd unit and read Python. Every command was actually run on this cluster; where something is a judgement call the reason is given, because the reasons matter more than the commands when your lab differs.
A locally hosted assistant in one LXC (jarvis, id 130, 192.168.86.226) that you talk to from your phone (or later by voice). It answers questions about the lab and the house from live data, keeps a notebook, reminders and a shopping list, and can start, stop and restart guests, edit the backup job, run a command in a container, and create/check off/delete projects in OpenProject, but only after you say yes to anything consequential. The language model runs on the CPU of one node; nothing goes to a hosted AI service.
| Piece | Where | What it is |
|---|---|---|
jarvis-brain |
LXC 130, loopback :8080 | llama.cpp llama-server with Gemma 4 26B-A4B (Q4_0, about 14 GB), CPU only, about 25 tokens/s |
jarvis-bridge |
LXC 130 | one Python program (standard library only): chat over ntfy, commands, tools, guards, the policy gate, approvals, reminders, the daily briefing, /watch, a loopback API |
jarvis-web |
LXC 130, LAN :8765 | the only LAN-facing part: a login, a phone chat page with approve buttons, and the voice endpoint |
jarvis-stt, jarvis-tts |
LXC 130 (off) | Wyoming speech-to-text (Whisper) and text-to-speech (Piper), installed and measured, not exposed |
jarvis-exec |
on every Proxmox node | the one command Jarvis' ssh key may run |
| ntfy | LXC 127 (existing) | the chat transport |
These are the decisions that shaped everything. Each was measured or learned the hard way (see section 18).
approvals table. The row is executed only after your own chat message says yes: not a message the bot wrote, not a stale one, once, with everything re-checked at that moment.VM.PowerMgmt and no access at all to the chat server (LXC 127) and Jarvis itself (LXC 130); the ssh key can run one script that accepts five verbs; the web service holds no secrets except its own.pct stop 130. Nothing else depends on Jarvis.| Need | Here |
|---|---|
| Proxmox VE (a cluster is not required) | VE 9.2, three nodes; Jarvis on SuperPC |
| A CPU with AVX2 (AVX-512 is better) | Ryzen 9 9900X (Zen 5), 60 GB RAM, 28 GB given to Jarvis |
| Shared or local storage for model files | CephFS directory /mnt/pve/cephfs/jarvis (re-downloadable, not backed up) |
| A chat transport you already run | ntfy (self-hosted), reachable at http://192.168.86.202:8080 and publicly via the reverse proxy |
| Things to read | Home Assistant (a token), Prometheus/Uptime Kuma, Frigate, a wiki, OpenProject |
| Time | built across three working sessions, 2026-10-09 to 2026-10-11: the chat assistant with read-only eyes first, then the approvals, then the web page, voice groundwork and OpenProject |
Why one unprivileged LXC with nesting: it is a boundary you can stop in one command, it cannot touch the host, and the CPU model needs no GPU. Why cpuunits 50: Frigate and Ceph on the same host must win under contention. Why swap 0: swapped model pages ruin speed.
On the Proxmox host:
# model directory on CephFS, owned by the unprivileged container's root mapping (uid/gid 100000)
mkdir -p /mnt/pve/cephfs/jarvis && chown 100000:100000 /mnt/pve/cephfs/jarvis
pct create 130 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \
--hostname jarvis --ostype debian --arch amd64 \
--description "Jarvis local AI assistant. Stop this LXC = kill switch." \
--cores 12 --memory 28672 --swap 0 --cpuunits 50 \
--rootfs boot-disks:30 \
--mp0 /mnt/pve/cephfs/jarvis,mp=/srv/models,backup=0 \
--net0 name=eth0,bridge=vmbr0,firewall=1,gw=192.168.86.1,ip=192.168.86.226/24,type=veth \
--nameserver 8.8.8.8 --unprivileged 1 --features nesting=1,keyctl=1 \
--onboot 0 --tags "jarvis;llm" --start 0
pct start 130
(backup=0 keeps 14+ GB of re-downloadable weights out of the backup job. The root disk is on Ceph: model decoding never touches disk, only loading does, so the slow replicated pool is fine. Expect a cold start to take about 2 minutes.)
Inside the container:
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get install -y --no-install-recommends build-essential cmake git curl ca-certificates \
libcurl4-openssl-dev pkg-config python3 python3-venv python3-pip jq sqlite3 htop vmtouch ccache openssl
useradd --system --no-create-home --shell /usr/sbin/nologin jarvis-llm
useradd --system --no-create-home --shell /usr/sbin/nologin jarvis-bridge
mkdir -p /etc/jarvis/secrets /opt/jarvis/{bin,bridge,docs,eval,ops,tests} && chmod 700 /etc/jarvis/secrets
Then the secret helper, so no secret is ever typed on a command line or printed (/opt/jarvis/bin/set-secret):
#!/bin/bash
# usage: set-secret <name> Reads a secret hidden from the terminal and stores it root-only in /etc/jarvis/secrets/<name>.
set -eu
name="${1:-}"; [[ "$name" =~ ^[a-z0-9_]+$ ]] || { echo "usage: set-secret <name>"; exit 1; }
dir=/etc/jarvis/secrets; mkdir -p "$dir"; chmod 700 "$dir"
read -r -s -p "Paste the value for '$name' (hidden), then Enter: " v; echo
[ -n "$v" ] || { echo "Empty value, nothing saved."; exit 1; }
umask 077; printf '%s' "$v" > "$dir/$name"; chmod 600 "$dir/$name"
echo "Saved '$name' ($(wc -c < "$dir/$name") characters)"
Verify: pct exec 130 -- nproc shows 12, free -m shows about 28 GB, touch /srv/models/.w works.
mkdir -p /opt/src && cd /opt/src
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DLLAMA_CURL=ON -DGGML_CCACHE=ON
nice -n 10 cmake --build build --config Release -j 10 --target llama-server llama-bench llama-cli llama-gguf-split
GGML_NATIVE=ON makes it use every instruction set this CPU has (AVX-512 on Zen 5). Never run language models on the integrated GPU of this host: it hung once and needed a hard reboot.
Candidates were mixture-of-experts models because only a few billion parameters are active per token, which is what a CPU's memory bandwidth can feed. Measured here with llama-bench (8 threads, flash attention on):
| Model (GGUF) | Size | Prompt processing (tokens/s) | Generation (tokens/s) | Tool-call quality |
|---|---|---|---|---|
| Qwen3.5-4B Q4_K_M | 2.5 GiB | 261 | 20 | good, but weaker at following long rules |
| gpt-oss-20b MXFP4 | 11.3 GiB | 142 | 26 | decent |
| Gemma 4 26B-A4B Q4_0 | 13.6 GiB | 209 | 25 | best in our tool-call evaluation |
Rules of thumb that held: (a) 8 threads beat 12 (the memory bus is the limit, and Ceph and Frigate need the rest), (b) prompt processing, not generation, is the bottleneck because the tool definitions and the system prompt are about 3,000 tokens, so keep the start of the prompt identical between requests so the server can reuse its cache, (c) thinking mode off (--reasoning off) for an assistant.
Download (resumable) with /opt/jarvis/bin/fetch-models.sh:
dest=/srv/models/gguf; mkdir -p $dest
get() { curl -fL --retry 5 --retry-delay 5 -C - -o "$dest/$2" "https://huggingface.co/$1/resolve/main/$2"; }
get ggml-org/gemma-4-26B-A4B-it-GGUF gemma-4-26B-A4B-it-Q4_0.gguf
/etc/systemd/system/jarvis-brain.service:
[Unit]
Description=Jarvis brain (llama.cpp, Gemma 4 26B-A4B, CPU)
After=network-online.target
Wants=network-online.target
[Service]
User=jarvis-llm
Group=jarvis-llm
ExecStart=/opt/src/llama.cpp/build/bin/llama-server \
-m /srv/models/gguf/gemma-4-26B-A4B-it-Q4_0.gguf --alias jarvis-brain \
-t 8 -c 16384 -np 2 --jinja -fa on --reasoning off \
--cache-reuse 256 --cache-ram 1024 --ctx-checkpoints 3 \
--host 127.0.0.1 --port 8080
Restart=on-failure
RestartSec=5
TimeoutStartSec=420
Nice=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
[Install]
WantedBy=multi-user.target
What the flags do: -np 2 two parallel slots (a reply and a forced tool call can overlap), -c 16384 total context, --jinja the model's own tool-call template, --cache-reuse 256 --cache-ram 1024 --ctx-checkpoints 3 keep the processed prompt prefix so the second message of a conversation does not re-read 3,000 tokens, --host 127.0.0.1 nothing else can talk to it.
systemctl daemon-reload && systemctl enable --now jarvis-brain
curl -s http://127.0.0.1:8080/health # {"status":"ok"} after ~2 min cold, ~12 s with the file cached
Verify: a chat completion with a tool definition returns a tool_calls entry. (eval/toolcall_eval.py does this systematically; it is how the model was chosen.) Expect about 25 tokens/s and about 15 GB of RAM.
One shared ntfy topic, jarvis, that you and the bot both read and write. Why one topic: it is the simplest thing that works and the phone already shows it. The bot tags its own messages from-jarvis and ignores anything carrying that tag.
On the ntfy host (LXC 127, ntfy runs in Docker):
# the bot's password is generated inside the Jarvis container and travels via stdin only
pct exec 130 -- bash -c 'umask 077; f=/etc/jarvis/secrets/ntfy_bot_password; [ -s $f ] || openssl rand -base64 30 | tr -d "/+=\n" | cut -c1-32 > $f; chmod 600 $f'
pct exec 130 -- cat /etc/jarvis/secrets/ntfy_bot_password | pct exec 127 -- docker exec -i ntfy sh -c 'NTFY_PASSWORD="$(cat)" ntfy user add --role=user jarvis-bot'
pct exec 127 -- docker exec ntfy ntfy access jarvis-bot jarvis read-write
pct exec 127 -- docker exec ntfy ntfy access cody jarvis read-write # your own account
ntfy's default per-client rate limit is too low for a chatty bot; the compose file got NTFY_VISITOR_REQUEST_LIMIT_BURST=150 and NTFY_VISITOR_REQUEST_LIMIT_REPLENISH=2s (a backup of the compose file was kept next to it).
Phone note: the ntfy iPhone app can only receive; it has no message box. Type from the ntfy web app (https://ntfy.codycodez.com/app, add it to the home screen) or, from v0.9, from the Jarvis web page (section 11). The Android app can send.
The bridge is plain Python with no third-party packages (nothing to compromise but our own code). It lives in /opt/jarvis/bridge, owned by root and run as python3 -I by an unprivileged user, so the service cannot edit its own code.
| File | Role |
|---|---|
ntfy_bridge.py |
subscribes to the topic; slash commands; the tool list and the policy gate (dispatch); guards and the claim verifier; the approval flow; reminders, briefing, /watch; the loopback API plumbing |
jarvis_tools.py |
weather (Open-Meteo, keyless), Uptime Kuma via Prometheus, Frigate, a safe calculator, the reminders store |
jarvis_lab.py |
read-only Proxmox, Home Assistant, wiki index, host sensors, the /problems roll-up |
jarvis_notes.py |
the sandboxed, git-versioned markdown notebook |
jarvis_actions.py |
tier T2: proposals, the approvals table, executing an approved row |
jarvis_api.py |
the chat log and the loopback API the web service talks to |
jarvis_projects.py |
OpenProject |
/etc/jarvis/policy.json is re-read on every message. A tool exists for the model only if it is listed there as T0 or T1 with mode auto (or as the T2 proposal tool with mode ask). Excerpt:
{ "limits": { "messages_per_hour": 60, "max_tool_calls_per_message": 8, "note_writes_per_hour": 20, "project_writes_per_hour": 30 },
"tools": { "get_cluster_status": {"tier":"T0","mode":"auto"}, "save_note": {"tier":"T1","mode":"auto"},
"project_request": {"tier":"T1","mode":"auto"}, "propose_action": {"tier":"T2","mode":"ask"} },
"actions": { "modes": { "start":"ask", "restart":"ask", "run_command":"ask", "project_delete":"ask" },
"protected_guests": [127,130], "exec_denied_guests": [130], "approval_ttl_s": 900, "max_pending": 5 } }
Set a mode to off to switch something off at once; auto is honoured only for start/shutdown/restart, never for commands, force-stop, backup edits or deletions, whatever the file says.
/etc/systemd/system/jarvis-bridge.service (secrets come as systemd credentials, so the bridge user cannot read /etc/jarvis/secrets directly):
[Service]
User=jarvis-bridge
ExecStart=/usr/bin/python3 -I /opt/jarvis/bridge/ntfy_bridge.py
LoadCredential=ntfy_bot_password:/etc/jarvis/secrets/ntfy_bot_password
LoadCredential=pve_token:/etc/jarvis/secrets/pve_token
LoadCredential=ha_token:/etc/jarvis/secrets/ha_token
LoadCredential=pve_actions_token:/etc/jarvis/secrets/pve_actions_token
LoadCredential=exec_key:/etc/jarvis/secrets/exec_key
LoadCredential=web_api_key:/etc/jarvis/secrets/web_api_key
StateDirectory=jarvis-bridge
Restart=always
MemoryMax=256M
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/srv/notes
RestrictSUIDSGID=yes
State lives in /var/lib/jarvis-bridge/ (state.json, reminders.json, chatlog.jsonl, and audit.db, an SQLite file with the audit log and the approvals table).
On this model: it invented the time 5-15% of the time; said "I added that line" with no tool call 10 out of 10 times for one phrasing; dropped the calculator on word problems once the conversation grew (8/8 to 0/6); after being caught claiming a reminder it apologised instead of setting it; and sometimes returned an empty reply. So:
pveum user add jarvis@pve --comment "Jarvis assistant (LXC 130)" --enable 1
pveum acl modify / --roles PVEAuditor --users jarvis@pve
pveum user token add jarvis@pve bridge --privsep 1 --comment "Jarvis bridge, read-only" --output-format json \
| python3 -I -c "import sys,json; sys.stdout.write(json.load(sys.stdin)['value']+'\n')" \
| pct exec 130 -- /opt/jarvis/bin/set-secret pve_token
pveum acl modify / --roles PVEAuditor --tokens 'jarvis@pve!bridge'
pct push 130 /etc/pve/pve-root-ca.pem /etc/jarvis/pve-root-ca.pem --perms 644 # TLS is verified against the cluster CA
Writes with this token return HTTP 403 (tested). Trap, later: a more specific ACL path replaces the roles inherited from / for the same user, so the moment you grant another role on /vms you must grant PVEAuditor there too, or the read-only token becomes blind. After touching ACLs always run pveum user permissions 'jarvis@pve!bridge' --path /vms/125.
Create a long-lived access token (profile page), then set-secret ha_token. The code allows exactly three things: GET /api/states, todo.get_items (read) and todo.add_item pinned to todo.shopping_list. (Cody accepted an owner/admin token; the code, not the token, limits what is done.)
jarvis_tools.py.tests/fixtures)./srv/notes is a second CephFS directory (/mnt/pve/cephfs/jarvis-notes) shared with the bridge user through a group (jarvis-notes, setgid), mounted with pct set 130 -mp1 /mnt/pve/cephfs/jarvis-notes,mp=/srv/notes,backup=0. Notes are markdown, create and append only, sandboxed to that directory in code, every change a git commit. Deliberately not in the backup job (Cody's choice).pvesh set /cluster/backup/<jobid> --vmid "<old list>,130". Note: the root disk contains /etc/jarvis/secrets, so protect the backup storage accordingly./restart jellyfin).propose_action) only writes a row in approvals with status pending, an expiry (15 minutes, 10 for commands), and a summary that says exactly what would happen. Cody receives a message built from that row.yes, no, yes 7 or yes all. Only his own fresh message counts. A bare yes answers only what Jarvis' last message asked; a casual ok is enough for start/shutdown/restart but never for commands, force-stops, backup edits or deletions.approved -> running atomically (so it runs at most once), re-checks the live state and the policy, does it, and reports the result built from what the system answered.pveum role add JarvisOps --privs "VM.PowerMgmt"
pveum user token add jarvis@pve actions --privsep 1 --comment "Jarvis actions: power only" --output-format json \
| python3 -c "import json,sys; sys.stdout.write(json.load(sys.stdin)['value'])" \
| pct exec 130 -- sh -c 'umask 077; cat > /etc/jarvis/secrets/pve_actions_token; chmod 600 /etc/jarvis/secrets/pve_actions_token'
pveum acl modify /vms --roles PVEAuditor,JarvisOps --users jarvis@pve --propagate 1 # (the auditor role is the trap from 8.1)
pveum acl modify /vms --roles JarvisOps --tokens 'jarvis@pve!actions' --propagate 1
for id in 127 130; do # the chat server and Jarvis itself: carved out, enforced by Proxmox
pveum acl modify /vms/$id --roles NoAccess --users jarvis@pve --propagate 1
pveum acl modify /vms/$id --roles NoAccess --tokens 'jarvis@pve!actions' --propagate 1
done
The code can POST to exactly four endpoint shapes (status/start|shutdown|stop|reboot on a guest's own node); a regular expression allow-list refuses everything else.
Editing a vzdump job through the API needs Sys.Modify on /, which is far too broad. So instead:
# in the Jarvis container: a key whose private half never leaves /etc/jarvis/secrets
ssh-keygen -q -t ed25519 -N "" -C "jarvis-exec@130" -f /etc/jarvis/secrets/exec_key
# on a Proxmox node: ONE cluster-wide line in /etc/pve/priv/authorized_keys (keep a backup of the file first)
restrict,from="192.168.86.226",command="/usr/local/sbin/jarvis-exec" ssh-ed25519 AAAA... jarvis-exec@130
# the host keys of the three nodes go into /etc/jarvis/known_hosts (root-owned; ssh uses StrictHostKeyChecking=yes)
/usr/local/sbin/jarvis-exec (root, on each node) is the second lock. sshd hands it the client's command line as data; it accepts only ping, exec <vmid> <timeout> <base64> (a command inside a running container, never on the hosts, never in container 130, 60 KB output cap, 120 s max), and backup-show|backup-add|backup-remove <job> <vmid> (changes only the guest list; verifies its own work and restores the job on mismatch). Everything is logged to the node's journal (journalctl -t jarvis-exec).
status/current), because the cluster-wide list is about 10 seconds stale.Why: the iPhone ntfy app cannot send; approvals should be one tap; and a voice agent needs an HTTP endpoint to ask Jarvis questions. One small service does all three.
phone/PC --HTTP :8765--> jarvis-web (unprivileged user, no capabilities, holds NO Proxmox/HA/ntfy secrets)
| loopback + shared key
v
jarvis-bridge: 127.0.0.1:8091 (loopback only)
jarvis_web.py serves a fixed set of files (no directory access) and requires a login for everything except the empty page shell. Password: set by you, stored only as an scrypt hash. Sessions: random 256-bit tokens (stored hashed), HttpOnly + SameSite=Strict cookie, idle 7 days / 30 days max, at most 8. Login throttling: 5 wrong passwords lock that address for 5 minutes, plus a global cap of 60 attempts an hour. State-changing requests need a custom header, JSON, and a matching Origin (CSRF). Strict Content-Security-Policy; the page script builds text with textContent only.yes 7, everything), answered silently (priority 1), and mirrored into the ntfy topic so the topic stays a complete record. The Approve buttons send exactly what typing yes 7 would, and each needs a second tap.POST /api/ask with a bearer token; no slash commands; a spoken yes or no never approves or cancels anything; a policy auto is treated as ask; an approval request for a spoken command goes to the chat and the speaker only hears "I've sent that to your chat for approval". Replies are trimmed to speakable text (no markdown, at most about 420 characters, the rest stays in the chat log).192.168.86.226:8765), plain HTTP. Anyone on the LAN can reach the login; nobody without the password can do anything. For HTTPS (needed for a browser microphone) put a proxy host in front; that is a decision for later.useradd --system --no-create-home --shell /usr/sbin/nologin jarvis-web
head -c 48 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 40 > /etc/jarvis/secrets/web_api_key # key between web and bridge (umask 077)
: > /etc/jarvis/secrets/voice_token # empty = the voice endpoint does not exist
# files: /opt/jarvis/web/{jarvis_web.py,index.html,manifest.webmanifest,static/*} (root-owned)
# units: jarvis-web.service (sandboxed: ProtectSystem=strict, NoNewPrivileges, CapabilityBoundingSet=, RestrictAddressFamilies=AF_INET AF_UNIX,
# SystemCallFilter=@system-service, MemoryDenyWriteExecute, MemoryMax=192M) and the bridge unit with LoadCredential=web_api_key
bash /opt/jarvis/bin/set-web-password # hidden prompt, twice; stores only the scrypt hash, enables and starts jarvis-web
Then open http://192.168.86.226:8765 and sign in; on the iPhone use Safari > Share > Add to Home Screen (it then looks like an app). The chat log is /var/lib/jarvis-bridge/chatlog.jsonl (mode 0600).
/opt/jarvis/ops/run_all_suites.sh (offline), then the live test eval/web_e2e9.py run on the Proxmox host: it starts the real unit with a temporary password, drives it over the LAN like a phone, approves and denies real requests on a test container, checks that ports 8091/8080/10300/10200 are not reachable from the LAN, that the service runs as an unprivileged user with no capabilities, that nothing leaks into any log, and then tears everything down.
The standard Home Assistant route: satellite (microphone, speaker, wake word) -> Home Assistant Assist pipeline -> speech-to-text -> a conversation agent (built-in intents first, Jarvis for the rest) -> text-to-speech -> satellite.
python3 -m venv /opt/jarvis/voice/venv
/opt/jarvis/voice/venv/bin/pip install faster-whisper piper-tts wyoming wyoming-faster-whisper wyoming-piper "av>=14,<16" # PyAV 19 breaks faster-whisper
# models go on the CephFS mount: /srv/models/voice/{whisper,piper} (small.en for Whisper; Piper voice en_GB-alan-medium)
Units jarvis-stt.service (port 10300) and jarvis-tts.service (port 10200) bind 127.0.0.1 until WYOMING_BIND in /etc/jarvis/voice.env is set to the LAN address. The Wyoming protocol has no login, so opening it to the LAN is a deliberate step. Measured on this CPU while the language model was busy: Piper 40-60x faster than real time (first audio in 0.07 s); Whisper small.en about 0.5 s per short command, 4-9% word error with a vocabulary hint, 16-18% without ("jellyfin" becomes "gelafin", "Ceph" becomes "Seth"; the unit passes --initial-prompt with the lab's words). Expect 6-10 s from the end of your sentence to the first spoken word; the language model is most of that.
192.168.86.226:10300 (speech-to-text) and :10200 (text-to-speech).POST http://192.168.86.226:8765/api/ask with the bearer token in /etc/jarvis/secrets/voice_token. Two ways: a small custom conversation agent (a custom component that calls that endpoint), or an Ollama-compatible shim in front of it so HA's built-in Ollama integration can use Jarvis with Prefer handling commands locally switched on (sentence triggers only fire for the default agent or with that option).| Option | Roughly | Notes |
|---|---|---|
| The Home Assistant app on a phone or tablet | free | the Assist button uses the phone's microphone through your pipeline; test the whole chain before buying anything |
| M5Stack ATOM Echo | about US$14-25 | flash ESPHome from the browser; tiny speaker and a weak microphone, fine for a desk or a test, usually paired with a real speaker |
| ESP32-S3-BOX-3 | about US$70-130 | screen, two microphones, speaker; ESPHome support exists but the firmware choice has rough edges |
| Home Assistant Voice Preview Edition | about US$59 (stock varies) | dual microphones with echo cancellation, speaker plus a 3.5 mm output, hardware mute switch, on-device wake word including "Hey Jarvis"; the least work and the best microphone for the money |
| Raspberry Pi Zero 2 W + ReSpeaker 2-mic HAT + a speaker | about US$60-80 all in | wyoming-satellite on the Pi; more setup and some users report stalls on the Zero 2 W |
Placement matters more than the model: one satellite in the room where you actually talk. "Hey Jarvis" ships ready-made in openWakeWord and on the Voice Preview Edition.
Goal: create, track, check off and remove projects and tasks by chat. OpenProject here is OpenProject (LXC 118, http://192.168.86.214:8080, version 17.5), configured for host name project.codycodez.com behind the reverse proxy.
jarvis, name "Jarvis", any e-mail, status active. Give it a role that may create and edit work packages in your projects. To let Jarvis create projects it needs the global permission "Create project"; to let it delete projects it must be an administrator (OpenProject allows project deletion only to administrators). Delete is always behind your yes, so an administrator account is a reasonable choice; if you prefer less power, leave it non-admin and deletions will simply be refused by OpenProject with a clear message.pct enter 130 then /opt/jarvis/bin/set-secret openproject_token. then systemctl restart jarvis-bridge: the bridge gets the token as a systemd credential (LoadCredential=openproject_token:... in its unit), and credentials are read when the service starts. (v0.9 said "nothing to restart" and the unit had no such line; found 2026-10-13, the first time a real token existed.)python3 -I /opt/jarvis/bridge/jarvis_projects.py selftest creates a throw-away project, adds a task, checks it off, marks the project finished, deletes both, and prints PASS/FAIL per step (and tells you if something has to be removed by hand).jarvis_projects.py uses REST API v3 with HTTP Basic (apikey:<token>) against the container's own LAN address, sending Host: project.codycodez.com and X-Forwarded-Proto: https (OpenProject only answers under its configured host name and redirects plain http, so the request must look like it came through the proxy). Only fixed endpoints are used (the facts they rely on were checked against OpenProject 17.5's own source: the project status finished, the closed-status filter operator c, administrator-only project deletion), no path or URL ever comes from a message, names that come back are collapsed to one line and capped, and the token appears in no URL, message, log or approval.
| You say | What happens | Tier |
|---|---|---|
/projects, "what projects do I have?", "how's the Garden project going?" |
read: projects with open/done counts, progress, overdue, the open tasks | T0 |
create a project called Garden, /newproject Garden |
creates it, answers with the link | T1 |
add a task buy mulch to the Garden project due friday, /addtask Garden: buy mulch |
adds the task (a due date is taken from the end of the sentence: due friday / tomorrow / 2026-10-20 / in 3 days) | T1 |
check off buy mulch, /done Garden: buy mulch |
closes the task (prefers the status "Closed", retries on edit conflicts) | T1 |
mark the Garden project as done, /finish Garden |
sets the project status to finished | T1 |
delete the project Garden, /rmproject Garden, /rmtask Garden: buy mulch |
asks you first: shows exactly what would go (with the task counts) and waits for yes; re-checks the name at the moment you approve | T2 |
The reply to every change is built by code from OpenProject's own answer; if the model claims a change that did not happen it is replaced by an honest message. Vague wording ("I'd like to track the deck rebuild as a project") makes the model's first move a required project_request call.
This is the part to copy.
| Layer | What | Size |
|---|---|---|
| Offline suites | every module against fakes: a fake Proxmox, a fake ssh/jarvis-exec, captured real HA/wiki responses, a fake OpenProject (auth, host header, locking, workflow, permissions), real HTTP over loopback for the web service |
1617 checks, seconds |
| Mutation testing | sabotage the safety code on purpose (let any password in, drop the CSRF check, let a spoken yes approve, execute a row nobody approved, skip the name re-check before a deletion, ...) and confirm a test fails each time | 159 sabotaged versions, all caught (a mutation nobody notices means a property is not really tested) |
| Live sweeps | the real model through the full pipeline, real data sources, scenarios written in wording the code does not recognise; run with an empty and a 20-message history because the empty run hides failures | 93 scenarios |
| End-to-end | on the Proxmox host: the real ntfy, real Proxmox, real ssh, real web service on the LAN, a test container, every outcome verified independently with pct/pvesh |
50 + 50 checks |
How to run them: section 15. Findings that only these found: the model fabricating an "Approval #27" that was never recorded, the force-stop chosen for "power cycle it", a stale status in the cluster list, the ACL that blinded the read-only token, a rate-limit that triggered the end-to-end test itself.
| Want to | Do |
|---|---|
| Stop everything | pct stop 130 on the Proxmox host (pct start 130 brings it back in ~2 minutes) |
| Stop only the web page | systemctl stop jarvis-web (the password hash stays; systemctl disable jarvis-web keeps it off at boot) |
| Stop only chat | /pause in chat, or systemctl stop jarvis-bridge |
| Switch an action off | set its mode to off in /etc/jarvis/policy.json (effective on the next message) |
| Revoke power over Proxmox | pveum user token remove jarvis@pve actions |
| Revoke the command/backup channel | delete the jarvis-exec@130 line from /etc/pve/priv/authorized_keys |
| Change the web password | run set-web-password again (sessions end) |
| Revoke OpenProject | delete Jarvis' token in OpenProject, or remove /etc/jarvis/secrets/openproject_token |
Deploy and roll back (from the Proxmox host): build a staged copy in /opt/jarvis/staging-vNN, run ops/run_all_suites.sh, mutate_doors.py and mutate_actions.py, the live sweeps (LONG=1 as well) and the end-to-end tests, then bash deploy_v09.sh; bash rollback_v09.sh restores the previous version. The deploy script backs up the old version once, installs, re-runs every offline suite against the deployed files before restarting, then restarts.
Logs: journalctl -u jarvis-bridge -u jarvis-web -u jarvis-brain; the audit log is /audit in chat (or audit.db); what Jarvis asked Proxmox to do is in the task log as user jarvis@pve with token actions; what ran on a node is journalctl -t jarvis-exec.
jarvis_lab.py through the one patchable network function, return a short plain-text report, register the tool in TOOLS, TOOL_SCHEMAS, policy.json (T0), add a slash command and, if the answer must be live, a read guard. Test with captured real responses.jarvis_actions.py (it only records a row), execute it only from execute(), give it its own mode in policy.json, and the narrowest permission the other system offers. Write the offline and the mutation tests first.Tokens for the planned read-only integrations (create the token yourself, then pct enter 130 and /opt/jarvis/bin/set-secret <name>; Jarvis never needs it on a command line):
| Service (version here) | Create | Name | Read-only? |
|---|---|---|---|
| Paperless-ngx (latest, LXC 100) | create a user jarvis (Administration > Users) with view-only permissions on documents; sign in as it > My Profile > API auth token (header Authorization: Token <token>); if the button is greyed out the token can be made in the Django admin |
paperless_token |
yes, by the user's permissions |
| Immich (v2, LXC 107) | Account Settings > API Keys > New; tick only the read permissions (for example asset.read, album.read, library.read, timeline.read); header x-api-key |
immich_token |
yes (scoped keys); check the names the dialog offers |
| Jellyfin (12.1, LXC 125) | Dashboard > Advanced > API Keys > + ; keys are administrator-level (they cannot be scoped), so Jarvis' code would only ever call read endpoints; the alternative is a non-admin user jarvis with an access token |
jellyfin_token |
no: limited by our code, not by Jellyfin |
| Pi-hole (v6) | Settings > Web interface / API > app password; the API turns a password into a session (POST /api/auth) |
pihole_password |
no (app password = the API); our code would read /api/stats/summary only |
| n8n (latest, LXC 105) | Settings > n8n API > Create an API key (label, expiry); header X-N8N-API-KEY, base path /api/v1 |
n8n_token |
scopes only on the Enterprise plan; otherwise full access, so limited by our code |
| Baikal calendar (0.10.x, LXC 100) | create a Baikal user jarvis in the admin panel and share your calendars with it (check that the sharing dialog offers read-only); CalDAV with that user |
baikal_password |
depends on the sharing option; verify before relying on it |
| OpenProject (17.5, LXC 118) | section 13.1 | openproject_token |
as discussed there |
Menu names follow the vendors' documentation and can differ slightly by version.
set-secret installed, curl/ssh work. (verify: pct config 130)jarvis-brain healthy, ~25 tokens/s. (verify: /health, a tool-call completion)ntfy-selftest.sh)/status answers. (verify: run_all_suites.sh)/guests, /ceph, /backups work; an attempted write returns 403./home, /shopping. Wiki, Kuma, Frigate reachable.vzdump 130 restored in a scratch test if you like.JarvisOps role, actions token, ACLs with NoAccess on 127 and 130; jarvis-exec on every node, the authorized_keys line, known_hosts. (verify: effective permissions for both tokens on /vms/125, /vms/127, /vms/130; the end-to-end test eval/bridge_e2e7.py on a test container)set-web-password. (verify: eval/web_e2e9.py)voice/run_wyoming_check.sh); then, with your OK, LAN binding, Home Assistant pipeline, a satellite.selftest./vms made the read-only token see zero guests until PVEAuditor was added on /vms too. (Two minutes of blindness, caught by looking at the effective permissions.)av>=14,<16.or True); mutation testing is what exposes those.pkill -f kills your own shell when the pattern appears in its command line; use ports, pids or scripts.jarvis-exec in the wrong place; nothing was restarted and the fix took a minute.small.en with greedy decoding once heard "How full are my disks?" as "Half full on my discs". A bigger beam, the medium model, or a small clean-up map of known mis-hearings are the first things to try.18.1 Put the container on the tailnet. On the Proxmox host add to /etc/pve/lxc/130.conf (same two lines as any container that runs Tailscale; keep a copy of the file first), then pct reboot 130:
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
In the container: add the official apt repository (https://pkgs.tailscale.com/stable/debian/<codename>.noarmor.gpg to /usr/share/keyrings/tailscale-archive-keyring.gpg and the matching .tailscale-keyring.list to /etc/apt/sources.list.d/), apt-get install tailscale, systemctl enable --now tailscaled, then tailscale up --hostname=jarvis --accept-dns=false. It prints a login link; the owner opens it. Afterwards disable key expiry for the machine in the admin console. tailscale ip -4 is the address used below.
18.2 The web page without a password. Set in jarvis-web.service: Environment=WEB_BIND=<tailnet ip> WEB_PORT=8765 WEB_AUTH=tailscale, After=tailscaled.service, and remove the web_password_hash credential. Write /etc/jarvis/web_access.json: {"logins": ["you@example.com"], "nodes": ["iphone-14", "personal-pc"], "hosts": ["jarvis", "jarvis.<tailnet>.ts.net", "<tailnet ip>"]} (device names exactly as tailscale status prints them; tailscale whois <ip> shows login and name). Why a device list and not "anyone on the tailnet": a tailnet usually also holds service containers, and one of those being compromised must not be able to approve actions. Why the Host list: DNS rebinding. tailscaled is asked who is on the other end through its unix socket (/var/run/tailscale/tailscaled.sock, world-readable, whois is a read-only call): GET /localapi/v0/whois?addr=<ip>:1 with Host: local-tailscaled.sock. Test with tests/test_web_tailnet_offline.py (a fake tailscaled socket and fake devices on 127.0.0.x) and sabotage-test with tests/mutate_tailnet.py; deploy with ops/deploy_v10.sh.
18.3 Speech servers and the firewall. /etc/jarvis/voice.env: WYOMING_BIND=<tailnet ip>; systemctl enable --now jarvis-stt jarvis-tts. The Wyoming protocol has no login, so add an nft table that drops everything but loopback and Home Assistant's tailnet address on ports 10200/10300 (/etc/jarvis/jarvis-fw.nft + the oneshot unit jarvis-fw, a table of its own so Tailscale's rules are not touched). Do not rely on systemd IPAddressAllow=: it is silently not enforced in an unprivileged container. Verify from another tailnet host that the port is dropped and from the Home Assistant container that it still connects.
18.4 Home Assistant. Generate a voice token (head -c 48 /dev/urandom | base64 | tr -dc A-Za-z0-9 | head -c 40 > /etc/jarvis/secrets/voice_token, mode 600; restart jarvis-web). Home Assistant's Ollama integration takes a URL and an optional API key (sent as Authorization: Bearer) and calls only GET /api/tags and POST /api/chat; jarvis_web.py implements exactly those (and /api/version), using the last user message. Run ops/ha_voice_setup.py inside the HA container (it has aiohttp; the HA token and the voice token come on stdin): it adds Wyoming twice, the Ollama integration with a conversation agent Jarvis (model jarvis:latest) and the Assist pipeline Jarvis (preferred; prefer_local_intents so Home Assistant keeps its own devices). Test with ops/ha_voice_test.py (text, then speech synthesised with Piper through Whisper). Undo with ops/ha_voice_undo.py.
18.5 Lessons. (1) A test that feeds the access list a string instead of a list found a parser bug; test the config file's bad shapes. (2) systemd IP filters do nothing in unprivileged containers: prove every network restriction from the outside (the Proxmox host connected to a port that the unit claimed to restrict). (3) Gemma's sliding-window cache cannot reuse anything after a changed token: keep the start of the prompt byte-identical, and measure with the real prompt before blaming idle time. (4) The first self-test against a real service can fail on timing, not logic: look for a leftover object before running it again.
19.1 The allow-list. The idea: one narrow lane for "run a command inside a container" that needs no yes. Keep propose() as the only place a command row is created; in it, after the usual checks (a running container, not 127/130, not a VM), match the whole command against the allow-list; on a match store the NORMALISED string, mark the row approved with decided_by = 'policy:allowlist', and let the existing "auto" path start it. Matching rules that matter: whole-string equality or a template with validated placeholders ({unit} and {name} a few safe characters that never start with -, {n} 1-200); entries containing shell characters are dropped when the policy is read, and entries are never "tidied" (an untidy entry is dropped, not rewritten); anything multi-line is refused before normalising (df + newline + -h would otherwise collapse into the listed df -h); protected_guests and exec_denied_guests always win; a spoken request gets the allow-list switched off in acting_cfg; and execute() re-checks the stored command against the live policy so turning the list off is a real kill switch. Show the reply as "Running #N now" (not as an approval request) in every path, including the one where the model proposed it. Test every one of those as a sabotage (mutate_v11.py).
19.2 The warm-up. Measure first: send the same long prompt twice, changing only the date at its end, and read timings.cache_n / prompt_n from the server. If a changed token near the start of the prompt makes the cache useless (it did: 0 of 3,470 reused), and the change happens on a schedule, warm the cache on that schedule instead of pinging. warm_cache() builds exactly the message list a real turn builds (system_prompt() + the remembered history + a throw-away user message) with the same tool list and max_tokens: 1, so the next real message shares everything except its own words. Run it from the scheduler at a quiet time with a catch-up window and a once-per-day marker in the state file.
19.3 Keeping the web conversation off ntfy. A per-thread flag set while a web message is handled (CHAN.ch = 'web'), consulted by publish() (chat log yes, ntfy no). Threads that outlive the handler (the approval execution, the "working on it" timer) must carry the flag explicitly. An action started while a message is handled waits on a gate that publish() releases, so "Running #N now" always precedes "#N done".
19.4 The menus. Keep the catalog as data (jarvis_commands.py), serve it and the live lists from the bridge over the loopback API (/v1/commands, /v1/choices?kind=...), proxy them in the web service with a whitelist of kinds and parameters, and make every button build the same text a person would type: no new power, nothing to audit beyond what exists. Test that every menu command exists in the bridge and that the text built from sample values is what the bridge parses (a round trip for /restart and /run).
19.5 Push-to-talk over HTTPS. (1) A certificate: tailscale cert <host>.<tailnet>.ts.net (HTTPS must be enabled for the tailnet; the name becomes public in certificate-transparency logs). Use your own listener, not tailscale serve: the device list needs the real connecting device. A weekly timer re-runs the command and restarts the page only if the certificate changed. (2) A second listener with a TLS context (TLS 1.2+, do_handshake_on_connect=False so handshakes happen in the request's thread). (3) The page records with getUserMedia + a ScriptProcessor, resamples to 16 kHz 16-bit mono, and POSTs the raw bytes (application/octet-stream plus the custom header: same CSRF rules). (4) The service speaks Wyoming with a few lines of stdlib (header line, data bytes, payload bytes) to the speech servers: transcribe, then the voice door, then synthesize. (5) The firewall rule for the speech servers must admit the web service's own tailnet address. (6) Headers: Permissions-Policy: microphone=(self), media-src 'self' blob: data:.
19.6 Lessons. (1) A sabotage script that cannot find its anchor, or its tests, reports "caught": read the failure text of the first few results and print anchor misses as errors. (2) Compare the staged files with the working files by checksum before every deploy: the gate tests what is staged, not what you meant to stage. (3) Test doubles that sleep are defeated by suites that stub time.sleep globally: wait on an Event instead. (4) The database lock in this project is not re-entrant: compute values before taking it. (5) Never rewrite configuration entries while normalising them; accept or drop.
20.1 First, audit by hand. Before writing code, find out what is true: the node's own API (/api/sno/, /api/sno/satellites, /api/sno/estimated-payout, /api/heldamount/paystubs/<from>/<to>, /api/heldamount/held-history on port 14002), the disk (findmnt, smartctl, du with and without --apparent-size), the logs, and the chain (a public explorer's token-transfer list for the wallet). Reconcile the money three ways: node estimate, satellite statements, chain. Here the estimate was 24% low (stale prices in the node's priceModel) and the statements matched the chain to the cent. Write the findings down (docs/STORJ.md) before coding: the code then encodes known facts instead of guesses.
20.2 One module, one network primitive. jarvis_storj.py has a single function that touches the network (_fetch: GET only, redirects refused, answer capped) and everything else is pure: parse() turns raw answers into a snapshot, the report functions turn a snapshot into text. Tests replace _fetch, or (better) run a real local HTTP server so redirects, size caps and outages are tested for real. Config comes from the policy with every value checked (enabled only if exactly true; the node address must match a private-address pattern with fullmatch; numbers clamp or fall back; booleans are not numbers).
20.3 Wire it the standard way (checklist from ARCHITECTURE.md). A DATA_COMMANDS entry (/storj), a tool in TOOLS + TOOL_SCHEMAS + the policy (get_storj_status, T0/auto), a LIVE_GUARDS entry so a question that names Storj fetches the numbers before the model answers (and DATA_TOOLS so it is skipped for write requests), a menu group in jarvis_commands.py, HELP, and collect_problems for /problems and the watch (add a severity word to SEVERE for the alerts that must be loud). Adding a tool changes the prompt prefix, so the first message after the deploy is slow (once).
20.4 The payout watch. Detect what you care about at its source: the chain shows the money arriving days before the node's own statement updates. Rules that keep it honest: match the token by CONTRACT address (a look-alike token can call itself USDC), the sender by an allow-list (dust from address-poisoners lands on the wallet), the recipient by exact address; build the message only from numbers (never from explorer text); the first pass records what exists and says nothing; remember what was announced (capped list, saved with the state); run it in its own thread with a non-blocking lock; one notice after two days of failures, not one per failure. The per-thread web flag means the notice reaches ntfy even while a web chat is in progress: prove that with a test that has a real web message in flight.
20.5 Test and sabotage. 212 checks, then mutate_storj.py (85 broken copies, one property each: read-only, private address, token contract, sender list, no replay, quiet web chat). Before sabotaging, look at what the tests do NOT notice (here: lookalike private ranges, a change under a cent, a shrinking node, a boolean as a number) and add those checks first. Check that a sabotage is caught by the test you meant, not by a crash (7 of the 85 were first caught only by an exception; they are exercised by real checks too).
20.6 Lessons. (1) Run the suites inside the container before deploying: the staged policy there enables Storj, and one older test would have looked at the real node; tests that copy the policy must strip anything that points at a real system. (2) A claim about behaviour ("web chat stays off ntfy") is checked on the real topic, not only in a test. (3) When the user reports a problem you believe is fixed, look at the evidence first (here: the topic held no web chat since the fix) before changing anything. (4) The node's own dashboard numbers are estimates; the satellites' statements and the chain are the truth for money. (5) Deploy scripts should MERGE new settings into the live policy (add the tool and the block if missing) instead of installing a fixed copy.
20.7 Night tweaks: a better voice, and a misreport found by speaking to it. Swapping a text-to-speech engine is easy when everything speaks one protocol (Wyoming): run the new engine on the same port, reload Home Assistant's Wyoming entry so it re-reads the voice list, set the pipeline's voice, point the Console at the same name. Keep the old engine installed but disabled and write the rollback first. Install a new engine in its OWN virtual environment so it cannot change the versions the working speech servers rely on. Report the same language spelling the old engine used (en_GB), or the pipeline will not offer the voice. Then test through the real pipeline (ops/ha_voice_test.py), because that is what found the bug: the storage report printed bare percentages (percent USED) under "Plenty of room", and the speaking model read them as free space. Any number a model will read aloud must carry its unit word ("54% used").