The master log for the Jarvis project: what we are building, the rules we work by, where we are, and what happened when. Technical reference: Jarvis · how to use it: User Guide · problems solved on the way: Lessons Learned · AI projects overview: AI Projects. A copy of the plan lives with the code in the Jarvis container (
/opt/jarvis/docs/). How it was built, step by step (so you could rebuild it): Jarvis: how it was built. Last updated 2026-10-14 (bridge v0.12, Jarvis Console v0.11). Every version is listed in Jarvis: version log.
A locally hosted, Iron-Man-style assistant that Cody can grant permissions to, or let act on its own, deliberately broad: homelab operations, smart home, personal data, business/projects. Chat over ntfy and the Jarvis Console (a web page on the tailnet: no password, command menus, a desktop layout and a microphone button). Voice works from the Home Assistant phone app and in the Console; speakers and microphones around the home (a "satellite") are the next goal. Everything is proven on the current hardware first (CPU only); a GPU is a later, optional hot-add.
jarvis) is the chat channel; Discord is deferred.authorized_keys.The model can never run anything itself: its only power is to write a proposal, and the message Cody reads is built by code from the stored row.
| Tier | Meaning | What |
|---|---|---|
| T0 observe | read-only, always allowed | clock, own status, weather, Uptime Kuma, cameras, Proxmox / Ceph / backups / storage / IPs / temperatures, Home Assistant, wiki search, notes, calculator, /problems roll-up, OpenProject projects and tasks · the Storj node, read-only (health, earnings and hold-back, forecast, payments received: /storj) |
| T1 reversible | automatic, audited, rate-limited | save/append notes, reminders, add to the shopping list, /mute, /watch alerts, create an OpenProject project or task, check a task or project off |
| T2 consequential | Jarvis proposes, Cody says yes | start / shut down / force-stop / restart a container or VM · add or remove a guest in the weekly backup job · run any other command inside a container · delete an OpenProject project or task |
| Runs without asking | the read-only command allow-list (since v0.11) | 27 reviewed read-only commands (uptime, df -h, free -h, docker ps, systemctl status <service>, journalctl -u <service> -n 50, ...) in any running container; never on the Proxmox hosts, never in ntfy (127) or Jarvis (130), never for a spoken request |
| T3 never | no tool exists | deleting data or backups, Ceph/cluster config, credentials, firewall, shell on the Proxmox hosts, anything inside ntfy (127) or Jarvis (130) |
| Phase | What | Status |
|---|---|---|
| A | CPU-only: LXC, local model, ntfy chat, read-only tools, backups | Done (2026-10-09/10) |
| B | GPU (RX 580 in the x4 slot) | Not started; hardware not bought. Measure first, pin Frigate to the iGPU by /dev/dri/by-path, soak-test |
| C | Permission broker, approvals, first writes | Done for Proxmox in v0.7; hardening ideas in the backlog |
| D | Voice | Live (2026-10-13): through Home Assistant (phone app; speech servers on the tailnet, only Home Assistant may connect) and in the Jarvis Console (push-to-talk over HTTPS). Needs next: a satellite (hardware), wake word, Whisper mishearings. See the plan below; since 2026-10-14 night the voice is Kokoro bm_daniel (British male, picked by ear) at 1.05x, for Home Assistant and the Console |
| E | Autonomy and proactivity | Started: /watch, daily briefing, reminders, commands that run without asking (allow-list), a midnight warm-up so the first message of the day is fast; the Storj node is a watched source and Jarvis announces payouts (v0.12) |
| G | Phone / web door | The Jarvis Console, live (v0.11): passwordless over Tailscale (your devices only), desktop layout, grouped command menus with live dropdowns, container and project panels, push-to-talk and read-aloud on https://jarvis.tailbacb1d.ts.net:8766; its conversation stays off ntfy |
| F | Scale decision (bigger GPU or a dedicated box) | Open, decide with measured bottlenecks |
| Date | What happened |
|---|---|
| 2026-10-14 night | v0.12 tweaks: the British butler voice, a clearer storage report. Cody picked Kokoro bm_daniel by ear from nine samples (the closest to the films' Jarvis) and asked for it about 1.1x faster: now 1.05x (0.95 x 1.1), live for Home Assistant's Jarvis pipeline and the Console's read-aloud; Piper stays as a disabled fallback (ops/rollback_voice_v12.sh). The end-to-end test through Home Assistant passes. Found by that test: Jarvis answered "how much disk space is left" with the numbers backwards ("boot-disks 2% free") because the storage report printed percent USED under the heading "Plenty of room"; every figure now says "used" (tested). 2,056 offline checks. |
| 2026-10-13 late evening | v0.12: Storj, read-only, and a payout watch. After a full audit of the Storj node (healthy; payouts correct to the cent against the chain; just small and young; docs/STORJ.md), Jarvis got /storj, /storj money, /storj forecast, /storj payouts (also a model tool and a Storj group in the Console), Storj problems in /problems and the watch (disqualified, suspended, falling scores, outdated, QUIC, silent satellites, nearly full, dashboard down) and a payout watch: every 2 hours it checks the public block explorer for STORJ/USDC arriving in the node's wallet from the Storj payout address list (matched by token contract and sender, so address-poisoning dust is ignored) and sends one ntfy message per payment; the first pass only learns. Web chat stays off ntfy (verified on the topic); watch alerts and payout notices still reach it, even while a web message is being answered (tested). Read-only by construction (GET only, private node address, redirects refused; a test scans the source). 2,056 offline checks, 85 new sabotage copies all caught. Found on the way: an older test would have looked at the real node (fixed: tests strip the Storj policy). Cody ran the Storj log-rotation script and turned off key expiry; the OpenProject admin token stays. The British-butler voice followed the same night (next row). |
| 2026-10-13 evening | v0.11: commands without asking, a midnight warm-up, a quiet web page, the Jarvis Console, talking in the browser. (1) Read-only command allow-list ON for every running container: 27 reviewed commands run without a yes (/run storj df -h answers "Running #N now" and then the output); anything else still shows the exact command and waits; never hosts, 127, 130 or a spoken request; re-checked when it runs. (2) Midnight warm-up (00:05): the first message of each new day used to take 30-70 s because the date at the end of Jarvis' prompt changes at midnight and the model's cache cannot be reused after a changed word; Jarvis now reads the new day's prompt in the quiet hour. (3) The web page no longer touches ntfy. (4) The Jarvis Console: full-width desktop layout with a command sidebar; nine grouped menus with live dropdowns (containers, projects, open tasks, notes, reminders, the allow-listed checks); "Pick a container..." and "Pick a project..." panels; push-to-talk and read-aloud on an HTTPS address with a real Tailscale certificate. (5) Storage: hanna-properties-backend moved to Ceph (verified file by file), local-lvm 93.8% -> 53.8%. 1,842 offline checks; 243 deliberately broken copies of the code, all caught. |
| 2026-10-13 | v0.10: no password, voice through Home Assistant. Jarvis joined the tailnet; the web page's login is the Tailscale device (right login AND a named device, because the tailnet also holds internet-facing containers); Host-header rule against DNS rebinding; the voice door speaks Ollama's dialect so Home Assistant's Ollama integration uses Jarvis; speech servers on the tailnet behind a firewall rule (systemd's IP filters are not enforced in an unprivileged container); Home Assistant wired (Wyoming x2, Ollama, pipeline "Jarvis"); OpenProject connected (and a bug found: the bridge's unit never loaded its token); Minecraft world moved from local-lvm to Ceph; Storj root grown to 25 GB. |
| 2026-10-11 | v0.9: a phone web page, a voice endpoint, and OpenProject. (1) Web page on the LAN (http://192.168.86.226:8765): login (Cody sets the password; only a hash is stored), chat, approve/deny buttons (each needs a second tap), add-to-home-screen on the iPhone. It is a separate service with no Proxmox/Home Assistant/ntfy secrets, talking to the bridge over a keyed loopback API; web messages enter the same pipeline as ntfy and are answered silently and mirrored into the topic. (2) Voice endpoint (/api/ask, bearer token, off until a token is set): a spoken request can never approve or cancel anything, policy auto counts as ask, approval requests go to the chat. The speech services exist as units (loopback only); a loopback demo of the whole chain (speech, Whisper, Jarvis, Piper) measured about 4-5 s from the end of a spoken question to the first spoken word of the answer when the prompt cache is warm (speech-to-text 0.5 s, Jarvis 3.5-4.4 s, first audio 0.1 s); the first request after a restart is a cold read of about 20 s; one of three questions was misheard by Whisper. (3) OpenProject: create, track, check off and (with a yes) delete projects and tasks; replies are built from OpenProject's own answer. Tested offline against a fake OpenProject (259 checks) and live with the model (19 scenarios); the live OpenProject has not been touched yet: it needs Cody's API token (steps in the build guide). Also: a proposal for read-only commands that may run without asking (not enabled), the replication guide, and the token instructions for the next integrations. Checks: 1617 offline checks, 159 deliberate sabotages all caught, 93 live scenarios (the 74 older scenarios stayed at 100% with an empty and with a 20-message history after the prompt and tool changes; the 19 OpenProject scenarios are 19 of 19 at four runs each in both histories after the fixes the long-history runs forced), live end-to-end on the LAN: 50 of 50 checks over the real LAN service (login, approvals on a test container, the voice door, ports not reachable from the LAN, no secret in any log), everything restored afterwards. Findings by testing: (1) a false "I've checked off the task" that a check skipped because it guessed what was asked: first-person claims are now caught whatever the message looked like; (2) people say "pruning the roses" for the task "Prune roses": tasks are matched by word stems and a failed match lists the open tasks; (3) the project overview hid overdue tasks and the model answered "none": the data now carries them; (4) a wrong "I cannot delete projects" and a false "I've asked for your approval" are challenged once; (5) the deploy gate refused the first deploy because the suite runner looked for a file in the wrong place (nothing was restarted); (6) the live end-to-end test needed no fix; (7) Whisper misheard "disks" in the voice demo (the weak link of voice) |
| 2026-10-10 evening | v0.7: Proxmox actions with approval. Start / shut down / force-stop / restart containers and VMs, add or remove a guest in the backup job, and run one command inside a container, each only after Cody replies yes. Power actions use a new API token that can only manage power and has no access to ntfy (127) or Jarvis (130); backup-job edits and commands use an ssh key that every node restricts to one script (jarvis-exec). Verified live on a test container: start, command output with tricky quoting, restart (uptime reset), graceful shutdown, a hung command stopped at 60 s, backup-job add and remove with the job config and jobs.cfg byte-identical afterwards, forged "yes" messages ignored, no secret in any log. 406 checks in the new suite (1,085 offline checks in all), 64 deliberate sabotages all caught by the tests, 23 new live scenarios. Incident, fixed in minutes: the new ACL made the read-only token blind to all guests (a more specific Proxmox ACL replaces inherited roles). Voice groundwork measured. Stale NAS address (.100) corrected in the wiki: .100 is the Fios extender, the NAS is .103. |
| 2026-10-10 | v0.6: read-only Proxmox / Ceph / backups / storage / IPs / temperatures, Home Assistant (presence, battery, plugs, shopping list; adding to the list is the one write), wiki search, /problems, opt-in /watch, /mute, ~20 more commands. LXC 130 added to the Sunday backup job (notebook excluded by choice). Latency work: writes 22-40 s -> 3-5 s. Fixes from Cody's own use of the bot. |
| 2026-10-09 | v0.1-v0.5: LXC 130 created, llama.cpp built, Gemma 4 26B-A4B chosen after a bake-off, 42-minute soak, ntfy topic and bot user, bridge with policy gate / audit log / notebook / weather / monitors / cameras / calculator / reminders / briefing, and the guards that stop the model from inventing times and claiming saves it never made. |
bm_daniel, 1.05x, 2026-10-14). Possible tweaks: another speed (KOKORO_SPEED) or voice (ops/switch_voice_v12.sh); streaming the reply into the speech engine sentence by sentence (the biggest latency win).https://jarvis.tailbacb1d.ts.net:8766, microphone) and the HA app (three-dots menu > Assist). (Done since: Storj log rotation, Tailscale key expiry off. Decided: the OpenProject admin token stays.)/storj in a week: the gap between "stored" and "paid for" should shrink.warmup at 00:05 each night)./addbackup <name> with a yes); local-lvm is at 53.8%.Standard Home Assistant route: a satellite (microphone, speaker, wake word) -> HA Assist pipeline -> speech-to-text (Whisper on LXC 130) -> built-in intents first, otherwise Jarvis -> text-to-speech (Piper on LXC 130) -> satellite speaker.
small.en transcribes a short command in about 0.5 s (4-9% word error on synthetic speech once it is given a vocabulary hint; 16-18% without, "jellyfin" and "Ceph" are the hard words). Through the real Wyoming protocol on loopback it works end to end. Expect roughly 6-10 s from the end of speech to the first spoken word today (the language model is most of it)./api/chat in Ollama's dialect for Home Assistant, /api/ask, /api/voice for the Console; never approves anything), the two speech services on the tailnet (a firewall rule admits only Home Assistant and Jarvis' own page), the Home Assistant pipeline and the Console's microphone. Older note: the Jarvis side of the hand-off (POST /api/ask with a bearer token: speakable answers, never approves anything) and the two speech services as units, bound to loopback until Cody agrees to open them./opt/jarvis/docs/VOICE.md.| Date | Decision | Why |
|---|---|---|
| 2026-10-09 | Gemma 4 26B-A4B (Q4_0) on CPU | best tool use and speed inside the 28 GB cap |
| 2026-10-09 | Plain-Python bridge on one ntfy topic | simple, auditable, nothing else to compromise |
| 2026-10-09 | Guards + claim verifier instead of trusting the model | measured: invents times, claims saves it never made |
| 2026-10-10 | Wiki read from public pages, no wiki credentials | less to protect |
| 2026-10-10 | Proxmox read token = PVEAuditor | writes proven to be HTTP 403 |
| 2026-10-10 | Approvals are executed by code, never by the model | the model cannot be the gatekeeper |
| 2026-10-10 | Power actions: own token, VM.PowerMgmt only, NoAccess on 127 and 130 | Proxmox enforces the boundary |
| 2026-10-10 | Backup edits and commands: forced-command ssh key, not a root shell | the API needs Sys.Modify on / for job edits, far too broad |
| 2026-10-10 | Commands only inside containers, never on hosts, never in 127/130 | Jarvis must not be able to edit its own code, policy or secrets |
| 2026-10-10 | Admin Home Assistant token accepted | Cody's call; the code still limits what it does |
| 2026-10-11 | The LAN door is a separate service that holds no secrets but its own and talks to the bridge over a keyed loopback API | a bug in the LAN-facing code must not expose the Proxmox, Home Assistant or ntfy credentials |
| 2026-10-11 | The web page sends exactly what typing yes 7 sends; voice can never approve |
one approval path to audit; a voice in the room is not Cody |
| 2026-10-11 | Plain HTTP on the LAN first, HTTPS later through the proxy | editing NPM is Cody's call; login is throttled and sandboxed meanwhile |
| 2026-10-11 | OpenProject: a dedicated user's API token; creating and checking off are T1, deleting is T2; what Cody is told is built from OpenProject's own answer | OpenProject's permissions are the outer limit; the model cannot misreport a change |
| 2026-10-11 | Cody creates tokens for new integrations himself (instructions provided) | he keeps control of credentials |
| 2026-10-13 | The web page's login is the Tailscale device (right login AND a named device), not a password; the LAN port is closed | the tailnet also holds internet-facing containers; a plain "anyone on the tailnet" rule would let a broken-into one approve actions |
| 2026-10-13 | Voice goes through Home Assistant's Ollama integration; Jarvis answers Ollama's /api/chat |
no code in Home Assistant; HA keeps its own devices (prefer_local_intents) |
| 2026-10-13 | The speech servers listen on the tailnet and a firewall table admits only Home Assistant and Jarvis itself | Wyoming has no login; systemd's IPAddressAllow= is not enforced in an unprivileged container |
| 2026-10-13 | The command allow-list is ON for every running container (not hosts, 127, 130 or voice) | Cody reviewed the 27 entries; each is one read-only command with no way to read a secret and no shell |
| 2026-10-13 | A midnight warm-up replaces the keep-warm ping | the slow message is the first of a new day (the date in the prompt), not idle time |
| 2026-10-13 | A conversation on the web page stays off ntfy | Cody: the notifications were unnecessary; the page is the record |
| 2026-10-13 | HTTPS by our own listener with a tailscale cert certificate, not tailscale serve |
the device list needs the real connecting device, which a Serve proxy would hide |
| 2026-10-13 | Push-to-talk goes through the voice door | what is spoken can never approve or run anything unasked |
| 2026-10-14 | Jarvis only READS the Storj node (GET, fixed paths, private address); a payment is recognised by token contract and a sender list, never by names | the node's wallet receives address-poisoning dust and look-alike tokens; the node and the explorer are outside text that must never steer a message |
| 2026-10-14 | The payout watch looks at a public block explorer with the wallet address only (switchable to the satellites' own totals) | the chain shows money days before the node's statement updates; the address is public chain data |
| 2026-10-14 | Web chat stays off ntfy; watch alerts, payout notices, reminders and the briefing still go to ntfy | Cody: chat pings are annoying, the important notices are wanted |
| 2026-10-14 | The voice is Kokoro bm_daniel (British male) at 1.05x; Piper stays installed as a disabled fallback |
Cody picked it by ear as the closest to the films' Jarvis and asked for it ~1.1x faster than the 0.95x samples |
| 2026-10-14 | Every percentage in the storage report says "used" | a spoken answer read bare percentages under "Plenty of room" as free space and got it backwards |
| 2026-10-14 | The OpenProject token stays the built-in admin's | Cody: not worth a dedicated user for now |
/opt/jarvis/README.md, /opt/jarvis/docs/{ROADMAP,ARCHITECTURE,OPERATIONS,CHANGELOG,VOICE,BUILD-GUIDE,EXEC-ALLOWLIST-PROPOSAL,STORJ}.md; the code is a git repo in /opt/jarvis. To rebuild from scratch: Jarvis: how it was built. Storj node notes and numbers: STORJ.md. Version history: Jarvis: version log.LONG=1 history; then the end-to-end tests. Procedure and commands: Jarvis -> Update Procedure.jarvis_lab.py; small reversible write -> T1 with a limit; anything consequential -> a new action in jarvis_actions.py behind the approval flow (never a new tool the model can call directly).pct stop 130.