Started 2026-10-09 with the AI-assistant work. Add new sections as problems get solved.
Don't trust a small model to act — verify. Measured on the CPU-hosted Gemma 4 26B-A4B: it invented the current time ~5–15% of the time, and said "I've added that line" without calling the tool 10 times out of 10 for one phrasing. Fix: run required tools before the model answers (clock, health), and check every "I saved it" claim against what actually happened (nudge, then force the right tool, then admit it). See Jarvis → Known Issues.
Prompt wording interacts. Adding one rule ("things said in this conversation count as memory") fixed recall (21/30 → 30/30) but silently cut first-try note appends from 42/50 to 17/50. A single clarifying sentence restored it (50/50). Re-run the live tests after any prompt change; never judge a prompt edit by one example.
Measure the rate, not the example. Single runs hid every one of these problems; 10–15 repeats per case exposed them. Run each prompt variant side by side against the same cases before adopting it.
A test that can't fail proves nothing. Two tests of mine passed no matter what (a leftover or True; a stubbed function that was never the real one). Check that each new test fails when the thing it guards is broken.
Don't make the model scan long dumps. Listing ~20 matches out of a 2,000-token state dump was fully correct only 53–63% of the time. Give it small, filtered tools instead.
On a CPU, prompt processing is the bottleneck, not generation. ~185 tokens/s means a 4,000-token prompt takes ~22 s. Keep prompts small, keep the prefix stable (prefix caching hit 98.6%), and answer simple voice commands without the model.
Server defaults can be hostile to a shared host. llama-server's default 8 GB prompt cache would have blown the RAM budget (--cache-ram 1024). ntfy's default request limit (burst 60, then one per 5 s per client IP) returned HTTP 429s during testing. Read the real memory split (anonymous vs file-backed) before judging a footprint: 27 GB RSS was really ~15 GB.
Loading models is hard on HDD-backed Ceph. A cold 14 GB model load drove I/O pressure to 40–50% for ~2 minutes, and 28 GB of downloads pushed it to ~80%. Don't hot-swap models and rate-limit big pulls.
Look at the evidence before blaming the cause you expected. The "prompt cache is making it skip tools" theory was wrong (cache on 19/20, off 17/20); testing it directly saved a wrong fix.
Never run LLM workloads on the integrated GPU. On 2026-09-29 Frigate's hardware video decoding hung the iGPU's video block, froze guests and forced a hard power-off (Frigate has decoded in software since). Any new GPU gets a soak test first, with Frigate pinned to the iGPU render node by /dev/dri/by-path so a new card cannot renumber it.
Do arithmetic in code, including dates and clock times. Small models slip on "what is 15% of 80" and worse on "tomorrow at 8am". Jarvis gets an exact calculator (an allow-list parser: numbers and a few math functions, nothing else) and the reminder tool resolves 17:30, tomorrow 08:00 and full dates itself, so the model only has to pass the words through. All 88 calculator and reminder runs in the live test (11 scenarios x 8) were correct.
Cheap, read-only guards beat pleading in the prompt. For narrow, recognisable questions (time, weather, "is anything down", "who's at the door", plain arithmetic) the bridge simply runs the tool first. A false positive costs one harmless read, while a model that skips the tool costs a wrong answer. Keep guards read-only; anything that changes something goes through the policy gate and the claim verifier instead.
Build "at most once" delivery on purpose. The scheduler marks a reminder as fired before sending it, so a crash can never send it twice; one that was due while the service was down arrives late with a "was due" note instead of vanishing. Likewise, the daily briefing refuses to fire if it is more than 6 hours late and never fires retroactively when you change its time.
Test under a realistic conversation, not an empty one. The first sweep of the v0.5 abilities was perfect on an empty chat; replaying it behind a 20-message history exposed two real failures (the model stopped using the calculator; after being caught it apologised instead of acting). Run both.
Give the assistant read-only access first, and prove it is read-only. Jarvis reads Proxmox through an API token with only the built-in PVEAuditor role, so Proxmox itself refuses writes (verified: stop, start and config changes all return HTTP 403). In Home Assistant, where a token cannot be scoped, the code allows exactly two services and pins the one write to the shopping list. Then test the limits, not just the features: deliberately widening an allow-list or sending a POST must make a test fail.
Live data beats documentation. Within hours of reading the wiki, Jarvis gave the NAS a wrong IP, because four wiki pages still said 192.168.86.100 (a July audit; the real address is .103, and .100 is the Wi-Fi extender; corrected in the wiki on 2026-10-10). Container and node IPs now come from Proxmox first, and Jarvis says so when the wiki disagrees. Documentation that a tool can quote needs the same care as code.
An error class that inherits from another can hide errors. ToolError is a ValueError; an except ValueError around "fetch then parse JSON" silently turned "Home Assistant is unreachable" into "unexpected format". Parse in a separate step from fetching, and test the error paths (the offline suite caught this the first time it ran).
Deliberately break the code to check the tests. Sabotaging 25 safety and logic properties (weakening an allow-list, dropping a rate limit, skipping a check) showed 4 tests that would have passed anyway: one only checked that something raised, one covered a flag no test set. A flaky test (message ids built from Python's randomised hash()) was found the same way.
Small models sometimes answer with nothing. About one request in sixteen (and, for one question, six in eight) ended after ~4 tokens with an empty reply. Retry (the prompt is cached, so it costs about a second), then nudge, then say so, instead of showing the user nothing.
Measure latency in a real, growing conversation, not just in repeated tests. A safeguard that took 3 s in the repeated test sweeps took 60 s in real chat: it asked the model for a tool call from a shortened tool list, which changed the start of the prompt and made the model re-read the whole conversation twice (the sweeps reused one identical prefix, so the cache hid it). Replaying one 18-message conversation and logging tokens re-processed per message showed it at once; keeping the tool list identical cut write requests from 22-40 s to 3-5 s and the mean from 10.5 s to 5.2 s. Anything that changes the start of the prompt is expensive on a CPU.
Read the audit log of real use, and never bulk-delete "test" data without checking whose it is. While the work was idle, Cody tried Jarvis from his phone; the audit log showed three problems no test had found (a pasted metrics table triggered the calculator guard and took 56 s; /clear didn't exist; "save some metrics from the lxc or proxmox" had nothing to read). It also showed that one of the notes in the notebook was his, so clean-up deleted only the test notes.
Proactive alerts need a spam budget. /watch speaks only about problems that persisted across two checks, never repeats one it already reported, treats an unreadable source as "unknown" rather than "fixed", caps itself at 6 messages an hour and stays silent during Home Assistant's quiet hours. Off by default.
Let the model propose and let code dispose. For anything consequential the model's only power is writing a proposal row; the text Cody reads is built by code from that row, an approval is Cody's own fresh message that answers one pending row exactly once, and everything (does the guest exist, is it in the right state, is the action still allowed) is re-checked at the moment of execution. Whatever the model says afterwards is replaced, so "Done, restarted!" can never be the only thing Cody sees.
Use the platform's own permission model where it is narrow enough, and a tiny forced-command wrapper where it is not. Start/stop/restart fits a custom Proxmox role with one privilege (VM.PowerMgmt) plus NoAccess on the chat channel and Jarvis itself, so Proxmox enforces the boundary whatever our code does. Editing a backup job needs Sys.Modify on / in the API, far too broad, so that and "run a command in a container" go through an ssh key that sshd restricts to one script with five validated verbs. The script verifies its own work and restores the old state on a mismatch.
A more specific Proxmox ACL replaces inherited roles. Granting JarvisOps on /vms to the Jarvis user removed its inherited audit rights below /vms, and the read-only token (capped by its user) suddenly saw zero guests. It was caught within two minutes by checking the effective permissions right after the change; always check pveum user permissions <token> --path <path> for both tokens after touching ACLs.
Test the live model with wording you did not write. Three real failures came from phrasing nobody had scripted: "power cycle it" made the model pick force-stop; "the wiki container is hung" named no guest exactly; the model passed crm (101) copied from our own listing as a target. Fixes: Cody's own words decide the action when they name exactly one; partial names resolve; the target resolver accepts our own output format.
A bare "yes" is dangerous; scope it. It counts only for what the assistant's last message asked about; anything else needs an explicit id (yes 7); talking about something else closes the window; a casual "ok" never approves a command or a force-stop.
Sabotage the safety code and demand a failing test. 64 deliberate breakages (skip the approved check, widen the POST allow-list, let auto apply to commands, accept the bot's own "yes", store command output in the log ...). The first pass caught 52 of 54 and exposed a dead check, a property no test covered, and three tests that "caught" a sabotage by crashing instead of asserting. Later live findings added more; all 64 are caught now.
A test that depends on the clock is a bug waiting for an evening. A /remind test picked "the last pending reminder" and failed only after 17:30, when another test's reminder had rolled over to tomorrow. Select by content, not position.
Check whose address a stale entry really is before "fixing" it. .100 still answered ping, so it was not simply dead: the MAC address differed from the NAS's, the NAS's own interface showed .103, and the IP plan already said .100 is the Fios extender. Only then were the wiki pages corrected (and a fifth page, reference/ports, which still said .100-.102).
Pin the dependencies of a service venv. PyAV 19 (pulled in automatically) broke faster-whisper's audio decoding with a cryptic unexpected keyword argument; av>=14,<16 fixed it. Also: do not pkill -f a name that appears in your own command line.
A model will imitate any special message format it has seen in its own conversation. After a few real approval requests were in the history, the model answered "terraria-ai is using too much RAM, shut it down" with a perfectly formatted "Approval #27: shutdown ..." that did not exist (no row, no audit entry; the end-to-end test caught it only because it checked the database, not just the text). Fixes: any model-written text that looks like an approval request is rejected (the real request is built by code and replaces the model's words); approval requests and their results are kept out of the model's conversation memory entirely (a first attempt, a neutral note in the history, was imitated too: a live sweep caught the model writing "(I sent Cody an approval request ...)" with no proposal); particle verbs ("shut it down", "turn it off", "bring it up") are recognised as change requests. Lesson: verify that the thing exists, not only that the reply says so.
Look at the proxy's access log before guessing why a client "can't". One read-only look at which clients read and which publish on the topic settled the phone question: the phone is an iPhone running the ntfy iOS app (it reads, 750+ requests) and all publishing comes from a desktop browser. The iOS app has no message box.
auto is read as ask, the approval request goes to the chat. A voice in the room is not the owner.yes 7; there is no second way to approve, so there is nothing new to audit. The table records which door it came through.Host and X-Forwarded-Proto), and enforces optimistic locking (lockVersion, retry once). Read the vendor's documentation for permissions before designing what a token may do.or True, an assertion that a sabotaged rate limiter still satisfied). Re-run it for every new safety property; an equivalent mutant (one that cannot change behaviour) is dropped, not tolerated.web_e2e9.py starts the real, sandboxed service with a temporary password, drives it like a phone, checks that the other ports are not reachable from the LAN, and removes everything again; it refuses to run if a real password exists.(to be added)
cache_n / prompt_n. Then warm the cache on the schedule on which it goes cold.IPAddressAllow= does nothing inside an unprivileged container (no cgroup BPF). Prove every network restriction from the outside: the Proxmox host connected to a port the unit claimed to restrict. Use the bind address and an nft table.whois) and keep a list of allowed device names, plus a Host-header rule against DNS rebinding.ls\nid into the valid command ls id; accept entries as written or drop them. Likewise refuse multi-line commands before normalising (df + newline + -h collapses into the listed df -h).tailscale cert./api/sno/ (disk) with the satellites' daily numbers before suspecting anything.du with and without --apparent-size, xfs_io -c stat). It changes when the drive really fills.