I Saw a Video About Hermes. Here's What Happened.

Four days ago I watched about 11 minutes of a 29-minute YouTube video — Tina Huang’s “Hermes Agent Fundamentals” — about a self-hosted, terminal-native AI agent runtime. I wasn’t looking for a project, and I didn’t even finish the video. I watched enough of it, half paying attention, to get curious — then closed the tab and went to go actually try the thing instead of watching someone else use it.

Here’s what happened next.

Where I was 4 days ago

I’ve been running a homelab for a few years — the usual DIY home automation path: Home Assistant, a Proxmox cluster, OPNsense doing router duty, UniFi APs and switches, the whole self-hosted stack. It works. It also accumulates debt the way every side project competing with a full-time job does: things you know are wrong, that you mean to fix “this weekend,” and then don’t, for a year. And some of it isn’t even debt you know about — it’s just running, quietly, until something goes and looks.

A few concrete examples of what had been sitting, untouched, for longer than I’d like to admit:

• My SSO provider (authentik) was running on Ubuntu 23.10 — end-of-life since July 2024. Thirteen-plus months with zero security patches, on the one box that gates login to everything else.
• My UniFi Network controller was 15 months and six minor versions behind, stuck because upgrading past 8.5.6 needed a newer MongoDB that requires AVX CPU instructions — and the Proxmox node it lived on doesn’t have AVX. That’s not a “run apt upgrade” problem, that’s a “read the MongoDB changelog and understand your CPU’s instruction set” problem, and it’s exactly the kind of rabbit hole that eats a weekend and then gets shelved.
• Outside the homelab entirely, two more things had been quietly stuck: a public tool I’d written and open-sourced (HA-Chromium-Kiosk) hadn’t shipped a release in over a year, sitting on 12 open GitHub issues from real users I’d never gotten back to. And this very blog had been stuck on WordPress.com’s Premium plan for years, unable to build the structured watch-collection catalog I actually wanted because the platform’s pricing tier flatly doesn’t allow custom plugins or themes.

Some of this I knew about and had been putting off. Some of it I genuinely didn’t know was there until an agent went looking — a good reminder that “I know my homelab has debt” and “I know what the debt is” aren’t the same claim.

What changed: how this actually built up, step by step

Four days ago this was just me and one AI assistant talking in a terminal. What’s running today didn’t arrive all at once — it built up in a specific order, and the order matters if you’re thinking about trying this yourself:

1. Started with just the two of us. No persistence beyond a chat session, no standing infrastructure — just me directing an AI assistant through the recon/fix work above, one conversation at a time.
2. Added memory. Facts that should survive across sessions (how my network is laid out, my own conventions and preferences, standing decisions already made) stopped needing to be re-explained every time.
3. Built a structured ontology. Before letting anything act autonomously, I wanted a defined, explicit model of what actually exists in my homelab — not a pile of loose notes, but named domains and categories with rules for what belongs where. More on exactly what’s in it below.
4. Stood up specific agent bots, each in its own OS user space, running autonomously. This is where it stopped being “me driving a chat” and became standing infrastructure: an ops agent, a vault auditor agent, and a coding agent — each its own Linux user, own credentials, own process, running independently of whether I’m actively at the keyboard.
5. Wired them to Telegram — to me first. Each agent got its own bot, locked to just my Telegram ID, so I could direct and check in on any of them from my phone, not just a terminal.
6. Then wired them to each other. A peer-to-peer mesh — any agent can message any other agent directly and get a synchronous reply, without going through me as a relay.
7. Started building the policy foundation that governs how they coordinate. Once agents can act autonomously and talk to each other, the real question becomes: who’s allowed to decide what, and who has to ask first. That’s now a standing set of written policies, not tribal knowledge in my head. Detail below.

What got fixed in 4 days

The honest framing isn’t “hours saved” — it’s that several of these had a low probability of ever getting fixed without dedicated, patient effort grinding through platform-specific gotchas I wasn’t going to sit down and learn on a Tuesday night. What actually happened, on the infrastructure side and on the coding side at the same time:

Worth being specific about how this actually ran day to day, because it’s easy to picture me hunched over a terminal for 4 straight days, and that’s not what happened. All three agents were working in parallel — one on infra, one on the vault, one on code — with me supervising as needed rather than driving every step. When I was home, that meant checking in from the desktop. When I wasn’t, it meant the same thing from Telegram on my phone. This landed over what was almost a normal long weekend for me — I wasn’t chained to a desk for it; I went and did the things I’d normally do, and came back to review, redirect, or approve whatever needed a human call.

Day 1 — First full recon pass. Mapped the entire network, identified unlabeled devices, found things I’d genuinely forgotten existed (a decommissioned pihole, a deleted Nightscout instance still showing as a VM).

Day 2 — The big one. A full infrastructure audit surfaced ~30 issues in a single day, most fixed same-day: stale DNS entries, a dead VLAN, backup jobs that were silently misconfigured, IoT devices offline for reasons nobody had diagnosed. The ops agent came online this day. First security pass found some real gaps that had been running quietly.

Day 3 — The vault auditor and coding agents came online, and all three got wired into a working mesh. Also: the authentik EOL rebuild (fresh LXC, Postgres data migrated, zero downtime cutover), and the UniFi controller finally jumped from 8.5.6 to 10.6.101 — 15 months of version debt closed in one sitting, including figuring out the AVX/MongoDB workaround properly instead of just hacking around it.

Same day, on the coding side: a full review-and-fix pass on HA-Chromium-Kiosk, the year-stale public tool from earlier in this post. A single review-and-fix session:

• Full read-through plus static analysis (bash -n, ShellCheck) surfaced 8 real bugs beyond the 5 already-open GitHub issues — including the actual root cause of a long-standing install-breaking bug and a full-abort bug where declining to overwrite one config file killed the entire install.
• Fixed and shipped 9 of 12 open issues, with commits, not just diagnosis.
• A clean bash -n/ShellCheck pass doesn’t prove a shell script actually works. That pushback led to standing up a real Docker container running genuine systemd (not a mock), which caught a bug static analysis structurally could not see: a heredoc-generated runtime script whose network-check loop had its guard variables expand at install time instead of kiosk-boot time, so it silently reported “Home Assistant reachable” even when nothing was listening. Found, root-caused, fixed, verified against both an unreachable and a real live Home Assistant instance.
• A 6-level test plan with a regression table, then a 19-assertion integration harness wired into CI on GitHub’s own hosted runners — so the class of bug above can’t silently come back.
• A year-old abandoned feature branch got reviewed, the genuinely good parts (color output, install summary/confirmation) kept, and rebuilt clean on top of the fixed code instead of merging bugs back in — shipped as a real PR with CI green.
• Three tagged releases, checksums verified end-to-end against the actual downloaded artifact — not just written into a changelog and trusted.
• A full real-world validation pass: a real Home Assistant instance stood up, the kiosk installed against it for real, watched it report success against the live server, screenshotted the whole thing, and used those genuine artifacts — not mockups — to write a how-to guide.
• One real compatibility break found along the way — a plugin the how-to guide had assumed worked no longer functions against current Home Assistant versions, a well-documented, recurring pattern on the plugin’s own tracker. The honest call was to document the limitation and exclude it from the demo, not paper over it or pin an old version to fake a clean result.

None of the individual kiosk issues were hard. What made them pile up for over a year is the same dynamic as the infra debt: fixing a shell script correctly means reading the whole thing carefully, understanding what each prompt/flag does at install time versus at kiosk-boot time, and then actually testing it — none of which fits into a stolen 20 minutes between meetings. Nothing here needed novel engineering; it needed someone to sit with the whole script, the whole issue backlog, and a real test environment for an extended, uninterrupted stretch.

Day 4 — The big structural project: physically swapping two Proxmox boxes between rooms (so Home Assistant’s USB Zigbee/Z-Wave dongles never had to move), building a new network bridge, standing up UniFi OS Server, and splitting an old catch-all VM into three properly isolated containers. All done with the cluster staying live and every step independently verified before moving to the next. Same day: migrated all 5 physical UniFi devices (2 APs, 3 switches) off the legacy controller onto UniFi OS Server — hit and fixed three real failures along the way (a corrupted migration attempt, a stuck adopt flag causing a connectivity flap loop, and a config restore that reported success but silently never reached one switch’s actual hardware, caught only by verifying against the device directly instead of trusting the API response). Legacy controller decommissioned once backups were confirmed.

Same day, on the coding side: this blog got migrated off WordPress.com, the other piece of stuck debt from earlier. Wanting a proper structured watch-collection catalog (not just tagged posts) had run straight into WordPress.com’s Premium plan limits — no custom plugins, no custom themes, full stop. In about a day of agent-driven work:

• Rebuilt from scratch as an Astro static site — real content collections for both blog posts and a proper watch catalog with structured fields (brand, reference, movement, case dimensions, acquisition story, photo gallery), not tags standing in for a database.
• Migrated the real content: every existing post pulled off the live WordPress site — including their original inline images, not just the text — converted to clean Markdown, verified against the actual rendered output rather than assumed correct.
• Migrated the watch collection from a personal spreadsheet, including the embedded photos (Google Sheets export → xlsx → parsed the drawing XML to map each image to its correct row — not something obvious the first time through).
• Deployed as a Cloudflare Worker, then a second, already-owned domain’s DNS migrated to Cloudflare and wired up as a custom domain — verified live end-to-end from outside the network, not just “the API call returned success.”
• The whole deploy pipeline (build + deploy) now runs directly with a scoped API token, so future posts and watches ship without a manual dashboard click.

Same shape as the kiosk work: not one hard problem, several small platform-specific frictions (WordPress.com’s login bot-check refusing automated access, mapping spreadsheet images to rows, Cloudflare cert provisioning delays) that would each cost real focused time to work through, none exciting enough to actually block a weekend for. You’re reading the result of that migration right now — including this post itself, drafted the same way as everything else.

Where I am today

• A rebalanced 4-node Proxmox cluster with no node overcommitted on paper, the router node reduced to a single purpose (minimizing blast radius on the one box where downtime kills the whole LAN).
• UniFi fully migrated — current software, all physical devices (APs + switches) live on UniFi OS Server, legacy controller decommissioned.
• authentik off an EOL OS.
• The security gaps I knew about (and a few I didn’t) tracked and mostly closed, with a fresh, more thorough audit now open and scoped rather than an ad-hoc “I should really look into that” list.
• A structured knowledge base of my entire homelab — over a hundred notes, auto-maintained by a dedicated auditor agent — that didn’t exist before and now can’t silently rot the way tribal knowledge in my own head does.
• A public tool (HA-Chromium-Kiosk) brought current after a year stale, with real releases and real test coverage.
• This blog, off WordPress.com and onto a self-managed static site on a custom domain.
• Three AI agents that keep working on this even when I’m not actively driving, escalating to me only when something genuinely needs my judgment or elevated access.

The AI setup, for anyone who wants to replicate this

• All three agents run on Hermes, an open-source, terminal-native agent runtime — self-hosted, one shared install on my own box, each agent as its own isolated OS user and profile (own credentials, own memory, own skill library).
• Each agent got its own Telegram bot, so I can direct or check in on any of them from my phone, not just a terminal — bot identities aren’t shareable between agents, a platform constraint, not a design choice.
• The ops agent has scoped API credentials to Proxmox, OPNsense, TrueNAS, Home Assistant — not raw root everywhere, though tightening that scoping further is itself one of the open follow-ups.
• Documentation lives in Obsidian, plain markdown files, so any agent (or me) can read/write it directly without a proprietary format or API in the way.
• Escalation policy: agents without elevated access ask the one that has it, and wait for a reply, rather than acting alone.

The ontology: what’s actually in it

Here’s the part I think matters most for anyone trying this themselves, and it’s the piece that’s easy to skip: none of the above works for more than a week without something keeping the knowledge base honest.

Every homelab person has tried the “write it all down in Notion/Obsidian/a wiki” project at some point. It works for the first week. Then a device gets replaced, an IP changes, a service gets migrated, and the note doesn’t get updated — because updating docs is always the first thing that slips when you’re heads-down actually fixing something. Eighteen months later the wiki actively lies to you, which is arguably worse than no wiki at all.

So before any agent was allowed to act autonomously, I wanted an explicit ontology — a defined structure for what kinds of things exist, where each kind of thing lives, and what “done” looks like for a note about it. It’s set up as a root index (currently one domain, Homelab, with room for more — Code, People, whatever comes next — following the same pattern) and each domain owns its own detailed structural index underneath.

What’s actually in the Homelab domain today:

• Infrastructure — split Physical (the actual boxes: Proxmox nodes, switches, APs, storage) from Virtual (the VMs/LXCs running on them) as separate categories, on purpose — a physical box and the guest running on it are different entities with different lifecycles, and conflating them is exactly the kind of drift the ontology exists to prevent.
• Applications — one note per running service (Home Assistant, OPNsense, Nginx Proxy Manager, Jellyfin, Frigate, and so on), each tagged to the guest it runs on.
• Network — LAN, WAN, WiFi, VLAN, Firewall, Routing as their own notes, not folded into a single “networking” catch-all.
• Automation — Home Assistant-specific structure: Areas and Rooms, Automations and Scenes, People and Presence, Device Catalog, Voice Control, Automation Platform.
• Agents — one note per AI agent plus a roster index, covering identity, access, credentials held, and role.
• Policies — the standing rules governing how work gets done (see below) — deliberately separate from Issues (specific problems) and asset notes (what a thing is).
• Issues — the working issue tracker, Open/Resolved, one file per issue, sequentially numbered, the place all day-to-day investigation and fix work actually gets logged.
• Code — a newer category for anything that’s software development rather than infrastructure (this blog post started life here, in fact).

And critically, there’s a dedicated agent whose only job is enforcing that structure — not fixing my homelab, just keeping the map of it accurate.

It doesn’t touch Proxmox or OPNsense or any live system — it has vault read/write and nothing else, deliberately provisioned with zero elevated access. What it does:

• Checks every wikilink in the vault actually resolves to a real note (catches typos and renames that would otherwise silently break navigation)
• Finds orphan notes — pages nothing links to, which in practice means pages nobody will ever find again
• Validates the topology canvas — every device box on the visual map actually points at a real note, no duplicate diagram elements
• Checks each category folder actually contains everything that fits its own stated definition
• Watches naming conventions hold as new notes get added
• Catches “entity vs. implementation” conflation — a subtle one: is a page about the UniFi controller as a concept getting mixed up with the specific LXC it happens to run in this week, which matters because the LXC changes over time and the concept doesn’t
• Flags cross-domain naming collisions before they become two different pages that mean different things

When it finds something mechanical — a broken link, a stale canvas reference — it just fixes it directly. When it finds something structural that needs a judgment call, it opens an issue for whichever agent owns that domain to decide, rather than silently restructuring content it doesn’t own.

The practical effect after 4 days: over a hundred notes, and I trust every link in that vault to actually go somewhere real, without having spent any of my own time policing it. That’s the difference between a wiki and a wiki that stays true. The ontology is what makes the timeline in this post citable — every claim above traces back to a dated note or issue, because the structure that makes that possible is itself actively maintained, not a one-time setup.

The policy foundation — how the agents coordinate with each other and with me

Standing up autonomous agents that can act on real infrastructure and talk to each other directly raises an obvious question: who’s allowed to decide what, without asking me first? I didn’t want that living as tribal knowledge in my own head or buried inside each agent’s private instructions — so it’s written down as its own category, binding on every agent, not just a suggestion:

• Change Approval — draws the line between what needs a written plan reviewed and approved by me before execution (VLAN/topology changes, firewall rules, anything touching network segmentation, new access grants with real security implications) versus what an agent just does directly (service restarts, config corrections, DNS/proxy updates — anything with an easy rollback and low blast radius). Also sets a “one issue at a time” convention rather than batching unrelated fixes together.
• Peer Agent Escalation — governs how an agent without elevated access handles hitting something it can’t do itself: message the agent that does by name, self-identify clearly, and actually wait for a reply before proceeding — not notify-and-continue.
• Issue Tracking Conventions — the mechanical rules (numbering, frontmatter, open/close/reopen) that keep 50-plus issues consistent enough to actually search and trust.

This is early — the mesh and the escalation rule are only a day old as of this post, and I’ve already found a real gap worth being honest about: in a synthetic test, one agent didn’t fully trust a self-identified message from another over the peer channel, and defaulted to escalating to me directly instead. Real-world precedent shows the intended pattern working correctly, but the synthetic test is a reminder that a written policy and an agent reliably following it under adversarial-looking conditions are two different things. I’m treating that as a known open question, not something to paper over.

How much effort was this, actually

I don’t want to just say “AI saved me a ton of time” and leave it vague — that’s the kind of claim that means nothing. So here’s a real estimate, scoped honestly:

• Calibrated to my own skill level, not a professional IT admin’s. I’m a software engineer by trade — fast at anything adjacent to that, noticeably slower at rarely-touched, platform-specific internals (Proxmox’s LXC/VM quirks, OPNsense’s split REST/legacy-PHP auth model, UniFi’s MongoDB/AVX packaging, WordPress.com’s plan restrictions, the internals of an xlsx file’s embedded images). Those platform-specific internals are also exactly where the agents themselves spent the most time — the boring, grinding part, not the interesting part.
• Two categories worth calling out on their own, because they’re easy to undercount: standing up and configuring the three agents themselves (own OS users, credentials, Telegram bots, escalation policy, a live-tested peer mesh) was roughly 10–16 hours — a real, mostly one-time cost most “AI saved me time” claims don’t mention. And finding problems — full inventory sweeps, security audits, root-causing ambiguous devices — was a separate 15–22 hours from fixing them. That discovery work is exactly the part that doesn’t happen at all in most homelabs; issues get found reactively, one at a time, when something breaks, not via a deliberate audit that produces dozens of dated, structured write-ups.
• Infrastructure side: roughly 85–131 hours of hobbyist-adjusted effort in total — agent setup, issue discovery, device/DNS/IoT cleanup, the authentik/UniFi/security-gap tech debt, and the full cluster rebalance + physical hardware swap.
• Coding-agent side: roughly 43–71 hours — the HA-Chromium-Kiosk review/fix/test/release cycle from Day 3, and the blog rebuild + content migration + domain cutover from Day 4.
• Combined: roughly 128–202 hours, done across 4 days of actual wall-clock time — but not 4 days of my time. That gap is the whole point: the agents ran this in parallel while I supervised asynchronously, from my desktop when home and from Telegram when I wasn’t, over what was otherwise close to a normal long weekend.

The hour count isn’t really the point, though. The more honest framing is this: a meaningful chunk of that time wasn’t “would’ve taken me N hours eventually” — it was work that had already proven it wasn’t going to happen. authentik sat on an OS unpatched for 13+ months. UniFi was 15 months and six versions behind, blocked on an obscure CPU-instruction-set incompatibility I wasn’t going to research on a Tuesday night. A public tool I’d open-sourced sat with zero releases and zero closed issues for over a year, despite none of the individual bugs being hard — they just needed someone to sit with the whole thing, uninterrupted, which a side project competing with a full-time job structurally doesn’t get. And this blog sat on a platform that had a hard “no” built into its pricing tier for a feature I actually wanted.

Some of that debt was visible to me the whole time, correctly triaged as “not this weekend” for months, sometimes over a year, against a job that doesn’t leave slack for grinding through platform gotchas. Some of it — the security gaps in particular — I didn’t know existed until an agent went looking. Either way, the honest measure of what changed isn’t the hours — it’s that the probability of any of it getting found or fixed without dedicated, patient effort was low, and now it’s done.

What’s next

The immediate open items: a deeper security audit now that the dust from the rebalance has settled, and a few pieces of debt that are next in the queue but not yet touched (my own router’s DHCP is running on a legacy system that’s itself heading toward end-of-life; a few IoT devices are still unexplained no-shows on the network).

But the more interesting “what’s next” is structural, not a punch list. This setup is 4 days old and already the simplest version of itself it will ever be. I can already see where it gets harder:

• More agents. Three is manageable to reason about by hand. I expect to spin off more as new domains show up (this blog is one early example — it started as something the coding agent did as part of its remit, but it could just as easily become its own agent later). Each new agent multiplies the coordination surface, not just adds to it.
• Policy enforcement, not just policy writing. Right now the Change Approval and Peer Agent Escalation policies are documents the agents are instructed to follow — and the one real gap I’ve already found (the synthetic peer-trust test above) shows that’s not the same thing as a policy that’s actually enforced. At some point “written down and followed most of the time” needs to become “structurally can’t be violated,” and I don’t yet know what that mechanism looks like.
• Ontology complexity. The current structure comfortably covers one domain (Homelab) plus a newer second one (Code). It was designed assuming a handful of domains. I don’t know yet whether the same flat structure holds up at 5-10 domains, or whether it needs its own hierarchy — an ontology of ontologies, which is exactly the kind of thing that sounds absurd until it’s suddenly necessary.
• Graphs and RAG. A hundred-plus notes is small enough that “does this link resolve” and “read the whole file” both work fine as strategies. That stops being true well before it gets to the thousands. At some point pure wikilink traversal and full-file reads won’t scale, and something closer to a real knowledge graph with retrieval will replace “an agent reads everything.”
• Loop engineering. Right now every agent is fundamentally reactive — something happens (I ask, an issue gets found, a peer message arrives), the agent responds, then it’s idle again. Real standing infrastructure eventually needs actual closed loops: scheduled re-checks, drift detection that runs without me or another agent triggering it, monitoring that feeds back into action on its own.
• Token/cost optimization. None of this has been cheap to run in terms of model usage, and I haven’t measured it carefully yet. At the current scale that’s fine to ignore. It won’t stay fine to ignore once there are more agents doing more continuous work rather than bursty, human-triggered sessions.

None of this is a plan yet — it’s the honest list of things I expect to become real problems, in roughly the order I expect to hit them. I’d rather say that plainly now than pretend the current setup is a finished system instead of an early one.

And yes, I notice the irony. Multi-agent orchestration, policy enforcement, knowledge graphs at scale, closed feedback loops, cost optimization under load — that list above is basically an enterprise-architecture roadmap, and the CTO in me can’t unsee it. At what point does this stop being a homelab and start being an actual platform? I don’t know yet, and I’m in no hurry to find out. For now it’s my sandbox — the one place I get to make those calls just for the fun of it, with nobody’s quarter depending on the answer.

A note on how this post itself was made, in the spirit of the rest of it: this was drafted collaboratively by the three agents described above — pulling from their own real logs, commits, and issue tracker entries — and then edited, fact-checked, restructured, and approved by me before publishing. Human in the loop, the whole way through.