At 5:30 on a Monday morning in August, somebody in my lab was flashing firmware onto a PIC18 dev board, watching it answer over serial, and reading the results to see if it worked. It wasn't me. I was asleep. It was an AI agent, a few hours into a 25-task shift that ended around lunchtime with the whole fleet validated — 16 build targets on 15 physical boards, 1,390 passing tests.
That wasn't a stunt. In general, that's how part of my lab runs now.
I wrote in August that you can't manage agents like people. They're tools. This is the follow-up: you don't manage tools, you build the shop around them. Here's what that shop looks like for firmware, what it cost, and the three times the rules mattered more than the agents did.
The infrastructure
It takes a bit more infrastructure to run agents against hardware on a budget, but it's way more accessible than most people think.
It started the way it does for most of us: running agents on my laptop in a VM. The more agents you run continuously, the less you get to power off your laptop. Agents are memory hungry, especially when you're running a lot of them. I needed 64 GB of RAM to keep all of them running well, and that makes AWS or similar platforms prohibitively expensive — an on-demand x86 instance with 64 GB is over $300 a month. So I built a separate computer to run agents on, all the time.
That machine deserves its own article (it has a couple of GPUs for running and training local models), so I'll leave it there for now.
The lab
The microcontrollers and the custom USB hub are not plugged into the agent computer. I wanted a layer of isolation. So I built a network service that manages the hub and the boards, and I run it on a ~$200 mini PC. Honestly, any low-cost mini PC would work, but I'd stick with x86. I tried a Raspberry Pi first and hit problems with some of the toolchains and programmers, so I switched back.
Day to day, the lab is just a network service to the agents. They need network access and a credential, nothing else. An agent submits a build and a test script; the service power-cycles the board, flashes it, runs the script against the serial port, and hands back the results. The agents work on firmware without ever touching a programmer or a USB port.
The rig itself: fifteen microcontroller dev boards on a custom PCB USB hub, spanning six ISA families — Arm Cortex-M, RISC-V, Xtensa, AVR, MSP430, and Microchip PIC/dsPIC. One job at a time; the server queues and serializes everything. You can see the results from many of the benchmarks and experiments at /fleet.
The enclosure the rig lives in was designed by an agent in a single overnight run — 33 tasks, 10 hours 42 minutes of machine time, about an hour of mine — and printed right the first time.
The rules live in the service, not the prompt
This is the part I'd want another engineering leader to take away. Every rule that kept the lab safe in August is enforced by the infrastructure. Not by an agent's good behavior, and not by me watching.
- One job at a time. The service has a queue. Agents can't parallelize onto the hardware even if they try.
- Every job has a deadline. Fifteen minutes by default. A hung board or a runaway script gets killed by the lab PC, and the agent gets an exit code that says so.
- Bad boards are quarantined. When a board keeps failing, the service stops accepting jobs for it. No agent can submit to it. The service made that decision, and it stays made until I get a chance to take a look.
- Results are tied to the job. When a job goes to the rig, the service hands back a handle for that run. If a run fails or gets lost, the agent gets nothing, never some other run's results. So when two agents hit the rig at once, or one hallucinates a job it never submitted, neither can misread the other's output as its own.
- Retries are capped. When a board won't take an upload, the agent gets three tries. Then it stops and leaves a note.
- Decisions are pre-made. Never ask an agent to "hey, run the benchmarks." Have it write up a formal plan first, then spin off subagents (or use an orchestrator) to run that plan. The order of operations, which boards to test, and what counts as evidence of pass or fail should all be decided up front.
- Make failure safe. Agents will make mistakes. The service makes sure a mistake can't brick the hardware. Whether it's a test harness or hardware-in-the-loop, an agent error should cost you a rerun, not a board.
None of that is management. It's a shop with safety glasses for the agents.
I've had agents duplicate work when the orchestration slipped. Several worked the same repos from separate worktrees. Two of them once produced byte-identical documentation commits independently, and a third noticed and adopted the pushed one instead of duplicating the work. When they both hit the hardware, the job simply ran twice, safely. I didn't have to referee it, just clean up git, because the rules were already in place.
Hardware is slow, and the wait belongs to the lab PC
A regression test in my lab means programming a chip and waiting for the physical world. A test takes minutes, not milliseconds. Coding agents aren't built for that. Claude Code cuts a shell command off at 10 minutes by default (it's configurable), and Codex defaults to something like 60 seconds or 5 minutes depending on the version.
The service is asynchronous: submit a job, poll for status, fetch the results. The long wait happens on the mini PC, and the agent checks back. Before I had that, I watched an agent write a wrapper that printed a dummy "still running" line every nine minutes so the harness wouldn't kill it. Hacks like that are clever, but every heartbeat burns a turn.
The longest single unattended run in my logs is 16 tasks in 25 hours and 49 minutes. That only works because waiting costs an agent nothing.
Three times the rules mattered
The USB bus that locked up. Uploads fail, boards refuse to enumerate, and a bad power cycle can leave the whole hub latched in a fault. Once the lockups started, I had a separate agent remote into the lab machine and do a deep dive while it worked on the service code. It confirmed the USB bus was locking up, and it could reproduce the lockup on demand. It proposed a fix. What it couldn't do was apply the fix, because it didn't have root on the lab PC. So the next day I validated the fix myself and gave the service permission to reset the entire USB bus. That one change has handled every bus lockup since, including the ones a board with bad firmware can inflict on the rig. The agent found the hole and wrote the patch. The infrastructure made it wait for me, and a couple of days later the rig was back up.
Three tries on a board that gets one good shot. The worst board was a little CH552 (an 8051-class part) that would not take an upload. Iteration three landed a real, verified flash, and then the board went silent, because the firmware it had just proven flashable didn't bring up USB. The agent's note read like a senior engineer's: no fourth iteration, the cap says three, and nothing further could be validated without a hand on the BOOT button. It then spent 25 minutes running regression controls on the live rig to prove it hadn't disturbed anything, and surfaced an unrelated fault while it was at it: four hub ports powering up with nothing enumerating. Overnight the agents built a corrected 16 MHz image and a throwaway 24 MHz control build to prove the clock flag was load-bearing, and by morning the verdict was written down. FAIL, root cause a 24 MHz build running on a 3.3 V supply. The board's quarantined now. Three days of machine time to conclude "stop submitting to this board, here's exactly why" is not a failed run. It's the honest outcome, documented.
The green checkmarks that lied. Two boards spent a while quietly running at the wrong clock. The dsPIC was on its 8 MHz reset oscillator instead of the configured 200 MHz; the MSP430 was at half its intended rate. Every test was green. Every number was internally consistent. The data was just measuring a board running twenty-five times slower than the label said. The dsPIC jumped from rank 14 to rank 2 on one operation once it was fixed. I only caught it because the dsPIC's top-line numbers looked wrong. Once I checked the rest of the fleet, I found the MSP430 too. No service and no agent can put a probe on a pin. That stayed my job, and it's the one rule I can't bolt onto the rig.
What it costs
This is the part I'd put in front of management. It's a budget line and a list of rules, not headcount.
| Item | Cost | Notes |
|---|---|---|
| Agent workstation | $1000-1500 (more if you want GPUs) | Workstation with 64 GB RAM. The alternative is >$300/month on AWS, forever. |
| Lab PC | ~$200 one-time | Any x86 mini PC. |
| Custom USB hub PCB | ~$150 | 16 ports with per-port power control. |
| 15 dev boards | Under $500 | Most under $30 each. |
| My time | Under 2 hours a month | Triaging issues with the hardware. |
Swap the custom hub and the dev boards for your own hardware-in-the-loop setup and that's the starter infrastructure for agents doing consistent, meaningful firmware work. The only other cost is tokens, and that depends on how hard your team runs them. I've written about mine.
What you're actually buying
Looking back through a month of logs, the moments I trust most aren't the thousand passing tests. They're the stops: the agent that diagnosed the bus lockup and then stopped because it didn't have root, the retry loop that quit at three, the job that gets killed at fifteen minutes. Every one of those came from a rule in the infrastructure, not from an agent deciding to be careful.
If your agents never stop and tell you something's wrong, that's not a good sign. It usually means the gates aren't there.
So the ask to your management isn't a team of agents to manage. It's a cheap PC between the agents and anything that can break, a short list of rules the service enforces, and one person who decides what "done" means and owns the probe. If you're figuring out what that looks like for your firmware, I'm happy to talk.
The benchmark numbers these agents captured live at /fleet.