Automated Firmware Testing with AI

When a deadline gets tight and problems start piling up, one of the first things to get thrown overboard is firmware testing.

I don't mean end-to-end product testing — companies rarely skimp on that. I mean testing the firmware on its own, in isolation. And it's one of those things that's almost invisible to leadership. You can't tell that nobody wrote any unit tests if the product seems to work fine. The trouble is, some of those problems won't surface until you've shipped 100,000 units. Now you've potentially got a recall on your hands.

Firmware testing is the kind of thing that builds quality over time. It lets you hold a really high bar without having to make big, repeated investments — and AI can do a lot of the heavy lifting on the harnesses and tooling around it.

There are many types of full product testing that are great, but they are all limited by the fact that they are testing the product as a whole.

They cannot test every possible edge case in the firmware — often they cover very few edge cases. They are also very costly to run, so they are run infrequently.

Types of Firmware Testing

The best way to catch these edge cases is to have a robust firmware testing process that runs continuously during development.

This isn't an exhaustive list — it's the six types of testing where AI tooling can help you right now. I'm leaving static/dynamic analysis, compliance testing, and testing done outside of engineering off the table here.

Unit tests on a PC. You compile part of your program into a PC application and run it there.
Unit tests on the real device. Same idea — testing submodules and subcomponents — but running on the actual target hardware.
Simulation. You run a simulator of the processor and execute your actual firmware binary against it, either for unit tests or to simulate full end-to-end system behavior.
Regression testing. You run a set of tests that you know should pass, and confirm they do. This can be done on a PC or in simulation, but ideally it's done on a device.
Hardware-in-the-loop. The hardest one, and I think the most important. You test your firmware fully end-to-end against the real hardware you plan to ship.
Full product testing. This is the one that all companies do to varying degrees.

My suggestion for most firmware projects: In addition to full product testing, do at least some unit testing on critical parts of your firmware, then build a hardware-in-the-loop test harness. Once you have the harness hardware, you can use it for continuous integration testing and regression testing.

That combination is the gold standard and saves money across the product lifecycle.

AI won't fix bad architecture

One thing up front: AI is not going to rescue firmware that's poorly architected and poorly written. If you've got massive globals shared between every routine in the application, there's not much it can do — in exactly the same way an engineer would want to clean up those code-quality problems before writing any tests.

Unit tests work best when there's real isolation between your modules and the modules are well-defined. You can't manufacture that after the fact, because the modules are the thing the tests are evaluating.

But assuming your system has at least decent bones, you can literally ask a frontier model to generate test cases — or really, test harnesses — for your code, at least to run on a PC. Running them on-device usually takes more work, and that's not something AI is going to do all of for you. It's still a great starting point.

Here's a concrete example. Say you've got a CRC routine with a bug that fails when it hits a null. It's pretty easy to have a unit test, running on your computer, that exercises essentially every possible value of an incoming byte — and it'll run in a few milliseconds. That's a bug that would never show up in product-level testing, but absolutely will in the field.

Hardware-in-the-loop

This is where you've got a simulation of the rest of your product talking to an actual board.

I say simulation deliberately. Sometimes there's a motor, or some physical reality, you don't actually want in the loop — say you're moving an arm, and you don't want a real arm in the rig. The point of hardware-in-the-loop is full visibility from your inputs to your outputs: you apply the same inputs and confirm the outputs don't change.

The really nice part is that you can often run these boards much faster than they run in production. You can exercise different components and subsystems, and you can replay the exact same scenarios over and over.

To pull it off you usually need a few pieces. You need a harness on the capture side and the output side, which often means designing a small piece of hardware — an Arduino-based board works well — plus firmware for it, plus a software tool to drive it.

I'll be completely honest: without too much work, you can get most of that out of AI boilerplate generation. As long as you're over-provisioning on the Arduino/hardware side, the model usually does a pretty good job here. And you're not really risking anything — this isn't the AI writing code that leads to a recall.

Worst case, you get some flaky tests you'll have to clean up later.

Why AI is actually good at this

Here's a useful way to think about it. The frontier models have ingested millions of unit tests across all kinds of software. In a lot of cases that gives them better instincts for writing unit tests than a firmware engineer has — not because the engineer isn't sharp, but because writing tests isn't the main focus of the job.

So you can hand it the shape of what you need: "I need these tests, I need to cover at least this much, here are the specs I'm going for, here are the cases I want covered." Then ask it — do you suggest any more? It'll usually add test vectors you hadn't thought of, ones you can run through your unit tests or your hardware-in-the-loop rig.

But your engineers are still the ones deciding: is this the right coverage? Is this the right test to run? Those are calls the AI can't make, because it doesn't understand your product the way you do. Its product knowledge will never catch up to a person's. Brand risk, recall risk, safety — it grasps those a little in the abstract, but most of what it's seen is business application software. It doesn't internalize hardware the way we have to.

Where I'd start

If you just want to test the waters, start with unit tests that run on a PC.

I often recommend Unity from Throw The Switch — it's a C-based unit test framework I've used plenty to run unit tests on a PC, and with more tooling you can run it on-device too.

For smaller applications — say 64K of flash or less — running on-device adds some complexity. Sometimes microcontrollers have too little RAM/Flash to be unit tested on board. It tends to rely on standard out being redirected to a serial port, plus some tooling to catch it. All of it is usually viable, just more work.

So if you've got no unit testing today and don't even know how to set it up, honestly: point a frontier model at that library and at your firmware and say, "Can you write some unit tests using this, to run on my PC?" It may come back and say it needs to make some modifications to your firmware to make that work — and then your engineers get to make some choices. But the concept is simple, and it's a real way to find bugs that would otherwise wait to bite you in the field.

The same goes for hardware-in-the-loop. My preferred stack is as much full end-to-end testing as I can get, with a hardware simulator standing in for the rest of the product. I usually do this in Python — a small piece of code runs on an Arduino (or similar), communicates over USB (sometimes through a USB isolator) to a PC, and talks continuously to Python on the other end. Then I get Python-based unit testing, regression testing, and a lot of other fun ways to exercise every part of the firmware.

Hardware in the Loop Diagram

The real engineering cost is the planning & build of the simulator itself. It requires custom software and often some custom hardware.

This type of test setup keeps providing value. It makes product pivots more stable, so you can test the new hardware easily. If you port to a new microcontroller — great, do it and re-run the regression test.

Want a new junior team to do the port — go ahead! The harness gives them guardrails to ensure quality and provide feedback.

Want an AI to do the port? The same rules apply.

Say an intermittent connection issue shows up in alpha testing. Great — add support in regression testing to simulate the disconnection to replicate the issue. Once we can simulate it, it's easy to fix the firmware.

More value: Years after shipping you get a report from the field that some bug happened. You can add a test to your hardware-in-the-loop setup to reproduce that bug in real hardware. Now it's easy to detect it, fix it, and prove you fixed it, without ever having to fully recreate the situation that caused it.

You can also re-run the regression tests off the latest production boards years later to ensure quality.

These are all things you can lean on AI for. Just remember what it is: a tool for your product team to use to build a better product.

Next week I'll show you what this looks like in practice: test-driven development on an 8-bit PIC, where the whole test rig — including the hardware — is software.