Data Faults — The Bugs That Hide in Plain Sight

I've reviewed hundreds of different firmware projects for a variety of organizations over the years, and I can tell you one of the most common faults I see is a data fault between an interrupt and the main loop.

The problem with these is that they pass everything. They pass a bench test. They pass code review. The thing seems to work flawlessly — and then it fails in the field.

Why your code can be "right" and still wrong

A data fault between the main loop and an interrupt is a subtle thing, because the compiler and the code itself are both correct. It's an architecture problem — a design problem. You're not sharing the data properly between the interrupt and the main loop.

Most of the time it works perfectly. There's no issue. But every once in a while, the interrupt fires at exactly the wrong moment — while the main loop is in the middle of modifying that variable or doing something with it. When that happens, your state machine can fail. Not because your logic is wrong, but because the compiler has no idea the interrupt exists.

And catching it is brutally hard. Say there are a few lines of code where an interrupt landing right there is a one-in-a-thousand event. If that interrupt fires maybe once an hour, you are never going to see it sitting at your desk. But ship 100,000 units, and now it's happening a hundred times an hour, out in the field. That's a disaster — and it's incredibly hard to track down, because, again, your C or C++ logic is actually correct. The flaw is in the design.

Clean code is a business decision

One of the biggest root causes I see is just bad code. When the code is a real mess, these bugs are very hard to even see, let alone fix.

So clean code isn't an aesthetic choice. It's a business decision. This class of bug can be relatively easy to find when you have clean isolation between your modules and you're not leaning on a ton of shared global variables.

It's most common in exactly the opposite situation — a pile of shared variables and flags scattered all through the program — because one part of the code might be touching a value in an interrupt while another part touches it in the main loop, or in a thread.

The same thing can happen between threads, but that's a much more well-understood problem, documented all over the place, with mutexes and semaphores to manage it. In firmware and embedded systems we also have to use interrupts, and an interrupt is a very different kind of execution than a thread.

The AI angle

AI writes a lot of code that would be correct if it ran on a computer. And because of that, AI can introduce exactly these problems — just as easily as an engineer can.

This is one of those things a quick diff won't catch. If you're eyeballing the change, the code is plausible. So the downside of letting AI write more of your firmware is that it can produce more faults like this one. The upside: if you keep your code clean and well architected, these stay easy to find and fix.

What it looks like in the disassembly

It can be hard to understand in practice. When firmware is not well architected, it may seem pretty simple and harmless to share data between the main loop and an interrupt.

Let me show you the smallest version of this bug I can fit on a page. A timer fires once a second. The main loop counts how many times it gets through its work in a second, and stores the most it ever got through. This is a basic throughput measurement you'd actually ship.

volatile uint32_t counter;      /* bumped by main, cleared by the ISR */
uint32_t maximum_count;

int main(void)
{
    /* timer already running: fires once per second */
    while (1) {
        data_processing();

        counter++;
        if (counter > maximum_count) {
            maximum_count = counter;
        }
    }
}

void timer_isr(void)
{
    counter = 0;                /* start a new one-second window */
}

Read that as C and it's fine. counter is even declared volatile, which is the thing everybody reaches for when they hear "shared with an interrupt." Now compile it for a Cortex-M4 — clang --target=arm-none-eabi -mcpu=cortex-m4 -mthumb -Os — and look at the loop body (prologue elided):

        movw    r4, :lower16:counter          @ r4 = &counter
        movw    r5, :lower16:maximum_count    @ r5 = &maximum_count
        movt    r4, :upper16:counter
        movt    r5, :upper16:maximum_count
.LBB0_1:
        bl      data_processing
        ldr     r0, [r4]        @ counter++  ── load
        adds    r0, #1          @            ── modify
        str     r0, [r4]        @            ── store
        ldr     r0, [r4]        @ re-read for the compare
        ldr     r1, [r5]
        cmp     r0, r1
        bls     .LBB0_1
        ldr     r0, [r4]
        str     r0, [r5]        @ maximum_count = counter
        b       .LBB0_1

There's the whole bug, in three instructions: ldr, adds, str.

counter++ is not one operation. It's a load, an add against a copy sitting in a register, and a store back. And volatile did not save us — volatile tells the compiler it may not cache or optimize away the access. It says nothing about atomicity. The read-modify-write is still three separate instructions, and an interrupt can land between any two of them.

How a lost counter reset doubles the measurement

Say the timer fires between the adds and the str. The main loop has already pulled 999 out of memory and bumped its register copy to 1000. The ISR runs and writes 0 to counter. Then main resumes exactly where it left off — at the str — and lays 1000 straight over the top. The reset didn't fail to happen. It happened, and then it was destroyed by a value the main loop had been holding since before the interrupt.

Now look at what that does to the number you actually care about. counter never went back to zero, so the next window doesn't start at zero — it starts at 1000 and counts up another full second of work on top. maximum_count latches around 2000. Your instrument now reports exactly double the real throughput.

That's the part that makes this one nasty. The corruption can only happen at the instant the timer fires, and the timer fires precisely when counter is at its peak. So a lost reset is never a small error — it's always a whole window doubled.

How often? The vulnerable gap is about three cycles wide. At 48 MHz with a loop body around 48,000 cycles, you're looking at roughly a 1-in-16,000 chance each time the timer fires — about once every four or five hours per unit. You will never catch that at your desk. Ship 100,000 units and it's happening several times a second across the fleet, and it reaches you as a support ticket about a throughput number that is inexplicably, suspiciously, precisely double.

And while you're in there: notice the second ldr before the compare. Because counter is volatile, the compiler re-reads it from memory, so the value being tested isn't necessarily the value that was just stored. Same class of bug, one line further down.

How I actually hunt them

When a customer brings me one of these — "we've got this weird issue, it's really rare, everything works fine except once in a while we see this bizarre fault, or it crashes, and 99.9% of the time it's flawless, and we can't generate the error" — here's roughly how I work it.

I break down the problem as they describe it. I get to understand the product and the architecture of the firmware, and then I try to understand what's actually going on.

I then confirm there is isolation where I expect it to be. If you look at the seams between the modules and the seams between execution environments (interrupts, main loop, threads, etc.) you'll start to see potential faults. Once I have a list of potential faults, I then try to figure out if those faults may explain the symptoms.

Once I have some fault candidates, if I'm trying to fix it, first I'll try to encourage the fault to happen. So if an interrupt occurs rarely that may trigger it, I'll then ensure the interrupt triggers significantly more often.

After you can trigger the symptoms, then you can work on a solution.

I then fall back on the same discipline that would have prevented it in the first place: I start cleaning things up. I create barriers between modules, between execution environments, to build a more understandable model of the software and prevent data faults.

So the fix for all of this is usually boring, and not much fun to talk about:

Fixing architecture issues
Cleaning the code
An experienced firmware engineer who understands the product

These are the same things that will keep these problems from ever happening in the first place. That's your best plan going forward.