From 25 million instructions to one faulty skip: AI-driven fault analysis of a secure boot

In part 1, we argued that the walls around firmware analysis are coming down when an AI agent drives the right instruments: esFirmware to run, trace and fault firmware, and Epoch to turn an execution into a time-travel trace. This post opens the case behind that claim.
The target is the boot ROM of a widely deployed ARM-based router SoC. It loads a second-stage bootloader (BL2) from flash and checks its authenticity and integrity before running it. We gave it a tampered BL2, which it correctly rejected. The question: can a single, well-placed fault turn that rejection into an acceptance?
The short answer is yes. One 2-byte instruction, skipped once, flips the verdict, despite several countermeasures in the code. The rest of this post shows how we got there, step by step, and what we could and could not prove.

The setup: an AI agent did the work
Every step below was carried out by an AI agent, Claude Opus 4.8, driving our tools directly. The analyst set the goals, chose the fault model and the success criterion, and reviewed the evidence. The bring-up, the reversing, the campaign set-up and the trace analysis were done by the agent.
A frontier model is not a requirement. On other projects, we saw open-weight models do the same kind of work. They are less capable, but they run entirely on our own infrastructure. For firmware that must not leave the organisation, that trade-off matters: the binaries, the traces and the findings stay in house.
At a glance

Each stage narrows the problem: the full trace locates the verdict, the snapshot cuts each run by a factor of about 600, the campaign finds the one vector, and two short traces prove it.
The target and the fault model
We started from two raw dumps and no documentation:
The boot ROM embeds many protections against the fault injection. It verifies the BL2 with an RSA-PSS signature (RSA-2048) over a digest of its content. The comparison itself is hardened, there are checks to make sure that the code followed the expected flow and a dedicated constant-time compare, and verdicts encoded as redundant 32-bit patterns (0xAAAAAAAA for accept, 0x55555555 for reject) rather than a simple 0 or 1. These are classic countermeasures against fault injection.
To create a meaningful test, we modified a few bytes of the BL2. The signature no longer matches, so a correct boot ROM must reject it. That rejection is our baseline.
The fault model is instruction skip: a single instruction is not executed, as if a glitch had made the CPU jump over it. It is the most common effect of clock and voltage glitches, and a good first model for a campaign. The success criterion is simple: the verdict word becomes 0xAAAAAAAA for an image that should have been rejected.
Step 1: bringing the boot ROM up in emulation
A boot ROM talks to hardware from its first instructions: timers, clock and power controllers, the flash controller, a hardware hash engine. None of it exists in a generic emulator, and none of it is documented. This is where emulation projects usually stall.
To get past this wall, we built a dedicated tool whose only aim is rehosting a binary: making it run outside its original hardware, on an emulator, with enough fidelity to execute its real code paths. It combines two sources of insight. Our TTD capabilities let the agent record each emulated run and walk back from a stall to the exact instruction and register access that caused it. Open-source static analysis tools give it the surrounding code, the cross-references and the functions that touch the same peripheral.
With this tool, the agent worked through the bring-up as a loop:
- Run the boot ROM in emulation and record the run until it stalls, typically spinning on a peripheral register that never changes.
- Walk back in the trace to the access that blocks execution, then read the code around it with static analysis to understand what the ROM expects from that register.
- Add a hook or a stub that answers what the ROM expects, then run again.
Each iteration moved execution further. In the end, a small library of hooks was enough: a ticking general-purpose timer, stubs for the hardware blocks the ROM only polls, and a software implementation of the hash the ROM normally offloads to a hardware engine. With these, the ROM runs its real secure boot path, on unmodified code.
The agent also located the BL2 in the flash dump by parsing its header and checked it was bit-for-bit identical to the image the ROM loads at runtime.
The workflow spans two emulators, by necessity rather than by choice: QEMU for fast, many-shot fault campaigns, Unicorn for recording time-travel traces. Hooks and stubs had to behave the same on both. The analyst never had to handle that plumbing directly. Everything was managed through the AI agent.
Step 2: recording the whole secure boot with Epoch
With the boot ROM running, we recorded one complete execution with the tampered BL2 as an Epoch time-travel trace: about 25 million instructions. It covers the whole story, from reset to the end: initialisation, loading the BL2, hashing it, verifying the signature, rejecting it, and halting.
Three landmarks in that trace frame everything that follows:
This trace is the ground truth for the rest of the work. Every hypothesis the agent made later, about where the verdict lives or how it is used, was checked against it. And because it is a recording, not a live session, the agent could query it as often as needed without ever re-running the boot.
Step 3: walking back from the verdict
With no symbols, the boot ROM is a flat block of code. Instead of reading it top-down, the agent started from the end: the halt. From there it navigated backward in the trace, asking at each step which instruction produced the value that led here.
That backward walk reconstructed the authentication chain, which we named as follows:
secure_memcmp constant-time compare of the computed and expected values
-> RSA_PSS_verify turns the compare result into a verdict word
-> verify_sbc_header reads the verdict and decides accept / reject
-> boot authentication -> download session -> jump to BL2Two facts from the trace shaped the fault campaign:
- RSA_PSS_verify always returns 0. The verdict is not in its return value but in a field of a descriptor structure, at offset 0x40.
- That field is written by one of two stores: 0xAAAAAAAA on the accept path, 0x55555555 on the reject path. A single conditional branch decides between them.
In a listing, these are easy to miss. In the trace, there is a query: "who wrote this memory word, and with which value?"
Step 4: snapshots, or why we never replayed 25 million instructions
A fault campaign runs the target thousands of times. If each run replays the full boot, 25 million instructions per run, the campaign becomes slow and expensive, and almost all of that time is spent before the code we want to fault.
The trace offers a shortcut. At the entry of RSA_PSS_verify, the agent reads the exact CPU registers and the memory it uses from the Epoch trace: the descriptor, the computed digest, the expected value, the stack. That state became a snapshot loaded into esFirmware. Each faulted run then starts right there and executes only the verification routine: about 43,000 instructions instead of 25 million, a reduction of roughly 600 times.
Two checks made this safe:
- Replayed from the snapshot without any fault, the routine reaches the same reject verdict as the full trace.
- The inputs are the real ones from the tampered boot, not synthetic values.
The agent proposed this optimization on its own once it understood the success criterion: the only thing that matters is the verdict word, and the verdict word is decided inside one function.
Step 5: the fault campaign and the single vector
The campaign ran in esFirmware, which handles fault injection natively. The analyst gave the fault model (skip one instruction) and the success criterion (the verdict word at descriptor offset 0x40 ends up as 0xAAAAAAAA). The agent turned that into an esFirmware campaign: start from the snapshot, sweep every instruction of secure_memcmp and of the RSA_PSS_verify code around it, inject one skip per run, and classify each run against the success criterion.
esFirmware brought three things a home-made script would not:
- The same rehosted environment. Faulted runs reuse the hooks and stubs from the bring-up, so a fault is tested on the same execution as the baseline, not on an approximation of it.
- Fault models as configuration. Instruction skip, its size, its location and the success check are parameters of the campaign, not code to write and debug. The agent spent its effort on choosing what to fault, not on building a harness.
- Reproducible results. Each run is defined by its fault and its starting state, so any interesting run can be replayed on its own, and recorded as a trace for step 6.
On this target the campaign stayed small. The benefit grows with heavier ones, such as multiple faults or longer code, where a hand-written harness quickly becomes the weak link.
One detail mattered more than expected. The code mixes 2-byte and 4-byte Thumb instructions. A naive skip that always replaces 4 bytes with a NOP can swallow the next instruction too, and hide a real vector. The campaign was rerun with skips matched to each instruction's real size.
The result: across the whole sweep, exactly one skip flips the verdict.
cmp r0, r5 ; r0 = compare result, r5 = 0xAAAAAAAA
bne reject_path ; mismatch -> go to the reject store <-- skipped
str r5, [r4, #0x40] ; fall through: verdict = 0xAAAAAAAA (accept)
...
str ..., [r4, #0x40] ; reject path: verdict = 0x55555555
The countermeasures did their job almost everywhere: no skip inside the constant-time compare produced an accept, and the redundant encodings rule out the trivial "flip a 0 into a 1". But the code is laid out so that the accept store sits directly after the conditional branch. Skip the branch, and execution falls straight into the accept path.
Step 6: proving it matters, and recording the evidence
Flipping a word in memory is not a bypass unless something trusts that word. Our first hypothesis was that the flip might be cosmetic, overwritten or re-checked later. The full Epoch trace settled it. At transition #15,988,640, the caller verify_sbc_header compares the verdict word directly against the accept pattern:
cmp r0, r10 ; r0 = [desc+0x40] = 0x55555555 (verdict)
; r10 = 0xAAAAAAAA (accept pattern)
beq accept ; not equal -> reject, failure flag set
With the skip, the verdict word equals r10, the branch to the accept path is taken, and the failure flag is never set, so a second gate further on is passed too. The roughly 460,000 instructions that follow are a timer wait, not a second verification. Nothing re-checks the content of the BL2.
To make this verifiable by others, we recorded two Epoch traces of the verification routine, from the same snapshot and the same tampered input:
Here are the executed instructions around the divergence, as recorded in each trace.
Baseline run, no fault:
#43,339 cmp r0, r5 ; r5 = 0xAAAAAAAA, first check: mismatch
#43,340 bne second_check
#43,341 ldr r0, [sp, #4]
#43,342 cmp r0, r5 ; second check: mismatch again
#43,343 b decide
#43,344 bne reject_path ; taken
#43,345 ldr r0, [sp, #4]
#43,346 cmp r0, r11 ; r11 = 0x55555555, confirms the reject
#43,347 bne fail
#43,348 str.w r11, [r4, #0x40] ; verdict = 0x55555555 (reject)
Faulted run, one skip:
#43,339 cmp r0, r5 ; first check: mismatch
#43,340 bne second_check
#43,341 ldr r0, [sp, #4]
#43,342 cmp r0, r5 ; second check: mismatch again
#43,343 b decide
#43,344 nop ; <-- the bne, skipped
#43,345 str r5, [r4, #0x40] ; verdict = 0xAAAAAAAA (accept)
#43,346 b exit
The two listings tell the whole story. The code checks the result twice, and both checks see the mismatch. But the decision rests on a single branch. And the two paths are not symmetric: the reject path confirms its value before storing it, while the accept path stores straight away.
Both traces are tagged at the decisive instructions. They are identical up to transition #43,344 and diverge exactly there. Anyone can open them side by side and step through the difference.
What we proved, and what we did not
Proven, on real inputs:
- A single 2-byte instruction skip turns the reject verdict for a tampered BL2 into accept, reproduced on two emulators.
- It is the only single-skip vector in the verification code we swept.
- The caller consumes that verdict directly, with no further check of the BL2 content.
- Faulted and baseline traces exist, diverge at one instruction, and can be inspected by anyone.
Not proven end to end: the final jump into the tampered BL2. On this boot path, the jump also depends on a download session with a host, which our emulation does not reproduce. The jump is gated only by the verdict we flipped, but we did not execute it.
The trace also corrected us once. A later gate, just before the jump, first looked like an easier target: skip it and a failed image boots. The Epoch trace showed that on this path, after a reject, the boot ROM halts earlier and never reaches that gate. It is not a single-fault bypass here, only on other boot paths we did not exercise. Without the trace, it would have gone into the report as one.
Finally, all of this is emulation. Real glitches produce messier effects than a clean skip, and confirming the vector on hardware is the natural next step. Emulation tells you exactly where to aim.
Takeaways
For firmware developers: countermeasures inside the crypto are not enough if the code around them is fragile. Here the compare and the encodings held; the layout of one branch did not. A cheap fix is to make the accept path need a positive confirmation, for example storing the accept pattern only after a second, independent comparison, and placing the reject store on the fall-through path.
For evaluators: the expensive parts of this work, bring-up, navigation and campaign set-up, were largely done by the agent. The analyst's time went into the decisions: the fault model, the success criterion, and what counts as proof.
On method: two ideas did most of the work. Start from the verdict in a time-travel trace and walk back, instead of reading the code forward. And use the trace to build snapshots, so a fault campaign only replays what it needs.
All of this ran on the firmware alone, with no board on the bench. That is what makes it usable early in development, when a finding still costs a code change rather than a recall.
Bring your own target to Epoch
Our team will show you how Epoch fits the way you work.
Want to explore first? Test the Epoch engine at ttd.eshard.com. Free, no sign-up, no credit card.
