Building voidwalk

I started voidwalk in April because I wanted to know what a disassembler actually has to do, and reading about it only got me so far. Writing one seemed like the quickest way to find out. Until July it was called DAT, short for dynamic analysis tool. For now it only does static analysis, but the dynamic side is still the plan.

What it does today

You give it a Linux or Windows executable. It reads the first few bytes to tell ELF from PE, parses the section table, and then walks through the .text section, decoding the x86 or x86-64 machine code back into assembly. Each of those steps turned out to be a different kind of problem.

The decoder was the hardest part, and it's still a work in progress. I've restructured it several times as it grew, once to add the 0F opcode map and again to fit in 64-bit support. To check it, I diff its output against GNU objdump. On /bin/ls from Debian 13, all 22,523 instructions start at the same addresses.

No third-party disassembler

Every x86 and x86-64 instruction voidwalk shows is decoded by code I wrote. Using an existing disassembly library would have been faster, but it would have skipped the part I wanted to learn. So far the decoder handles prefixes, the full one-byte opcode map including the x87 FPU instructions, the common 0F instructions, and the 64-bit additions: REX prefixes, the r8 to r15 registers and RIP-relative addressing.

The decoder is also kept apart from the file formats. The ELF and PE readers only work out which architecture a binary was built for and pass that on, and neither of them knows how to decode an instruction. Adding a new architecture means writing one Decoder subclass and adding a case to makeDecoder():

std::unique_ptr<Decoder> makeDecoder(Arch a) {
    switch (a) {
        case Arch::X86:     return std::make_unique<X86Decoder>(false);
        case Arch::X86_64:  return std::make_unique<X86Decoder>(true);
        case Arch::ARM32:   return std::make_unique<Arm32Decoder>();
        case Arch::AArch64: return std::make_unique<AArch64Decoder>();
        default:            return nullptr;
    }
}

The ARM cases are already there. For now they point at stub decoders, which is why ARM code still shows up as raw bytes.

Reading without copying

At first voidwalk read files the ordinary way, with an ifstream: seek to an offset, read the bytes the parser asked for, repeat. That was easy to write. Then I remembered the system call overhead. Every ordinary read is a system call, and the kernel copies the bytes from its page cache into my buffer before I can look at them. A full sweep of .text makes a lot of small reads, so on a big binary that cost adds up.

In August I switched AddressSpace over to memory mapping, with mmap on Linux and MapViewOfFile on Windows. The file's pages are mapped straight into the process, so after the first access a read is just a memory load, and the OS only loads the pages that actually get used. It meant writing two platform-specific versions instead of one portable stream, and I decided the speed on large files was worth that. All file access still goes through AddressSpace, which is also where the bounds checks live.

One core, three front ends

Decoding runs on a background thread in a shared core. The front ends only talk to it through a Session object, which owns that thread:

Every push runs a GitHub Actions job that builds the test suite and runs its 230+ tests on both Linux and Windows. The tests build their own input binaries in memory as they run.

Keeping the UI responsive

Large binaries used to freeze the desktop app. The first disassembly view was a QTableWidget, which creates an item for every cell, whether it's on screen or not. On a big program that meant building every row up front on the UI thread, and the window stopped responding until it was done.

The switch to memory mapping was part of fixing this. The bigger change was moving the decode to a background thread, so the UI never waits for the whole sweep. The worker publishes how many instructions are ready, and the UI picks up the new rows as they arrive, so the listing fills in while you're already scrolling through it. Opening another file stops the old worker before a new one starts. The terminal UI fills in the same way.

I also replaced the QTableWidget with a QTableView over my own model. The view only asks the model for the cells that are actually visible, so the table can hold millions of rows and still paint instantly. Rows are only ever appended during a decode, so each update just tells the view about the new rows at the end.

The background thread brought a problem of its own. The worker appends instructions while the UI thread reads them, and a std::vector that outgrows its capacity reallocates, freeing the buffer the UI may be in the middle of reading. My first fix was to reserve one slot per byte of .text before the sweep, since no instruction is shorter than a byte. That was safe but wasteful: /bin/ls has 91,566 bytes of .text and 22,523 instructions, so about three quarters of the reserved slots were never used. It also trusted the section size in the file header, so a malformed binary that lied about it could make voidwalk ask for an absurd amount of memory.

So I wrote a small container, ChunkStore, that keeps elements in fixed-size blocks of 8,192. Only the table of block pointers is reserved for the worst case, which makes that reservation 8,192 times smaller, and each block is allocated when the previous one fills up. Blocks never move once they exist, so the UI can keep reading while the worker appends, and memory use follows the number of instructions actually decoded. I looked at std::deque as well, since it also leaves existing elements in place, but its indexing goes through an internal map that can be reallocated as it grows, so reading from another thread would still race.

What it still cannot do

ARM binaries are detected, but their code shows up as raw bytes. SSE and AVX instructions aren't decoded, and only .text gets disassembled: voidwalk doesn't read symbol tables, imports or the entry point yet.

SSE and AVX support is up next. After that comes the debugger, with breakpoints, stepping and instruction tracing. The terminal UI and the desktop app already have run and step controls and register and stack panes, but there's no engine behind them yet. Control-flow graphs, ARM decoding and AI-generated explanations of the disassembly come later.