README.md

To reproduce

• Create or link a large (10s of MB) binary file to /tmp/big.zip.

• Run

• Fetch the data with curl:

  curl -O http://localhost:8080

==> fails in chunky parser at random points during download

• Fetch the data with wget:

  wget http://localhost:8080

==> works

(Chrome fails mid-stream similarly to curl. It may use libcurl.)

log.txt

 => curl --version
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 
________________________________________________________________________________
 /tmp @ 11:06 PM Wed Sep 08 [!549]
 => curl http://localhost:8080 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2936k    0 2936k    0     0  29.1M      0 --:--:-- --:--:-- --:--:-- 30.1M
curl: (56) Received problem 3 in the chunky parser
________________________________________________________________________________
 /tmp @ 11:06 PM Wed Sep 08 [!550]
 => curl http://localhost:8080 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  704k    0  704k    0     0  28.6M      0 --:--:-- --:--:-- --:--:-- 31.2M
curl: (56) Received problem 3 in the chunky parser
________________________________________________________________________________
 /tmp @ 11:06 PM Wed Sep 08 [!550]
 => curl http://localhost:8080 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.7M    0 54.7M    0     0  34.6M      0 --:--:--  0:00:01 --:--:-- 34.9M
curl: (56) Received problem 3 in the chunky parser

test.js

var http = require( 'http' ),
    spawn = require('child_process').spawn;

http.createServer(function (req, res) {

    filename = '/tmp/big.zip'
    
    res.writeHead( 200, {
        'Content-Type': 'application/zip',
        'Content-Disposition': 'attachment; filename="' + filename + '"'
    } )
    
    // Create the subprocess
    subproc = spawn( 'cat', [filename] )
    
    // Stream the data through to the response as binary chunks
    subproc.stdout.on('data', function (data) {
        res.write(data, "binary");
    });
    
    // End the response on exit (and log errors)
    subproc.on('exit', function (code) {
        if (code !== 0) {
            console.log('subprocess exited with code ' + code);
        }
        res.end()
    });
    
} ).listen(8080);

console.log('Server running at http://localhost:8080');

Replied on the Google Groups thread: http://groups.google.com/group/nodejs/browse_thread/thread/f66cd3c960406919

To be clear, cat isn't the point. I would not have used spawn('cat') just to read a file. That's just the simplest subprocess to use in this example. In reality I'm generating compressed streams (e.g. zip).

And no, I'm not setting content length, because it is not known ahead of time. That is intentional. In my understanding it is optional, and only required as a hint to aid e.g. download manager GUIs in displaying useful progress bars, not for the correctness of the actual transaction.

Fixed in 0.3.0 unstable.