stream_spec.md

a Stream spec

---

##UPDATE

I've now moved this stuff into it's own repo and added an executable checker, and some nifty diagrams of the stream state transitions state diagrams

This gist is no longer being maintained. goto dominictarr/stream-spec

---

This document defines the behaviour that a Stream must implement in order to be compatible with Stream#pipe. This is not an official document, but is intended as a guide to produce correct behaviour in user-land streams.

Stream

All streams must emit 'error' if writing to or reading from becomes physically impossible. 'error' implys that the stream has ended.

All streams may emit 'close'. 'close' means that any underlying resources have been disposed of. If a ReadableStream has ended normally, it must not emit 'close' before 'end'.

A Stream must not emit 'error' if the error is recoverable. (that is not in the stream spec)

All streams should implement destroy but a WritableStream must implement destroy.

emit('error')

All streams must emit 'error' when an error that is not recoverable has occurred. If it has become physically impossible to write to or read from the Stream, then emit 'error'.

A WriteableStream may throw an error if write has been called after end. (which should never happen, in correct usage)

otherwise, a stream must never throw an error. (always emit)

WritableStream

A WritableStream must implement methods write, end, and destroy, and writable must be set to true, and must inherit Stream#pipe

write(data)

write must return either true or false. (if false then the writer should pause) If write is called after end, an error may be thrown.

If write returns false,it must eventually emit 'drain'. write returning false means the stream is paused. paused means (or downstream) is at capacity, and the writer/upstream should attempt to slow down or stop. It does not mean buffer, although that is something a stream may reasonably do.

end()

Calling end may set writable to false. If the Stream is also readable, it must eventually emit 'end'.

destroy()

Used to dispose of a Stream.

Calling destroy must dispose of any underlying resources. Calling destroy must emit 'close' eventually, once any underlying resources are disposed of.

emit ('drain')

After pausing, a Stream must eventually emit 'drain'. For example, when if a call to write returns false Stream#pipe will call pause on the source and register a one-time listener on drain, that will call resume on the source.

If drain is not emitted correctly, it's possible for 'data' events to stop coming (depending on the source's behaviour when paused).

ReadableStream

A ReadableStream must inherit pipe from Stream, and set readable to true, and must emit zero or more 'data' events, followed by a single end event. A ReadableStream may implement pause and resume methods.

• I will not bother to specify the behaviour of pipe because I am attempting to document what must be done in order for your Stream to be compatible with pipe.

emit('data', data)

A ReadableStream may emit one or more 'data' events. A ReadableStream must not emit emit a 'data' event after it has emitted 'end'

emit('end')

A ReadableStream should emit an 'end' event when it is not going to emit any more 'data' events. 'end' must not be emitted more than once. A ReadableStream may set readable to false after it has emitted the 'end' event.

emit ('close')

A ReadableStream must emit a 'close' event after the 'end' event. 'close' must only be emitted once. if destroy is called, 'close' must be emitted, unless the stream has already ended normally. If 'close' is emitted before 'end' that signifies a broken stream, this should only happen if destroy was called.

Emitting close will cause pipe to call destroy on the down stream pipe, if it is emitted before end.

pause()

A readable Stream may implement the pause method. When pause is called, the stream should attempt to emit 'data' less often. (possibly stopping altogether until resume is called)

resume()

A ReadableStream may implement the resume method. If the Stream has been paused, it may now emit 'data' more often, or commence emitting data if it has stopped all together.

If a stream is also writable, and has returned false on write it must now eventually emit drain

destroy()

A ReadableStream should implement destroy.

Calling destroy must dispose of any underlying resources. Calling destroy must emit 'close' eventually, once any underlying resources are disposed of.

request for comment

thank you in advance!

from what I gather reading fs.js

• fs.WriteFile throws if not writable.
  https://github.com/joyent/node/blob/v0.6/lib/fs.js#L1336
• writable = false on end()
  https://github.com/joyent/node/blob/v0.6/lib/fs.js#L1373
• emits 'error' when on failed read. like 'close' ?
  https://github.com/joyent/node/blob/v0.6/lib/fs.js#L1126

hmm...
fs.ReadStream emits 'close' after 'end', and does not emit 'close' or 'end' after an error. it just emits 'error'.

• https://github.com/joyent/node/blob/v0.6/lib/fs.js#L1124-1135

so am thinking correct practice is:

readableStream
  .pipe(throughStream)
  .pipe(streamThatMayClose
     .on('close', function () { readableStream.destroy() })
  )

the current pipe implementation, or 0.7 do not do this for you. though, I understand that this may change in a future implementation.

"write must return either true or false" - I don't think that's technically true according to https://github.com/joyent/node/blob/v0.6/lib/stream.js#L38 - write only has to return false if the stream wants the source to pause. Otherwise it can return anything it wants (including not returning, yielding the default of undefined).

I realise that most of the core streams do actually explicitly return true - so it's probably regarded as best practice - but pipe streams do work fine without it.

you are correct. I decided to use "must" because it made the spec simpler, and all the core streams are like that. I'd be happy to change it if you can suggest another phrasing that is also simple.

this needs to be updated for streams that allow half duplex.

that is when one side of a stream closes.

for example, you call end() on an http request, but the server is still writing a response, so the stream isn't cleaned up (destroyed) untill after the writable side emits 'end'. this is the case where readable == writable == true