JavaScript Standard Input/Output: Unspecified
Some definitions. One important point is stdin, stdout, stderr do not
have to originate from a keyboard, or a TTY.
Let's hear what IBM has to say Understanding standard input, standard output, and standard error
Once a command begins running, it has access to three files:
It reads from its standard input file. By default, standard input is the keyboard.
It writes to its standard output file.
It writes error messages to its standard error file.
Using the shell: In the shell, the names for these files are:
stdin for the standard input file.
stdout for the standard output file.
stderr stderr for the standard error file.
The shell sometimes refers to these files by their file descriptors, or identifiers:
0 for stdin
1 for stdout
2 for stderr
From the online man pages stdout(3)
Under normal circumstances every UNIX program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user's terminal (see tty(4)) but might instead refer to files or other devices, depending on what the parent process chose to set up. (See also the "Redirection" section of sh(1).).
From 2011 Is there a way to read standard input with JavaScript?
It's not in the ECMAScript (standardized version of JavaScript) standard library.
From 2023 ECMAScript 2024 Language
ECMAScript is based on several originating technologies, the most well-known being JavaScript (Netscape) and JScript (Microsoft). The language was invented by Brendan Eich at Netscape and first appeared in that company's Navigator 2.0 browser.
You'll notice the absence of, see no mention of the terminology stdin, stdout, stderr in the standard/specification.
ECMA-262 does not specify reading stdin or stderr, nor writing to stdout.
JavaScript engine and runtime implementations if reading and writing to stardard input, writing to standard output, and writing to stardard error are not uniform; there's no specification.
WHATWG's Console mentions stdout several times,
does not mention stdin. Using console.log() in a Native Messaging host does not work,
as we are not printing to a TTY. That W.I.P. is not applicable reading stdin at all.
Common I/O (stdin/stdout/stderr) module specification #47.
You'll notice that no two JavaScript engines or runtimes implement standard input/output the same.
There's no controlling JavaScript specification to implement.
The code uses the JavaScript runtime executable only without any packages, no package.json file in the case of Node.js, no third-party libraries.
For the examples we will be using Native Messaging protocol as an example of JavaScript runtimes being launched and executed by an external program, and thereafter communication between the runtime and the local or remote application, in this case, the browser.
The protocol
On the application side, you use standard input to receive messages and standard output to send them.
Each message is serialized using JSON, UTF-8 encoded and is preceded with an unsigned 32-bit value containing the message length in native byte order.
The maximum size of a single message from the application is 1 MB. The maximum size of a message sent to the application is 4 GB.
The below applies to any application that communicates with the given JavaScript engine or runtime using standard input and output - that is not a TTY.
Node.js
V8 JavaScript/WebAssebly runtime. Written primarily in C++.
node v22.0.0-nightly202311151d8483e713
executable is 96.7 MB.
A few ways I've read stdin and writing to stdout using node executable.
Synchronously reading stdin.
This blocks some asynchronous operations in a node environment nodejs/node#49050 (comment)
... if readSync is sitting there waiting for input, its blocking the event loop right? that could be why its halting. the deno one is written without "sync" io, so it doesn't have this problem.
#!/usr/bin/env -S ./node --max-old-space-size=6 --jitless --expose-gc --v8-pool-size=1
// Node.js Native Messaging host
// guest271314, 10-9-2022
import {readSync} from 'node:fs';
// Node.js Native Messaging host constantly increases RSS during usage
// https://github.com/nodejs/node/issues/43654
process.env.UV_THREADPOOL_SIZE = 1;
// Process greater than 65535 length input
// https://github.com/nodejs/node/issues/6456
// https://github.com/nodejs/node/issues/11568#issuecomment-282765300
process.stdout._handle.setBlocking(true);
// https://github.com/denoland/deno/discussions/17236#discussioncomment-4566134
function readFullSync(fd, buf) {
let offset = 0;
while (offset < buf.byteLength) {
offset += readSync(fd, buf, { offset });
}
return buf;
}
function getMessage() {
const header = new Uint32Array(1);
readFullSync(0, header);
const content = new Uint8Array(header[0]);
readFullSync(0, content);
return content;
}
function sendMessage(json) {
let header = Uint32Array.from({
length: 4,
}, (_,index)=>(json.length >> (index * 8)) & 0xff);
let output = new Uint8Array(header.length + json.length);
output.set(header, 0);
output.set(json, 4);
process.stdout.write(output);
// Mitigate RSS increasing expotentially for multiple messages
// between client and host during same connectNative() connection
header = output = null;
global.gc();
}
function main() {
while (true) {
try {
const message = getMessage();
sendMessage(message);
} catch (e) {
process.exit();
}
}
}
main();
Asynchronously reading stdin.
Keep this in mind: process.stdout.fd !== "/proc/self/fd/1".
We can process asynchronous tasks without blocking reading stdin (e.g., full-duplex)
#!/usr/bin/env -S node --expose-gc --experimental-default-type=module
// Node.js Native Messaging host
// guest271314, 10-9-2022
import { open } from "node:fs/promises";
process.env.UV_THREADPOOL_SIZE = 1;
// https://github.com/denoland/deno/discussions/17236#discussioncomment-4566134
// https://github.com/saghul/txiki.js/blob/master/src/js/core/tjs/eval-stdin.js
async function readFullAsync(length, buffer = new Uint8Array(65536)) {
const data = [];
while (data.length < length) {
const input = await open("/dev/stdin");
let { bytesRead } = await input.read({
buffer
});
await input.close();
if (bytesRead === 0) {
break;
}
data.push(...buffer.subarray(0, bytesRead));
}
return new Uint8Array(data);
}
async function getMessage() {
const header = new Uint32Array(1);
await readFullAsync(1, header);
const content = await readFullAsync(header[0]);
return content;
}
async function sendMessage(message) {
const header = new Uint32Array([message.length]);
const stdout = await open("/proc/self/fd/1", "w");
await stdout.write(header);
await stdout.write(message);
await stdout.close();
global.gc();
}
async function main() {
while (true) {
try {
const message = await getMessage();
await sendMessage(message);
} catch (e) {
process.exit();
}
}
}
main();
Another way to read stdin in Node.js that I was trying to get away from
is using an event listener (Node.js-specific) pattern. https://github.com/simov/native-messaging/blob/8e99d2a345ae94426a502d05aa5d57b966f6bc78/protocol.js.
We'll wind up revisiting parts of this code later in another JavaScript runtime.
module.exports = (handleMessage) => {
var msgLen = 0,
dataLen = 0,
input = []
function sendMessage(msg) {
var buffer = Buffer.from(JSON.stringify(msg))
var header = Buffer.alloc(4)
header.writeUInt32LE(buffer.length, 0)
var data = Buffer.concat([header, buffer])
process.stdout.write(data)
}
process.stdin.on('readable', () => {
var chunk
while (chunk = process.stdin.read()) {
// Set message value length once
if (msgLen === 0 && dataLen === 0) {
msgLen = chunk.readUInt32LE(0)
chunk = chunk.subarray(4)
}
// Store accrued message length read
dataLen += chunk.length
input.push(chunk)
if (dataLen === msgLen) {
// Send accrued message from client back to client
handleMessage(JSON.parse(Buffer.concat(input).toString()))
// Reset dynamic variables after sending accrued read message to client
msgLen = 0,
dataLen = 0,
input = []
}
}
})
process.on('uncaughtException', (err) => {
sendMessage({
error: err.toString()
})
})
return sendMessage
}
Notice that we have to read from stdin more than once to retrieve 1 MB
of JSON data from the browser.
QuickJS
Written in C. 4.6 MB when built using make. 915.5 KB after strip.
We can read 1 MB from stdin in one read. Based on my tests no other JavaScript runtime in
these tests do that.
#!/usr/bin/env -S ./qjs --std
// QuickJS Native Messaging host
// guest271314, 5-6-2022
function getMessage() {
const header = new Uint32Array(1);
std.in.read(header.buffer, 0, 4);
const output = new Uint8Array(header[0]);
std.in.read(output.buffer, 0, output.length);
return output;
}
function sendMessage(message) {
const header = Uint32Array.from({
length: 4,
},
(_, index) => (message.length >> (index * 8)) & 0xff
);
const output = new Uint8Array(header.length + message.length);
output.set(header, 0);
output.set(message, 4);
std.out.write(output.buffer, 0, output.length);
std.out.flush();
}
function main() {
while (true) {
const message = getMessage();
sendMessage(message);
}
}
try {
main();
} catch (e) {
std.exit(0);
}
Synchronous. Though non-blocking when we read the message from the browser, execute the command,
then stream data output from the local application to the browser piped from QuickJS std.popen(), capture_system_audio.js
function main() {
const message = getMessage();
const size = 1764;
const data = new Uint8Array(size);
const pipe = std.popen(
JSON.parse(String.fromCharCode(...message)),
'r'
);
while (pipe.read(data.buffer, 0, data.length)) {
sendMessage(`[${data}]`);
pipe.flush();
std.gc();
}
}
txiki.js
Depends on QuickJS for JavaScript engine. Other dependencies include libuv, wasm3, curl,
libffi. 14.8 MB. Written in C.
#!tjs run
// txiki.js Native Messaging host
// guest271314, 2-10-2023
// https://github.com/denoland/deno/discussions/17236#discussioncomment-4566134
// https://github.com/saghul/txiki.js/blob/master/src/js/core/tjs/eval-stdin.js
async function readFullAsync(length) {
const buffer = new Uint8Array(65536);
const data = [];
while (data.length < length) {
const n = await tjs.stdin.read(buffer);
if (n === null) {
break;
}
data.push(...buffer.subarray(0, n));
}
return new Uint8Array(data);
}
async function getMessage() {
const header = new Uint8Array(4);
await tjs.stdin.read(header);
const [length] = new Uint32Array(
header.buffer
);
const output = await readFullAsync(length);
return output;
}
async function sendMessage(message) {
// https://stackoverflow.com/a/24777120
const header = Uint32Array.from({
length: 4,
},
(_, index) => (message.length >> (index * 8)) & 0xff
);
const output = new Uint8Array(header.length + message.length);
output.set(header, 0);
output.set(message, 4);
await tjs.stdout.write(output);
return true;
}
async function main() {
try {
while (true) {
const message = await getMessage();
await sendMessage(message);
}
} catch (e) {
tjs.exit();
}
}
main();
Deno
V8 JavaScript/WebAssembly engine. Version 1.38.1 is 126.6 MB. Written in Rust.
#!/usr/bin/env -S ./deno run --v8-flags="--expose-gc,--jitless"
// Deno Native Messaging host
// guest271314, 10-5-2022
// https://github.com/denoland/deno/discussions/17236#discussioncomment-4566134
// https://github.com/saghul/txiki.js/blob/master/src/js/core/tjs/eval-stdin.js
async function readFullAsync(length) {
const buffer = new Uint8Array(65536);
const data = [];
let n = null;
while (data.length < length && (n = await Deno.stdin.read(buffer))) {
data.push(...buffer.subarray(0, n));
}
return new Uint8Array(data);
}
async function getMessage() {
const header = new Uint32Array(1);
await Deno.stdin.read(new Uint8Array(header.buffer));
return readFullAsync(header[0]);
}
async function sendMessage(message) {
const header = new Uint32Array([message.length]);
await Deno.stdout.write(new Uint8Array(header.buffer));
await Deno.stdout.write(message);
}
async function main() {
while (true) {
const message = await getMessage();
await sendMessage(message);
gc();
}
}
try {
main();
} catch (e) {
Deno.exit();
}
Bun
JavaScriptCore JavaScript engine. Written in Zig. Version 1.0.11 is 87 MB.
I have not been able to get Bun.file() or BunFile to work as expected.
So we'll use Node.js's process.stdin. We don't use Node.js's open() because
Bun has not implemented "node:fs/promises" (node v22.0.0-nightly202311151d8483e713 does not implement Promise.withResolvers())
#!/usr/bin/env -S ./bun run --no-install --hot
// Bun Native Messaging host
// guest271314, 10-9-2022
async function getMessage() {
const { promise, resolve } = Promise.withResolvers();
// https://github.com/simov/native-messaging/blob/8e99d2a345ae94426a502d05aa5d57b966f6bc78/protocol.js
let messageLength = 0,
bytesWritten = 0,
input = [];
process.stdin.on("readable", () => {
let chunk;
while ((chunk = process.stdin.read())) {
// Set message value length once
if (messageLength === 0) {
[messageLength] = new Uint32Array(chunk.buffer.slice(0, 4));
chunk = chunk.subarray(4);
}
// Store accrued message length read
bytesWritten += chunk.length;
input.push(...chunk);
if (bytesWritten === messageLength) {
// Send accrued message from client back to client
resolve(new Uint8Array(input));
}
}
});
return await promise;
}
function sendMessage(json) {
// https://github.com/denoland/deno/discussions/17236#discussioncomment-4566134
const header = new Uint32Array([json.length]);
/*
// Long form
const header = new Uint32Array([
((uint32) =>
// https://stackoverflow.com/a/58288413
(uint32[3] << 24)
| (uint32[2] << 16)
| (uint32[1] << 8)
| (uint32[0])
)(Array.from({
length: 4,
}, (_,index)=>(json.length >> (index * 8)) & 0xff)
)
]);
*/
process.stdout.write(new Uint8Array(header.buffer));
process.stdout.write(json);
Bun.gc(true);
}
async function main() {
while (true) {
try {
const message = await getMessage();
sendMessage(message);
} catch (e) {
process.exit();
}
}
}
main();
I have tried to implement a JavaScript Native Messaging host using V8 via d8 and
SpiderMonkey using js without a runtime, unsuccessfully, so far.
- Why is this SpiderMonkey Native Messsaging host not sending stdout to the client?
- d8 Native Messaging host
Conclusion
This is a glaring omission from the Ecmascript Language specification.
There is no uniformity, no single code pattern we can use to read stdin (synchronously or asynchronously)
in all JavaScript engines and runtimes.
There's no way to write to stdout in different JavaScript runtimes using the same runtime/engine agnostic JavaScript source code.
There's no standard input/output specification to reference for implementation because such an Ecmascript specification section does not exist.