Arrow JS IPC compression interoperability

arrow-js-zstd-ipc-reproducer.md

Arrow JS IPC compression interoperability

This example demonstrates two different results for Arrow IPC streams with compressed body buffers:

1. Arrow JS 21.2.0 can read a compressed IPC stream written by PyArrow.
2. PyArrow cannot read a compressed IPC stream written by Arrow JS 21.2.0.

The same behavior occurs with both allowed codecs (ZSTD and LZ4 Frame). The example below uses ZSTD.

The example uses a highly compressible 10,000-row table. This is important because Arrow IPC permits a writer to leave an individual buffer uncompressed when compression would not make it smaller. A very small example could therefore conceal the Arrow JS writer problem.

Prerequisites

• Python 3.12
• Node.js 20 or newer
• uv

Setup

mkdir arrow-js-zstd-repro
cd arrow-js-zstd-repro

uv venv --python 3.12
source .venv/bin/activate
uv pip install pyarrow==23.0.0

npm init -y
npm install apache-arrow@21.2.0 zstd-codec@0.1.5

Arrow JS does not bundle ZSTD, so the JavaScript code registers zstd-codec.

1. Write the stream with Python

Create write_with_python.py:

from pathlib import Path

import pyarrow as pa
import pyarrow.ipc as ipc


ROW_COUNT = 10_000
OUTPUT = Path("python-zstd.arrows")

table = pa.table(
    {
        "id": pa.array(range(ROW_COUNT), type=pa.int32()),
        "label": pa.array(["a highly compressible value"] * ROW_COUNT),
    }
)

options = ipc.IpcWriteOptions(compression="zstd")
with pa.OSFile(str(OUTPUT), "wb") as sink:
    with ipc.new_stream(sink, table.schema, options=options) as writer:
        writer.write_table(table)

print(f"Python wrote {OUTPUT}: {ROW_COUNT} rows, {OUTPUT.stat().st_size} bytes")

python write_with_python.py

2. Read and write with Arrow JS

Create read_and_write_with_js.mjs:

import assert from "node:assert/strict";
import { readFile, writeFile } from "node:fs/promises";

import {
  CompressionType,
  compressionRegistry,
  tableFromArrays,
  tableFromIPC,
  tableToIPC,
} from "apache-arrow";
import zstdCodecPackage from "zstd-codec";


const ROW_COUNT = 10_000;
const { ZstdCodec } = zstdCodecPackage;

const zstd = await new Promise((resolve) => ZstdCodec.run(resolve));
const simple = new zstd.Simple();

compressionRegistry.set(CompressionType.ZSTD, {
  encode: (data) => simple.compress(data),
  decode: (data) => simple.decompress(data),
});

// Read and validate the compressed stream from Python.
const pythonBytes = await readFile("python-zstd.arrows");
const pythonTable = tableFromIPC(pythonBytes);

assert.equal(pythonTable.numRows, ROW_COUNT);
assert.equal(pythonTable.numCols, 2);
assert.equal(pythonTable.getChild("id").get(0), 0);
assert.equal(pythonTable.getChild("id").get(ROW_COUNT - 1), ROW_COUNT - 1);
assert.equal(pythonTable.getChild("label").get(0), "a highly compressible value");

console.log(
  `Arrow JS read python-zstd.arrows: ${pythonTable.numRows} rows, ` +
    `${pythonTable.numCols} columns`,
);

// Write an equivalent ZSTD-compressed stream with Arrow JS.
const jsTable = tableFromArrays({
  id: Int32Array.from({ length: ROW_COUNT }, (_, index) => index),
  label: Array(ROW_COUNT).fill("a highly compressible value"),
});
const jsBytes = tableToIPC(jsTable, "stream", CompressionType.ZSTD);

await writeFile("js-zstd.arrows", jsBytes);
console.log(`Arrow JS wrote js-zstd.arrows: ${ROW_COUNT} rows, ${jsBytes.length} bytes`);

node read_and_write_with_js.mjs

3. Read with Python

Create read_with_python.py:

import pyarrow as pa
import pyarrow.ipc as ipc


try:
    with pa.memory_map("js-zstd.arrows", "r") as source:
        table = ipc.open_stream(source).read_all()
except OSError as error:
    if "Destination buffer is too small" not in str(error):
        raise
    print("Expected failure while Python reads js-zstd.arrows:")
    print(f"{type(error).__name__}: {error}")
else:
    raise SystemExit(
        f"Unexpected success: Python read {table.num_rows} rows. "
        "Arrow JS may have fixed the writer bug."
    )

python read_with_python.py

Expected output

Byte sizes may differ.

Python wrote python-zstd.arrows: 10000 rows, 52560 bytes
Arrow JS read python-zstd.arrows: 10000 rows, 2 columns
Arrow JS wrote js-zstd.arrows: 10000 rows, 23864 bytes
Expected failure while Python reads js-zstd.arrows:
OSError: ZSTD decompression failed: Destination buffer is too small

Why the Python read fails

Each compressed IPC body buffer begins with an eight-byte signed integer that must contain the buffer's uncompressed length. Arrow JS 21.2.0 instead writes the compressed length. PyArrow uses that value to allocate the output buffer, so the ZSTD decompressor reports that its destination is too small.

Arrow JS can read its own output because its reader only uses the prefix to distinguish compressed buffers from buffers marked -1 (not compressed); its ZSTD codec determines the decompressed size from the ZSTD frame itself. This is also why an Arrow JS-to-JS round trip does not expose the writer bug.