TextEncoder & TextDecoder shenanigans

ten-tdec.md

TextEncoder & TextDecoder shenanigans

This gist is to remind developers that resizable or growable buffers don't play nice with TextEncoder or TextDecoder instances.

Setup

// setup
const ref = new TextEncoder().encode("hello");

// SharedArrayBuffer (static) View
const sab = new Uint8Array(new SharedArrayBuffer(5));
// optional pre-populate
sab.set(ref, 0); // ✅

// SharedArrayBuffer (growable) View
const gsab = new Uint8Array(new SharedArrayBuffer(5, { maxByteLength: 16 }));
// optional pre-populate
gsab.set(ref, 0); // ✅

// ArrayBuffer (static) View
const buff = new Uint8Array(new ArrayBuffer(5));
// optional pre-populate
buff.set(ref, 0); // ✅

// ArrayBuffer (resizable) View
const rbuff = new Uint8Array(new ArrayBuffer(5, { maxByteLength: 16 }));
// optional pre-populate
rbuff.set(ref, 0); // ✅

TextDecoder

Only an ArrayBuffer, resizable or not, can be used. SharedArray needs to be sliced as static ArrayBuffer.

const decoder = new TextDecoder;

decoder.decode(buff); // ✅
decoder.decode(rbuff); // ✅
decoder.decode(sab.slice(0)); // ✅ - not shared
decoder.decode(gsab.slice(0)); // ✅ - not shared

decoder.decode(sab); // ❌
// Failed to execute 'decode' on 'TextDecoder':
// The provided ArrayBufferView value must not be shared.

decoder.decode(gsab); // ❌
decoder.decode(gsab.subarray(0)); // ❌
// Failed to execute 'decode' on 'TextDecoder':
// The provided ArrayBufferView value must not be shared.

TextEncoder encodeInto

Only static ArrayBuffer can be used, making it impossible to use for linear, resizable, serializations purposes.

const encoder = new TextEncoder;

encoder.encodeInto("hello", buff); // ✅

encoder.encodeInto("hello", gsab); // ❌
encoder.encodeInto("hello", gsab.subarray(0)); // ❌
// Failed to execute 'encodeInto' on 'TextEncoder':
// The provided Uint8Array value must not be shared.

encoder.encodeInto("hello", rbuff); // ❌
encoder.encodeInto("hello", rbuff.subarray(0)); // ❌
// Failed to execute 'encodeInto' on 'TextEncoder':
// The provided Uint8Array value must not be resizable.

---

Related topic in TC39 around the fact we can't even know AOT the size of a string before deciding to use awkward magic behind the scene to make encodeInto happen: https://es.discourse.group/t/string-bytelength-count/2315