Files and IPC Bytes

Two helpers work without a q process: one reads q's binary table files and one encodes q IPC messages.

Task	Python	Node.js
Read a q table file	read_binary6(filepath, backend="pyarrow", symbol_encoding="strict") returns a Narwhals DataFrame	await readBinary6(path, { symbolEncoding }) resolves to an Arrow Table
Encode an IPC message	serialize_as_ipc_bytes6(msg_type, enable_compression, any) returns bytes	await serializeAsIpcBytes6(messageType, compress, value) resolves to a Buffer

Read q table files

read_binary6 reads a table that q saved as a single file, for example with `:/data/trade set trade. It reads both plain and keyed tables. The DataFrame backend and the text policy work the same way as for a connection.

Python

import narwhals as nw
from xqdb import read_binary6

trade = read_binary6("data/trade")  # PyArrow-backed Narwhals DataFrame
print(nw.to_native(trade).schema)

legacy = read_binary6("data/legacy", backend="polars", symbol_encoding="lossy")

Node.js

import { readBinary6 } from "@xbbg/xqdb";

const trade = await readBinary6("data/trade"); // apache-arrow Table
console.log(trade.numRows, trade.schema.fields.map((field) => field.name));

const legacy = await readBinary6("data/legacy", { symbolEncoding: "lossy" });

What 0.1.8 reads, and what it does not:

• Single-file tables only. A splayed table (a directory with one file per column) and a partitioned database are directories. The helper rejects them with an error saying the path is not a regular file. Read those through a q process instead.
• Compression. Files compressed by q with LZ4 (algorithm 4) are decompressed automatically. Other algorithms raise Not supported compression algo.
• Memory. The whole file is read into memory. There is no fixed size limit, and running out of memory produces an error instead of ending the process.
• Errors. A missing file raises XqdbIOError (XQDB_IO in Node.js). A file that holds something other than a table raises XqdbError with the message Not a table.
• Node.js paths. Windows UNC paths (\\server\share\...) and device paths are rejected with XQDB_CONVERSION. Copy the file locally first.

The Node.js helper parses the file on a libuv worker thread, so the event loop keeps running.

Encode IPC messages

serialize_as_ipc_bytes6 turns a value into one complete q IPC message: an 8-byte header followed by the encoded value. The value follows the same conversion rules as a query argument, so a list becomes a general list and a DataFrame becomes a table. Use it when you write to a transport that xqdb does not manage, or when you store messages to replay later. On the q side, -9! decodes these bytes.

Python

import pyarrow as pa
from xqdb import serialize_as_ipc_bytes6

rows = pa.table({"sym": pa.array(["AAPL"]).dictionary_encode(), "price": [10.5]})
message = serialize_as_ipc_bytes6("async", True, ["upd", "trade", rows])

assert message[0] == 1  # little-endian
assert message[1] == 0  # message type: 0 async, 1 sync, 2 response
compressed = message[2] == 1  # small messages are never compressed

Node.js

import { serializeAsIpcBytes6 } from "@xbbg/xqdb";
import { Float64, Table, vectorFromArray } from "apache-arrow";

const rows = new Table({
  sym: vectorFromArray(["AAPL"]),
  price: vectorFromArray([10.5], new Float64()),
});
const message = await serializeAsIpcBytes6("async", true, ["upd", "trade", rows]);
console.log(message.length, message[1]); // message type byte: 0 async, 1 sync, 2 response

The message type must be "async", "sync", or "response". Other values raise ValueError in Python. In TypeScript the parameter type allows only those three strings.

Compression is a request, not a guarantee. xqdb compresses only messages of at least 2,000 bytes, and only when the compressed form is under half the original size. Otherwise it returns the message uncompressed, with header byte 2 set to 0.

Release 0.1.8 has no matching function to decode IPC bytes in Python or Node.js.