xqdb changelog

Release notes for every version with a docs/release-notes entry, newest first, generated from tag v0.1.8. All releases, including earlier ones, are listed on GitHub Releases.

v0.1.8

Released 2026-09-03. Release notes source

Fixed — empty dictionaries round-trip

An empty q dictionary decoded to {} on Python and Node, but sending that {} back failed with Not supported empty dictionary, so a value the library handed out could not be passed back as a query argument.

{} now serializes as (`symbol$())!() — the symbol-keyed frame this library already emits for every non-empty dictionary and the frame q itself produces for an emptied symbol dictionary — and nested empty dictionaries serialize the same way. The wire form is 99 11 0 <0 keys> 0 0 <0 values>, 13 bytes, matching the length pass exactly.

Decoding also accepts q's canonical empty literal ()!(), whose keys are an empty general list rather than an empty symbol list. It used to fail with Only support symbol keys dictionary or keyed table, got k type 0; it now decodes to {} alongside (`symbol$())!() and typed-value forms such as 0#`a`b!1 2. A general list that holds any keys is still rejected with the same error, and a dictionary frame truncated before its key type or its values now reports a deserialization error instead of an internal parser panic.

v0.1.7

Released 2026-09-01. Release notes source

Added — opt-in lossy decoding for q text that is not valid UTF-8

q stores symbols and strings as raw bytes and never validates them, while Arrow string columns and Rust strings must be valid UTF-8. Until now a single stray Latin-1 or binary byte anywhere in a response failed the whole result:

"q symbol-list value is not valid UTF-8: invalid utf-8 sequence of 1 bytes from index 3"

A connection can now opt in to transcoding instead. The policy is symbolEncoding on Node (QOptions and readBinary6(path, { symbolEncoding })) and symbol_encoding on Python (Q(...), the Q.symbol_encoding property, and read_binary6(path, symbol_encoding=...)), accepting "strict" or "lossy":

• "strict" (default) keeps the existing behaviour: the first invalid byte sequence fails the response with an error naming the offending value.
• "lossy" replaces each maximal invalid subsequence with U+FFFD REPLACEMENT CHARACTER — the same substitution String::from_utf8_lossy performs — so every other value in the response survives intact. A Latin-1 caf\xe9 decodes as caf\ufffd.

The policy applies to every place wire text becomes a string: symbol atoms, symbol lists and columns, nested symbol lists, char columns, string columns, lambda source and context, dictionary keys, and table column names. String column rows are transcoded individually, so a multi-byte code point split across two rows is replaced in both rather than silently validated as a whole. Valid text decodes identically under both policies.

Any other value ("latin1", "Lossy", …) is rejected at construction: a RangeError on Node, a ValueError on Python, and XQDB_CONVERSION from the native readBinary6.

Changed

• q error messages are now always decoded lossily. A server error whose text contained a non-UTF-8 byte used to surface as a deserialization failure that hid the actual error; it now surfaces as the server error with replacement characters, under either policy.
• Symbol lists are validated once instead of twice: the pass that measures a list's wire length no longer decodes each symbol, leaving that to the pass that materializes it. Strict-mode errors for invalid symbols are unchanged in type and still name the offending value.

Every buffer the lossy path allocates is sized from wire data and reserved fallibly, in line with 0.1.5 and 0.1.6.

Note on 0.1.6

The 0.1.6 npm release built and validated successfully but was never approved for publication, so @xbbg/xqdb@0.1.6 does not exist on npm; that run has been cancelled in favour of this release. 0.1.7 contains everything 0.1.6 did.

Python users upgrading from 0.1.3

This is the first PyPI release since 0.1.3, so it also delivers the 0.1.4 through 0.1.6 changes. One of them is breaking:

q timestamp atoms are naive. 0.1.3 returned UTC-aware datetime values; 0.1.4 and later return naive values with tzinfo is None, matching Arrow timestamp[ns] columns, so the same q value compares equal whether read as an atom or out of a column and can be passed back as a query argument. If you need an aware value, attach the zone yourself — and only when the data really is UTC, because a process that stores .z.P holds local wall-clock times:

ts = conn.sync("first exec time from trade")   # naive datetime
ts = ts.replace(tzinfo=timezone.utc)

Query arguments accept both shapes: a naive datetime keeps its wall clock and an aware one is normalized through its UTC offset.

The rest is additive: no fixed size ceiling on results, every wire-sized allocation reports an error instead of aborting the interpreter, 40-bit framing for messages above 4 GiB, and the dependency advisories patched in 0.1.4. See docs/release-notes/v0.1.4.md through v0.1.6.md for the details.

v0.1.6

No GitHub Release or registry upload exists for this version. Release notes source

Fixed — the last receive-side allocations that could abort the process

0.1.5 made the allocations that dominate a large q response fallible, so a result too big for the machine returns an error rather than killing the host process. Four smaller per-element sites were missed and are fixed here:

• Dictionary key strings, copied one per entry.
• String values decoded through K::try_from_any_value, one per element of a dictionary or mixed list.
• Symbol atoms.
• The column vector handed to the DataFrame constructor.

Each is individually small, but a list decodes one per element and the aggregate scales with the payload, so a large enough frame could still reach Rust's aborting allocator instead of returning XqdbError. Every allocation on the receive path sized from wire data is now fallible.

No API, wire-format or behavioural change beyond the failure mode: work that succeeded in 0.1.5 succeeds identically here.

Note on 0.1.5

The 0.1.5 npm packages published successfully, but the release job failed its post-publish readability check while the registry was still propagating, so no GitHub Release was created for that tag and the Python distribution was never published. docs/release-notes/v0.1.5.md has been corrected: it claimed every process-sized allocation was fallible, which was true only of the sites listed there. Users on 0.1.5 lose nothing by staying there except the four sites above.

v0.1.5

Released 2026-08-29. Release notes source

Fixed — large q results no longer fail

A legitimate q response is no longer refused for being large. 0.1.4 applied a hardcoded 512 MiB ceiling to every IPC message, so an ordinary query could fail with:

Decompressed IPC message length 706440911 exceeds the 536870912-byte safety limit

The same constant also capped q binary table reads and Kxzip decompression at 512 MiB. All of these ceilings are gone. A result is now bounded only by available memory and by the q header's own 40-bit length field, and a result that does not fit returns an error rather than terminating the host process.

Verified against kdb+ over IPC: an 800,000,014-byte frame, and the same value as a q-compressed frame declaring 800,000,028 decompressed bytes, both of which 0.1.4 rejected.

What replaced the ceilings. Bounds that remain are either the wire format's own or proportional to bytes actually received, so nothing legitimate is refused:

• A declared length reserves at most 32 MiB up front. Beyond that the response buffer grows eightfold only once the peer has filled the previous reservation, so committed memory stays within eight times the bytes actually delivered.
• A compressed response is rejected when its declared decompressed size is unreachable from the bytes received. The q IPC decompressor emits at most 2056 bytes per 17 input bytes, so 121x is a strict bound; this client's own compressor reaches 120.935x on maximally compressible input.
• A Kxzip file is rejected when its declared output exceeds what its own LZ4 blocks can hold, taken per block as the smaller of the block's uncompressed size and 255x its actual compressed bytes. The previous block-count product was no real bound: eight bytes of footer metadata bought 4 MiB of permitted claim, so an 8 KiB crafted file could demand a multi-gigabyte allocation.

Fixed — outgoing messages above 4 GiB were framed incorrectly

The request encoder hardcoded the q header's length-extension byte to zero and wrote a 32-bit length, so a message above 4 GiB declared its length modulo 2^32 and desynchronized the peer. Both the connector and serializeAsIpcBytes6 / serialize_as_ipc_msg6 now write the 40-bit form, bounded by the format's own maximum of 1,099,511,627,775 bytes.

Fixed — compressed frames above 4 GiB were unparseable

When a message exceeded 4 GiB the compressor selected compression mode 2 but wrote the eight-byte decompressed-length prefix at the wrong offset, emitting a frame no q peer could decode. This path was unreachable while the 512 MiB ceiling existed and became reachable once it was removed.

Fixed — a result too large for the machine aborted the process

Allocations sized from wire data went through Rust's aborting allocator, so a payload larger than available memory killed the host process — the Node worker or the Python interpreter — with no catchable error. The allocations that dominate a large response are now fallible and report an error instead: message body, decompression target, symbol offsets and values, char-column views, nested-list payloads and offsets, table column metadata and column decode, dictionary values and entries, q datetime and time conversion, GUID/byte/char payload copies, and Arrow validity bitmaps.

Four smaller per-element sites were missed and still allocate infallibly in this release: dictionary key strings, string values decoded through K::try_from_any_value, symbol atoms, and the final table column vector. 0.1.6 completes the set.

Arrow bitmaps keep polars' word-at-a-time packing; only their buffer allocation changed, measured at parity (mean ratio 1.004 over five rounds against Bitmap::from_trusted_len_iter).

Behaviour to be aware of

• Error text changed for oversized and malformed frames. Code matching on the string "safety limit" will no longer match; the messages now name the offending value and the bound it failed.
• A compressed frame whose declared decompressed size is unreachable from its payload now fails as a deserialization error and closes the connection, in line with every other malformed-frame path.
• A q binary table file whose reported size is zero but which holds content is now read completely instead of being rejected.

Performance

No regression. Measured against kola on 100,000-row fixtures, 50 iterations, comparing within each run to cancel machine drift: read wide 43.3 ms versus 96.3 ms, read trade 12.4 ms versus 15.7 ms, read depth 14.3 ms versus 17.1 ms, send trade 17.9 ms versus 22.4 ms. The scalar round trip remains a tie.

v0.1.4

Released 2026-08-28. Release notes source

Breaking change — Python timestamp atoms are now naive

q timestamp atoms returned by the Python binding are now naive datetime values with tzinfo is None. Version 0.1.3 returned UTC-aware values.

A q timestamp carries no timezone, and Arrow timestamp[ns] columns were already naive. In 0.1.3 the same q value therefore compared unequal depending on whether it was read as an atom or out of a column, and a value read from a column could not be passed back as a query argument at all. Both now work.

Migration. If you need an aware value, attach the zone yourself:

ts = conn.sync("first exec time from trade")   # 0.1.4: naive datetime
ts = ts.replace(tzinfo=timezone.utc)           # only if the data really is UTC

Do not assume UTC unless your q process guarantees it. A process that stores .z.P holds local wall-clock times, and 0.1.3 labelled those UTC — so an astimezone() call on a 0.1.3 value could shift it silently.

Argument handling. Query arguments now accept both shapes. A naive datetime keeps its wall clock. An aware one is normalized through its UTC offset, covering fixed offsets and zoneinfo zones including DST transitions. A tzinfo whose utcoffset() returns None counts as naive, matching Python's own definition of awareness.

Precision. Sub-microsecond digits that pandas.Timestamp keeps in .nanosecond are preserved when such a scalar is passed as an argument, so a value read from a timestamp[ns] column round-trips exactly. Whole Series and DataFrame round-trips remain nanosecond-exact on every backend.

Dependencies

Patched two advisories that were active in the compiled native graph: crossbeam-epoch 0.9.20 (RUSTSEC-2026-0204) and memmap2 0.9.11 (RUSTSEC-2026-0186). Dropped the unused Polars serde feature, which removes the unmaintained bincode (RUSTSEC-2025-0141) from every shipped binary. Bumped pyo3 to 0.29.2 and raised the napi floor to 3.12.2.

Raised the apache-arrow floor to ^21.2.0: Arrow only gained cross-copy Symbol.hasInstance identity for Table and Vector in 21.2, so earlier 21.x releases could fail instanceof against values this package returns.