DataFrames and Arrow

xqdb keeps q tables columnar from end to end. The Rust core decodes each column into an Arrow array. Python then wraps the result in a Narwhals object for the backend you chose. Node.js gets the same columns as apache-arrow objects. Rows are never expanded into per-row Python or JavaScript objects.

Python

What you get back

q value	Python result
Table	narwhals.DataFrame
Keyed table	narwhals.DataFrame with the key columns followed by the value columns
Typed list, for example 1 2 3	narwhals.Series
Dictionary	dict. Tables inside it are converted too.
General list	tuple. Tables inside it are converted too.
Atom	A Python scalar. See the type mapping.

Choose a backend

The backend decides which native frame the Narwhals object wraps:

backend	Native object	Install
"pyarrow" (default)	pyarrow.Table, pyarrow.ChunkedArray	Installed with xqdb
"pandas"	pandas.DataFrame, pandas.Series	pip install pandas
"polars"	polars.DataFrame, polars.Series	pip install polars

Set it when you create the connection, or change conn.backend between calls. An unknown name raises ValueError. A backend that is not installed raises ModuleNotFoundError. xqdb never falls back to a different backend.

import narwhals as nw
import xqdb

conn = xqdb.Q("localhost", 1800, backend="polars")
conn.sync("trade:([] sym:`AAPL`MSFT`AAPL; price:10.5 11.0 10.75; size:100 200 300)")

frame = conn.sync("select from trade")
print(type(nw.to_native(frame)))  # polars.DataFrame

# Narwhals expressions run on whichever backend you chose.
summary = (
    frame.filter(nw.col("sym") == "AAPL")
    .select(nw.col("price").mean().alias("avg_price"), nw.col("size").sum())
)
print(summary.to_dict(as_series=False))  # {'avg_price': [10.625], 'size': [400]}

conn.backend = "pandas"
print(type(nw.to_native(conn.sync("select from trade"))))  # pandas.DataFrame
conn.disconnect()

read_binary6() takes the same backend argument. See Files and IPC Bytes.

Backend differences

• Symbols. q symbol columns become dictionary-encoded string columns. That is dictionary<values=string_view, indices=uint32> in PyArrow, category in pandas, and Categorical in Polars.
• Integer nulls in pandas. pandas converts an Arrow integer column that contains nulls to float64, so a q long column with a null loses exactness above 2⁵³. This was checked with pandas 3.1 and Narwhals 2.28. If you need exact int64 values alongside nulls, use the pyarrow or polars backend.
• Nanosecond scalars. Whole frames and Series round-trip at nanosecond precision on every backend. A single timestamp scalar taken out of a frame depends on the backend:

Backend	Scalar type	Digits below a microsecond
pyarrow with pandas installed	pandas.Timestamp	Kept in .nanosecond, and xqdb sends them on to q
pyarrow without pandas	None. PyArrow raises instead of converting.	Not applicable
pandas	pandas.Timestamp	Kept in .nanosecond, and xqdb sends them on to q
polars	datetime.datetime	Dropped by Polars before xqdb sees the value

When nanoseconds matter, pass the frame or Series rather than a scalar.

Send frames and Series

Any eager DataFrame or Series from PyArrow, pandas, or Polars can be an argument, wrapped in Narwhals or not. It becomes a q table or typed list. Data crosses from Python into Rust through the Arrow C Stream interface, with no Arrow IPC serialization in between. Lazy frames such as polars.LazyFrame raise TypeError. Collect them first. A bare pyarrow.Array is not accepted, so wrap it in pa.chunked_array([...]).

import pandas as pd
import pyarrow as pa
import xqdb

conn = xqdb.Q("localhost", 1800)
conn.sync("quotes:([] sym:`symbol$(); bid:`float$(); note:())")

# pandas: a categorical column becomes a symbol column; object strings become q strings.
frame = pd.DataFrame({
    "sym": pd.Series(["AAPL", "MSFT"], dtype="category"),
    "bid": [10.5, 11.0],
    "note": ["first", "second"],
})
conn.sync("insert", "quotes", frame)

# PyArrow: dictionary-encode strings to send a symbol column.
more = pa.table({
    "sym": pa.array(["GOOG"]).dictionary_encode(),
    "bid": [12.25],
    "note": ["third"],
})
conn.sync("insert", "quotes", more)
print(conn.sync("exec sym from quotes").to_list())  # ['AAPL', 'MSFT', 'GOOG']
conn.disconnect()

String columns are not symbols

An Arrow string column, a pandas object or string column, or a Polars String column becomes a general list of q strings (char vectors). To send a symbol column, dictionary-encode or categorize it: pyarrow.Array.dictionary_encode(), pandas category, or Polars Categorical.

Columns of these Arrow types can be sent:

Arrow type	q column
bool	boolean
uint8	byte
int16, int32, int64	short, int, long
float32, float64	real, float
string, large_string, string_view	general list of strings
dictionary<string>	symbol
fixed_size_binary(16), or binary where every value is 16 bytes	guid
date32	date
timestamp[ms]	datetime (z)
timestamp[us], timestamp[ns]	timestamp (p)
duration (any unit)	timespan
time32, time64	time, with millisecond precision required
list of booleans or numbers	general list of typed lists

Other types, such as int8 or uint16 to uint64, raise an error. Arrow nulls become q nulls: 0N for longs, 0n for floats, and the null symbol for symbols.

Node.js

What you get back

Tables are apache-arrow Table objects and typed lists are Vector objects, in both cases decoded from an Arrow IPC stream that the native layer produces. The data stays columnar and is never turned into row objects. xqdb does not claim the transfer across the N-API boundary is zero-copy.

import { Q } from "@xbbg/xqdb";
import { Table } from "apache-arrow";

const q = await Q.connect({ host: "localhost", port: 1800 });
try {
  await q.sync("trade:([] sym:`AAPL`MSFT; price:10.5 11.0; size:100 200; time:2#.z.p)");
  const trade = await q.sync("select from trade");
  if (trade instanceof Table) {
    console.log(trade.numRows); // 2
    console.log(trade.schema.fields.map((f) => `${f.name}: ${f.type}`));
    // sym: Dictionary<Uint32, LargeUtf8>, price: Float64, size: Int64, time: Timestamp<NANOSECOND>

    const size = trade.getChild("size");
    console.log(size?.get(0)); // 100n: int64 cells are bigint

    const time = trade.getChild("time");
    console.log(time?.get(0)); // a millisecond number: Arrow JS rounds when reading timestamps
    console.log(time?.data[0]?.values[0]); // the exact bigint nanoseconds since the Unix epoch
  }
} finally {
  await q.disconnect();
}

apache-arrow's get() returns timestamp cells as millisecond numbers. Read the column's underlying BigInt64Array, as shown above, when you need every nanosecond. The type mapping lists the Arrow type for every q column type.

xqdb requires apache-arrow 21.2 or newer. From 21.2 onward, instanceof Table and instanceof Vector hold even when your application and xqdb load different copies of the library.

Send Arrow data

Pass a Table to send a q table, or a Vector to send a typed list.

import { Q } from "@xbbg/xqdb";
import { Float64, Int64, Table, Utf8, vectorFromArray } from "apache-arrow";

const q = await Q.connect({ host: "localhost", port: 1800 });
try {
  await q.sync("quotes:([] sym:`symbol$(); bid:`float$(); size:`long$(); note:())");

  const rows = new Table({
    sym: vectorFromArray(["AAPL", "MSFT"]), // inferred Dictionary<Int32, Utf8>: a symbol column
    bid: vectorFromArray([10.5, 11.0], new Float64()),
    size: vectorFromArray([100n, 200n], new Int64()),
    note: vectorFromArray(["first", "second"], new Utf8()), // plain Utf8: q strings
  });
  await q.sync("insert", "quotes", rows);

  const symbols = vectorFromArray(["AAPL", "GOOG"]); // a symbol list
  console.log(await q.sync("{[s] select from quotes where sym in s}", symbols));
} finally {
  await q.disconnect();
}

Arrow JS dictionary-encodes strings by default

vectorFromArray() and tableFromArrays() infer Dictionary<Int32, Utf8> for JavaScript strings, so those columns reach q as symbols. Pass new Utf8() as the type to send q strings. The same rule applies in Python: dictionary-encoded strings become symbols and plain strings become q strings.