xbbg

Appearance

Catalogs & Documents ​

The works lane answers a query with catalog records: undated bibliographic entries with identifiers (ISBN, DOI, content hash), availability signals, and links. It sits beside the news, data, and events lanes with its own machinery, and connects to file download and dependency-free text extraction.

Search catalogs ​

ts
import { internetArchiveSource, openLibrarySource, searchWorks, searchWorksAcross } from "@xbbg/xnews";

const { items, results } = await searchWorksAcross([openLibrarySource(), internetArchiveSource()], {
  title: "The Intelligent Investor",
  author: "Graham",
  limit: 10,
});

for (const record of items) {
  console.log(record.provider, record.title, record.publishedYear, record.identity.isbn13, record.availability);
}
for (const result of results) console.log(result.provider, result.status, result.recordCount);

// One catalog, with the same non-throwing envelope.
const single = await searchWorks(openLibrarySource(), { isbn: "9780060555665" });
console.log(single.status, single.items[0]?.title);

WorksQuery takes at least one of query, title, author, isbn, or doi (an empty query is a config error), plus page (1-based), maxPages (default 1), limit, and the usual transport options. searchWorks returns a WorksResult (status, items, recordCount, page, warnings, requestUrls, error); searchWorksAcross queries catalogs concurrently and merges their records while keeping each catalog's envelope.

CatalogFactoryOriginPolicy notes
open-libraryopenLibrarySource()Official APISend a User-Agent naming your app and a contact; identified callers get a higher allowance
internet-archiveinternetArchiveSource()Official APINo published rate ceiling, but heavy anonymous traffic is throttled. Borrowable items are not redistributable
libgenlibgenSource({ mirrors })Caller-supplied mirrorsUnofficial; no terms or uptime contract; queries need at least 3 characters
annas-archiveannasArchiveSource({ mirrors })Caller-supplied mirrorsUnofficial; metadata search is keyless; fast downloads need a paid membership key

WORKS_PROVIDER_POLICIES holds the same facts as data. You are responsible for complying with copyright and each catalog's terms; xnews grants no rights to any file.

Records, identity, and dedupe ​

A WorkRecord has provider, sourceId, title, subtitle?, authors, publisher?, publishedYear?, edition?, series?, language?, format? (file extension), pageCount?, sizeBytes?, identity, availability, url (a detail page, not a file), addedAt?, modifiedAt?, warnings, and provenance.

  • Identity holds isbn13, isbn10, doi, oclc, lccn, openLibraryId, and md5, plus origin ("record" when the catalog stated them, "resolved" when inferred) and confidence (always 1 for "record").
  • Availability is the catalog's access signal: open-access, public-domain, preview, borrow, metadata-only, or unknown when the catalog says nothing.
  • Dedupe. mergeWorkRecords (used by searchWorksAcross) merges on the strongest shared identifier (ISBN-13, then DOI, then content MD5) and falls back to a normalized title, author, year, and format key.

Scraped catalogs often state no identifiers. resolveWorkIdentity scores a record against an authoritative catalog and returns the best match only above a confidence floor:

ts
import { openLibrarySource, resolveWorkIdentity, type WorkRecord } from "@xbbg/xnews";

async function enrich(record: WorkRecord): Promise<WorkRecord> {
  const resolution = await resolveWorkIdentity(record, openLibrarySource(), { minConfidence: 0.9 });
  if (resolution.status !== "ok" || !resolution.matched) return record; // no match, or the lookup failed
  console.log(resolution.identity.origin, resolution.identity.confidence); // "resolved", 0.9..1
  return { ...record, identity: resolution.identity };
}

minConfidence defaults to 0.82 and maxCandidates to 10. candidates always lists the scored near misses, and status distinguishes "the catalog answered with no match" from "the catalog never answered" (error or disabled).

ISBN and DOI helpers are exported: normalizeIsbn, isbn10To13, extractIsbns (checksum-validated, from free text), isbnIdentity, normalizeDoi, and workMatchScore.

Mirror pools ​

libgen and annas-archive rotate domains faster than a release cycle, so every URL builder takes an origin you supply. Keep the list in a deployment-local file, mirrors.local.txt by default or the path in XNEWS_MIRRORS_FILE:

text
# mirrors.local.txt
[libgen]
https://first-mirror.example        # a trailing comment becomes the label
second-mirror.example               # a bare host is read as https://

[annas-archive]
https://another-mirror.example
ts
import { libgenSource, loadMirrorList, mirrorBaseUrls, searchWorks } from "@xbbg/xnews";

const list = await loadMirrorList(); // or loadMirrorList("config/mirrors.txt")
console.log(list.warnings); // rejected entries, for example http:// origins

const result = await searchWorks(libgenSource({ mirrors: mirrorBaseUrls(list, "libgen") }), {
  title: "Security Analysis",
});
console.log(result.status, result.warnings);
  • Mirrors are tried in order, and only a failed request advances to the next one. A mirror that answers with zero results ends the search, so one mirror's honest empty page is never replaced by another's results. Mirrors that failed first are named in warnings.
  • An empty pool reports status: "disabled" with a config error, never an empty catalog.
  • parseMirrorList is pure and rejects non-HTTPS origins at parse time. loadMirrorList throws a config error when the file cannot be read. withMirrorFailover applies the same failover rule to your own calls.

Download files ​

WorkRecord.url is a detail page. resolveWorkFiles walks the provider's own indirection to candidate files, downloadFile fetches one, and downloadWork does both and returns the first candidate that answers:

ts
import { downloadWork, extractText, internetArchiveSource, searchWorks } from "@xbbg/xnews";

const result = await searchWorks(internetArchiveSource(), { title: "The Wealth of Nations", limit: 5 });
const record = result.items.find((item) => item.availability === "public-domain");

if (record) {
  const file = await downloadWork(record);
  console.log(file.fileName, file.contentType, file.sizeBytes);

  const book = await extractText(file);
  console.log(book.format, book.characterCount, book.sections.length);
}
CatalogHow a file is reached
internet-archive/metadata/<id> lists files; /download/<id>/<name> serves one. Restricted items answer 401
libgenThe mirror's ads.php page mints a single-use key for get.php, so that page is fetched first
annas-archiveThe documented fast_download JSON API; pass annasArchiveKey (a membership key)
open-libraryA catalog, not a file host: resolves to no files

Downloads raise the response ceiling to DEFAULT_DOWNLOAD_MAX_BYTES (2 GiB) unless you set maxResponseBytes, and set allowCrossOriginRedirects, because catalogs mint a token on their own host and redirect to a CDN. A redirect into a private network address is still refused. Bytes are never decoded, so any file type passes through intact.

Extract text ​

extractText({ bytes, fileName }) decodes these formats with no runtime dependency:

FormatHow
EPUBZIP plus the OPF spine, so chapters come out in reading order and cover/navigation documents stay out of the body
PDFNative decoder: object scan, /ObjStm object streams, Flate/ASCIIHex/ASCII85/RunLength filters, PNG predictors, text operators, /ToUnicode CMaps
DjVuNative decoder for the hidden text layer (IFF, Z′-Coder, BZZ), written from the published DjVu v3 specification
MOBI / AZW / AZW3PalmDOC LZ77 inside the PDB container
CBZOne empty section per page image, in natural order, plus a warning: comics have no text layer
TXT / HTMLText is decoded as UTF-8; HTML has its tags stripped
CBRRefused with a config error naming RAR as the missing decoder
ts
import { readFile } from "node:fs/promises";
import { extractText } from "@xbbg/xnews";

const bytes = new Uint8Array(await readFile("annual-report.pdf"));
const extracted = await extractText({ bytes, fileName: "annual-report.pdf" }, { maxCharacters: 2_000_000 });

console.log(extracted.format, extracted.characterCount, extracted.warnings);
for (const section of extracted.sections.slice(0, 3)) {
  console.log(section.href, section.title, section.text.slice(0, 120));
}

The format is inferred from the file name; pass format when the bytes came without one, and maxCharacters to stop early. The PDF decoder does not rasterize or guess: a PDF with no text layer (a scan) throws a config error that says OCR is needed, instead of returning an empty string, and so does a DjVu without hidden text. Oversized PDF streams are recorded in warnings rather than materialized. For lower-level access, extractPdfText, extractPdfImages, and extractDjvuText are exported too.

OCR for scanned PDFs ​

Scanned PDF pages are usually JPEG images (/DCTDecode), which xnews extracts directly without a PDF renderer and sends to any OpenAI-compatible /v1/chat/completions vision endpoint:

ts
import { readFile } from "node:fs/promises";
import { extractText } from "@xbbg/xnews";

const bytes = new Uint8Array(await readFile("scanned-prospectus.pdf"));
const book = await extractText(
  { bytes, fileName: "scanned-prospectus.pdf" },
  { ocr: { baseUrl: "http://127.0.0.1:10000", model: "Unlimited-OCR", pagesPerRequest: 8 } },
);
console.log(book.text.slice(0, 1_000), book.warnings);

Defaults follow Baidu's Unlimited-OCR recipe (temperature: 0, its image modes and prompts); serve it with that project's vLLM or SGLang instructions. On SGLang, customLogitProcessor enables its repeat suppression; vLLM rejects that field, so it is only sent when you set it. Any other OpenAI-compatible vision endpoint works: set baseUrl to OPENROUTER_BASE_URL and pass apiKey to route through OpenRouter. Pages are batched (pagesPerRequest, default 8) so a long book does not overrun a model's context, and a failed batch becomes a warning with its page range rather than a silently shorter text. ocrImages exposes the OCR step on its own.

Spreadsheets, archives, and CSV ​

The readers behind the data lane are public, in @xbbg/xnews/parsers and the root export:

ts
import { readFile } from "node:fs/promises";
import { excelSerialDateToIso, parseCsvTable, readXlsx, readZipEntries } from "@xbbg/xnews/parsers";

const workbook = await readXlsx(new Uint8Array(await readFile("e16.xlsx")));
const firstSheet = workbook.sheets[0];
console.log(firstSheet?.name, firstSheet?.rows[3]?.[2]); // rows are dense: blank cells keep their column

const cell = firstSheet?.rows[1]?.[0];
if (typeof cell === "number") console.log(excelSerialDateToIso(cell)); // dates arrive as Excel serials

const entries = await readZipEntries(new Uint8Array(await readFile("bundle.zip")));
console.log(entries.map((entry) => `${entry.name} ${entry.bytes.length}`));

console.log(parseCsvTable('name,value\n"a, b",1\n'));
  • readXlsx resolves sheets in workbook order, concatenates rich-text runs, honors xml:space="preserve", and keeps blank cells positional. Formula cells yield their cached value; styles, charts, and formula evaluation are out of scope.
  • readZipEntries rejects multi-disk archives, supports ZIP64 sizes, requires each entry to inflate to exactly its declared size, and refuses any entry that declares more than MAX_ZIP_UNCOMPRESSED_BYTES (512 MiB) before decompressing.
  • parseCsvRecords and parseCsvTable follow RFC 4180 (quoted fields, escaped quotes, embedded newlines) and throw on text that ends inside a quoted field.

xbbg is independent open-source software and is not affiliated with, endorsed by, sponsored by, or approved by Bloomberg Finance L.P.

Bloomberg and related marks are trademarks or service marks of Bloomberg Finance L.P. or its affiliates. Use of Bloomberg data, APIs, and SDKs requires appropriate Bloomberg licenses and entitlements.