Transport & Security

Every request xnews makes goes through one internal chokepoint, and that chokepoint calls options.fetch, which defaults to globalThis.fetch. Nothing else opens a connection. That gives you one place to add proxies, pacing, metering, or test doubles, and one set of guarantees about what xnews will and will not do.

Transport options

Every fetcher, feed builder, and watcher accepts SourceFetchOptions:

Option	Default	Effect
fetch	globalThis.fetch	The function every request goes through
signal	none	Aborts pending requests and stops watchers
timeoutMs	15,000 per request	Covers the whole request including its redirect chain; exceeding it fails with code timeout
maxResponseBytes	32 MiB	Streamed body cap. Must be a non-negative safe integer. Heavy sources raise their own ceiling (see below); an explicit value always wins
redirect	"follow"	"follow" follows at most 10 policy-checked hops; "manual" and "error" refuse to follow
allowCrossOriginRedirects	false (true for file downloads)	Lets a redirect leave the origin even when credentials, sensitive query values, a body, or extra headers are present
userAgent	an xnews identifier	Overrides the User-Agent on every request
secUserAgent	none	Required for sec.gov (falls back to userAgent)
msrbAcceptTermsOfUse	false	Must be true before any MSRB EMMA request
limit, since, until	none	Result count and inclusive date window

import { buildCompanyNewsFeedResult } from "@xbbg/xnews";

const controller = new AbortController();
setTimeout(() => controller.abort(), 60_000);

const result = await buildCompanyNewsFeedResult({
  ticker: "RGA",
  companyName: "Reinsurance Group of America",
  secUserAgent: "my-app/1.0 ops@example.com",
  timeoutMs: 20_000,
  maxResponseBytes: 16 * 1024 * 1024,
  signal: controller.signal,
});
console.log(result.providers.map((provider) => `${provider.provider}:${provider.status}`));

What xnews enforces

Request URLs. Only http: and https: URLs are requested, URLs with embedded credentials are refused, and sec.gov and emma.msrb.org must be HTTPS. Violations fail before any I/O with code config.

Redirects. With redirect: "follow", xnews follows redirects itself so it can re-check every hop: your fetch always receives redirect: "manual". A hop is refused when it:

• points at localhost or a non-public literal IP address (private, loopback, link-local, and other special-purpose ranges). This is never overridable.
• downgrades HTTPS to HTTP,
• or crosses origins while carrying a caller User-Agent identity, a sensitive query parameter (names such as api_key, token, auth, secret, password, signature), a non-GET/HEAD body, or caller-supplied headers, unless allowCrossOriginRedirects is set.

Credential-free public HTTPS GET redirects are followed normally. SEC identity and EMMA consent are re-applied on every hop. xnews does not resolve DNS itself, so a hostname that resolves to a private address is your transport's policy to enforce (see Governed retrieval).

Response size. Bodies are read as a stream and cut off at maxResponseBytes, failing with code network and a message that names maxResponseBytes. Sources that legitimately serve large files raise their own ceiling: file downloads 2 GiB (DEFAULT_DOWNLOAD_MAX_BYTES), FFIEC CDR bulk data 512 MiB, CRA, census, and FR Y-9 archives 256 MiB, NIC 64 MiB, and E.16 8 MiB.

Decoding. A payload that passed the byte cap can still expand. Each ZIP entry must inflate to exactly its declared size, entries declaring more than MAX_ZIP_UNCOMPRESSED_BYTES (512 MiB) are refused before decompression, oversized PDF streams become warnings, and CSV that ends inside a quoted field throws. Malformed or truncated JSON or XML is a provider error, not an empty result, and response bytes are never copied into diagnostics.

Diagnostics. URLs in warnings and errors (including ProviderError.url and XnewsFetchError.url) have credentials removed and sensitive query values replaced with <redacted>. In requestUrls, the OpenAlex feed provider and fredDataSource also redact their keys. Two places are not redacted: the URLs your injected fetch receives (an API key in a query string reaches your function in clear text), and the requestUrls of the Socrata-backed data sources (CFTC COT, CDC wastewater), which include an appToken you pass verbatim as $$app_token. Redact before you log or persist either.

Inject a fetch

The injected function has the shape of fetch. Pass init through unchanged so xnews's headers and redirect: "manual" reach the network.

Pace requests per provider

xnews does not sleep between requests or enforce provider rate limits; PROVIDER_POLICIES documents them and your transport applies them. This wrapper spaces requests to rate-limited hosts:

import { buildWatchlistNewsFeedResult, type SourceFetchOptions } from "@xbbg/xnews";

type Fetch = NonNullable<SourceFetchOptions["fetch"]>;

/** Spaces requests to each matching domain (and its subdomains) at least `ms` apart. */
function pacedFetch(spacing: Readonly<Record<string, number>>, base: Fetch = globalThis.fetch): Fetch {
  const nextSlot = new Map<string, number>();
  return async (input, init) => {
    const host = new URL(input instanceof Request ? input.url : input.toString()).hostname;
    const rule = Object.entries(spacing).find(([domain]) => host === domain || host.endsWith(`.${domain}`));
    if (rule) {
      const [domain, ms] = rule;
      const now = Date.now();
      const slot = Math.max(now, nextSlot.get(domain) ?? 0);
      nextSlot.set(domain, slot + ms);
      if (slot > now) await new Promise((resolve) => setTimeout(resolve, slot - now));
    }
    return base(input, init);
  };
}

const fetch = pacedFetch({
  "sec.gov": 100, // SEC: at most 10 requests per second across all sec.gov hosts
  "arxiv.org": 3_000, // arXiv: 3 seconds between legacy API calls
  "gdeltproject.org": 5_000, // GDELT: about one request per 5 seconds per IP
});

const result = await buildWatchlistNewsFeedResult({
  subjects: [
    { kind: "company", ticker: "RGA", companyName: "Reinsurance Group of America" },
    { kind: "company", ticker: "MET", companyName: "MetLife" },
  ],
  sources: ["sec-edgar", "gdelt", "google-news"],
  secUserAgent: "my-app/1.0 ops@example.com",
  fetch,
  timeoutMs: 60_000, // time spent waiting for a slot counts against the timeout
});
console.log(result.partial, result.warnings);

Waiting inside the injected function counts toward timeoutMs, so raise it when queues can get long.

Meter and log

import { buildTopicNewsFeed, type SourceFetchOptions } from "@xbbg/xnews";

type Fetch = NonNullable<SourceFetchOptions["fetch"]>;

const meteredFetch: Fetch = async (input, init) => {
  const url = new URL(input instanceof Request ? input.url : input.toString());
  for (const key of [...url.searchParams.keys()]) {
    if (/key|token|secret|auth|sig/i.test(key)) url.searchParams.set(key, "redacted");
  }
  const started = performance.now();
  const response = await globalThis.fetch(input, init);
  console.log(init?.method ?? "GET", response.status, `${Math.round(performance.now() - started)} ms`, url.href);
  return response;
};

await buildTopicNewsFeed({ query: "repo market", sources: ["google-news", "federal-reserve"], fetch: meteredFetch });

The same hook works for tests: return canned Response objects and every parser runs exactly as in production.

Governed retrieval

If your platform requires every outbound request to go through its own client (DNS pinning, SSRF allow-lists, robots.txt, central rate limiting), use xnews as endpoint knowledge plus parsers and do the I/O yourself. @xbbg/xnews/catalog cannot open a connection, even in principle: its import graph never reaches the fetch layer.

import { googleNewsRssUrl } from "@xbbg/xnews/catalog";
import { parseGoogleNews } from "@xbbg/xnews/parsers";

async function governedGet(url: string): Promise<string> {
  // Replace with your platform's client.
  const response = await fetch(url, { redirect: "error", signal: AbortSignal.timeout(10_000) });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  return response.text();
}

const query = '"Federal Reserve" balance sheet';
const xml = await governedGet(googleNewsRssUrl(query));
const items = parseGoogleNews(xml, query, 20);
console.log(items.map((item) => item.title));

Every provider has a URL builder in /catalog and a pure parser in /parsers, so this pattern covers the whole news lane. Feed-level behavior (subject normalization, merging, date windows, annotation) lives in the builders; reproduce what you need with mergeNewsItems, classifyMarketEvent, and subjectMatcher.

Related

• Provider Policies: the rate limits and access rules to encode in your transport.
• Errors & Statuses: XnewsFetchError codes and how they map to provider statuses.