Watchlists & Watchers

Watchlist feeds

A watchlist feed runs one feed per subject concurrently and merges everything into a single list:

import { buildWatchlistNewsFeedResult } from "@xbbg/xnews";

const result = await buildWatchlistNewsFeedResult({
  subjects: [
    { kind: "company", ticker: "RGA", companyName: "Reinsurance Group of America" },
    { kind: "company", ticker: "MET", companyName: "MetLife" },
    { kind: "topic", query: "insurance regulation" },
  ],
  secUserAgent: "my-app/1.0 ops@example.com",
  limit: 50,
});

console.log(result.items.length, result.partial);

for (const { subject, result: feed } of result.subjects) {
  const failing = feed.providers.filter((provider) => provider.status === "error");
  console.log(subject.displayName, feed.items.length, failing.map((provider) => provider.provider));
}

WatchlistNewsFeedResult contains:

Field	Meaning
subjects	One { subject, result } per input subject, where result is a full NewsFeedResult
items	Items from every subject, merged and deduplicated across subjects, newest first, capped by limit
providers	Every subject's ProviderResult rows, flattened
warnings	Every subject's warnings, flattened
partial	true when any subject's feed was partial
fetchedAt	ISO timestamp of the request

Behavior to know:

• Options other than subjects (sources, since, secForms, credentials, transport settings) apply to every subject. Without sources, each subject gets its own default: company subjects use sec-edgar, yahoo-finance, google-news, and finviz; topic subjects use google-news.
• Per-subject feeds are fetched without limit; the limit applies to the merged list.
• Subjects run concurrently, and each subject fans out to its providers concurrently. A 50-company watchlist with the default sources issues about 200 requests at once, including 50 to sec.gov. SEC asks for at most 10 requests per second, GDELT allows about one request per five seconds per IP, and TickerTick allows 10 requests per minute. Split large watchlists into batches and pace them, or inject a rate-limited fetch (Transport & Security).
• buildWatchlistNewsFeed returns only the merged items and honors strict: true.

Watchers

Each feed kind has a watcher, an async generator that polls and yields only items it has not yielded before:

Watcher	Options
createNewsWatcher	Company feed options plus intervalMs, seenIds
createTopicNewsWatcher	Topic feed options plus intervalMs, seenIds
createWatchlistNewsWatcher	Watchlist options plus intervalMs, seenIds

import { createWatchlistNewsWatcher } from "@xbbg/xnews";

const controller = new AbortController();

try {
  for await (const fresh of createWatchlistNewsWatcher({
    subjects: [
      { kind: "company", ticker: "RGA", companyName: "Reinsurance Group of America" },
      { kind: "topic", query: "reinsurance pricing" },
    ],
    sources: ["google-news", "yahoo-finance", "pr-newswire", "globenewswire"],
    intervalMs: 5 * 60_000,
    signal: controller.signal,
  })) {
    for (const item of fresh) console.log(item.publishedAt, item.provider, item.title);
  }
} catch (error) {
  if (!controller.signal.aborted) throw error;
}

Semantics, from the implementation:

• Interval. intervalMs defaults to 60,000 ms and is the sleep after each poll.
• What is yielded. An array of unseen items, newest first. The first poll yields everything currently in the feed. Polls with nothing new yield nothing.
• What counts as seen. An item is skipped if its id or its canonical URL key was already seen, so a story re-published with a new title but the same URL is not repeated.
• Outages. If no provider finishes ok, empty, or partial on a poll, the watcher throws Error("… unavailable: …") with the provider warnings, rather than looking like a quiet feed. With strict: true it also throws when any provider warns.
• Stopping. Aborting signal stops the loop, but how it stops depends on timing: an abort during the sleep rejects with the signal's reason, and an abort during a poll fails the in-flight requests with code aborted, which either ends the loop after that poll or throws the "unavailable" error if no provider had finished. Wrap the loop as shown and treat any error after an abort as a normal shutdown.
• Memory. The seen set lives in memory and grows with every distinct item for the life of the generator.

Survive restarts

The watcher's memory is lost when your process exits. Persist the ids you have processed and pass them back as seenIds, so a restart does not replay old items. This recipe also restarts the watcher with a delay after an outage:

import { readFile, writeFile } from "node:fs/promises";
import { createNewsWatcher, type NewsItem } from "@xbbg/xnews";

const STATE_FILE = "seen-ids.json";

async function loadSeenIds(): Promise<string[]> {
  try {
    return JSON.parse(await readFile(STATE_FILE, "utf8")) as string[];
  } catch {
    return [];
  }
}

async function publish(items: readonly NewsItem[]): Promise<void> {
  for (const item of items) console.log(item.publishedAt, item.title, item.url);
}

const controller = new AbortController();
process.once("SIGTERM", () => controller.abort());

const seenIds = new Set(await loadSeenIds());

while (!controller.signal.aborted) {
  try {
    for await (const fresh of createNewsWatcher({
      ticker: "RGA",
      companyName: "Reinsurance Group of America",
      secUserAgent: "my-app/1.0 ops@example.com",
      seenIds,
      intervalMs: 5 * 60_000,
      signal: controller.signal,
    })) {
      await publish(fresh);
      for (const item of fresh) seenIds.add(item.id);
      // Keep the newest 5,000 ids so the state file stays bounded.
      await writeFile(STATE_FILE, JSON.stringify([...seenIds].slice(-5_000)));
    }
  } catch (error) {
    if (controller.signal.aborted) break;
    console.error("watcher stopped; restarting in 60 s", error);
    await new Promise((resolve) => setTimeout(resolve, 60_000));
  }
}

Persist ids, not URLs: seenIds is compared against NewsItem.id (and the watcher's internal URL keys). Record NEWS_ITEM_ID_SCHEME_VERSION alongside the ids so you notice if a release changes how ids are derived.

Agents and graphs

Infinite generators are a poor fit for agent tools and LangGraph nodes. @xbbg/xnews-langgraph provides finite, checkpoint-friendly equivalents: createXnewsNewsNode performs one poll and returns the updated seenIds and addedItems. See LangGraph Tools.

Related

• Economic Data Releases: createDataReleaseWatcher for dated releases.
• Live Markets & World Data: createEventWatcher for active alerts.
• Errors & Statuses: what each provider status means.