Provider Policies
Every source is a public feed or API, but access rules differ. xnews enforces the preconditions it can check before any network I/O and records the rest as data in PROVIDER_POLICIES (news providers), WORKS_PROVIDER_POLICIES (catalogs), and per-source *_PROVIDER_POLICY constants. A missing entry means no special requirement is known, not that no limit exists. xnews never sleeps between requests: apply the rate limits in your scheduler or injected fetch (Transport & Security).
Access requirements
| Provider | Requirement | How to satisfy it | Without it |
|---|---|---|---|
sec-edgar, sec-fulltext, sec-current, sec-press | Declared User-Agent with a real contact | secUserAgent (or userAgent) | disabled, code config |
msrb-emma | Recorded acceptance of the EMMA Terms of Use | msrbAcceptTermsOfUse: true | disabled, code config |
openalex | API key | openAlexApiKey (feeds) or apiKey | disabled, code config |
| FRED | API key | apiKey | Required by the type |
| Alpha Vantage transcripts | API key | apiKey | Required by the type |
| WFP HungerMap | WFP-issued key | hungerMapDataSource({ apiKey }) | disabled, code config |
libgen, annas-archive | Caller-chosen mirror origins | mirrors (see mirror pools) | disabled, code config |
| Anna's Archive downloads | Membership key | annasArchiveKey | No fast-download candidates |
| CFTC COT, CDC wastewater | Optional Socrata app token for sustained polling | appToken | Shared IP throttling pool |
| Crossref | Optional contact for the polite pool | mailto | Public pool |
Use example values such as my-app/1.0 ops@example.com only in documentation; in production identify your application and a monitored address.
News provider policies
Generated from PROVIDER_POLICIES in @xbbg/xnews 0.2.2.
| Provider | Max req/s | Min interval | Declared UA | API key | Terms acceptance | Terms URL |
|---|---|---|---|---|---|---|
sec-edgar | 10 | yes | ||||
sec-fulltext | 10 | yes | ||||
sec-current | 10 | yes | ||||
sec-press | 10 | yes | ||||
arxiv | 3 s | |||||
openalex | 100 | yes | ||||
bis-research | terms | |||||
bis-research-hub | terms | |||||
nber | ||||||
ssrn | ||||||
crossref | ||||||
world-bank | ||||||
biorxiv | ||||||
medrxiv | ||||||
europe-pmc | terms | |||||
hf-papers | ||||||
osf-preprints | ||||||
msrb-emma | EMMA | |||||
cftc-cot | ||||||
ffiec-cdr | ||||||
dtcc-sdr | ||||||
hf-transcripts |
sec-edgar: SEC fair-access policy: declare a User-Agent with contact information (set secUserAgent) and stay at or under 10 requests per second across all sec.gov endpoints combined.sec-fulltext: SEC fair-access policy: declare a User-Agent with contact information (set secUserAgent) and stay at or under 10 requests per second across all sec.gov endpoints combined.sec-current: SEC fair-access policy: declare a User-Agent with contact information (set secUserAgent) and stay at or under 10 requests per second across all sec.gov endpoints combined.sec-press: SEC fair-access policy: declare a User-Agent with contact information (set secUserAgent) and stay at or under 10 requests per second across all sec.gov endpoints combined.arxiv: arXiv asks legacy API clients making consecutive calls to leave at least three seconds between requests. Adapter availability does not grant a license to redistribute paper content.openalex: OpenAlex requires an API key. Requests consume the key's operation-dependent daily budget, while response rate-limit headers are authoritative for short-term scheduling.bis-research: Review BIS terms and obtain permission where required before reproducing or redistributing content. Adapter availability is endpoint access, not a content-redistribution license.bis-research-hub: Review BIS terms and obtain permission where required before reproducing or redistributing content. Adapter availability is endpoint access, not a content-redistribution license.nber: The JSON listing is nber.org's own site search API — undocumented and subject to change; the RSS feed is the stable official surface. Metadata and abstracts link to NBER landing pages; the papers themselves are not generally redistributable.ssrn: Unofficial, undocumented Elsevier endpoint behind papers.ssrn.com browsing; it may change or be restricted without notice. Listings return metadata without abstracts. Landing pages and PDFs remain governed by SSRN terms; is_paid records are metadata-only pointers.crossref: Free public REST API. Set mailto to join the polite pool and honor the X-Rate-Limit response headers. Bibliographic metadata is openly reusable, but abstracts are publisher-supplied and may carry publisher rights.world-bank: Free, keyless Documents & Reports (WDS) API. Most World Bank publications are CC BY 4.0, but each record's own license statement is authoritative; attribute the World Bank when reproducing content.biorxiv: Free, keyless details API shared by bioRxiv and medRxiv; responses page 100 records per cursor and the endpoint can take tens of seconds to answer. Preprint licenses vary per record; only the metadata is CC0.medrxiv: Free, keyless details API shared by bioRxiv and medRxiv; responses page 100 records per cursor and the endpoint can take tens of seconds to answer. Preprint licenses vary per record; only the metadata is CC0.europe-pmc: Free REST API with cursorMark paging. EBI asks bulk users to keep request rates modest; abstracts and full texts carry per-record licenses.hf-papers: Undocumented but widely used huggingface.co endpoint listing community-curated daily arXiv papers; availability and response shape may change without notice.osf-preprints: OSF JSON:API v2. Unauthenticated traffic is throttled; pages cap at 100 records. Preprint licenses vary per provider and per record.msrb-emma: EMMA gates data behind Terms-of-Use acceptance; after accepting, set msrbAcceptTermsOfUse: true to send the acceptance cookie.cftc-cot: Socrata Open Data API for CFTC Commitments of Traders. Unauthenticated requests share an IP throttling pool; for sustained polling create a free app token (https://publicreporting.cftc.gov/profile/edit/developer_settings) and pass it as appToken. Weekly cadence: positions as of Tuesday, normally published Friday 15:30 ET.ffiec-cdr: FFIEC CDR bulk data distribution is a stateful ASP.NET page: the postback chain needs the session cookies from the initial page load, and call-report archives run about 6 MB per quarter. Quarterly cadence; the page's Call Updated stamp moves when late filers revise a period.dtcc-sdr: DTCC Public Price Dissemination is free and keyless. The intraday slice catalog only retains the most recent days; older data must come from the cumulative end-of-day files, published in the evening US time and skipped on weekends and US holidays. Reported notionals of large trades are capped (trailing '+') under CFTC/SEC block-trade rules.hf-transcripts: Hugging Face datasets-server over an MIT-licensed earnings-call transcript snapshot (~685 US large-cap issuers — much of, but not all of, the S&P 500 — 2005 onward). Pages cap at 100 rows and transcript rows run ~50 KB each; new quarters appear when the dataset is republished, not in real time.libgen: Unofficial mirrors rotate hostnames and publish no terms or uptime contract. Records expose no availability metadata and therefore use availability 'unknown'. Queries must be at least three characters.open-library: Open Library asks frequent callers to send a User-Agent naming the application and a contact address; identified requests get a higher allowance than anonymous ones. Catalog metadata only — availability reflects Open Library's own access signals and is never an assertion that a file may be redistributed.internet-archive: Official advancedsearch.php and/metadata/<id>APIs, keyless and free; the Archive publishes no numeric rate ceiling but throttles heavy anonymous traffic, so identify the caller and keep concurrency low. Availability reflects the Archive's own stated rights and lending signals — a borrowable item is not a redistributable one.annas-archive: Unofficial mirrors rotate hostnames and publish no terms or uptime contract, so every URL requires a caller-supplied origin. Metadata search is keyless; the fast_download JSON API needs a paid membership key (annasArchiveKey) and allows a limited number of downloads per day.
Data source policies
Generated from the *_PROVIDER_POLICY constants. Research policies such as NBER_PROVIDER_POLICY are also exported by name from @xbbg/xnews/catalog.
| Constant | Max req/s | Declared UA | API key | Terms URL |
|---|---|---|---|---|
ARXIV_PROVIDER_POLICY | ||||
BIS_PROVIDER_POLICY | terms | |||
CRA_PROVIDER_POLICY | no | terms | ||
FFIEC_002_PROVIDER_POLICY | no | terms | ||
FFIEC_CENSUS_PROVIDER_POLICY | yes | no | terms | |
FFIEC_E16_PROVIDER_POLICY | no | terms | ||
FRED_PROVIDER_POLICY | 2 | yes | terms | |
FRY9_PROVIDER_POLICY | no | terms | ||
HMDA_PROVIDER_POLICY | yes | no | ||
NIC_PROVIDER_POLICY | no | terms | ||
OPENALEX_PROVIDER_POLICY | 100 | yes |
ARXIV_PROVIDER_POLICY: arXiv asks legacy API clients making consecutive calls to leave at least three seconds between requests. Adapter availability does not grant a license to redistribute paper content.BIS_PROVIDER_POLICY: Review BIS terms and obtain permission where required before reproducing or redistributing content. Adapter availability is endpoint access, not a content-redistribution license.CRA_PROVIDER_POLICY: FFIEC says CRA data are generally released annually by August following the activity year, with annual prior-year resubmission files beginning in February 2026. The original 2024 ZIPs were 33,707 bytes (transmittal), 5,610,753 bytes (aggregate), and 22,816,732 bytes (disclosure). Aggregate and disclosure flat files reproduce derived report tables, not loan-level records. The CRA product and flat-file pages document no numeric request-rate ceiling.FFIEC_002_PROVIDER_POLICY: NPW publishes keyless institution-level FFIEC 002 CSVs, one RSSD ID and reporting quarter per request; it does not offer a bulk 002 archive. Board information is public domain unless indicated otherwise, should cite the Board, and is supplied without warranty. No numeric request-rate ceiling is documented.FFIEC_CENSUS_PROVIDER_POLICY: FFIEC publishes the census flat file annually for use with the matching HMDA/CRA activity year; the census year identifies that year's geography and income denominator, so substituting a stale file changes tract classifications and denominators. The 2026 ZIP was 95,033,841 bytes and inflated to a 301,371,095-byte, 1,212-field CSV. The archive and geomap rejected xnews's bot-shaped default User-Agent during live probes, so this adapter uses a browser-shaped default. Geomap derives a public ArcGIS token from its service manifest rather than requiring a caller key. FFIEC documents no numeric request-rate ceiling.FFIEC_E16_PROVIDER_POLICY: FFIEC publishes the keyless E.16 country-exposure release quarterly from aggregated FFIEC 009 filings. It is a cleansed population-level product, not filer-level data: All Banks contains the LFI and All Others populations, so those groups must not be summed. The verified 2026-03-31 bare XLSX was 202,996 bytes; the 2024-12-31 ZIP wrapper was 175,217 bytes and contained one roughly 181 KiB XLSX. Filenames and link labels are irregular, so consumers must discover links from the index and use the workbook's stated period.FRED_PROVIDER_POLICY: FRED limits API traffic to 120 requests per minute and requires compliance with its API terms. A FRED API key identifies the caller but does not replace source-specific attribution or use restrictions.FRY9_PROVIDER_POLICY: NPW publishes keyless combined quarterly BHCF ZIPs by direct GET and refreshes them around 05:00 EST each weekday. FR Y-9C and Y-9LP are quarterly; FR Y-9SP is semiannual. The page warns that filings may remain incomplete or change until the 45-calendar-day Y-9 deadline. Live 2025-09-30 and 2025-12-31 probes found 664,462-byte and 1,472,610-byte ZIPs expanding to 2,455,902 and 11,226,479 bytes. FR Y-9C is consolidated while FR Y-9LP is parent-only; they are different accounting scopes and must not be summed. No API key, session cookie, authentication, or numeric request-rate ceiling is documented, but ffiec.gov's edge answers a first cold request with an HTML block page under HTTP 403 and then serves subsequent requests, so treat an isolated 403 as retryable rather than as a missing credential.HMDA_PROVIDER_POLICY: Free, keyless HMDA Data Browser API. Its documentation states no numeric request-rate ceiling or API-specific terms; schedule conservatively because exports can be very large. The CFPB edge rejects xnews's bot-shaped default User-Agent, so this adapter uses a browser-shaped default. Public loan-level exports are the modified LAR: CFPB documents 27 fields redacted and 6 fields modified for applicant and borrower privacy.NIC_PROVIDER_POLICY: NIC says Structure Data Download is refreshed daily and describes the snapshots as non-confidential public data, without stating a redistribution license. The five keyless direct-GET CSV ZIPs require neither an ASP.NET postback nor a caller-managed session cookie and currently span about 0.6-14 MiB; xnews allows 64 MiB. No numeric request-rate ceiling is documented. The page's last-update date is the snapshot as-of date; Attributes #ID_RSSD and Relationships parent/offspring RSSD IDs join to FFIEC Call Reports' IDRSSD.OPENALEX_PROVIDER_POLICY: OpenAlex requires an API key. Requests consume the key's operation-dependent daily budget, while response rate-limit headers are authoritative for short-term scheduling.
Works catalog policies
| Catalog | Declared UA | Terms URL |
|---|---|---|
libgen | ||
open-library | yes | terms |
internet-archive | terms | |
annas-archive |
Fixed feeds
Every FIXED_FEEDS entry carries suggestedMinPollSeconds (5 minutes for most market feeds, 10 for regulators and newspapers, 15 for central-bank news, 6 hours for research feeds), a scheduling suggestion rather than an enforced limit. Registry access is endpoint access only; follow each publisher's terms before reproducing full content.
Licensing
- arXiv: API metadata is CC0; PDFs and source files are not generally redistributable. Leave 3 seconds between legacy API calls.
- OpenAlex: data is CC0; budgets and rate limits apply per key.
- FRED: follow the FRED API Terms of Use and display: "This product uses the FRED® API but is not endorsed or certified by the Federal Reserve Bank of St. Louis."
- BIS, NBER, SSRN: metadata and links only; papers remain governed by their publishers' terms.
- Crossref, World Bank, Europe PMC, bioRxiv/medRxiv, OSF: metadata is openly reusable; abstracts and full texts carry per-record licenses.
- Transcripts: dataset licenses do not change the rights in the underlying calls.
xnews provides access, not a license. You are responsible for complying with each source's terms.