Skip to content

ADR 0021: Capture on first request, refresh while in use

Status: accepted on 2026-11-15 by the owner. Product authority: consumer product requirements 3.1 (markdown/argus-consumer-product-requirements-3.1.md), section 6.7, which this record changes, and decisions D16 and D17.

Context

  1. Coverage was a fixed list. config/consumer_coverage.yaml named 30 companies and 2 funds. Every periodic job read a fixed list, and a symbol outside it answered not_covered or nothing.
  2. The owner wants the whole market. On 2026-11-15 the owner asked for all US-listed stocks, exchange-traded funds and the main crypto assets, and for capture to follow what people ask about instead of a fixed job list.
  3. Production on that day. The security master held 10,404 securities of 7,997 companies from the SEC ticker list. Filed financial facts existed for 28 companies and end-of-day bars for 33 symbols. The SEC bulk archives of ADR 0012 had never been loaded.
  4. Fixed jobs do not reach the market. A per-company SEC job reads one company a slot, so the five jobs reach about 1,200 companies. Most of the 10,404 symbols are never asked about, and each still costs a request a day.
  5. What one security costs differs by kind of data.
  6. The whole daily price history of a symbol is one EODHD request.
  7. A company's filing index is one SEC request; its recent filings, events, corporate actions and insider reports are a few tens of requests.
  8. A large filer's first capture of financial facts does not fit the 300-second task limit. The bulk archive holds every filer and is read in one night.
  9. A ticker can be called unknown only against the whole SEC list.

Decision

  1. Each kind of data has one way in.
Data How it is captured
Identity The whole SEC ticker list, every day. Unchanged.
Filed financial facts The SEC bulk archive, every night (ADR 0012). Never on request.
End-of-day prices and split history On the first request for the security, then daily while it is in use.
Filings, company events, corporate actions, insider and ownership reports On the first request for the company, then daily while it is in use.
Fund facts The funds of the coverage file. On request once the fund list of the SEC is in the security master.
Crypto assets Fixed jobs. They are a handful of assets.
  1. The coverage file says what is captured on request. It gains a section market with three switches: financials (filed financial facts of every listed company, from the bulk archive), company_data_on_request (SEC company data) and prices_on_request (end-of-day prices and splits). The stocks and funds lists stay and now mean "captured ahead of any request". The file remains the one statement of coverage: the coverage resource, the website and the tools read it, and a switch changes only with a release.

  2. A first request starts a capture and never waits long.

  3. A data tool that resolves a listed US security outside the lists records that the security was asked about, at most once a day per process, and wakes the worker. The record names the security and nothing else.
  4. The worker submits the captures through the governed connector sync under the scheduler's identity, as a periodic job does. One capture per security, kind and day: a second request the same day submits nothing.
  5. A tool that finds no data for a security whose capture has not finished waits a few seconds for it. If the data arrives it answers with it. Otherwise it answers temporarily_unavailable with a retry time and says the security is being captured for the first time.
  6. A capture that finished and found nothing is answered as before: empty, or not_covered for prices.

  7. A security in use is refreshed every day. A security is in use when someone asked about it in the last 90 days, or an active account holds it, watches it or has an alert on it. The worker's capture pump refreshes the captured kinds of that set, a few tasks a minute. A security that leaves the set keeps its data; the next request captures what is missing.

  8. Limits. New first captures are capped per account and per day and for the whole service per day, counted before anything is sent. An account over its cap is told so with a retry time. The caps are settings; zero stops capture on request without a release.

  9. Privacy. The capture state records the security, the kind and times. It holds no account. The set of held, watched and alerted securities is read across accounts as the held-reference job of release 1.1.37 reads holdings: it returns securities only, never who, and never a count.

  10. Prices keep their license gate. Capturing prices on request offers the end-of-day price of any listed symbol. Requirements 3.1 allowed a wider price coverage only with EODHD's written authorization or with the owner's attestation of D6 extended to all listed symbols. On 2026-11-15 the owner extended it (D18), so prices_on_request ships on with release 1.1.39. If EODHD's terms come to require otherwise, the switch goes off with a release.

  11. The deferral of AC-4 ends. The owner's decision of 2026-11-14 to wait two weeks after the first bulk load is withdrawn (D17). The first load ran on 2026-11-15.

Consequences

  • Any listed company answers its filed financials once the bulk rows are visible, and its filings and events within about a minute of the first question. Capture volume follows use, not the size of the market.
  • find_security gets a third coverage value, on_request, next to full and identity_only. The coverage resource and the website stop listing coverage as symbols only and state the three tiers. Tests that pin the old wording are updated, not deleted.
  • The first answer about a new security is often "being captured, ask again in N seconds". Agents retry; the retry time is part of the answer.
  • The worker gets more tasks. A capture on request shares the worker slots with periodic jobs, so a busy queue delays first captures. Worker concurrency is a deployment setting to watch.
  • A new table and a forward-only migration (0116_security_capture_state). The MCP processes add and touch request rows; only the worker records tasks and their outcome.
  • The fixed per-company SEC jobs and the coverage price jobs stay for the lists of the coverage file.
  • Not decided here: the fund list of the SEC (series and classes) in the security master, and reading the bulk filing index to refresh only the companies that filed. Both follow.