Skip to content

ADR 0012: A current-value projection for filed facts, and SEC bulk ingestion

Status: accepted on 2026-11-13 by the owner. Product authority: consumer product requirements 3.1 (markdown/argus-consumer-product-requirements-3.1.md).

Context

Requirements 3.0 phase 2 asks for current-state projections, independent ingestion and a light read path, with snapshot tools at p95 300 ms and at most three database round trips per read. It was not started. The review of 2026-11-12 found why it now comes first.

  • Reads are slow. One tool call makes several child calls: eight for get_overview, up to thirty for compare_securities. Each child call rebuilds authentication, the license service and the output policy, then queries a general fact table with three time conditions, a raw-record join and a window ranking. The listing notes record 30 to 45 seconds for the statements of the largest filers.
  • Coverage is a 30-company pilot. Ingestion asks the SEC for one company per request and one task may run 300 seconds. A large filer's first load does not fit. Two filers exceed the JSON item budget of the submissions endpoint on every run.
  • Citations point at the wrong filing. The fact store keeps, for each concept, unit and period, only the version filed last. A net income figure for a fiscal year is then cited to a later proxy statement instead of the annual report that first disclosed it.
  • The fact table grows by repetition. A row's identity includes the license scope and the parser version, so a license revision or a parser upgrade stores the same fact again. Nothing prunes it.

Decision

  1. Projection table. Add a table that holds the current value of each filed fact: one row per company, taxonomy, concept, unit, period end and period type. A row carries the value and its provenance: form type, accession number, filing date, whether it comes from an amendment, whether it restates an earlier value, and the earlier value's accession number. It also carries the source and the license id.
  2. Citation rule, applied when the row is written.
  3. An annual value cites the annual report (10-K, 20-F or 40-F). A quarterly value cites the 10-Q.
  4. If an amendment exists, the latest amendment is cited and the value is marked restated.
  5. A later filing that repeats the same value does not change the citation.
  6. A later filing that states a different value replaces the value, is cited, and marks it restated; a note cites the original.
  7. Governance stays. The projection is read inside the core service boundary. The license is checked at delivery against the license id on the row, so a license revision needs no rewrite. Audit records are kept. Within one tool call the principal, license service and output policy are built once and reused by the child calls.
  8. Switch-over. A setting moves get_financials, compare_securities, get_overview and what_changed to the projection. Before the switch the two paths are compared value by value for the companies already covered. The only accepted difference is the citation change in item 2.
  9. Citations. Projection rows get a new reference prefix. Existing evidence:external_fact_… references keep working and no published evidence hash changes.
  10. Bulk ingestion. Financial facts and filing indexes for all filers are refreshed each night from the SEC bulk archives. One archive is registered as one raw record (address, hash, size, fetch time) and is the evidence anchor for the rows read from it. Members are processed per company in shards that fit the task limit, and write the projection directly. The per-company path stays for same-day updates.
  11. Daily bars. Daily bars get a projection on the same idea, so a tool can read a year of bars in one query.
  12. History is kept. The existing fact table is not compacted and no row is deleted. A license revision does not trigger a full re-ingestion.
  13. SEC access rules. Requests declare a user agent and stay under ten per second.

Consequences

  • Financial coverage can grow from the pilot to every filer with XBRL data in one step, because the archives already contain them.
  • Evidence granularity changes for bulk rows: the anchor is the archive record plus the accession number, not one raw record per fact. The governance documents state this before the switch.
  • Tests that pin "the later filing wins" are updated to the rule in item 2. They are not deleted.
  • The migration is new and forward-only. The pinned migration head moves with it in the maintainer documentation and in two tests, and the role script is re-run after it.
  • The backfill reads the existing table through its indexes only, one company at a time, with a statement timeout. It never scans the table.
  • Growth of the fact table falls to the same-day path. Compacting old rows remains a separate decision.