Stream Catalog and Bounded Incremental Data¶
P6 exposes read-only structured data events through five registry-defined tools:
stream_catalog, stream_peek, stream_collect, stream_refresh, and
stream_subscribe. CLI subscription is a single cursor/poll window; REST and MCP
use the same finite result contract. None of these tools creates a callback,
webhook, notification, schedule, trading trigger, or human console.
The built-in topic and event records are local contract fixtures. The current
production bindings have no configured live stream provider and return
unavailable with fixture_data_disabled and production_source_not_configured
metadata. The protocol rules below describe the supported bounded contract;
they do not imply current production event coverage.
Catalog contract¶
Each topic declares its topic id, event schema version, provider, license grant, TTL, watermark, latest sequence, opaque cursor protocol, retention time/event limits, maximum subscription duration, and maximum events per request. Catalog and event models are closed schemas.
Every event carries a stable event_id for deduplication plus sequence,
event_time, observed_at, known_at, provider, schema version, quality codes,
license, evidence, lineage, and structured data. Payload fields associated with
prompts, commentary, recommendations, signals, alerts, notifications, scores,
orders, or trade actions are rejected.
Cursor and replay rules¶
Cursors are opaque HMAC-authenticated records bound to authenticated caller,
institution and topic. Request-supplied identity fields never establish this
binding. They
contain the next sequence and an expiry derived from topic retention. Tampering,
cross-tenant/cross-topic use, invalid sequences, expiration, or a sequence older
than retained data fails with cursor_invalid or cursor_expired.
Repeated reads with the same cursor return the same retained event ids. A client
persists the returned cursor only after processing the window; after disconnect it
reuses that cursor. It deduplicates by event_id. Sequence holes return gap and
explicit ranges. Data beyond TTL returns stale without events.
Bounds and failure states¶
The protocol caps each response at 500 events, subscriptions at 60 seconds, provider batches at 500 events, and pending retained events at 2,000. Same tenant/topic refreshes share one in-flight provider request. A full topic buffer rejects refresh before calling the provider. There is no unlimited history replay or subscription.
Stable statuses are ok, stale, gap, partial, out_of_order,
cursor_expired, cursor_invalid, license_denied,
resource_limit_exceeded, and provider_failure. Provider failure is retryable;
if retained usable events exist, refresh returns them as partial and retains the
provider failure code. License and resource violations fail closed.
Monitoring¶
The prepared durable implementation stores topic state and event windows in
PostgreSQL. Topic row locks allocate sequences atomically; compact event
identities survive payload expiration so replay cannot manufacture fresh data.
Provider refresh leases expire after 60 seconds and use ownership tokens to
reject writes from a stale leader. Clients retry a coalesced partial response
after the active leader commits. This storage implementation is not yet evidence
that a production supplier/topic is registered or that its data coverage passed.
The production fixture isolation described above remains in force until the
real-provider bridge and its acceptance are complete.
Operator-configured minimum refresh intervals persist with each topic, so releasing a leader does not reset the provider request budget for other workers. Topics with the same provider share a transaction lock, in-flight lease and the strongest persisted interval across sources, tenants and workers. A restarted reader or newly registered topic cannot reset this budget. Increasing an interval also respects the last request already made. Distinct providers have independent budgets. This conservative grouping does not establish a supplier subscription or authorize provider I/O: the real-provider bridge still must set intervals from verified supplier limits and enforce any additional shared account/product quota. A full retained window rejects refresh before provider I/O; ordinary read calls continue to consume its events.
The migration is additive. A production application rollback preserves these tables and their audit/data history; destructive schema downgrade is restricted to isolated restore rehearsals. Payload retention follows each configured topic; deduplication identities must be retained until a separately approved cleanup policy can demonstrate that upstream IDs will not be replayed.
Prometheus exposes argus_stream_operations_total,
argus_stream_provider_requests_total, and argus_stream_flow_control_total.
The last metric uses bounded kinds such as coalesced, backpressure, and
cursor_rejected; event values and cursor contents never appear in labels.