Skip to content

Version 1.1.8 ยท Last updated

Observability and asynchronous work

Argus separates liveness, readiness, metrics, audit, and task status. They answer different questions and should not be used interchangeably.

Health checks

  • /health/live proves the API process can answer and reports package/version.
  • /health/ready checks required dependencies and returns 503 unless healthy.
  • Container health also verifies database migration heads, Redis, object storage, Celery worker ping, scheduler heartbeat, API loopback, or an MCP protocol round trip according to the service role.

A live but unready service should stay out of traffic. Do not restart it blindly before reading the failed dependency entry.

Metrics

API metrics are Prometheus text at /metrics; staging/production refresh operational metrics before responding. MCP has a separate authenticated metrics listener. Monitoring covers API/MCP availability, HTTP error rate and latency, governance decisions, Celery queue age/failures, connector failures, container restarts, certificate expiry, backup age, and audit integrity.

Every Registry tool has a bounded metric surface: operation latency, argus_tool_freshness_seconds, argus_tool_completeness_ratio, coded provider errors, cache hit/miss, complete/partial/unavailable status, license denial, and closed-schema errors. Labels contain only Registry identifiers, interfaces, providers, and coded states; facts, cursors, source text, and caller input are never used as labels.

Bounded incremental data additionally exposes argus_stream_operations_total by topic/operation/coded status, argus_stream_provider_requests_total by provider outcome, and argus_stream_flow_control_total for request coalescing, backpressure, and cursor rejection. Cursor values and event payloads are never metric labels.

External model capabilities expose argus_model_calls_total by provider/model/operation/status, argus_model_call_latency_seconds, argus_model_call_degradations_total by reason, argus_model_call_retries_total, argus_model_tokens_total by direction, and argus_model_pending_review_candidates_total by task. Model call audit events (event="model_call") carry the model snapshot, prompt version, input SHA-256, source ids, status, and task correlation id - never prompt text, fragment text, or credentials. See Model Capabilities for enable/disable runbooks.

Use a dedicated metrics token through X-Argus-Metrics-Key or Bearer auth. The metrics endpoint is not a customer data API and exposes no dashboard or control UI.

Audit

The core boundary appends a start record before governed work and a terminal record for success or failure. Audit records include identity, interface, tool, purpose, decisions, safe summaries, hashes, and status while redacting credentials and sensitive payloads. Use audit-export with institution/caller/time/tool/status/ policy filters; production retention runs daily and preserves complete chains.

Celery work

Task types include filing ingestion/parsing/fact extraction, event extraction, data quality checks, point-in-time export, and connector sync. Celery Beat schedules audit retention, bounded vector-index sweeps, and configured market sessions.

Messages are persistent, acknowledged late, rejected on worker loss, and subject to Redis visibility redelivery. Work units are bounded by time limits. Design task handlers and callers for idempotent replay: a worker can finish work but lose its acknowledgement before Redis observes completion.

Task status

Task states are structured records rather than log inference. Poll the tool-specific status endpoint or service repository, retain task and audit ids together, and stop polling on terminal success/failure. A task submission response proves acceptance, not completion or data usability.

Operator triage order

  1. Check liveness, then readiness.
  2. Identify the failed dependency or queue.
  3. Correlate request audit_id, task id, and structured logs.
  4. Inspect permission/license/policy metrics before treating denials as outages.
  5. Confirm migrations, Redis visibility, object storage, and provider health.
  6. Retry only operations documented as idempotent and only with bounded backoff.