Internal Data Source Ingestion Planning¶
This document is the step 50 design record for internal data source ingestion. It is a planning contract only: it does not introduce a new runtime connector engine, human data browser, dashboard, control console, chat surface, webhook, Kafka, Elasticsearch, Airflow, or any investment-advice capability.
Internal data sources are allowed only when they enter Argus through the same
contracts as external sources. The authoritative boundary remains
CoreServiceBoundary, the shared DataPackage and SourceEvidence schema,
license checks, credibility levels, data quality and provenance checks, tenant
isolation, and append-only audit logging. Client investment intent does not block
otherwise permitted facts.
Ingestion Contract¶
Every internal source ingestion proposal must map to a complete machine-readable package before it can become queryable.
| Contract area | Required internal-source behavior | Required metadata | Test matrix id |
|---|---|---|---|
| Data package | Internal facts use the same DataPackage fields as public filings, market data, events, and agent metadata. |
package_type, package_version, facts, source_evidence, data_quality, data_license, audit_id, output_policy_version. |
internal-package-contract |
| Source evidence | Each fact cites SourceEvidence with source_type=internal_system or another explicit source type. |
Source system id, source file or record id, excerpt or structured field value, location, retrieved time, parser version, confidence, credibility level. | internal-source-evidence-contract |
| License model | Internal data is checked by the same license model before output. | License id, status, allowed uses, prohibited uses, redistribution status, authorized institutions, restricted fields. | internal-license-contract |
| Credibility level | Internal data is not automatically above regulatory originals or company official announcements. | Credibility level, review status, configured review workflow id, reviewer role requirement. | internal-trust-contract |
| Quality checks | Missing fields, source conflicts, unit or currency conflicts, time conflicts, duplicates, low confidence, and license conflicts become QualityIssue entries. |
Quality level, issue type, severity, evidence ids, review requirement. | internal-quality-contract |
| Audit flow | Register, ingest, parse, map, validate, reject, conflict, publish, and export events remain audit-addressable. | Audit id, caller id, institution id, source connector id, parser version, policy versions, result status. | internal-audit-contract |
Source Connector Registry¶
Source Connector Registry is the unified registry for both internal and external data sources. It records metadata and governance state; it does not grant permission by itself and does not replace core permission, license, quality, evidence, or output-policy checks.
| Registry field | Purpose | Required for audit |
|---|---|---|
source_id |
Stable source identifier. | Yes |
source_name |
Human-readable source label for audit review. | Yes |
source_scope |
internal or external. |
Yes |
source_type |
Filing, vendor feed, internal research database, CRM export, risk system, accounting system, or caller-submitted file. | Yes |
owning_institution_id |
Tenant that owns or authorizes the source. | Yes |
license_id |
Versioned license policy applied to the source. | Yes |
license_status |
Authorized, restricted, prohibited, or unknown. | Yes |
redistribution_status |
Allowed, restricted, prohibited, or unknown. | Yes |
allowed_uses |
Permitted use cases. | Yes |
prohibited_uses |
Forbidden use cases. | Yes |
update_frequency |
Batch, hourly, daily, event-driven, manual upload, or ad hoc. | Yes |
field_coverage |
Fields and field categories the source claims to provide. | Yes |
field_catalog_version |
Field dictionary version used for mapping. | Yes |
failure_status |
Healthy, degraded, failed, paused, revoked, or unknown. | Yes |
last_retrieved_at |
Last extraction or fetch time. | Yes |
known_at_policy |
Rule for computing known-at time. | Yes |
parser_version |
Parser or mapping version. | Yes |
quality_level |
Current source-level quality state. | Yes |
retention_policy |
Retention class and deletion requirements. | Yes |
redistribution_restrictions |
Downstream sharing constraints. | Yes |
audit_id |
Audit id for the latest registry change. | Yes |
Registry rows must be versioned. A registry update can make a source unavailable or more restricted immediately, but it cannot silently broaden prior package outputs without a new audit record and a license decision.
Trust And Review Rules¶
Internal data defaults to a lower or equal credibility level than external primary evidence unless an institution explicitly configures both a credibility rule and a review workflow.
| Scenario | Default credibility behavior | Required review behavior | Test matrix id |
|---|---|---|---|
| Internal research note conflicts with regulatory filing | Regulatory filing remains higher priority. | Internal value is marked conflicting and requires review. | trust-regulatory-over-internal |
| Internal accounting export supports a public filing value | Internal source may be additional evidence but not the sole top-tier source. | Link both evidence records and preserve source priority. | trust-internal-supporting-evidence |
| Institution configures reviewed internal golden source | May be elevated to configured internal credibility, never above regulatory original. | Record review workflow id, reviewer role, and effective time. | trust-reviewed-internal-source |
| Caller-submitted spreadsheet lacks review | Treat as caller-submitted unverified. | Mark pending review and block ordinary fact output when evidence is insufficient. | trust-caller-submitted-unverified |
Field Mapping¶
Internal source fields must map into the machine-readable Field Catalog before ordinary output.
| Mapping item | Required behavior | Test matrix id |
|---|---|---|
| Source system | Preserve source system id and owning institution. | field-map-source-system |
| Source field | Preserve original field name and path. | field-map-source-field |
| Canonical field | Map to a field catalog entry, including unit, currency, period, aliases, license tier, and quality rules. | field-map-canonical-field |
| Extraction time | Preserve when the internal system emitted or exported the data. | field-map-extraction-time |
| Known-at time | Preserve when the data became knowable to the authorized institution. | field-map-known-at-time |
| License restriction | Preserve source and field-level license limits. | field-map-license-restriction |
| Parser version | Preserve mapping and parser version. | field-map-parser-version |
Unmapped fields remain in a non-queryable or review-only state. They cannot be returned as ordinary facts or used by data preflight results.
Internal Source Sample¶
| Sample item | Value |
|---|---|
source_id |
internal-risk-system-v1 |
source_scope |
internal |
source_type |
risk_system |
owning_institution_id |
institution-alpha |
license_id |
internal-risk-license-v1 |
update_frequency |
daily |
field_coverage |
risk.internal_rating, risk.exposure_bucket |
failure_status |
healthy |
last_retrieved_at |
2026-06-19T01:00:00Z |
parser_version |
internal-risk-parser-v1 |
quality_level |
requires_review |
retention_policy |
retain-7-years-delete-on-revocation |
redistribution_restrictions |
prohibited_outside_owning_institution |
Design-level package mapping:
| Data package field | Internal-source mapping |
|---|---|
request_subject |
Company, security, account-independent entity, or source record id. |
facts |
Canonical field catalog facts only; no account positions, portfolio weights, costs, P&L, target prices, or trading intent. |
source_evidence |
Source system id, source record id, extracted field path, structured excerpt, extraction time, parser version, confidence, and credibility level. |
known_time |
The time the owning institution could know the source value, not necessarily the extraction time. |
data_quality |
Includes missing mapping, source conflict, stale source, license conflict, low confidence, or review-required issues. |
data_license |
Internal license policy, allowed uses, prohibited uses, redistribution restriction, and restricted fields. |
audit_id |
Audit id for register, ingest, validate, and publish stages. |
Conflict Handling Matrix¶
Internal data never silently overwrites external data. Conflicts become quality issues and preserve both evidence records.
| Conflict case | Required outcome | Quality issue | Test matrix id |
|---|---|---|---|
| Internal value differs from regulatory filing | Return regulatory fact when authorized and mark internal fact as conflicting or review-only. | source_evidence_conflict |
conflict-internal-regulatory |
| Internal value differs from company announcement | Preserve both evidence records and require review before using internal value. | source_evidence_conflict |
conflict-internal-company-announcement |
| Internal value differs from authorized vendor | Apply configured source priority and record conflict details. | source_evidence_conflict |
conflict-internal-vendor |
| Internal value has newer known-at time than as-of query | Exclude it from historical output. | time_conflict |
conflict-internal-future-known-at |
| Internal source license forbids requested use | Deny or redact according to license; do not leak restricted values. | data_license_conflict |
conflict-internal-license |
Permission And License Matrix¶
| Scenario | Required behavior | Test matrix id |
|---|---|---|
| Caller belongs to owning institution and has field permission | Data may proceed to license, evidence, quality, output policy, and audit gates. | perm-owning-institution-allowed |
| Caller belongs to another institution | Deny before returning field values or evidence excerpts. | perm-cross-institution-denied |
| Caller lacks tool whitelist entry | Deny with machine-readable reason and audit id. | perm-tool-whitelist-denied |
| Caller lacks field category permission | Deny or redact restricted fields. | perm-field-category-denied |
| License allows internal use but forbids redistribution | Return only within allowed context and mark redistribution restriction. | license-internal-use-only |
| License status is unknown or prohibited | Fail closed for ordinary fact output. | license-unknown-fail-closed |
Source Mapping Example¶
| Internal fact metadata | Required preserved value |
|---|---|
source_system |
internal-risk-system-v1 |
source_record_id |
Stable internal record id or object path. |
source_field_path |
Original internal field path, such as ratings.current.internal_grade. |
canonical_field |
Field catalog id, such as risk.internal_rating. |
extracted_at |
Time the connector extracted the record. |
known_at |
Time the owning institution could know the value. |
license_id |
Internal source license policy id. |
parser_version |
Connector parser or field-mapping version. |
audit_id |
Audit record for the mapping decision. |
Source Material And Output-Origin Matrix¶
Internal data does not turn Argus into a judgment or execution engine. Argus returns facts and labeled source material; client Agents remain free to perform independent analysis and output. Account credentials and automated orders remain outside this data service.
| Scenario | Required behavior | Test matrix id |
|---|---|---|
| Request mentions a buy, sell, or hold view using internal data | Return otherwise permitted facts and preserve source labels; Argus itself does not generate the view. | strategy-client-analysis-independent |
| Unlabeled Argus-origin output presents a target price or position size as its own judgment | Output-origin denial before any access layer returns it; labeled quoted source material is preserved. | strategy-output-origin-denied |
| Internal data is requested for neutral field catalog mapping | Allowed only through structured metadata, permission, license, and audit gates. | strategy-neutral-metadata-allowed |
Release Gate For Step 50¶
Step 50 is complete only when design-level tests confirm:
- Internal data source samples map to a complete
DataPackageandSourceEvidencecontract. - Source Connector Registry can express source type, license, update frequency, field coverage, failure status, last retrieval time, parser version, quality level, retention policy, and redistribution limits.
- Internal source facts preserve source system, extraction time, known-at time, field mapping, and license restriction metadata.
- Internal data and external data conflicts produce quality issues and never silently overwrite higher-trust sources.
- Permission and license matrices deny cross-institution, field, tool, unknown license, and redistribution violations without leaking restricted values.
- License, tenant, permission, schema, provenance, and audit controls still apply to internal data; client-Agent reasoning remains independent.
- No human data browser, web page, dashboard, control console, chat surface, account self-service page, manual form, or prohibited financial capability is added.