Skip to content

Archive v0.7: scheduled catalog observer - #22

Merged
WangEn merged 16 commits into
mainfrom
archive/v0.7-catalog-observer
Oct 1, 2026
Merged

WangEn merged 16 commits into
mainfrom
archive/v0.7-catalog-observer

Conversation

@WangEn

@WangEn WangEn commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Goal

Turn the v0.6 observeFirstPartyIdentity ingestion primitive into a periodic first-party Catalog Observer that preserves vendor snapshots and continuously feeds the identity-drift engine.

Collection model

Migration 0009_catalog_observer.sql adds:

  • catalog_observer_sources: source registry, cadence, parser, credential reference and due time;
  • catalog_collection_runs: durable attempt/status/diagnostic history;
  • catalog_source_snapshots: append-only raw response snapshots linked to source_records.

Due work is claimed with FOR UPDATE SKIP LOCKED and next_run_at is advanced before HTTP I/O, so multiple runner processes can coordinate without holding database locks across network requests.

Observer

PgCatalogObserver:

  1. lazily registers current OpenAI / DeepSeek first-party model-list and docs sources;
  2. collects due sources on their own 6h / 24h cadence;
  3. hashes and stores every successful first-party response before identity parsing;
  4. parses OpenAI-compatible model lists;
  5. maps only remote model IDs that match an existing sourced current first-party binding;
  6. reuses the fetched source_record when calling v0.6 observeFirstPartyIdentity;
  7. records unknown remote IDs and missing known IDs as evidence without auto-creating, retiring, or remapping canonical Models.

Docs are snapshot-only in v0.7: they create source/snapshot evidence but do not make parser-derived identity claims yet.

Runner scheduling

Queue-mode runner enables the observer by default and launches collection beside Run processing. The runner polls PostgreSQL every 60s for due sources; source rows own the longer collection cadence.

Optional runtime controls:

  • MODELAPSE_CATALOG_OBSERVER_ENABLED
  • MODELAPSE_CATALOG_OBSERVER_POLL_MS
  • MODELAPSE_CATALOG_OBSERVER_TIMEOUT_MS
  • MODELAPSE_CATALOG_OBSERVER_MAX_RESPONSE_BYTES

A one-shot catalog-admin collect-first-party-catalog command uses the same implementation for operator collection / diagnosis.

v0.6 invariants retained

  • every retrieval gets a source-backed immutable snapshot;
  • alias history remains append-only;
  • execution-binding history remains immutable except one-way close;
  • an unchanged model list can append provenance without creating false binding drift;
  • lack of an explicit remote snapshot preserves the current known Snapshot;
  • unknown remote IDs never become canonical identity merely because they appeared in a list.

Coverage

Adds parser unit coverage and PostgreSQL integration coverage showing two scheduled model-list snapshots with changing provider snapshot IDs produce source-backed alias + execution-binding drift through the existing v0.6 engine while an unmapped remote ID remains evidence only.

Deployment and operator semantics are documented in docs/catalog-observer.md and the Coolify runner guide.

@WangEn
WangEn merged commit 21d37d7 into main Oct 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant