From d02e5a6a01b4c89ee6353d60ccb34e3522156c76 Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 12:56:59 -0400 Subject: [PATCH 1/8] Define stored-data access and domain event contract --- docs/stored-data-and-events-contract.md | 76 +++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/stored-data-and-events-contract.md diff --git a/docs/stored-data-and-events-contract.md b/docs/stored-data-and-events-contract.md new file mode 100644 index 0000000..6b20732 --- /dev/null +++ b/docs/stored-data-and-events-contract.md @@ -0,0 +1,76 @@ +# Stored Data Access and Change Events Contract + +## Purpose + +The memory service exposes the memory-domain data it actually persists independently of search and derived retrieval mechanisms. It also publishes domain-change notifications through a technology-neutral publication port, with a no-op implementation available by default. + +This document defines intended contract behavior. It does not claim that every current backend already implements every capability. + +## Stored-data fidelity + +The authoritative source is the data persisted by the selected memory implementation. Search results are not a substitute for stored data. + +A conforming implementation SHALL provide stable logical access to persisted memory records and SHALL NOT require a search query, lexical index, vector index, or ranking operation to retrieve a record by its identifier. + +The contract SHALL distinguish persisted data from derived representations. Depending on the backend and configured capabilities, persisted data may include: + +- Original record content in its stored form. +- Stored record metadata and provenance. +- Persisted chunks, including their IDs, content, ordering/offset information where available, parent-record identity, and source revision where supported. +- Persisted embeddings/vectors and their model identity, dimensions, and links to the represented record or chunk where stored. +- Other memory-domain representations that the implementation persists. + +A service SHALL expose the data it actually stores. It SHALL NOT fabricate a representation that is not persisted and present it as stored data. Capability metadata or documentation SHOULD make backend-specific persistence differences discoverable. + +## Required access capabilities + +The public contract SHOULD support: + +1. Get one record by stable ID, including its complete stored content and metadata. +2. Enumerate records with deterministic pagination and documented filtering/order semantics. +3. Retrieve persisted chunks and other representations by stable identity and/or parent record. +4. Retrieve stored embedding values and associated metadata when embeddings are persisted. +5. Export all supported persisted memory-domain data without relying on search or ranking. +6. Preserve resource identity and available lineage across reads and exports. + +Bulk export SHOULD define a consistent snapshot or explicitly document its consistency boundary. Export formats and pagination tokens SHOULD be documented and suitable for large datasets. A complete export MUST include every persisted memory-domain record and representation covered by the export contract, not merely records returned by a search. + +A backend that does not persist chunks or embeddings can still conform to the core record-access contract. It SHALL report that those representations are unsupported or absent rather than silently creating them during a stored-data read. + +## Search is a separate capability + +Text search, semantic vector search, hybrid search, ranking, summaries, and retrieval policies are separate capabilities. They MAY use derived indexes and representations. + +A stale, unavailable, or rebuilding search index SHALL NOT make an otherwise persisted record inaccessible through direct retrieval or complete enumeration/export. Search MAY be temporarily degraded while stored-data access remains available. + +## Domain change publication + +The memory service SHALL define and invoke a technology-neutral publication port for successful domain state changes. A no-op publisher SHALL be available and MAY be the default. Core memory operations SHALL not require a deployed broker or messaging service. + +Initial event types SHOULD include: + +- `memory.record.created` +- `memory.record.updated` +- `memory.record.deleted` + +Events SHALL identify the record and the change; include an event ID, event type, envelope version, occurrence time, producer identity, and record revision where available; and carry a schema-defined payload. Event payloads SHOULD remain compact and avoid copying large content unless the event contract explicitly requires it. Consumers can fetch the full persisted record through the stored-data API. + +The publisher call SHALL occur only after the relevant state transition has succeeded. A no-op publisher SHALL preserve ordinary operation behavior. A future real publisher MUST document delivery and failure semantics. If losing a notification is unacceptable, a durable outbox or equivalent mechanism SHOULD be considered rather than claiming that an ordinary post-write publish is atomic. + +Events are notifications, not the source of truth. Consumers SHALL tolerate duplicates and SHOULD support reconciliation through direct access or export. The event contract does not currently promise durable delivery, replay, global ordering, or exactly-once processing. + +## Decorator composition + +Stored-data access and event publication SHOULD be implementable as composable contract implementations or decorators where appropriate. For example, an event-publishing decorator can delegate to a storage implementation, then publish the successful change without coupling the storage implementation to a broker. + +Decorator order and failure semantics MUST be documented where they affect observable behavior. A decorator SHALL NOT claim stronger persistence/publication atomicity than its underlying implementation provides. + +## Acceptance criteria + +- A record can be retrieved by ID when search/index services are unavailable. +- Enumeration/export does not depend on search results and covers all supported persisted memory-domain data. +- Exported records preserve stored content and metadata; persisted chunks and vectors preserve identity and source links where available. +- Backends clearly identify unsupported derived representations. +- Successful create/update/delete operations invoke the publication port with the appropriate event; failed state changes do not announce a successful change. +- The no-op publisher requires no external infrastructure and does not change operation outcomes. +- Tests verify duplicate-safe event identity and record-fetch follow-up behavior where those fields are supported. From 2a5babc22c43768d6c7f47bad0d70aa620a1d80e Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 12:57:17 -0400 Subject: [PATCH 2/8] Link stored-data and event contract from README --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index fc473fd..71ed3bf 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,10 @@ Run tests with `python -m unittest discover -s tests`. The filesystem store remains authoritative in a RAM-cache/filesystem composition. The cache is read-through and write-through; puts update it only after the delegate write succeeds, and deletes evict cached values. Cache coherence applies to operations through the decorator. Changes made directly to the delegate are not observed by an existing cache. The outermost decorator owns and closes its delegate. +## Stored-data access and change notifications + +The intended service contract is documented in [Stored Data Access and Change Events Contract](docs/stored-data-and-events-contract.md). It distinguishes direct access/export of persisted memory-domain data from search over derived indexes, and defines a technology-neutral event publication port with a no-op default. This contract document describes target behavior; it does not imply that all listed API capabilities are implemented yet. + ## Architectural boundaries - Storage provides persistence mechanisms; memory defines memory-specific semantics. From 4a3d9111e798fe60d1e6d3cea9a298c76453b7ac Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 12:58:11 -0400 Subject: [PATCH 3/8] Use shared messaging contract for memory events --- docs/stored-data-and-events-contract.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/stored-data-and-events-contract.md b/docs/stored-data-and-events-contract.md index 6b20732..c40e833 100644 --- a/docs/stored-data-and-events-contract.md +++ b/docs/stored-data-and-events-contract.md @@ -2,7 +2,7 @@ ## Purpose -The memory service exposes the memory-domain data it actually persists independently of search and derived retrieval mechanisms. It also publishes domain-change notifications through a technology-neutral publication port, with a no-op implementation available by default. +The memory service exposes the memory-domain data it actually persists independently of search and derived retrieval mechanisms. It also publishes domain-change notifications through the shared FlossWare/messaging contract, using its no-op implementation when no transport is configured. Memory SHALL NOT define a competing local event envelope. This document defines intended contract behavior. It does not claim that every current backend already implements every capability. @@ -45,7 +45,7 @@ A stale, unavailable, or rebuilding search index SHALL NOT make an otherwise per ## Domain change publication -The memory service SHALL define and invoke a technology-neutral publication port for successful domain state changes. A no-op publisher SHALL be available and MAY be the default. Core memory operations SHALL not require a deployed broker or messaging service. +The memory service SHALL publish successful domain state changes through the shared FlossWare/messaging contract. It SHALL use the contract's no-op publisher when no transport is configured. Core memory operations SHALL not require a deployed broker or standalone network messaging service. Initial event types SHOULD include: From 6103cee4056a392d2f6f8e514eb9cb36aeab493e Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 13:08:19 -0400 Subject: [PATCH 4/8] Specify fire-and-forget semantics for memory events --- docs/stored-data-and-events-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stored-data-and-events-contract.md b/docs/stored-data-and-events-contract.md index c40e833..2cb291a 100644 --- a/docs/stored-data-and-events-contract.md +++ b/docs/stored-data-and-events-contract.md @@ -45,7 +45,7 @@ A stale, unavailable, or rebuilding search index SHALL NOT make an otherwise per ## Domain change publication -The memory service SHALL publish successful domain state changes through the shared FlossWare/messaging contract. It SHALL use the contract's no-op publisher when no transport is configured. Core memory operations SHALL not require a deployed broker or standalone network messaging service. +The memory service SHALL publish successful domain state changes through the shared FlossWare/messaging contract using fire-and-forget semantics. It SHALL use the contract's no-op publisher when no transport is configured. Memory operations SHALL NOT wait for consumer processing or acknowledgements, and SHALL not require a deployed broker or standalone network messaging service. Initial event types SHOULD include: From 48a31dc78328cd3e1ba3afe5b9d413734b570851 Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 13:13:19 -0400 Subject: [PATCH 5/8] Focus memory contract on persisted data access --- docs/stored-data-and-events-contract.md | 25 ++----------------------- 1 file changed, 2 insertions(+), 23 deletions(-) diff --git a/docs/stored-data-and-events-contract.md b/docs/stored-data-and-events-contract.md index 2cb291a..c4b304a 100644 --- a/docs/stored-data-and-events-contract.md +++ b/docs/stored-data-and-events-contract.md @@ -2,7 +2,7 @@ ## Purpose -The memory service exposes the memory-domain data it actually persists independently of search and derived retrieval mechanisms. It also publishes domain-change notifications through the shared FlossWare/messaging contract, using its no-op implementation when no transport is configured. Memory SHALL NOT define a competing local event envelope. +The memory service exposes the memory-domain data it actually persists independently of search and derived retrieval mechanisms. Messaging and change-event publication are deferred. This document defines intended contract behavior. It does not claim that every current backend already implements every capability. @@ -43,27 +43,9 @@ Text search, semantic vector search, hybrid search, ranking, summaries, and retr A stale, unavailable, or rebuilding search index SHALL NOT make an otherwise persisted record inaccessible through direct retrieval or complete enumeration/export. Search MAY be temporarily degraded while stored-data access remains available. -## Domain change publication - -The memory service SHALL publish successful domain state changes through the shared FlossWare/messaging contract using fire-and-forget semantics. It SHALL use the contract's no-op publisher when no transport is configured. Memory operations SHALL NOT wait for consumer processing or acknowledgements, and SHALL not require a deployed broker or standalone network messaging service. - -Initial event types SHOULD include: - -- `memory.record.created` -- `memory.record.updated` -- `memory.record.deleted` - -Events SHALL identify the record and the change; include an event ID, event type, envelope version, occurrence time, producer identity, and record revision where available; and carry a schema-defined payload. Event payloads SHOULD remain compact and avoid copying large content unless the event contract explicitly requires it. Consumers can fetch the full persisted record through the stored-data API. - -The publisher call SHALL occur only after the relevant state transition has succeeded. A no-op publisher SHALL preserve ordinary operation behavior. A future real publisher MUST document delivery and failure semantics. If losing a notification is unacceptable, a durable outbox or equivalent mechanism SHOULD be considered rather than claiming that an ordinary post-write publish is atomic. - -Events are notifications, not the source of truth. Consumers SHALL tolerate duplicates and SHOULD support reconciliation through direct access or export. The event contract does not currently promise durable delivery, replay, global ordering, or exactly-once processing. - ## Decorator composition -Stored-data access and event publication SHOULD be implementable as composable contract implementations or decorators where appropriate. For example, an event-publishing decorator can delegate to a storage implementation, then publish the successful change without coupling the storage implementation to a broker. - -Decorator order and failure semantics MUST be documented where they affect observable behavior. A decorator SHALL NOT claim stronger persistence/publication atomicity than its underlying implementation provides. +Stored-data access SHOULD be implementable through composable contract implementations or decorators where appropriate. Decorators MUST preserve direct access to the delegate's authoritative stored data and document observable ordering or failure behavior. ## Acceptance criteria @@ -71,6 +53,3 @@ Decorator order and failure semantics MUST be documented where they affect obser - Enumeration/export does not depend on search results and covers all supported persisted memory-domain data. - Exported records preserve stored content and metadata; persisted chunks and vectors preserve identity and source links where available. - Backends clearly identify unsupported derived representations. -- Successful create/update/delete operations invoke the publication port with the appropriate event; failed state changes do not announce a successful change. -- The no-op publisher requires no external infrastructure and does not change operation outcomes. -- Tests verify duplicate-safe event identity and record-fetch follow-up behavior where those fields are supported. From c77e036c825b389e66a2481af26bfa8209c6f5f9 Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 13:13:25 -0400 Subject: [PATCH 6/8] Defer messaging from memory contract documentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 71ed3bf..95b2010 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ The filesystem store remains authoritative in a RAM-cache/filesystem composition ## Stored-data access and change notifications -The intended service contract is documented in [Stored Data Access and Change Events Contract](docs/stored-data-and-events-contract.md). It distinguishes direct access/export of persisted memory-domain data from search over derived indexes, and defines a technology-neutral event publication port with a no-op default. This contract document describes target behavior; it does not imply that all listed API capabilities are implemented yet. +The intended stored-data contract is documented in [Stored Data Access Contract](docs/stored-data-and-events-contract.md). It distinguishes direct access/export of persisted memory-domain data from search over derived indexes. Messaging and event publication are deferred. ## Architectural boundaries From c2ec0231cedbc3722b5034dc42706c06a80547d8 Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 13:17:00 -0400 Subject: [PATCH 7/8] Clarify stored-data contract scope --- docs/stored-data-and-events-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/stored-data-and-events-contract.md b/docs/stored-data-and-events-contract.md index c4b304a..0b030ff 100644 --- a/docs/stored-data-and-events-contract.md +++ b/docs/stored-data-and-events-contract.md @@ -1,4 +1,4 @@ -# Stored Data Access and Change Events Contract +# Stored Data Access Contract ## Purpose From 2c7342cc243eb93a65c78487c7e13b86dc2c471e Mon Sep 17 00:00:00 2001 From: "Scot P. Floess" Date: Fri, 23 Oct 2026 13:17:10 -0400 Subject: [PATCH 8/8] Link stored-data access contract from current README --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 95b2010..b43e0b8 100644 --- a/README.md +++ b/README.md @@ -56,9 +56,13 @@ Run tests with `python -m unittest discover -s tests`. The filesystem store remains authoritative in a RAM-cache/filesystem composition. The cache is read-through and write-through; puts update it only after the delegate write succeeds, and deletes evict cached values. Cache coherence applies to operations through the decorator. Changes made directly to the delegate are not observed by an existing cache. The outermost decorator owns and closes its delegate. -## Stored-data access and change notifications +## REST service -The intended stored-data contract is documented in [Stored Data Access Contract](docs/stored-data-and-events-contract.md). It distinguishes direct access/export of persisted memory-domain data from search over derived indexes. Messaging and event publication are deferred. +See [service/README.md](service/README.md) for the initial PostgreSQL-backed REST service, direct record retrieval, pagination, export, and lexical search. Messaging is deferred. + +## Stored-data access contract + +See [Stored Data Access Contract](docs/stored-data-and-events-contract.md) for direct access, representation fidelity, and complete paginated export requirements. Search is a separate capability. ## Architectural boundaries