Skip to content

Persist FalkorDB propositions and chunk history atomically - #162

Draft
jimador wants to merge 3 commits into
codex/scoped-read-documentationfrom
codex/falkordb-store
Draft

jimador wants to merge 3 commits into
codex/scoped-read-documentationfrom
codex/falkordb-store

Conversation

@jimador

@jimador jimador commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Add durable proposition and chunk-history stores backed by FalkorDB through Drivine. Complete propositions and native query fields update atomically, with guarded revisions that preserve concurrent evidence.

Native structured and context-set reads apply filters, grants, ranking and limits in the database. Fixed identity constraints become operational before writes; initialization can recover a failed constraint after duplicate data is repaired. Incompatible caller transactions are rejected.

@jimador

jimador commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@jasperblues, could you review the FalkorDB proposition and chunk storage?

Stack order: #161 → #162 → #163 → #164 → #165.

@jasperblues jasperblues left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimador most of this store can move to Drivine's StatelessGraphObjectManager. Much of the PR rebuilds features Drivine already has.

Why use the object manager

  • Neo4j, FalkorDB, Memgraph and Oracle all come for free.
  • Drivine owns the engine quirks (async constraints, temporal coercion, no rollback on FalkorDB), so this module doesn't have to.
  • Far less code to review and maintain.

Already in Drivine

This PR Drivine 0.1.6
FalkorDbStorageSchema: create the index, poll for OPERATIONAL, handle FAILED, recover after data repair @Unique (composite at class level) plus SchemaCatalog.fromFragments. ConstraintManager already waits for FalkorDB's UNDER CONSTRUCTION → OPERATIONAL/FAILED, drops a failed constraint and reports the rows that violate it
A version UUID compared on write, with up to 64 retries (readVersion / compareAndSet / mutate) @NodeStamp plus update(id, attempts) { … }: reloads and reapplies on StaleObjectException, and returns null if the node was deleted
pin / unpin / addProvenance / setProvenance update { it.withPinned(true) } and so on
touchAccessed: a read and a write per id patchAll: named fields of many nodes in one statement
delete / clear* using FOREACH/CASE delete(id) and deleteAll(class, where)
findById / findByEntity / findByStatus / findByMinLevel / count load plus the typed DSL (eq, isIn, contains, gte, anyOf, not, orderBy, limit)

The proposition's JSON aggregate, with query fields copied out beside it (createdS/createdN, legacyText, contextOrder), is what forces custom Cypher. As a @NodeFragment (@Unique id, @NodeStamp, @JsonPacked provenance), the table above applies directly.

The three places that needed Cypher are now Drivine issues, 78 to 80 inclusive, and will be there:

The chunk nodes are keyed on (context, hash) and (context, source), so they need a single @NodeId: a synthetic id, with the composite @Unique still enforcing the pair.

Smaller

  • Statements are built with " " + concatenation; Embabel uses multiline strings.

@jasperblues

Copy link
Copy Markdown
Contributor

@jimador Drivine 0.1.7 has the three features from liberation-data/drivine4j#78–#80, so the remaining hand-written Cypher can go. It reaches Maven Central within the hour.

Create without losing a concurrent first write (#78)

val stored = sgom.create(proposition)                                   // null when the id is already there
    ?: sgom.update<Proposition>(proposition.id) { it.merge(proposition) }

Rank by effective confidence in the database (#79)

sgom.loadAll<Proposition> {
    where { proposition.context isIn scope.contextIdValues }
    orderBy {
        case(
            (proposition.validFrom above asOf) or (proposition.validTo atMost asOf) then 0.0,
            otherwise = proposition.confidence *
                exp(-proposition.decay * k * floor((epochSeconds(asOf) - proposition.anchorSeconds) / 86400)),
        ).desc()
        proposition.id.asc()
    }
    limit(limit)
}

On FalkorDB, keep the anchor as a numeric seconds property, as the PR already does. epochSeconds of a bound Instant works everywhere, but epochSeconds(property) is opt-in (@TemporalOrder), and FalkorDB refuses it.

Hash and bookmark written together (#80)

sgom.saveAll(listOf(ChunkHash(record), Bookmark(record)))               // one statement on FalkorDB: all or nothing

A batch of fragments only is one UNWIND statement. Set falkorDbTransactionMode: STRICT and a batch that can't be one statement is refused with NonAtomicBatchException before anything is written. PersistenceManager.atomicBatches tells you which behaviour you get.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
@jimador
jimador changed the base branch from codex/falkordb-query-foundation to codex/scoped-read-documentation October 20, 2026 23:09
@jimador
jimador force-pushed the codex/scoped-read-documentation branch from 028629c to ef4e7a0 Compare October 20, 2026 23:10
@jimador
jimador force-pushed the codex/falkordb-store branch from dabff50 to c44fcba Compare October 20, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants