The filter tree is the only filter grammar - #2785
Merged
Merged
Conversation
A filter as plain data: Expr for values and FilterExpr for predicates, one node per operation of the typed API, with expressions allowed on both sides of a comparison. compile() replays a tree onto the typed factories, so a tree built anywhere runs as the same filter everywhere. The type-erased factories move out of the python bindings into the core model, since the compiler needs them without the python feature.
GqlFilter gains an `expr` variant carrying the tree as GraphQL input types that mirror it one to one, so a filter with expressions on both sides of a comparison can be sent. The legacy grammar is converted onto the tree on arrival and nothing executes it any more: GqlFilter, DynFilter and DynView all compile from the tree. A tree can be applied to a graph directly.
A python filter is one value: the tree. Local filtering compiles it, the remote client sends it, repr prints it, so there is nothing to keep in step. With no consumer left, the composite filter enums, FilterTree, the wire recorder and the engine-to-wire converters are deleted. The one filter built from in-process state, by_state_column, is an opaque leaf that runs locally and refuses to serialise.
qp3.rs was a probe that only printed, and py was a stray file swept in by a bulk add.
… become trees
The lowering now produces the tree grammar itself, so a policy placeholder
({"var": …}, {"claim": …}) stays a wire value and is substituted later as
before; membership takes one list value for the same reason.
GqlFilter::into_tree_grammar is the one conversion a store calls, and a
tree-grammar `and` in a row filter applies its parts in order like the
legacy `and`. Data::access_filter_applies lets a policy tell a caller's
value the graph cannot compare from a grant that is wrong.
The per-entity input types (GqlNodeFilter, GqlEdgeFilter, GqlExplodedEdgeFilter, GqlGraphFilter, their wrapper structs) and the lowering that converted them onto the tree are removed. GqlFilter is now the tree type itself, published as FilterExpr, and every filter and select argument takes it. The view collections' nodeFilter and edgeFilter entries collapse into one filter entry, since the tree names the entity it tests. The remote client declares its variables as FilterExpr. filtering.rs keeps only the view collections, the access-filter types and the PropCondition grammar the namespace metagraph filter still uses. DegreeDirection moves next to the tree input types. The tree input structs reject unknown fields like every other input object. Tests spell their filters in the tree grammar; the schema and stubs are regenerated.
The page described filter_nodes, filter_edges and the removed builder classes. It now explains the expression tree in plain terms: where a filter starts, what it can read, how it compares, combining, reading through a view, property history, edges, and how filter applies on a graph, a node collection and a node. Every example runs against a small inline graph and asserts its answer.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 07e3ee4 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
6 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
6 ns/iter (± 0) |
1 ns/iter (± 0) |
6 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
7680537 ns/iter (± 53574) |
1564816 ns/iter (± 35303) |
4.91 |
lotr_graph_window_100/num_nodes |
19 ns/iter (± 0) |
5 ns/iter (± 0) |
3.80 |
lotr_graph_window_100_materialise/materialize |
7747316 ns/iter (± 21943) |
1669150 ns/iter (± 10700) |
4.64 |
lotr_graph_window_10/has_node_existing |
137 ns/iter (± 8) |
62 ns/iter (± 11) |
2.21 |
lotr_graph_window_10_materialise/materialize |
3374027 ns/iter (± 17398) |
971980 ns/iter (± 4278) |
3.47 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
2249104 ns/iter (± 34866) |
334634 ns/iter (± 1287) |
6.72 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
144 ns/iter (± 9) |
62 ns/iter (± 14) |
2.32 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1486269 ns/iter (± 10727) |
230399 ns/iter (± 2617) |
6.45 |
lotr_graph_window_50_layered/num_edges_temporal |
144764 ns/iter (± 2458) |
70121 ns/iter (± 7586) |
2.06 |
lotr_graph_window_50_layered/has_node_existing |
399 ns/iter (± 22) |
129 ns/iter (± 12) |
3.09 |
lotr_graph_window_50_layered/graph_latest |
82933 ns/iter (± 1314) |
36649 ns/iter (± 916) |
2.26 |
lotr_graph_window_50_layered_materialise/materialize |
30045339 ns/iter (± 219570) |
3488825 ns/iter (± 24948) |
8.61 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
587510 ns/iter (± 6543) |
192686 ns/iter (± 1569) |
3.05 |
lotr_graph_persistent_window_50_layered/has_node_existing |
407 ns/iter (± 388) |
174 ns/iter (± 83) |
2.34 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
3350197 ns/iter (± 75347) |
1659940 ns/iter (± 19402) |
2.02 |
lotr_graph_persistent_window_50_layered/graph_latest |
132252 ns/iter (± 2456) |
57549 ns/iter (± 4809) |
2.30 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
51487010 ns/iter (± 199687) |
5298035 ns/iter (± 147912) |
9.72 |
This comment was automatically generated by workflow using github-action-benchmark.
The converted degree tests used f-strings with nested double quotes, which Python 3.11 rejects at compile time. The GraphQL benchmark built its name filter in the removed grammar; it is now a tree.
A view inside a compiled and, or or not used to vanish: the intersection of entity filters never saw the nodes a window hides through time semantics, so window & pred returned pred alone and ~window returned everything. The RBAC row filter papered over this with a sequential special case for a grant's top-level and, which gave the same text two meanings depending on where it sat. Now the tree compiler applies the view legs of the top-level and first (nested ands flatten) and runs the other legs inside that view, the way graph.window(..).filter(pred) does. A view under or or not is refused when the filter is compiled and, in Python, where it is written. The row filter applies the tree as is. Reads through an exploded edge's endpoint are refused too, since the tree has no wrapper for them, and the opaque filter's refusal to serialise is pinned by a test. The at/after/before window arithmetic is shared between the wrapper ops, the erased edge factory and the view compiler instead of being copied. The collection and parity tests pin the composition rule; the set-algebra expectations no longer cover view legs, which are not set algebra.
The remote client raises ValueError for a filter that has no server-side form, as its methods document; the twelve client docstrings that still described the removed NodeFilter and EdgeFilter wire types say so too. latest: false and snapshotLatest: false in a ViewOp are refused instead of being applied. Set members and presence tests are checked when a filter is written, like comparisons already were. The warning on get_graph_unchecked is back on that method, and access_filter_applies carries its own. DegreeDirection uses the same serde casing as its siblings. The four schema types that lacked a description have one, the component resolvers and graph.nodes/edges select arguments describe the tree instead of the removed grammar, and the namespace filter's PropCondition helpers are private again. Dead field-name constants, two unused type aliases and stale comments about the deleted composite path are removed. The parity test for expression right-hand sides reads degrees at the deferred sites, so a remote that dropped the conjunct would fail it. The schema and stubs are regenerated.
The benchmark's TypeScript types are generated from the schema by genql, which makes a field's args object mandatory when an argument description contains a colon. The reworded select and filter descriptions had one, so every nodes and edges selection in the benchmark stopped type-checking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2643 (
filter_expr_wip); merging this lands there.What changes
#2643 gave every filter a typed expression, but a filter still had to travel to a server, or sit in a stored permission grant, in the old per-entity description grammar (
node: { property: { name, where } },window: { …, expr }). Python kept two representations per filter object, one to run and one to send, and anything without a description form (an expression on the right-hand side) could not be sent at all.This PR makes the expression tree the only form a filter has:
filter/model/tree.rs:Expr(aconst, areadof a field/degree/property/metadata, ortemporal/sum/any/… over one) andFilterExpr(comparisons with anExpron both sides, string ops, presence, membership, structural tests, views,and/or/not). It compiles onto the erased factories indyn_factory.rs, so one compiler serves Rust, Python and GraphQL.repr()prints it. The composite filters and the wire recorder are gone.FilterExpr, with one input type per tree node (Expr,Read,Scope,ViewOp,Target,Cmp,Membership). Everyfilter(expr:)/select(expr:)takes it. The view collections'nodeFilterandedgeFilterentries collapse into onefilterentry, since the tree names the entity it tests.GqlNodeFilter,GqlEdgeFilter,GqlExplodedEdgeFilter,GqlGraphFilter, their wrapper structs, and the lowering that converted them.GqlFilterin Rust is the tree type. The namespace metagraph filter keeps its ownwheregrammar; it filters graphs by metadata and was never part of this.FilterExpr!variable.degree() > in_degree()now travels like anything else.Breaking
GraphQL documents and stored permission grants written in the per-entity grammar must be rewritten as trees; there is no conversion on load. The ~450 test fragments in this repo were converted by a script whose output was checked against the old lowering before that lowering was deleted. The pometry-storage side is Pometry/pometry-storage (branch
filter-wire-rbac), which must merge after this.Other fixes folded in
Prop::compare/equals), as the property filters always did;risk <= 2**63 + 5matched nothing before.Edge.src().property(...)) keeps the edge's views; the window was dropped before.Verification
cargo test -p raphtory-graphqltest_base_install(incl. GraphQL e2e and local/remote parity)raphtory-auth137 passed (incl. the restriction-equivalence property tests); RBAC Python 344 passed · 1 xfailed