feat(amber): add models to the hub and unified search - #7930
feat(amber): add models to the hub and unified search#7930tanishqgandhi1908 wants to merge 1 commit into
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #7930 +/- ##
============================================
- Coverage 93.24% 93.23% -0.02%
+ Complexity 4677 4674 -3
============================================
Files 1177 1179 +2
Lines 47685 47763 +78
Branches 5312 5317 +5
============================================
+ Hits 44466 44531 +65
- Misses 1743 1751 +8
- Partials 1476 1481 +5
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 414 | 0.253 | 22,482/36,755/36,755 us | 🔴 +21.1% / 🔴 +165.5% |
| 🔴 | bs=100 sw=10 sl=64 | 912 | 0.556 | 106,753/132,312/132,312 us | 🔴 +11.3% / 🔴 +41.2% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,108 | 0.676 | 910,543/937,527/937,527 us | ⚪ within ±5% / 🔴 -7.6% |
Baseline details
Latest main 444fc58 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 414 tuples/sec | 466 tuples/sec | 894.86 tuples/sec | -11.2% | -53.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.253 MB/s | 0.284 MB/s | 0.546 MB/s | -10.9% | -53.7% |
| bs=10 sw=10 sl=64 | p50 | 22,482 us | 18,560 us | 11,139 us | +21.1% | +101.8% |
| bs=10 sw=10 sl=64 | p95 | 36,755 us | 33,738 us | 13,843 us | +8.9% | +165.5% |
| bs=10 sw=10 sl=64 | p99 | 36,755 us | 33,738 us | 16,839 us | +8.9% | +118.3% |
| bs=100 sw=10 sl=64 | throughput | 912 tuples/sec | 973 tuples/sec | 1,166 tuples/sec | -6.3% | -21.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.556 MB/s | 0.594 MB/s | 0.711 MB/s | -6.4% | -21.8% |
| bs=100 sw=10 sl=64 | p50 | 106,753 us | 100,445 us | 87,509 us | +6.3% | +22.0% |
| bs=100 sw=10 sl=64 | p95 | 132,312 us | 118,829 us | 93,675 us | +11.3% | +41.2% |
| bs=100 sw=10 sl=64 | p99 | 132,312 us | 118,829 us | 102,153 us | +11.3% | +29.5% |
| bs=1000 sw=10 sl=64 | throughput | 1,108 tuples/sec | 1,104 tuples/sec | 1,198 tuples/sec | +0.4% | -7.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.676 MB/s | 0.674 MB/s | 0.731 MB/s | +0.3% | -7.6% |
| bs=1000 sw=10 sl=64 | p50 | 910,543 us | 905,689 us | 859,766 us | +0.5% | +5.9% |
| bs=1000 sw=10 sl=64 | p95 | 937,527 us | 969,921 us | 902,897 us | -3.3% | +3.8% |
| bs=1000 sw=10 sl=64 | p99 | 937,527 us | 969,921 us | 937,957 us | -3.3% | -0.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,482.59,200,128000,414,0.253,22481.53,36755.07,36755.07
1,100,10,64,20,2194.10,2000,1280000,912,0.556,106753.09,132312.44,132312.44
2,1000,10,64,20,18051.19,20000,12800000,1108,0.676,910542.52,937526.96,937526.96Models had metadata and access APIs but no discovery: they were absent from the hub's counts, likes, views, tops and user-access endpoints, and from unified search. Adding the entity type is one registry entry plus one descriptor, since the hub and the search layer are now resource-agnostic. New schema carries the engagement tables and the full-text index: model_user_likes, model_view_count, and model in the PGroonga block. texera_ddl.sql only builds that index for a fresh database, so the migration creates it too. ModelSearchQueryBuilder is a projection over the shared query logic, adding framework and format to the union; every other slot is shared with the other versioned resources. getTops needed no change at all -- the descriptor path picks models up on its own. Model search matches on name and description, so the indexed expression is identical to the predicate the builder renders. framework and format are projected for display but not indexed.
805c251 to
bd1f962
Compare
|
/request-review @aicam |
aicam
left a comment
There was a problem hiding this comment.
This PR enables search for models and cause global search also show models. Although we don't have models but to prevent unexpected behavior, lets merge this after frontend fully merged.
What changes were proposed in this PR?
Models had metadata and access APIs but no discovery: they were missing from the hub's
counts, likes, views, tops and user-access endpoints, and from unified search.
model_user_likes,model_view_count, andmodeladded to the PGroonga block.texera_ddl.sqlbuilds that index only for a fresh database, so42.sqlcreates it aswell — closing for models the gap datasets still have on migrated databases.
EntityType.Modelplus oneEntityTablesregistry entry wires up every hub endpoint.VersionedResourceTables.ModelTablesandModelSearchQueryBuilderadd search. Thebuilder is a projection only;
frameworkandformatare the sole new union slots, andthe rest are shared with the other versioned resources.
DashboardResourcegains themodelentry slot, amodelIdfilter, the query andtoEntrycases, and models inhasMismatch.getTopsneeded no change — the descriptorpath picks models up on its own.
Model search matches on name and description, so the indexed expression is identical to the
predicate the builder renders;
frameworkandformatare projected for display, notindexed.
Any related issues, documentation, discussions?
Part of #6501 (umbrella #6494). Builds on #7928, and takes migration 42 because #6872 and
#7922 hold 41 — their commits show in the diff until they merge.
How was this PR tested?
WorkflowExecutionService/testOnly *dashboard*— 498/498 across 27 suites, 31 of them new:a
ModelSearchQueryBuilderSpecpinning the projection, access scoping and id filter; modelcases in
VersionedResourceTablesSpec,EntityTablesSpec,EntityTypeSpecandHubEntityModelSpec; and hub coverage inHubResourceSpecfor likes, unlikes, view counts,getCounts,getTopsanduser-access, including id collisions across all three entitytypes.
42.sqlwas applied twice against a local Postgres for idempotence, andidx_model_pgroongarenders the same shape asidx_dataset_pgroonga.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)