Skip to content

feat(amber): add models to the hub and unified search - #7930

Draft
tanishqgandhi1908 wants to merge 1 commit into
apache:mainfrom
tanishqgandhi1908:feat/models-in-hub
Draft

feat(amber): add models to the hub and unified search#7930
tanishqgandhi1908 wants to merge 1 commit into
apache:mainfrom
tanishqgandhi1908:feat/models-in-hub

Conversation

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Models had metadata and access APIs but no discovery: they were missing from the hub's
counts, likes, views, tops and user-access endpoints, and from unified search.

  • Schema: model_user_likes, model_view_count, and model added to the PGroonga block.
    texera_ddl.sql builds that index only for a fresh database, so 42.sql creates it as
    well — closing for models the gap datasets still have on migrated databases.
  • EntityType.Model plus one EntityTables registry entry wires up every hub endpoint.
  • VersionedResourceTables.ModelTables and ModelSearchQueryBuilder add search. The
    builder is a projection only; framework and format are the sole new union slots, and
    the rest are shared with the other versioned resources.
  • DashboardResource gains the model entry slot, a modelId filter, the query and
    toEntry cases, and models in hasMismatch. getTops needed no change — the descriptor
    path picks models up on its own.

Model search matches on name and description, so the indexed expression is identical to the
predicate the builder renders; framework and format are projected for display, not
indexed.

Any related issues, documentation, discussions?

Part of #6501 (umbrella #6494). Builds on #7928, and takes migration 42 because #6872 and
#7922 hold 41 — their commits show in the diff until they merge.

How was this PR tested?

WorkflowExecutionService/testOnly *dashboard* — 498/498 across 27 suites, 31 of them new:
a ModelSearchQueryBuilderSpec pinning the projection, access scoping and id filter; model
cases in VersionedResourceTablesSpec, EntityTablesSpec, EntityTypeSpec and
HubEntityModelSpec; and hub coverage in HubResourceSpec for likes, unlikes, view counts,
getCounts, getTops and user-access, including id collisions across all three entity
types. 42.sql was applied twice against a local Postgres for idempotence, and
idx_model_pgroonga renders the same shape as idx_dataset_pgroonga.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

@github-actions github-actions Bot added feature engine ddl-change Changes to the TexeraDB DDL platform Non-amber Scala service paths labels Aug 24, 2026
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Neilk1021, @Yicong-Huang
    You can notify them by mentioning @Neilk1021, @Yicong-Huang in a comment.

@codecov-commenter

codecov-commenter commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.09524% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.23%. Comparing base (444fc58) to head (bd1f962).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...era/web/resource/dashboard/DashboardResource.scala 55.55% 1 Missing and 3 partials ⚠️
...b/resource/dashboard/VersionedResourceTables.scala 88.88% 2 Missing ⚠️
.../resource/dashboard/user/model/ModelResource.scala 71.42% 2 Missing ⚠️
...b/resource/dashboard/ModelSearchQueryBuilder.scala 94.11% 1 Missing ⚠️
...xera/web/resource/dashboard/hub/EntityTables.scala 95.83% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7930      +/-   ##
============================================
- Coverage     93.24%   93.23%   -0.02%     
+ Complexity     4677     4674       -3     
============================================
  Files          1177     1179       +2     
  Lines         47685    47763      +78     
  Branches       5312     5317       +5     
============================================
+ Hits          44466    44531      +65     
- Misses         1743     1751       +8     
- Partials       1476     1481       +5     
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø)
agent-service 99.32% <ø> (ø) Carriedforward from 444fc58
amber 89.38% <88.09%> (-0.03%) ⬇️
computing-unit-managing-service 73.67% <ø> (ø)
config-service 86.73% <ø> (ø)
file-service 86.70% <ø> (ø)
frontend 95.67% <ø> (-0.01%) ⬇️ Carriedforward from 444fc58
notebook-migration-service 79.31% <ø> (ø)
pyamber 97.78% <ø> (ø) Carriedforward from 444fc58
workflow-compiling-service 77.19% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 10 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main 444fc58 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 414 0.253 22,482/36,755/36,755 us 🔴 +21.1% / 🔴 +165.5%
🔴 bs=100 sw=10 sl=64 912 0.556 106,753/132,312/132,312 us 🔴 +11.3% / 🔴 +41.2%
bs=1000 sw=10 sl=64 1,108 0.676 910,543/937,527/937,527 us ⚪ within ±5% / 🔴 -7.6%
Baseline details

Latest main 444fc58 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 414 tuples/sec 466 tuples/sec 894.86 tuples/sec -11.2% -53.7%
bs=10 sw=10 sl=64 MB/s 0.253 MB/s 0.284 MB/s 0.546 MB/s -10.9% -53.7%
bs=10 sw=10 sl=64 p50 22,482 us 18,560 us 11,139 us +21.1% +101.8%
bs=10 sw=10 sl=64 p95 36,755 us 33,738 us 13,843 us +8.9% +165.5%
bs=10 sw=10 sl=64 p99 36,755 us 33,738 us 16,839 us +8.9% +118.3%
bs=100 sw=10 sl=64 throughput 912 tuples/sec 973 tuples/sec 1,166 tuples/sec -6.3% -21.8%
bs=100 sw=10 sl=64 MB/s 0.556 MB/s 0.594 MB/s 0.711 MB/s -6.4% -21.8%
bs=100 sw=10 sl=64 p50 106,753 us 100,445 us 87,509 us +6.3% +22.0%
bs=100 sw=10 sl=64 p95 132,312 us 118,829 us 93,675 us +11.3% +41.2%
bs=100 sw=10 sl=64 p99 132,312 us 118,829 us 102,153 us +11.3% +29.5%
bs=1000 sw=10 sl=64 throughput 1,108 tuples/sec 1,104 tuples/sec 1,198 tuples/sec +0.4% -7.5%
bs=1000 sw=10 sl=64 MB/s 0.676 MB/s 0.674 MB/s 0.731 MB/s +0.3% -7.6%
bs=1000 sw=10 sl=64 p50 910,543 us 905,689 us 859,766 us +0.5% +5.9%
bs=1000 sw=10 sl=64 p95 937,527 us 969,921 us 902,897 us -3.3% +3.8%
bs=1000 sw=10 sl=64 p99 937,527 us 969,921 us 937,957 us -3.3% -0.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,482.59,200,128000,414,0.253,22481.53,36755.07,36755.07
1,100,10,64,20,2194.10,2000,1280000,912,0.556,106753.09,132312.44,132312.44
2,1000,10,64,20,18051.19,20000,12800000,1108,0.676,910542.52,937526.96,937526.96

Models had metadata and access APIs but no discovery: they were absent from
the hub's counts, likes, views, tops and user-access endpoints, and from
unified search.

Adding the entity type is one registry entry plus one descriptor, since the
hub and the search layer are now resource-agnostic. New schema carries the
engagement tables and the full-text index: model_user_likes,
model_view_count, and model in the PGroonga block. texera_ddl.sql only
builds that index for a fresh database, so the migration creates it too.

ModelSearchQueryBuilder is a projection over the shared query logic, adding
framework and format to the union; every other slot is shared with the other
versioned resources. getTops needed no change at all -- the descriptor path
picks models up on its own.

Model search matches on name and description, so the indexed expression is
identical to the predicate the builder renders. framework and format are
projected for display but not indexed.
@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

/request-review @aicam

@github-actions github-actions Bot removed the platform Non-amber Scala service paths label Aug 26, 2026
@github-actions
github-actions Bot requested a review from aicam August 26, 2026 18:13

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR enables search for models and cause global search also show models. Although we don't have models but to prevent unexpected behavior, lets merge this after frontend fully merged.

@aicam
aicam marked this pull request as draft August 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ddl-change Changes to the TexeraDB DDL engine feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants