Skip to content

refactor(amber): make the hub and unified search resource-agnostic - #7928

Merged
aicam merged 3 commits into
apache:mainfrom
tanishqgandhi1908:refactor/hub-resource-registry
Aug 26, 2026
Merged

refactor(amber): make the hub and unified search resource-agnostic#7928
aicam merged 3 commits into
apache:mainfrom
tanishqgandhi1908:refactor/hub-resource-registry

Conversation

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The hub dispatched on EntityType in four separate places, and DatasetSearchQueryBuilder
carried its own copy of the access-scoped join, the keyword/date/id filter and the
LakeFS-sized hydration. Adding a second resource type meant four dispatch edits plus a
clone of the query logic, and two of those sites compiled cleanly but failed at runtime.

  • EntityTables now exposes one EntityTableSet per entity type, with cloneTable as an
    Option and a new AccessTable that removes the non-exhaustive match in
    HubResource.userAccess.
  • VersionedResourceTables names the columns of a LakeFS-backed resource;
    VersionedResourceSearchQueryBuilder holds the single copy of FROM / WHERE / hydration.
    DatasetSearchQueryBuilder keeps only its projection.
  • The five dataset-only slots in UnifiedResourceSchema become shared versioned-resource
    slots, so the next resource type widens the union by two columns instead of seven.

Two existing bugs are fixed as a result: getTops?entityType=dataset with no actionTypes
no longer 500s on a throwing CloneTable(Dataset), and getCounts' clone guard now asks
whether the entity type has a clone table instead of whether it is not a dataset.

No user-facing behavior changes.

Any related issues, documentation, discussions?

Prepares the amber side of #6501 (part of #6494): adding models becomes one descriptor plus
one registry entry. Contains #6872 and #7922, so their commits show in the diff until they
merge.

How was this PR tested?

Existing suites, unmodified except where they pin the renamed projection aliases:
WorkflowExecutionService/testOnly *dashboard* — 467/467, and FileService/test — 376/376.
New VersionedResourceTablesSpec covers the descriptor contract and the access-join
predicate; EntityTablesSpec covers the registry; regression tests added for the two bugs
above (getTops with no actionTypes, and the clone bucket for a cloneless entity type).

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

@github-actions github-actions Bot added engine ddl-change Changes to the TexeraDB DDL refactor Refactor the code platform Non-amber Scala service paths labels Aug 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Mrudhulraj, @VuMartin
    You can notify them by mentioning @aglinxinyuan, @Mrudhulraj, @VuMartin in a comment.

@codecov-commenter

codecov-commenter commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.25%. Comparing base (5de9bf9) to head (fa7fda3).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...exera/web/resource/dashboard/hub/HubResource.scala 33.33% 11 Missing and 1 partial ⚠️
...b/resource/dashboard/VersionedResourceTables.scala 91.42% 1 Missing and 2 partials ⚠️
...ashboard/VersionedResourceSearchQueryBuilder.scala 95.65% 1 Missing ⚠️
...xera/web/resource/dashboard/hub/EntityTables.scala 97.43% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7928      +/-   ##
============================================
+ Coverage     93.22%   93.25%   +0.03%     
+ Complexity     4666     4661       -5     
============================================
  Files          1175     1176       +1     
  Lines         47645    47566      -79     
  Branches       5319     5300      -19     
============================================
- Hits          44417    44358      -59     
+ Misses         1749     1735      -14     
+ Partials       1479     1473       -6     
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø) Carriedforward from 34eb44b
agent-service 99.32% <ø> (ø) Carriedforward from 34eb44b
amber 89.37% <87.50%> (+0.03%) ⬆️
computing-unit-managing-service 73.67% <ø> (ø) Carriedforward from 34eb44b
config-service 86.73% <ø> (ø) Carriedforward from 34eb44b
file-service 86.72% <ø> (+0.02%) ⬆️ Carriedforward from 34eb44b
frontend 95.67% <ø> (+<0.01%) ⬆️ Carriedforward from 34eb44b
notebook-migration-service 79.31% <ø> (ø) Carriedforward from 34eb44b
pyamber 97.78% <ø> (+0.04%) ⬆️ Carriedforward from 34eb44b
workflow-compiling-service 77.19% <ø> (ø) Carriedforward from 34eb44b

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 6 better · 🔴 1 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 5de9bf9 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 563 0.344 17,051/20,933/20,933 us 🟢 -49.4% / 🔴 +56.3%
🟢 bs=100 sw=10 sl=64 1,222 0.746 82,837/94,687/94,687 us 🟢 -18.4% / ⚪ within ±5%
bs=1000 sw=10 sl=64 1,404 0.857 713,163/764,485/764,485 us ⚪ within ±5% / 🟢 -16.5%
Baseline details

Latest main 5de9bf9 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 563 tuples/sec 501 tuples/sec 908.02 tuples/sec +12.4% -38.0%
bs=10 sw=10 sl=64 MB/s 0.344 MB/s 0.306 MB/s 0.554 MB/s +12.4% -37.9%
bs=10 sw=10 sl=64 p50 17,051 us 14,730 us 10,907 us +15.8% +56.3%
bs=10 sw=10 sl=64 p95 20,933 us 41,331 us 13,604 us -49.4% +53.9%
bs=10 sw=10 sl=64 p99 20,933 us 41,331 us 16,697 us -49.4% +25.4%
bs=100 sw=10 sl=64 throughput 1,222 tuples/sec 1,250 tuples/sec 1,188 tuples/sec -2.2% +2.9%
bs=100 sw=10 sl=64 MB/s 0.746 MB/s 0.763 MB/s 0.725 MB/s -2.2% +2.9%
bs=100 sw=10 sl=64 p50 82,837 us 78,895 us 85,003 us +5.0% -2.5%
bs=100 sw=10 sl=64 p95 94,687 us 116,081 us 91,599 us -18.4% +3.4%
bs=100 sw=10 sl=64 p99 94,687 us 116,081 us 99,202 us -18.4% -4.6%
bs=1000 sw=10 sl=64 throughput 1,404 tuples/sec 1,435 tuples/sec 1,221 tuples/sec -2.2% +15.0%
bs=1000 sw=10 sl=64 MB/s 0.857 MB/s 0.876 MB/s 0.745 MB/s -2.2% +15.0%
bs=1000 sw=10 sl=64 p50 713,163 us 693,725 us 837,593 us +2.8% -14.9%
bs=1000 sw=10 sl=64 p95 764,485 us 762,229 us 881,015 us +0.3% -13.2%
bs=1000 sw=10 sl=64 p99 764,485 us 762,229 us 915,550 us +0.3% -16.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,355.32,200,128000,563,0.344,17050.72,20933.24,20933.24
1,100,10,64,20,1636.89,2000,1280000,1222,0.746,82836.56,94686.58,94686.58
2,1000,10,64,20,14243.99,20000,12800000,1404,0.857,713162.98,764485.31,764485.31

…aliases

The projection assertions added upstream in apache#7855 pin the pre-refactor alias
names. The id, publicity, downloadable, privilege and cover slots are now
shared with the other versioned resources, so their aliases are
resource-neutral while the columns feeding them stay the dataset's.

Also drops a stray indent in fetchDashboardVersionedResourcesByIds left by
the merge.
@tanishqgandhi1908
tanishqgandhi1908 force-pushed the refactor/hub-resource-registry branch from 25ebef8 to 2b40030 Compare August 26, 2026 04:42
@github-actions github-actions Bot removed ddl-change Changes to the TexeraDB DDL platform Non-amber Scala service paths labels Aug 26, 2026
@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

/request-review @aicam

@github-actions
github-actions Bot requested a review from aicam August 26, 2026 17:05

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aicam
aicam added this pull request to the merge queue Aug 26, 2026
Merged via the queue into apache:main with commit 444fc58 Aug 26, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine refactor Refactor the code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants