feat(notebook-migration): resolve the Jupyter URL and token per user

[zyratlo]

What changes were proposed in this PR?

notebook-migration-service read one Jupyter URL and token from configuration as process-wide values. That is safe only under the per-user-pod model, where each user runs their own pod. A single global instance would hand every user the same Jupyter and the same token. This resolves both per user instead.

This is stage 2 of the staged roadmap in the architectural note on #5258. Stage 1 (#7390) and stage 4 (#7671) are merged. Stage 3 (#8006) depends on this one.

Registry table
A new user_jupyter table holds one row per user whose Jupyter has been provisioned, keyed on uid. Both the internal and public URLs are stored rather than derived at read time, matching how workflow_computing_unit.uri is the recorded source of truth for a computing unit's address. Storing them lets the deployment change its addressing scheme without a code change, which is what #8006 needs in order to choose between per-user hostnames and a path prefix.

Derived tokens
Each user's token is HMAC-SHA256(secret, uid), computed from storage.jupyter.token-secret. No credential is stored at rest, any replica of the service derives the same value, and rotation is a secret change. The service refuses to start when per-user Jupyter is on with an empty secret: an empty HMAC key is publicly known, so tokens would still look distinct per user while anyone could derive anyone else's.

Fallback gated on a flag, not on row absence
With kubernetes.jupyter-enabled off, every user resolves to the statically configured Jupyter, which is how the single-node (#7932) and local-dev (#7934) deployments run one shared JupyterLab. With it on, a user with no row has nothing provisioned yet and gets an unavailable response. Falling back to the shared server there would hand an unprovisioned user somebody else's notebooks, and would let delete-notebook remove another user's file. The flag is deliberately separate from kubernetes.enabled, so a deployment can run computing units on Kubernetes without per-user Jupyter.

Provisioning
JupyterKubernetesClient mirrors the computing unit's KubernetesClient, including the hostname and subdomain pair that makes the pod FQDN resolve through a headless service. get-jupyter-iframe-url and set-notebook provision on demand; get-jupyter-url and delete-notebook resolve only, so reading a URL or deleting a file never starts a pod. A registered pod that stops answering is discarded and rebuilt, since the row would otherwise outlive the pod and point every later request at nothing. Two concurrent first requests are resolved by the primary key: the loser keeps the winner's row, which holds the same uid-derived addresses.

The uid always comes from the authenticated session and never from a request body, so one user cannot address another's Jupyter.

Two notes on the implementation. The Kubernetes client is built on demand rather than at object initialisation, unlike the computing unit's, because the single-node and local-dev deployments have no cluster to build one against and never provision. Adding io.fabric8:kubernetes-client also raises three transitive versions over Dropwizard's pins (slf4j-api to 2.0.13, jackson-dataformat-yaml and jackson-datatype-jsr310 to 2.17.0); LICENSE-binary claims are updated to match.

Out of scope, and left to #8006: the Helm templates, the NetworkPolicy, and the global Deployment. Also out of scope is reclaiming idle pods. A dead pod self-heals on next use, but nothing reaps one that is merely unused, and the row's ON DELETE CASCADE drops the registration without deleting the pod. Until a deployment supplies a browser-reachable address, kubernetes.jupyter-public-url-template defaults to empty and the public URL falls back to the in-cluster one.

Any related issues, documentation, discussions?

Closes #7665
Parent issue #4301

Stage 3 (#8006) is built directly on this branch, so every decision here has been exercised on a real cluster rather than only in unit tests. See the verification comment below.

How was this PR tested?

NotebookMigrationService/test passes (72) and Config/test passes (71). Test/compile across the whole repo is clean, as is scalafmtCheckAll.

New coverage: token derivation is deterministic per uid, distinct across 50 uids, and changes with the secret; the start-up guard rejects an empty secret only when the feature is on. Resolution falls back with the feature off, returns the row with it on, returns nothing for an unregistered user, and never returns one user's Jupyter to another. Provisioning is exercised against a stubbed Kubernetes client across every branch: fresh provision, reuse of a live pod, rebuild of a pod that stopped answering, cleanup when readiness never arrives, public URL templating, and the concurrent-registration race.

The existing withFakeJupyter suite is untouched and still passes on the stub bound to localhost:9100, which is the point of the flag-gated fallback: test users have no registry row and take the configured path exactly as before.

Applied sql/texera_ddl.sql to a scratch database the way CI does, confirmed the table shape, and confirmed that deleting a user removes their registration row. check_binary_deps.py passes under both the PR-mode and the strict nightly invocations, and NOTICE-binary matches generator output byte for byte.

Not tested on a live cluster. The chart has no notebook-migration or Jupyter templates yet, and the in-cluster pod address does not resolve from outside the cluster, so end to end Kubernetes validation belongs to #8006.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @tanishqgandhi1908, @aicam, @Ma77Ball
  You can notify them by mentioning @tanishqgandhi1908, @aicam, @Ma77Ball in a comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.25806% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.50%. Comparing base (bdc6d2a) to head (755bdaa).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...pache/texera/service/util/JupyterProvisioner.scala	83.33%	0 Missing and 10 partials ⚠️
...a/service/resource/NotebookMigrationResource.scala	94.11%	0 Missing and 1 partial ⚠️
.../apache/texera/service/util/JupyterEndpoints.scala	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8032      +/-   ##
============================================
- Coverage     93.51%   93.50%   -0.01%     
- Complexity     4709     4744      +35     
============================================
  Files          1181     1187       +6     
  Lines         47809    47938     +129     
  Branches       5327     5343      +16     
============================================
+ Hits          44708    44824     +116     
  Misses         1663     1663              
- Partials       1438     1451      +13     

Flag	Coverage Δ		*Carryforward flag
access-control-service	81.00% <ø> (ø)
agent-service	99.32% <ø> (ø)		Carriedforward from bdc6d2a
amber	89.58% <100.00%> (-0.02%)	⬇️
computing-unit-managing-service	73.67% <ø> (ø)
config-service	86.73% <ø> (ø)		Carriedforward from bdc6d2a
file-service	87.89% <ø> (ø)
frontend	95.92% <ø> (ø)		Carriedforward from bdc6d2a
notebook-migration-service	83.15% <91.72%> (+3.84%)	⬆️
pyamber	98.50% <ø> (ø)		Carriedforward from bdc6d2a
workflow-compiling-service	77.19% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 6 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main bdc6d2a benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	367	0.224	26,604/40,047/40,047 us	🔴 +12.3% / 🔴 +181.6%
🔴	bs=100 sw=10 sl=64	780	0.476	127,722/142,940/142,940 us	🔴 +7.6% / 🔴 +50.4%
⚪	bs=1000 sw=10 sl=64	925	0.565	1,076,944/1,143,226/1,143,226 us	⚪ within ±5% / 🔴 +24.7%

Baseline details

Latest main bdc6d2a from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	367 tuples/sec	401 tuples/sec	894.27 tuples/sec	-8.5%	-59.0%
bs=10 sw=10 sl=64	MB/s	0.224 MB/s	0.245 MB/s	0.546 MB/s	-8.6%	-59.0%
bs=10 sw=10 sl=64	p50	26,604 us	23,699 us	11,164 us	+12.3%	+138.3%
bs=10 sw=10 sl=64	p95	40,047 us	37,200 us	14,219 us	+7.7%	+181.6%
bs=10 sw=10 sl=64	p99	40,047 us	37,200 us	16,863 us	+7.7%	+137.5%
bs=100 sw=10 sl=64	throughput	780 tuples/sec	818 tuples/sec	1,158 tuples/sec	-4.6%	-32.6%
bs=100 sw=10 sl=64	MB/s	0.476 MB/s	0.5 MB/s	0.707 MB/s	-4.8%	-32.7%
bs=100 sw=10 sl=64	p50	127,722 us	118,709 us	88,711 us	+7.6%	+44.0%
bs=100 sw=10 sl=64	p95	142,940 us	150,636 us	95,062 us	-5.1%	+50.4%
bs=100 sw=10 sl=64	p99	142,940 us	150,636 us	102,275 us	-5.1%	+39.8%
bs=1000 sw=10 sl=64	throughput	925 tuples/sec	931 tuples/sec	1,188 tuples/sec	-0.6%	-22.1%
bs=1000 sw=10 sl=64	MB/s	0.565 MB/s	0.568 MB/s	0.725 MB/s	-0.5%	-22.1%
bs=1000 sw=10 sl=64	p50	1,076,944 us	1,074,529 us	874,204 us	+0.2%	+23.2%
bs=1000 sw=10 sl=64	p95	1,143,226 us	1,143,262 us	916,845 us	-0.0%	+24.7%
bs=1000 sw=10 sl=64	p99	1,143,226 us	1,143,262 us	947,186 us	-0.0%	+20.7%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,544.99,200,128000,367,0.224,26604.33,40046.62,40046.62
1,100,10,64,20,2564.54,2000,1280000,780,0.476,127722.06,142939.91,142939.91
2,1000,10,64,20,21616.42,20000,12800000,925,0.565,1076943.96,1143225.88,1143225.88

[zyratlo]

Flagging that tests are a large share of this PR: 728 of the 1380 added lines, about 53%, or a 1.40:1 ratio against production code.

Most of that came from a coverage pass after Codecov reported 42 missing lines on the first push. The biggest single file is JupyterKubernetesClientSpec at 188 lines, and much of it is Mockito scaffolding rather than assertions. fabric8's fluent API returns type variables, so RETURNS_DEEP_STUBS cannot be used and every step of each chain is stubbed by hand. That mirrors what KubernetesClientSpec in computing-unit-managing-service already does. The payoff is that the pod spec is now asserted field by field, including the hostname and subdomain pair that makes the pod FQDN resolve, which nothing covered before.

The six new files under util/ are at 100% line coverage with no missed lines. The branch partials that remain are scala-logging macro expansions, which would need the suite run at two log levels to cover, so I left them rather than contort the tests.

Worth noting the production side is smaller than the 520 figure suggests: NotebookMigrationResource.scala shows 49 added against 49 deleted because JupyterEndpoints and the reachability probe moved out into util/, so the genuinely new logic is the 451 lines across the six new files.

[zyratlo]

Worth recording that this has been exercised end to end, not only unit tested.

Stage 3 (#8006) is stacked on this branch, and deploying it to a local Minikube cluster ran every decision in this PR against real infrastructure. Calico was used deliberately, because Minikube's default CNI creates NetworkPolicy objects without enforcing them, which would make an isolation test look correct while enforcing nothing.

What that confirmed:

• Provisioning creates the pod on demand and the resource quota moves to match. The pod's labels, hostname and subdomain line up with what the headless service selects, so the FQDN this PR records actually resolves.
• The token a user receives matches an independent HMAC derivation of the same uid, computed outside the JVM from the configured secret. Two users receive different tokens.
• The stale row path works. Deleting a pod without touching its registry row makes the next request log the recorded address as unreachable, discard the row, and provision a replacement.
• Per-user isolation holds. One user's pod cannot reach another's on 8888 by pod IP or by DNS, in either direction, while the same pod reaches another namespace in the same probe. Removing the NetworkPolicy makes the connection succeed and restoring it blocks again, so the denial is attributable to the policy rather than to incidental CNI behaviour.

Three things only a real deployment surfaced, all fixed in the stage 3 branch rather than here, since none of them are defects in this code:

• Deriving the browser-facing origin from a gateway hostname alone is not enough. A deployment reached by port-forward or NodePort has no hostname, so the origin has to be settable directly.
• A proxy that rewrites the Host header breaks JupyterLab's own cross-origin check on cookie-authenticated requests, which silently prevents the kernel from starting.
• The gateway route serving the LLM endpoints had no explicit timeout, so Envoy's 15 second default cut off completions that take longer and were succeeding upstream.

On the Codecov result: the twelve lines it reports are partials, not uncovered lines. The per-file breakdown reads 0 Missing throughout. Ten are scala-logging macro branches, where logger.warn(...) expands to an isWarnEnabled check that would need the suite run at two log levels to cover both sides. The other two are a pattern-match dispatch branch and a case class synthetic.

[zyratlo]

/request-review @mengw15