Webapp: PET optimization view + per-model risk assessment by fazelehh · Pull Request #448 · aidotse/LeakPro

fazelehh · 2026-08-18T12:03:09Z

Top of the stack: #438 → #447 → this. Retarget to main as the rungs below merge (GitHub does it automatically).

Contains the merge of feature/leakage-risk-assessment (#445, @TheColdIce) with his original commits preserved — if #445 merges to main first, his commits drop out of this diff automatically; if this merges first, his work lands here with authorship intact and #445 closes as merged.

PET optimization in the webapp

View (Summary tab → "Optimize protection" per model): one slider (max acceptable quality loss, baseline and floor shown), live scatter filling in per tested setting (attack success TPR@1%FPR vs accuracy), finished state fades tested points and tags the best trade-offs "Safest"→"Best quality", side panel with the real hyperparameters, "Confirm this setting" runs a stronger verification attack, estimate vs verified side by side (amber badge if the estimate was >20% optimistic), "Adopt" writes an optimized row into the summary. No optimizer jargon anywhere; all copy in optimizationCopy.ts.
Backend: leakpro/optimization/adapters.py (arch file + tensor dataset → PETRecipe + splits; binary vs multiclass decided by probing the model head), pet_runner.py as a detached subprocess (survives backend restarts; JSONL is the resume state; kill = resumable pause; OOM auto-backoff halves the physical batch), five /jobs/{id}/pet/* endpoints placed before the SPA mount. ε=inf is serialized as null — Python's bare Infinity is rejected by every browser's JSON.parse.

Risk assessment made per model (on top of #445)

Declaring is per model because the population at risk follows the training split (f_train), not the dataset. Each summary row gets "Risk assessment" and "Optimize protection" buttons; submits go one model per request via RiskRequest.model_name; each row displays at its own declared α. Subjects left blank uses the model's exact num_train from metadata (provenance recorded); the wizard shows the derived count as placeholder. Known deferred gap (commented in code): two same-data models can be declared with different sensitivities; copy-from-model is the planned fix.

Fixes along the way

Records tab images for models loaded from previous sessions (sample endpoints now register the job's dataset_handler before unpickling).
Cross-session optimization targets the model's own job under its original name.
A failed optimization run is retryable instead of a dead end.

Verification

142 tests pass (optimization + risk suites), ruff clean, tsc 0 errors, vite build clean, and a 20-check end-to-end run of the PET flow (start → poll → verify → adopt → rejection paths) against a live TestClient with a BatchNorm model, exercising the ModuleValidator path in situ.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WotNsAFnfJsCerAiyoXLNM

Shared Opacus-compat helper

The webapp's DP-SGD loop carried a near-verbatim copy of the BatchNorm fix, in-place-activation pass and residual-forward patch that now live in leakpro.optimization.make_opacus_compatible (#447). It is removed here in favour of the shared helper, so a model trained through the wizard and the same model trained by a PET campaign are the same architecture — and the subclass-safety fix from #447's review covers both paths. This was the one review finding on #447 whose fix belongs in this PR.

Verification

155 tests pass (optimization + risk suites).
ruff check . clean; tsc --noEmit 0 errors; vite build clean.
Full 20-check end-to-end run of the PET flow against a live TestClient (start → poll → verify → adopt, plus the 404/409 rejection paths), using a BatchNorm fixture model so the ModuleValidator path is exercised in situ.

LeakPro measured attack success but had no defensible way to state risk. The webapp scored it as hardcoded AUC bands with no provenance, and AUC is an average-case measure that hides the worst-case per-record leakage which actually determines exposure. The core library had no risk concept at all, so CLI and notebook users got nothing. This adds leakpro/risk/, which reports a decomposition rather than a single opaque score. Measured quantities come from the audit and are reproducible; harm parameters are declared by the data controller and echoed verbatim in the output; combined figures keep every factor traceable to one of the two. Structure follows Sion et al. (IWPE 2019), LM = DTS x NR x DST x NDS, whose Loss Magnitude factors the paper deliberately leaves for the analyst to set — so they are user-supplied numbers defaulting to a neutral 1.0 and there is no combined risk band at all, only an advisory vulnerability band over measured lift, marked heuristic. Precision follows Jayaraman et al. (PoPETs 2021, Thm 4.2): balanced audits imply a prior of 0.5, and the same measurement can mean a very different threat at a realistic prior, so both are reported. assess_risk is a separate call rather than part of run_audit, so one expensive audit can be re-assessed cheaply under different operating points and priors, and run_audit keeps its signature. Four guards, each covering a way a naive version reports a confidently wrong number: an operating point the audit set cannot resolve is refused instead of reading as TPR 0; results with AUC < 0.5 are rejected by name rather than averaged in; the strongest attack is selected by TPR at the operating point rather than by AUC; and a percentage-valued fixed-FPR table is rejected rather than inflating a risk figure a hundredfold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

riskLevel() scored risk from hardcoded AUC thresholds (>=0.75 HIGH, >=0.60 MEDIUM) with no provenance, in TypeScript where it could not be tested. All scoring now happens in leakpro.risk behind POST /jobs/{id}/risk; the frontend posts a declared use-case profile and renders what comes back. The first view shows numbers, not prose: explanations live behind a reusable InfoButton, and RiskDiagram is an inline themed SVG of the model, encoding role by border style (given vs computed) and provenance by colour (measured vs declared). Warnings deliberately stay visible as a caveat chip — a caveat like "this operating point was not measurable" must not need a click to discover. Risk state lives in Step7Results rather than Summary because the tab bar unmounts the tab components, which discarded a completed assessment as soon as the user looked at the ROC curves. Assessments are keyed by job and model since the endpoint is per job and model names are not unique across compared jobs; previously only the first job's models were ever assessed. The worker forwards num_train, which the risk layer needs to scale exposure to the population and which saved results cannot recover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Adds a commented use_case: block to audit.yaml documenting every harm factor, and two notebook cells that call assess_risk on the audit results and save the JSON. Declaring the block changes nothing on its own: assessment is always an explicit call, so an existing config keeps working untouched. Sensitivity is left neutral here because CIFAR-10 is public benchmark data, which is also the honest default to show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The first pass showed the whole model at once: nine boxes in the schematic and info modals running four paragraphs. Correct, but it read as documentation and buried the parts a user has to act on. The schematic is now three boxes — the measured branch, the declared branch, and their product — with factor names spelled out instead of abbreviated, since the reader is meeting them for the first time. The input nodes, the role tags, the legend and the centre divider are gone; the column headings already carry the measured-versus-declared point that matters. Info modals now open with a sentence or two and put the depth behind a nested Detail disclosure: 174 words visible across all nine, down from about 1450, with the jargon (gamma, retention period, threat event frequency, the CNIL scale) one click further in rather than deleted. Deliberately unchanged: the caveat chip and the assumption list, which come from the library and are the auditable record rather than UI copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

review-notebook-app · 2026-08-18T12:03:15Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Implements the plan's structured recipe contract in the core module: - PETRecipe: make_model/make_optimizer/make_loader factories + criterion + epochs. Every factory receives the sampled knob config and picks what it needs, so further PETs (regularization bundle) need no interface change. - train_with_dpsgd: the single Opacus path (PrivacyEngine, BatchMemoryManager physical-batch cap, epsilon accounting into campaign_extras), previously duplicated per example. noise=0 is the non-private anchor (epsilon = inf). - confidence_logits: membership signal generalized to multiclass logits and binary sigmoid outputs. - build_campaign_fns: recipe + splits -> the Campaign's three callables, with the matched-reference attack (full mimicry) built in. The raw callable API remains the escape hatch for exotic loops. 10 new tests (CPU, tiny models), including accounting invariance under the physical-batch cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fy6fvLfZtfrSLBp5u8rMo

Both examples now only declare their recipe (model, optimizer, loader, loss) and data splits; the DP-SGD loop, matched-reference attack and utility evaluation come from the shared core path. The duplicated Opacus wiring (including the OOM fix) is gone from example code. validate_frontier.py revalidates through the same shared path. Smoke-verified both: same configs produce the same formal epsilon and consistent utility/TPR as before the port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fy6fvLfZtfrSLBp5u8rMo

GRU-D is the leakier LOS target versus the flat LR one: a recurrent net over the raw multivariate time series. Added as a PETRecipe, so the example only declares model/optimizer/loader/loss. Two GRU-D specifics handled: - raw-logit output (BCEWithLogitsLoss) -> new 'binary_logits' output_kind in confidence_logits (signed logit as the membership signal), with a unit test. - custom FilterLinear + manual recurrence -> Opacus on its functorch per-sample-gradient path (force_functorch=True), bn_flag off. Smoke test trains end to end under DP-SGD (eps computed, AUC ~0.66). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fy6fvLfZtfrSLBp5u8rMo

… accounting Two prerequisites for the webapp integration, both about the shared DP-SGD path being usable outside the three hand-picked example architectures. Opacus compatibility: train_with_dpsgd called make_private() on the model as submitted, so anything containing BatchNorm failed outright. The examples dodged this by construction (GroupNorm CNN, logistic regression, GRU-D with bn_flag=False), but both of the webapp's image presets are torchvision ResNet-18, so every image user would have hit it on the first config. Adds make_opacus_compatible(): ModuleValidator.fix() for BatchNorm, disabling in-place activations, and rewriting torchvision residual blocks whose `out += identity` ModuleValidator cannot see because it lives in forward rather than a submodule. Ported from the equivalent block in the webapp's own training loop, which the recipe refactor will replace. Only applied on the private path -- with no PrivacyEngine there is no reason to rewrite the user's model. Accountant: the accountant was hard-coded to "rdp" while the webapp uses "prv". RDP is the looser bound, so the same noise multiplier reported a larger epsilon via the campaign than via the webapp, and users comparing the two would have drawn a false conclusion. Now a parameter, defaulting to "prv" to agree with the webapp. delta and max_physical_batch are threaded through build_campaign_fns for the same reason; reference models share the target's settings so mimicry stays exact. Note this changes reported epsilon for existing campaigns: PRV is tighter, so epsilon goes down for identical noise. The empirical attack numbers are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

…ull mimicry Findings from an adversarial review of this PR. The theme is silent behaviour changes versus the code this PR replaced. train_with_dpsgd now fails closed: a missing or misspelled "noise_multiplier" raised nothing and trained a fully non-private model from a function whose only purpose is DP-SGD. The key is required; non-private runs pass an explicit 0. make_opacus_compatible runs on both branches, not just the private one. Rewriting BatchNorm to GroupNorm only for private configs left the noise=0 anchor a structurally different model, so its utility gap mixed the cost of DP noise with the cost of an architecture change — the frontier's utility axis was not comparable end to end. Documented that the submitted architecture is therefore not necessarily what trains. Full mimicry is enforced instead of silently downgraded: n_ref_train used min(target, ref_pool), so a short reference pool quietly trained weaker references and biased the audit optimistic. A short pool is now an error, which is what the old per-example code did loudly. _patch_residual_blocks only rewrites blocks that still use the stock forward. Subclasses with their own forward (SE, attention) were silently replaced by the vanilla residual path — changing the model rather than making it Opacus-safe. Such blocks are now left alone with a warning. The "auc" utility metric refuses multiclass output instead of reshape(-1)-ing it into a meaningless number. The accountant travels with the epsilon in campaign_extras. The prv default is deliberate (it matches the webapp's DP-SGD path), but PRV and RDP epsilons are not comparable — measured 4.22 vs 4.87 for identical noise here — so a record that does not name its accountant cannot safely be compared with another run's. Examples: GRU-D regains the max_physical_batch=128 cap the pre-PR code set deliberately (it is the memory-heaviest target in the repo, and the shared 256 default would OOM mid-campaign), and gains the small-dataset split guard its LR sibling already had. The LOS campaign imports target_models.LR instead of redeclaring an identical LRSigmoid, so the campaign trains the same class the audited target used. One earlier test asserted the now-incorrect behaviour (BatchNorm surviving the non-private path) and has been inverted to assert architecture consistency. 47 tests pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

Adds "Find the best protection" to the Results summary tab, one button per audited model, placed directly after the risk-card block. It opens a view that shows the utility-vs-attack trade-off as it is measured and lets the user pick an operating point. Flow: a single constraint control ("maximum acceptable quality loss", relative to the accuracy already shown for that model, with the resulting quality floor spelled out beside it) -> a scatter that fills in as settings are tested, with a live count -> the finished view, where everything tested fades to grey and the best trade-offs are highlighted, connected, and tagged in plain language from "Safest" to "Best quality". Clicking a highlighted point opens a panel with the actual hyperparameters, then "Confirm this setting" runs the full audit and shows estimate against verified side by side. A verified attack success more than 20% worse than the estimate is badged amber. "Adopt this configuration" writes the result back as a summary row tagged "optimized". Notes on choices that were not in the spec: - Attack success is reported at a 1% false-alarm rate everywhere, including verification. The search measures TPR at the proxy FPR, so a live axis labelled 0.1% would be showing a number the run never computed. - The quality constraint filters the display only; it does not gate the run. The slider stays adjustable after the fact instead of invalidating results. - The run returns as many non-dominated settings as it finds; the view keeps both ends of the trade-off and spreads at most five labels between them. - Copy lives in optimizationCopy.ts so the vocabulary stays reviewable in one place. No search-method jargon is exposed anywhere in the UI. Backend endpoints under /jobs/{id}/pet/ are consumed but not implemented here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The first fetch for a model that has never been optimized has nothing to return. Surfacing that as a red banner made the normal starting state look broken. Polling during a run stays silent for the same reason: one dropped request should not replace a live chart with an error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

Implements the endpoints the optimization view was written against. Three pieces: an adapter from an architecture file to a campaign, a detached runner, and five thin routes. leakpro/optimization/adapters.py turns "a .py defining an nn.Module, a tensor dataset, and the settings a model was trained with" into the PETRecipe and the five disjoint index sets build_campaign_fns needs. It deliberately does not go through AbstractInputHandler: a user handler exists to supply a custom train(), which is exactly what the shared DP-SGD loop replaces. Whether the head is binary is settled by instantiating the model and looking at its output, not by counting classes -- a two-class problem may have one sigmoid output or two softmax ones, and the loss, label dtype and attack all follow from which. pet_runner.py runs as a subprocess, never on the API's thread pool. A campaign is n_configs x (1 + n_refs) full trainings and the API demotes any job still marked running to failed on restart, so in-process was never viable. Campaign already appends one JSON line per setting and resumes from it, which makes killing the process a resumable pause rather than a lost run. It reuses the audit worker's dataset_handler registration, without which real uploaded datasets cannot be unpickled. Endpoints live in main.py before the SPA mount so the catch-all cannot shadow them: start (409 if one is already running), campaign (reads status + the JSONL off disk), verify as POST/GET pair, and adopt, which refuses anything not yet verified. Two things worth calling out: - json_safe() exists because Python writes a bare Infinity for float("inf"), which every browser's JSON.parse rejects outright. The non-private anchor records exactly that as its epsilon, so without it the frontend could not read the payload at all. - resolution_warning is computed once and carried through status.json into the view, which renders it in amber. A frontier drawn from a handful of events looks identical to a real one, so it must not be shown silently. carve_splits errors when a role would come out empty; "non-empty but too small to resolve an FPR" stays a warning, since that is a judgement call rather than a structural failure. Verified end to end against a fabricated job: start, detached campaign, poll, verify, adopt, optimized row in /results, plus the rejection paths. The fixture model carries BatchNorm, so the ModuleValidator fix from 8a54ff6a is exercised through the whole stack. 56 unit tests pass, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The optimization view was written on a branch that predated main's teal retint (PR #423), so it hard-coded a blue accent and slate dark surfaces. After rebasing the stack onto current main, retint to the shared tokens: amber primary accent on the chart, the app's bg-slate-700/border-primary primary-button convention, and surface/surface-2/surface-border for dark mode. No behaviour change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

…verywhere Two things surfaced while running a real campaign. Out-of-memory: train_with_dpsgd capped the physical batch at 256. Under DP-SGD that cap sets peak memory (Opacus holds per-sample gradients, ~batch x params), which is fine for the LOS logistic regression but OOMs a ResNet-scale image model on a 24 GB card. The runner now defaults the cap to 32 and exposes it as PETStartParams.max_physical_batch; it changes speed only, never the accounting. The runner subprocess is also launched with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, since fragmentation across back-to-back trainings triggers a spurious OOM well before the card is full. Model quality is now accuracy for every model, matching the baseline shown beside the slider. It was AUC for binary heads and accuracy for multiclass, so on a binary target (LOS) the y-axis and the "current quality" number sat on different scales and the quality-floor line was wrong. Fixing that exposed a latent bug: the accuracy metric used argmax, which on a single-logit binary head is always column 0 -- silently 0% or 100%. It now thresholds the logit at 0 for single-column outputs. Regression test included (fails on the old argmax path). The training.py hunk logically belongs on feature/pet-recipe; it sits here because that is the branch in play. Move it down when the recipe PR opens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

"Find the best protection" on a model loaded from a previous session failed with "No model named 'model_1' in this job": the view called the PET endpoints with the current session's job id, but a loaded model's dataset, arch.py and state entry live in the job it came from. Every ModelResult already carries job_id (its originating job), so the view now targets that. Also fixes the collision case behind it: when the compare view renames a loaded model (model_1 -> model_1_v2) to avoid a display clash, the backend still knows it only as model_1. The original name is now preserved as orig_model_name and used for the campaign; the display name is untouched. Both the current-session and loaded-model paths now route correctly, since a current model's job_id is simply its own job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The earlier OOM fix (max_physical_batch=32, expandable_segments) lived in main.py, so it only took effect once the FastAPI process was restarted -- a stale --reload backend kept training at batch 256 and kept OOMing with the same numbers. Two changes make it robust: - pet_runner sets PYTORCH_CUDA_ALLOC_CONF itself, before torch touches CUDA. The runner is a fresh process on every launch, so this cannot be defeated by an un-reloaded backend. - On a CUDA out-of-memory, the run halves max_physical_batch and retries down to a floor of 4. The batch cap is a memory measure only -- it never changes a result -- and the campaign resumes from its JSONL, so only the config that ran out of memory is re-run, not the whole sweep. Verification gets the same backoff. Full e2e still green (CPU path; the wrappers are exercised, the backoff is a no-op without CUDA). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The repeated "CUDA out of memory" the user kept seeing was not a new crash: it was the ORIGINAL failure's status.json being read back on every visit -- same dead pid, byte-identical numbers, GPU actually empty. Because the view treated status=failed as a terminal phase, it hid the Start button, so the only thing the page could ever do was replay the fossil. Failed now keeps the constraint panel live and shows "Try again" (the backend already permits starting over a failed run; it only rejects a running one). The error banner separates the plain-language line from the raw detail and says that finished settings are kept -- resume comes free from the JSONL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The webapp's DP-SGD loop carried a near-verbatim copy of the BatchNorm fix, in-place-activation pass and residual-forward patch that now live in leakpro.optimization.make_opacus_compatible. Sharing one implementation means a model trained through the wizard and the same model trained by a PET campaign are the same architecture, and the subclass-safety fix from the #447 review applies to both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

tpr_at_fpr now returns a TPRAtFPR named tuple rather than a 3-tuple (#438 review), so the runner's unpack was broken — verification would have crashed. Confirmed against the real function before fixing. While updating it, the verified result also carries realized_fpr and the resolution warning, and the panel renders that warning: a low attack number can mean "well protected" or "we could not measure it", and those must not look identical to a non-technical reader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

adapters.py had four things that already lived in leakpro/input_handler/user_imports.py — the file whose entire purpose is loading user-supplied models, which I never opened when writing it. - Module loading now uses import_module_from_file. Theirs is better than the copy it replaces: it caches by module name and restores sys.modules on failure, where mine created a fresh uuid-named entry per call and leaked one every time. - Named class lookup delegates to get_class_from_module. Only the "exactly one candidate, infer it" case stays local, which the webapp needs because an uploaded arch.py usually defines a single model and the user is never asked to name it. - The optimizer lookup uses get_optimizer_mapping() instead of a hardcoded three-entry dict, so every torch.optim optimizer works rather than adam/sgd/ adamw only. Criterion selection is deliberately NOT routed through get_criterion_mapping: that maps a user-supplied name string to a class, while this picks a loss from the model's head shape. Different jobs; forcing them together would be worse. Also fixes a regression I introduced while resolving a rebase conflict: taking --ours for training.py dropped binary_logits from PETRecipe's valid output_kinds, which GRU-D needs. Restored, with the docstring listing all three. Tests updated to the repo helpers' real contracts: get_class_from_module raises ValueError not KeyError, and rmsprop is now a valid optimizer, so the rejection test uses a name torch genuinely lacks. 71 tests pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

Recreated after rebasing onto the de-duplicated pet-recipe. Same conflict resolutions as the previous merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

Agreed with the risk module's author: declaring is per model, not per job, because the population at risk follows the training split (f_train), not the dataset -- two models on the same data with different splits expose different numbers of subjects. The global "Declare your use case" bar and the "Reduce the risk" block are replaced by two actions on every summary row: declare that model's use case (opens the wizard bound to the model) and find its best protection. Submits go one model per request via RiskRequest.model_name, which the backend already supported. Each row shows figures at its own declared alpha; the TPR column header is now neutral ("TPR@a") since rows can differ. Metadata as defaults: the subjects field left blank means the backend uses the model's exact num_train from its metadata (n_subjects_source records the provenance); the wizard now shows that model's estimated count as the placeholder so the default is visible instead of silent. Only the count is derivable -- sensitivity and records-per-subject have no metadata source and keep neutral defaults. New declarations are seeded from the last submitted profile, so judgement fields carry over across models of one comparison. PARKED (known gap, by decision): nothing stops two models on identical data being declared with different sensitivities; a copy-from-model affordance is the fix, deferred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

Actions column now carries two labeled buttons per row -- "Risk assessment" and "Optimize protection" -- instead of bare icons; the assessment button reads as active once that model is declared. A protection bar mirrors the risk bar: title, info button, one line. The info content explains the method in plain language (retrain many versions, attack each the same way as the audit, quality vs attack success, verify before adopt, epsilon recorded), with depth behind a Detail disclosure and no search jargon, matching the risk feature's presentation rule. The button label changes from "Find the best protection" to "Optimize protection" per PI request; the optimization view's own title still says "Find the best protection" -- flagged as a naming mismatch to resolve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

The sample_data/sample_image endpoints did a bare joblib.load, but dataset pickles reference the module they were written from (celebA_data_handler and friends). The audit worker registers the job's dataset_handler.py under that name as a side effect of running an audit -- so images worked for freshly-audited models and 500ed with "No module named ..." for models loaded from previous sessions on a backend that had not audited that job yet. Both endpoints now register the job's dataset_handler.py (same original-name guessing as the worker and the PET runner) and put the job dir on sys.path before unpickling. sample_image also returns a named 404 instead of a bare 500 when the dataset cannot load, so the failure is readable from the img request. Verified against the real celebA job that reproduced the bug: both endpoints 200, image bytes served. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6zdgpRhLbscVhC99K3UhY

Three copies of the same trick — register a job's uploaded dataset_handler.py under the module name its pickles were written with, or joblib fails with "No module named ...". The audit worker had the original; I added two more, to pet_runner.py and to the sample-data/sample-image endpoints. The endpoints' copy only existed because images broke for models loaded from an earlier session, which is precisely what drift between copies looks like from the outside. Now one function, leakpro/webapp/backend/dataset_modules.py, used by all three. It builds on user_imports.import_module_from_file rather than hand-rolling the spec loading a fourth time, and it also puts the job directory on sys.path, which only the endpoints' copy did — so the worker and the runner gain that. Verified against the real celebA job that produced the original bug report: sample_data 200 and sample_image returns the same 50366-byte PNG as before the refactor. 156 tests pass, ruff clean, tsc clean, full PET end-to-end green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QBPG7MprEqhqG95fFURk8h

TheColdIce and others added 4 commits August 17, 2026 15:17

fazelehh force-pushed the feature/pet-recipe branch from 172f691 to 692a9bd Compare August 19, 2026 06:14

fazelehh force-pushed the feature/pet-webapp branch 2 times, most recently from 49ec723 to a3fcfba Compare August 19, 2026 11:11

fazelehh mentioned this pull request Aug 19, 2026

PETRecipe: one shared DP-SGD training path #447

Open

fazelehh force-pushed the feature/pet-recipe branch from 18db5a2 to 6d20b70 Compare August 20, 2026 07:20

fazelehh force-pushed the feature/pet-webapp branch from a3fcfba to 7a4e0fa Compare August 20, 2026 07:20

fazelehh and others added 19 commits August 20, 2026 07:44

fazelehh and others added 2 commits August 20, 2026 07:54

fazelehh force-pushed the feature/pet-recipe branch from 6d20b70 to c72393a Compare August 20, 2026 07:58

fazelehh force-pushed the feature/pet-webapp branch from 7a4e0fa to 5dc859c Compare August 20, 2026 07:58

fazelehh force-pushed the feature/pet-recipe branch from c72393a to aba8a5f Compare September 21, 2026 12:33

Navigation Menu

Webapp: PET optimization view + per-model risk assessment - #448

Webapp: PET optimization view + per-model risk assessment#448
fazelehh wants to merge 25 commits into
feature/pet-recipefrom
feature/pet-webapp

fazelehh commented Aug 18, 2026 •

edited

Loading

Uh oh!

review-notebook-app Bot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fazelehh commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PET optimization in the webapp

Risk assessment made per model (on top of #445)

Fixes along the way

Verification

Shared Opacus-compat helper

Verification

Uh oh!

review-notebook-app Bot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fazelehh commented Aug 18, 2026 •

edited

Loading