Noise modelling fixes and data updates - #1
Merged
Merged
Conversation
…ner flexibility and less piping changes for pools requiring reserve inputs
…rades is the wrong shape and optional. Centralised in the new materialized function that is used by all dynamic input reserve calcs
…atio changes etc for gradient based optimisation methods
The merge of dev into reclamm-phase-1 reintroduced a reference to the old fees_array/gas_cost_array/arb_fees_array/trades_array parameters in the fused reserves guard. These were replaced by the DynamicInputArrays container in the dynamic inputs refactor. Replace the stale check with `dynamic_inputs is None`, which is the correct guard under the new API.
…phase-1 Reclamm phase 1
…pdates Replace the ad-hoc "keep overvalued, solve undervalued" virtual balance recalculation with the closed-form quadratic from ReClammMath.sol computeVirtualBalancesUpdatingPriceRatio. The old code silently drove centeredness to 1.0 for off-center pools. Add parametrized test mirroring the Foundry fuzz test testCalculateVirtualBalancesUpdatingPriceRatio__Fuzz, asserting that centeredness is preserved and the target price ratio is achieved.
Tests that construct run_fingerprint dicts directly (bypassing recursive_default_set) need the ste_temperature key now that the STE-enabled scan steps read it from the fingerprint.
…tio-change Price ratio change
The fees_gate STE was unnecessary — gamma (1 - fees) is either a static config value or a learnable param constrained to be nonzero, so the fee/zero-fee branch selection never benefits from soft gradients.
…ly forwarding - base_pool: add calculate_volatility_array and _handle_numeraire_ordering - optimal_n_pool_arb: use relative invariant check instead of absolute slack - balancer/balancer_reserves: forward lp_supply through scan functions - balancer: use materialized_inputs.lp_supply in dynamic_inputs path - TFMM_base_pool: use materialized_inputs.lp_supply, fix trade_array ref - core_simulator/__init__: enable JAX compilation cache
Three noise trade volume models for reCLAMM pools, each predicting non-arbitrage trading volume from pool TVL, fee tier, volatility, and arb volume. Used by the noise model dispatch in scan steps.
… steps Scan step carry now includes prev_lp_supply (pos 3) and price-ratio schedule state (pos 4-9). Scan inputs include price_ratio_update (7), lp_supply (8), and optional volatility (9). - reclamm_reserves: LP supply scaling, noise model dispatch (ratio, tsoukalas_sqrt, tsoukalas_log, loglinear, arb_only) in scan steps - reclamm: _prepare_dynamic_array for volatility, noise/LP kwargs threading through all pool method variants - tests: LP supply and noise volume coverage
Hierarchical Bayesian noise model calibration using numpyro: data pipeline, covariate encoding, model definitions, inference, BLUP postprocessing, diagnostic plotting, and CLI. Adds [calibration] optional deps (numpyro, arviz) to pyproject.toml and updates CI to install them.
…lot script run_period_sweep.sh: - New OVERFITTING_PENALTIES axis (defaults: ["" "1.0" "5.0"]); each value is passed as --overfitting-penalty to the tune script. - New ROBUST_TEMPS axis (defaults: ["" "0.5" "1.0" "0.1" "0.01"]); each value is passed as --robust-temperature. - New long_2021 period (2021-06-01 → 2025-01-01 train+val, → 2026-03-01 test) with optional --val-fraction parameter to give it a 40% val holdout matching its longer history. - Tags now combine objective + robust + penalty + period; existing result files are skipped on rerun. - Single-array overrides retained at the bottom of each axis block as runtime knobs. plot_sweep_results.py: - Refactor from single-pool BASE_FP/PERIODS into POOL_CONFIGS dict keyed by pool (aave_eth, cow_eth_mainnet, cow_eth_base) so the same script can plot sweep results for multiple pools. - Filenames now have suffix _mainnet/_base where needed. - Filename parsing in load_sweep_results recognises robust and penalty suffixes via _parse_variant; ALLOWED_ROBUST and ALLOWED_PENALTY constants control which combinations are loaded (defaults match the current run_period_sweep.sh active axes). - _short_label translates _robust0.5/_penalty5.0 to " r0.5"/" p5.0" for compact legend entries. - New --top N CLI flag ranks configs by test-period RoH and plots only the top N (plus on-chain baselines), saved with _topN suffix. - long_2021 added to aave_eth periods list.
Four scripts that together define the soup-to-nuts pipeline for a full reClAMM parameter sweep + selection + forward-simulation cycle. scripts/run_full_sweep.sh Bash entry point for the parameter sweep. Iterates over a CONFIGS list (pool × TVL × gas × fees), objectives, and overfitting penalties, with optional --method=optuna|cma_es and --pair=<prefix> filters. Throttles to MAX_PARALLEL workers; CMA-ES defaults to 4, Optuna to 8 (or the MAX_WORKERS env override). PR_MAX env var caps the price-ratio search via --pr-max for Optuna runs. Existing result files are skipped so reruns are incremental. scripts/evaluate_trials.py General-purpose loader and analyser for the result hash files written to results/. Walks the trial JSONs, deserialises params, and exposes load_reclamm_results which returns a list of dicts with study_id, tokens, initial_pool_value, params, train/val metrics, return_val (objective), and timestamps. Used by select_best_params and run_final_sims to discover candidate trials without re-running. scripts/select_best_params.py Picks the winning param set per (tokens, TVL). Ranks candidates by validation RoH (best_val_roh) and prints the chosen trial along with PR/margin/shift values, train/val metrics, and the source study_id. --all summarises every TVL config in one go. scripts/run_final_sims.py Forward-simulation runner. For each pair × TVL, loads selected params via select_best_params, builds noise arrays for both train and test periods (calling build_mm_simulator_arrays when not cached), runs do_run_on_historic_data, and dispatches to plot_reclamm_optuna_result for value / fee_revenue / volume / weight panels. Normalises all output by initial TVL so different TVL levels are directly comparable on a single plot, and records the initial_tvl per config so the volume panel can express cumulative volume as a multiple of TVL.
… keys _prepare_noise_arrays now returns a dict instead of a (vol, dow_sin, dow_cos) tuple; ReClammPool unpacks it via .get(...) to forward whatever keys the chosen noise model needs: - "ratio" / legacy paths: volatility, dow_sin, dow_cos - "market_linear": adds noise_base, noise_tvl_coeff - "mm_observed": adds noise_base, competitor_tvl This matches the dict-based API used downstream by _jax_calc_reclamm_reserves_with_dynamic_inputs and avoids forcing the caller to know which scalar arrays the active noise model expects.
prepare_dynamic_inputs builds lp_supply_array at minute resolution from the LP supply timeseries, but the pool scan loop iterates at arb_frequency-minute steps. materialize_dynamic_inputs then expects arrays whose length matches the scan_len, so a minute-resolution lp_supply_array silently mismatches for arb_frequency > 1. Fix by stride-subsampling lp_supply_array (and its test-period counterpart) by arb_frequency before they leave prepare_dynamic_inputs. No effect when arb_frequency == 1.
Previously only lp_supply_array was subsampled. fees_array, gas_cost_array, and arb_fees_array share the same shape contract — materialize_dynamic_inputs requires scan_len = (bout_length - 1) // arb_frequency, so any non-None minute-resolution array would have raised in _broadcast_dynamic_input_leaf once a caller populated those DataFrames with arb_frequency > 1. The bug never tripped because current callers use scalar fees/gas/arb_fees. Apply the same [::arb_freq] slice to all four (train + test) so the path is consistent and future callers can pass per-minute series safely.
Extend the noise-model dispatch already used in reCLAMM to BalancerPool's two reserve solvers (with_fees_using_precalcs and with_dynamic_inputs). Supports ratio, tsoukalas_sqrt / tsoukalas_log / loglinear, calibrated, market_linear, and mm_observed — the same set the reCLAMM path handles. Noise arrays are appended to the scan inputs at the same positions as in reclamm_reserves.py so the dispatch logic mirrors line-for-line, and noise volume is converted to noise-fee income (minus protocol_fee_split) and rebated to LPs via a uniform reserves scale, matching reCLAMM's treatment. _prepare_noise_arrays lives on BalancerPool but is a verbatim port of the reCLAMM helper — the noise model describes market-level organic volume, so the same arrays apply regardless of pool mechanics.
…msim into noise-modelling
Brings 65 commits of noise-modelling work onto noise-plus-plus, including: - Per-pool linear noise model (market_linear) and Michaelis-Menten with observed competitor TVL (mm_observed) - Noise-only lp_fee_revenue_usd reporting so optimisers can't monetise pool decay through sloshing arb volume - In-range gate on noise volume; protocol fee split applied to noise fees; blessed-arb experiment knob - Balancer noise model support - Subsampling of all minute-resolution dynamic input arrays (fees/gas_cost/arb_fees/lp_supply) to match arb_frequency Conflicts resolved (4 files): - pyproject.toml, setup.py: drop gdown, keep Historic-Crypto casing - reclamm.py: take the _resolve_noise_inputs / STE structure from noise-plus-plus, re-add mm_observed plus competitor_tvl_array threading at the call sites - reclamm_reserves.py: keep _BLESSED_ARB module flag and the noise_fee_income initializer; STE gates and noise-model dispatch are otherwise complementary Test failures pre-existing on noise-modelling and not introduced by this merge: 13 baseline/forward-pass tolerance failures (3 already fixed downstream on training-pipeline as 'update forward_pass_test_2 baseline after optimal_n_pool_arb relative invariant change'). Five lp_fee_revenue tests still fail post-merge because the test fixtures don't configure a noise model — addressed in a follow-up commit.
lp_fee_revenue_usd is noise-only by design (so optimisers can't monetise sloshing arb volume through a decaying pool). The five tests below were written when lp_fee_revenue_usd reported total inbound fees, and stopped passing once the semantic changed. Add a small _mm_observed_noise_kwargs helper that supplies constant noise_base and competitor_tvl arrays (the simplest noise model to wire — two scalars vs. tsoukalas' volatility + noise_params). With noise_base=13.8 and K=1e7, a $1M pool produces ~$0.20/step of noise fee income at 0.3% fees — positive and detectable without dominating the pool dynamics. Tests updated: - test_fee_revenue_positive_on_price_jump - test_higher_fees_more_revenue - test_protocol_split_reduces_lp_revenue - test_dynamic_inputs_fee_revenue - test_lp_supply_with_fee_revenue (uses K=1e10 so noise volume scales ~linearly with pool TVL, preserving the doubling check)
…us-plus Noise plus plus
…itting penalty Optuna's writer now persists train_objective, test_objective, and continuous_test_metrics as list[dict] with the full metric set, matching the shape save_multi_params produces for cma_es and bfgs. Rich metric dicts are captured in the optuna objective callback via set_user_attr. --overfitting-penalty is wired into cma_es_settings and applied inside eval_single using the same train-vs-val gap formula optuna uses, with val evaluation points drawn from the validation period. Penalty now factors into the run_fingerprint, so the 3 penalty variants of each (token, objective, TVL) produce distinct run_*.json files.
Move experiments/{fetch_competitor_tvl,run_mm_noise,tune_reclamm_calibrated_noise}.py
into scripts/ to live alongside run_full_sweep.sh and run_final_sims.py.
Update scripts/run_full_sweep.sh and scripts/run_period_sweep.sh to the new
paths.
Add RECLAMM_TRAINING.md at repo root — end-to-end guide covering data pull,
grid build, MM noise model, training sweep, selection and plot stages, plus
extending the calibration set to a new pair.
Carries the minimum data to run scripts/run_final_sims.py and
scripts/run_pr_sweep.py for AAVE/ETH and COW/ETH without re-running the
training sweep. Binance price parquets are not included; pull them with
scripts/download_data.py.
- results/mm_noise/{model.npz,meta.json}: frozen MM noise model artifact
- results/competitor_tvl/competitor_tvl.npz: DeFi Llama competitor TVL
- results/full_sweep/*_penalty1.0_*.json: per-config sweep summaries for the
6 winning (pair, TVL) configurations
- results/run_<hash>.json: trial trajectories for those 6 winners, read by
load_reclamm_results in run_final_sims.py
The PR-sweep entry point referenced by the training guide (RECLAMM_TRAINING.md) and the plot-reproduction runbook (RECLAMM_PLOTS.md, added in dd20627).
- Skip the BinanceDataDumper exchangeInfo ticker validation: the endpoint is
geo-switched to binance.us for US IPs, which hides pairs (e.g. RPLUSDT)
that exist in the binance.vision archive but aren't listed on Binance.US,
making download_data.py report 'no data' for those tokens.
- Convert the reindex grid to milliseconds explicitly in
forward_fill_ohlcv_data: int64 cast of a DatetimeIndex is unit-dependent
on pandas>=3, which collapsed every timestamp to the same value and made
update_historic_data fail its minute-grid check for every token.
- Use lowercase 'h' resample alias ('H' removed in pandas 3).
Verified end-to-end: update_historic_data('RPL') completes on pandas 2.3.3
(including with US geo-detection simulated) and pandas 3.0.3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix download_data.py for US users and pandas 3
cli.py uses PEP 604 unions (str | None) in signatures, which are evaluated at import time and raise TypeError on Python 3.9 (the CI version). This broke collection of all seven tests/noise files. Defer annotation evaluation with the __future__ import. Verified on a Python 3.9.x env with .[dev,calibration]: full-suite collection is clean (2198 tests) and tests/noise passes 176/176. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix noise_calibration import on Python 3.9
4c9eae7 changed the default protocol_fee_split from 0.0 to 0.25 to match reClAMM production configuration, but the stored forward-pass baselines (captured with split=0.0) were not re-captured, failing 12 baseline tests on any fingerprint that charges fees. Re-capture forward_pass_test_1/2 final values and returns with the production default; weights are unaffected. Also update test_excludes_training_fields: startDateString is now intentionally kept in the static dict (needed by the calibrated noise model, see _TRAINING_ONLY_FIELDS in jax_runner_utils). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update test baselines for protocol_fee_split=0.25 default
…nboarding skill - run_final_sims.py: PAIR_CONFIGS entries for btceth (WBTC/WETH pool 0xa6f548df93de92 from the MM artifact) and boldusdc (median-fallback noise); --pair choices derived from PAIR_CONFIGS; optional per-pair train/test window overrides (BOLD data starts 2025-07-09, CoinGecko free-tier limit) with the trial filter applied per pair; thread the --metric flag into select_best_params (was hardcoded). - scripts/prepare_bold_usdc_data.py: CoinGecko data prep for BOLD (liquity-bold-2) + USDC $1-peg rebuild over the union grid. - run_full_sweep.sh: btceth_5m config line. - Sweep artifacts: 50-trial returns_over_hodl runs for btceth_5m and boldusdc_1m (demo scale; production is 300 trials x 4 objectives). - .claude/skills/reclamm-pair-onboarding: step-by-step runbook for onboarding new pairs (data routes, pool-id lookup and median-fallback caveats, sweep command, registration, outputs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
onboard BTC/ETH + BOLD/USDC pairs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
mainup to date with the latest noise-modelling work, since external testers were told to run from this branch.Includes (see commit history for detail):
update_historic_dataKnown issue on this branch (fixed separately in a stacked PR):
scripts/reclamm/demo_run_reclamm.pyAAVE/ETH scenarios crash withKeyError: 'price_ratio'—_apply_active_noise_settingsbuilds a cfg withoutprice_ratio/centeredness_margin/daily_price_shift_exponent, whichnormalize_compare_run_cfg(added in 8331b7f) hard-indexes.🤖 Generated with Claude Code