Noise modelling fixes and data updates

.claude/skills/reclamm-pair-onboarding/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: reclamm-pair-onboarding
+description: Onboard a new token pair for reCLAMM simulations — fetch price data, run the Optuna parameter sweep, and produce final-sim CSVs/plots with candidate pool params. Use when asked to simulate a reCLAMM pool for a pair, gather candidate price_ratio/margin/shift params, or add a pair to run_final_sims.py.
+---
+
+# reCLAMM pair onboarding
+
+Pipeline: **price data → Optuna sweep → register pair → `run_final_sims.py`**.
+Worked examples on branch `btc-eth-pair-onboarding`: BTC/ETH (Binance data,
+real pool coefficients) and BOLD/USDC (CoinGecko data, median-fallback noise).
+
+**Never retrain the noise model.** The checked-in artifacts
+(`results/mm_noise/{model.npz,meta.json}`, `results/competitor_tvl/competitor_tvl.npz`)
+are fit across 38 pools and reused for every pair; per-pair arrays are built
+and cached automatically under `results/mm_noise/_sim_arrays/`.
+
+## Step 1 — price data (`quantammsim/data/<TOKEN>_USD.parquet`)
+
+`BTC_USD.parquet` is required for **every** pair (market features), plus one
+parquet per pool token (`TOKEN_MAP` in `quantammsim/calibration/market_features.py`
+maps WBTC→BTC, WETH→ETH, USDT→USDC, …).
+
+- **Binance-listed token**: `python scripts/download_data.py <TOKEN>`.
+- **Not on Binance**: copy `scripts/prepare_bold_usdc_data.py` (CoinGecko):
+  daily close+volume → minute grid by forward-fill, daily volume spread /1440
+  (so the daily resample recovers real volume), stables as flat $1.00 peg.
+  CoinGecko free tier = **last 365 days only** — this bounds the earliest
+  simulation start and usually forces a per-pair train window (Step 4).
+
+Check coverage: the parquet must span train start → test end.
+
+## Step 2 — pool id and the noise fallback
+
+Look the pair up in `results/mm_noise/meta.json` (`pool_ids` / `pool_tokens`).
+
+- **Pair present** (e.g. WBTC/WETH → `0xa6f548df93de92`): use that id; the
+  noise model gets real per-pool coefficients and competitor TVL.
+- **Pair absent**: any placeholder id works — the builder falls back to
+  median coefficients + K=$10M (`noise_model_arrays.py`). Median alpha is
+  tiny, so organic volume (and fee revenue) will be near zero. If realistic
+  fees matter, re-anchor: after `build_mm_simulator_arrays`, shift
+  `noise_base += log(V_max_daily) - mean(noise_base)` and optionally set a
+  constant competitor K — see `scripts/run_rpl_eth_sweep.py` (~lines 124-134)
+  for the pattern and the pair's real daily volume for V_max.
+
+## Step 3 — Optuna sweep
+
+Demo scale (~50 trials, minutes; the team's production scale is
+`scripts/run_full_sweep.sh`: 300 trials × 4 objectives × 3 penalties per TVL):
+
+```
+python scripts/tune_reclamm_calibrated_noise.py \
+  --noise-model mm_observed --artifact-dir results/mm_noise \
+  --tokens BTC ETH --pool-id 0xa6f548df93de92 \
+  --gas-cost 1.0 --fees 0.0025 --initial-pool-value 5000000 \
+  --objective returns_over_hodl --n-trials 50 --pr-max 5.0 \
+  --start-date "2025-01-01 00:00:00" --end-date "2025-10-05 00:00:00" \
+  --output results/full_sweep/returns_over_hodl_<pair>_<tvl>.json
+```
+
+- One sweep per (pair, TVL). Search space: price_ratio 1.01–200 (log),
+  margin 0.01–0.99, shift_exponent 1e-5–125 (log). Cap `--pr-max` sensibly:
+  ~5 for correlated majors, ~1.05 for stable/stable.
+- Dates: default train window is 2025-01-01 → 2025-10-05 (pre flash-crash);
+  keep it unless data starts later. `--end-test-date` sets the OOS span used
+  for validation metrics.
+- Side effect (the part `run_final_sims.py` actually reads): a trajectory
+  file `results/run_<hash>.json` written by `train_on_historic_data`.
+
+## Step 4 — register the pair and run final sims
+
+Add an entry to `PAIR_CONFIGS` in `scripts/run_final_sims.py` (tokens,
+pool_id, gas_cost, fees, the swept TVLs; `--pair` choices follow the dict).
+If the pair's data can't cover the default windows, add per-pair overrides:
+
+```python
+"train": ("2025-07-15 00:00:00", "2025-10-05 00:00:00"),   # optional
+"test":  ("2025-10-25 00:00:00", "2026-05-01 00:00:00"),   # optional
+```
+
+The sweep's train window **must match** the pair's train window — trials are
+filtered by it (`filter_trials_to_window`).
+
+```
+python scripts/run_final_sims.py --pair <name>    # or --all
+```
+
+Selection: best trial per TVL by OOS `returns_over_hodl` (override with
+`--metric`). Outputs in `results/final_sims/`:
+`run_{Value,Reserves,TokenValues}_<pair>_<tvl>_{train,test}_<hash>.csv`,
+themed plots `<pair>_{train,test}[_weights]_{light,dark}.png`, and
+`<pair>_sim_results.pkl`. The printed summary gives train/test RoH and fees.
+
+## Gotchas
+
+- `--method` on `run_final_sims.py` is currently a no-op.
+- First run per (pool, window) builds noise arrays (needs network-free local
+  parquets only); subsequent runs hit the `_sim_arrays` cache.
+- Stable pairs: expect PR near the 1.01 floor and near-zero fees under the
+  median-fallback noise — re-anchor (Step 2) before trusting fee numbers.
+- Sweeps and final sims are pure JAX forward passes — a laptop handles demo
+  scale; production sweeps want the parallel `run_full_sweep.sh` machinery.

.github/workflows/tests.yml

@@ -13,8 +13,6 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
-        with:
-          lfs: true
 
       - name: Set up Python 3.9
         uses: actions/setup-python@v5
@@ -25,7 +23,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -e ".[dev]"
+          pip install -e ".[dev,calibration]"
 
       - name: Run tests with coverage
         run: |

RECLAMM_PLOTS.md

@@ -0,0 +1,143 @@
+# Reproducing the AAVE/ETH and COW/ETH plots
+
+This branch (`noise-modelling-fixes-data`) carries the minimum data needed to
+produce the final train/test panels and PR-sweep heatmaps for both pairs
+without re-running the training sweep.
+
+## What's on the branch
+
+- `results/mm_noise/model.npz` + `meta.json` — the frozen MM noise model.
+- `results/competitor_tvl/competitor_tvl.npz` — DeFi Llama competitor TVL.
+- `results/full_sweep/<6 files>` — sweep summaries for the 6 winning configs.
+- `results/run_<6 hashes>.json` — trial trajectories for the same 6 winners
+  (read by `scripts/run_final_sims.py`).
+
+What's **not** included and must be pulled locally:
+
+- Binance minute parquets for `BTC`, `ETH`, `AAVE`, `COW`.
+
+## Prerequisites
+
+Conda env per the README:
+
+```
+conda activate qsim
+```
+
+## 1. Pull Binance price data
+
+```
+python scripts/download_data.py BTC ETH AAVE COW
+```
+
+Writes `quantammsim/data/<TOKEN>_USD.parquet` per token. `BTC` is required by
+the MM noise model's market features; `AAVE`/`COW`/`ETH` are the pair tokens
+the simulator reads prices from.
+
+## 2. Final-sims: train/test forward passes and selection
+
+```
+python scripts/run_final_sims.py --all
+```
+
+Loads the 6 committed `run_<hash>.json` files, picks the single candidate per
+`(pair, TVL)` tier, runs train and test forward passes at the picked params,
+and writes:
+
+- `results/final_sims/{aave,cow}_{train,test}.png` — share price, fee revenue,
+  cumulative volume per TVL tier.
+- `results/final_sims/{aave,cow}_{train,test}_weights.png` — effective weight
+  trajectories.
+- `results/final_sims/{aave,cow}_sim_results.pkl` — per-tier results cache.
+
+Per-tier picks for this branch's data (printed to stdout as `Params:` lines):
+
+| Tier      | PR      | margin | shift  |
+|-----------|---------|--------|--------|
+| AAVE 1m   | 1.068   | 0.0240 | 0.0225 |
+| AAVE 5m   | 1.296   | 0.0101 | 0.0675 |
+| AAVE 20m  | 1.348   | 0.0103 | 0.5411 |
+| COW 500k  | 99.665  | 0.8235 | 0.1418 |
+| COW 2m    | 54.549  | 0.8351 | 0.1422 |
+| COW 20m   | 172.463 | 0.9265 | 0.0521 |
+
+## 3. PR-sweep heatmaps
+
+One `run_pr_sweep.py` invocation per (pair, TVL), with that tier's
+`margin`, `shift`, `initial-pool-value`, and the selected PR as a marker.
+
+### AAVE/ETH
+
+Override the pair-specific flags (`run_pr_sweep.py` defaults are COW).
+
+```
+mkdir -p results/final_sims/for_fabio/aave/price_ratio_sweep
+
+python scripts/run_pr_sweep.py --tokens AAVE ETH --pool-id 0x9d1fcf346ea1b0 \
+    --gas-cost 1.0 --fees 0.0025 \
+    --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/aave/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 4.0 5.0 7.5 10.0 \
+    --margin 0.0240 --shift 0.0225 --initial-pool-value 1000000  --selected-pr 1.068
+
+python scripts/run_pr_sweep.py --tokens AAVE ETH --pool-id 0x9d1fcf346ea1b0 \
+    --gas-cost 1.0 --fees 0.0025 \
+    --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/aave/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 4.0 5.0 7.5 10.0 \
+    --margin 0.0101 --shift 0.0675 --initial-pool-value 5000000  --selected-pr 1.296
+
+python scripts/run_pr_sweep.py --tokens AAVE ETH --pool-id 0x9d1fcf346ea1b0 \
+    --gas-cost 1.0 --fees 0.0025 \
+    --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/aave/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 4.0 5.0 7.5 10.0 \
+    --margin 0.0103 --shift 0.5411 --initial-pool-value 20000000 --selected-pr 1.348
+```
+
+### COW/ETH
+
+The COW selections are at PR ≈ 50–200, so the `--prs` list extends past 10.
+The script's defaults already match the COW pair, so the pair-specific flags
+can be omitted.
+
+```
+mkdir -p results/final_sims/for_fabio/cow/price_ratio_sweep
+
+python scripts/run_pr_sweep.py --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/cow/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 5.0 10.0 30.0 50.0 80.0 100.0 120.0 150.0 200.0 \
+    --margin 0.8235 --shift 0.1418 --initial-pool-value 500000   --selected-pr 99.665
+
+python scripts/run_pr_sweep.py --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/cow/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 5.0 10.0 30.0 50.0 80.0 100.0 120.0 150.0 200.0 \
+    --margin 0.8351 --shift 0.1422 --initial-pool-value 2000000  --selected-pr 54.549
+
+python scripts/run_pr_sweep.py --multi-period --period-months 3 --onchain-pr 2.02 \
+    --output-dir results/final_sims/for_fabio/cow/price_ratio_sweep \
+    --prs 1.01 1.1 1.2 1.4 1.6 1.8 2.0 2.5 3.0 5.0 10.0 30.0 50.0 80.0 100.0 120.0 150.0 200.0 \
+    --margin 0.9265 --shift 0.0521 --initial-pool-value 20000000 --selected-pr 172.463
+```
+
+### Outputs
+
+Per-period single-config PNGs and one heatmap per invocation, named
+`pr_heatmap_<TOKENS>_3mo_m<margin>_s<shift>.png`, under
+`results/final_sims/for_fabio/{aave,cow}/price_ratio_sweep/`.
+
+## Gotchas
+
+- **zsh and shell variables**. `python ... $PRS_ARGS` with
+  `PRS_ARGS="--prs 1.01 1.1 ..."` does not word-split in zsh by default;
+  argparse receives the whole string as one token and emits
+  `unrecognized arguments: --prs ...`. Either inline the list (as above)
+  or use `${=PRS_ARGS}`.
+- **COW early-period failures**. The COW pool's data starts around 2024-12.
+  `run_pr_sweep.py --multi-period` tries each period from `2024-01-01`
+  onwards; the early periods raise inside `run_single_period`, the script
+  catches the exception and the final heatmap only contains the surviving
+  rows. Expected.
+- **Re-running with the same `(start, end)` reuses the cached noise array**
+  in `results/mm_noise/_sim_arrays/<pool_id>_<start>_<end>_mm.npz` —
+  rebuilding only the first time each period is touched.