Skip to content

feat: generalize data download to any CoinGecko-listed token - #7

Merged
tiago-hansen merged 1 commit into
mainfrom
feat/generalize-coingecko-download
Jul 10, 2026
Merged

tiago-hansen merged 1 commit into
mainfrom
feat/generalize-coingecko-download

Conversation

@tiago-hansen

Copy link
Copy Markdown
Collaborator

Context

The price-data pipeline had two disjoint entry points: download_data.py (Binance/Coinbase only) and a one-off prepare_bold_usdc_data.py (CoinGecko). The reCLAMM onboarding skill told users to copy that CoinGecko script every time a non-Binance token appeared. Every downstream consumer only cares about the canonical parquet schema, not the source — so the duplication was avoidable.

What changed

  • New quantammsim/utils/data_processing/coingecko_data.py — the CoinGecko source (registry + resolve_cg_id, daily → complete 1-minute canonical grid with the /1440 volume spread) and rebuild_stable_peg. Single source of truth for the COINGECKO_IDS / STABLECOINS / NATIVE_LST registries (moved out of fetch_token_mcaps.py, which now imports them back).
  • update_historic_data gains source / cg_id / cg_days. source="auto" (default) runs the exchange waterfall and falls back to CoinGecko only if nothing is found; source="coingecko" fetches directly. The finalization block is extracted into a shared _finalize_historic_data helper so every source produces identical parquet + daily/hourly CSV + plot outputs (CoinGecko tokens now get the daily CSV the old BOLD script never wrote).
  • download_data.py — new --source, --cg-id, --cg-days, --peg <STABLE> flags. The BOLD/USDC flow becomes one command:
    python scripts/download_data.py BOLD --cg-id liquity-bold-2 --peg USDC
    
  • Removed prepare_bold_usdc_data.py; rewrote reCLAMM SKILL.md Step 1.

Verification

  • Grid invariant: CoinGecko frame is an exact 1-min grid, passing the pipeline's == 60000ms assertion.
  • Volume invariant: daily resample-sum recovers real daily USD volume (the /1440 spread).
  • End-to-end source="coingecko" run wrote parquet (canonical columns, midnight-append), daily+hourly CSVs, and both plots with no KeyError from the new CoinGecko source color.
  • Peg builder round-trips at flat $1.00 over the token window with the date/close/Volume USD columns market_features reads.
  • fetch_token_mcaps.py, CLI --help, and the single-ticker guard for --cg-id/--peg all pass.

The only path not exercisable offline is the live exchange → CoinGecko auto-fallback (needs a real Binance miss); that branch is a three-line guard feeding the same get_coingecko_data the tests cover.

🤖 Generated with Claude Code

Unify the price-data pipeline behind a single entry point. Previously
download_data.py only handled Binance/Coinbase, and non-Binance tokens
required copying prepare_bold_usdc_data.py each time (per the reCLAMM
onboarding skill).

- Add quantammsim/utils/data_processing/coingecko_data.py: the CoinGecko
  source (registry + resolve_cg_id, daily->1-min canonical grid with the
  /1440 volume spread) and rebuild_stable_peg. Hosts the single-source-of-
  truth COINGECKO_IDS/STABLECOINS/NATIVE_LST registries.
- Wire CoinGecko into update_historic_data: source="auto" falls back to
  CoinGecko when the exchange waterfall finds nothing; source="coingecko"
  fetches directly. Extract the shared _finalize_historic_data helper so
  every source gets identical parquet + daily/hourly CSV + plot outputs.
- download_data.py: add --source, --cg-id, --cg-days, --peg flags. The
  BOLD/USDC flow becomes a single command.
- fetch_token_mcaps.py imports the registries from the shared module.
- Remove prepare_bold_usdc_data.py; update reCLAMM SKILL.md Step 1.
@tiago-hansen
tiago-hansen requested a review from mendesfabio July 10, 2026 18:24
@tiago-hansen
tiago-hansen merged commit 1c50f35 into main Jul 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants