Internet scale web crawler in Rust. Incremental and realtime, a tiered fetch ladder that handles anti-bot, an open verifiable fetcher protocol, compact columnar storage, published as Parquet to open-index on Hugging Face.
rust search-engine distributed-systems crawler information-retrieval web-crawler open-data tokio file-format dataset robots-txt parquet columnar-storage anti-bot web-archiving common-crawl huggingface web-index open-index incremental-crawler
-
Updated
Oct 26, 2026 - Rust