Hugging Face Hub Scraper 🚀

This project provides a collection of Python scripts designed to efficiently scrape metadata from the Hugging Face Hub, focusing on the two main categories: Models and Datasets. It leverages parallel processing to speed up data collection and includes functionality to save progress in a JSON file, allowing the process to resume seamlessly after interruptions. With these scripts, you can retrieve comprehensive metadata from both the Models Page and the Datasets Page on the Hugging Face Hub.

✨ Features

• Scrape Models and Datasets: Separate scripts to gather information on either models or datasets.
• Parallel Processing: Utilizes a ThreadPoolExecutor to fetch multiple resources in parallel, significantly speeding up the process.
• Checkpoint System: Automatically saves progress to a JSON file. If the script is interrupted, it can be restarted and will resume from where it left off.
• Configurable: All major parameters (like batch size, number of workers, and limits) can be configured via command-line arguments.
• Modular Code: Shared functionality is centralized in a utils.py file for easy maintenance and readability.

---

🛠️ Installation

1. Clone the repository:

   git clone <repository-url>
   cd <repository-directory>

2. Install the required Python libraries. It is recommended to use a virtual environment.

   pip install pandas huggingface_hub tqdm

---

📖 How to Use

1. Set Your Hugging Face Token

The use of a Hugging Face Token is not mandatory, but it is recommended if you want to access gated repositories. The recommended way is to set it as an environment variable.

For Linux/macOS:

export HF_TOKEN="your_hugging_face_token_here"

For Windows (Command Prompt):

set HF_TOKEN="your_hugging_face_token_here"

Alternatively, you can pass the token directly using the --hf-token command-line argument, but using an environment variable is more secure.

2. Running the Scripts

The project includes two main scripts: model_scraper.py for scraping models and dataset_scraper.py for scraping datasets.

Scraping Models 🤖

To scrape model information, run model_scraper.py.

Example usage:

python model_scraper.py --limit 50 --batch-size 10 --max-workers 5

This command will scrape the latest 50 models from the Hugging Face Hub, processing them in batches of 10 with 5 concurrent workers. The results will be saved to Output/models_dataset.csv.

Scraping Datasets 📚

To scrape dataset information, run dataset_scraper.py.

Example usage:

python dataset_scraper.py --limit 50 --batch-size 10 --max-workers 5

This command will scrape the latest 50 datasets from the Hugging Face Hub, processing them in batches of 10 with 5 concurrent workers. The results will be saved to Output/datasetsHuggingFace_dataset.csv.

Command-Line Arguments ⚙️

You can customize the behavior of the scripts using the following arguments:

• --hf-token: Your Hugging Face API token. Defaults to the HF_TOKEN environment variable.
• --max-workers: The maximum number of worker threads for parallel processing. (Default: 5)
• --batch-size: The number of items to process in each batch before saving a checkpoint. (Default: 100)
• --limit: The total number of items to fetch from the Hub. (Default: 1000)

3. Output 📊

Both scripts will:

1. Log their progress to the console.
2. Create a JSON checkpoint file (e.g., models_checkpoint.json) with the scraped data.
3. Once completed, they will create a CSV file inside the Output folder and print the head of a pandas DataFrame containing the final results.

Example Output 📊

Here's an example of the kind of data you can expect to find in the generated CSV files (e.g., Output/models_dataset.csv):

model_id	base_model	author	readme_file	license	language	downloads	likes	tags	pipeline_tag	library_name	created_at
bullerwins/FLUX.1-Kontext-dev-GGUF	[black-forest-labs/FLUX.1-Kontext-dev]	bullerwins	--- language: - en license: other license_...	other	[en]	51506	140	diffusion-single-file, gguf, image-generation,...	image-to-image	diffusion-single-file	2025-06-26 16:26:45+00:00
OmniGen2/OmniGen22	None	OmniGen22	--- license: apache-2.0 pipeline_tag: any-to...	apache-2.0	None	33558	322	diffusers, safetensors, any-to-any, arxiv:2506...	any-to-any	diffusers	2025-06-06 09:57:07+00:00
google/gemma-3n-E4B-it3	google/gemma-3n-e4b	google3	--- license: gemma library_name: transformer...	gemma	None	147609	401	transformers, safetensors, gemma3n, image-text...	image-text-to-text	transformers	2025-06-03 18:17:07+00:00
black-forest-labs/FLUX.1-Kontext-dev4	None	black-forest-labs4	--- language: - en license: other license_...	other	[en]	131465	1222	diffusers, safetensors, image-generation, flux...	image-to-image	diffusers	2025-05-28 22:23:43+00:00
tencent/Hunyuan-A13B-Instruct	None	tencent	--- license: other license_name: tencent-hun...	other	None	7807	681	transformers, safetensors, hunyuan, text-gener...	text-generation	transformers	2025-06-25 12:39:52+00:00

---

Here's an example of the kind of data you can expect to find in Output/datasetsHF_dataset.csv:

dataset_id	author	created_at	readme_file	downloads	likes	tags	language	license	multilinguality	size_categories	task-categories
black-forest-labs/kontext-bench	black-forest-labs	2025-06-26 06:04:00+00:00	--- license: mit language: - en pretty_nam...	0	30	[language:en, license:mit, size_categories:1K<...]	[en]	mit	None	[1K<n<10K]	None
FreedomIntelligence/ShareGPT-4o-Image	FreedomIntelligence	2025-06-16 07:53:06+00:00	--- language: - en license: apache-2.0 siz...	75	60	[task_categories:text-to-image, task_categorie...	[en]	apache-2.0	None	[10K<n<100K]	[text-to-image, image-to-image]
facebook/seamless-interaction	facebook	2025-06-23 05:17:45+00:00	--- license: cc-by-nc-4.0 configs: - config...	1	70	[license:cc-by-nc-4.0, modality:audio, modalit...	None	cc-by-nc-4.0	None	None	None
fka/awesome-chatgpt-prompts	fka	2022-12-13 23:47:45+00:00	--- license: cc0-1.0 tags: - ChatGPT task_...	22907	8132	[task_categories:question-answering, license:c...	None	cc0-1.0	None	[100K<n<1M]	[question-answering]
nvidia/OpenScience	nvidia	2025-06-03 19:47:29+00:00	--- configs: - config_name: OS-Q3-235B-4 ...	435	50	[license:cc-by-4.0, size_categories:1M<n<10M, ...	None	cc-by-4.0	None	[1M<n<10M]	None

---

📁 File Structure

.
├── Output                      # Folder to store output CSV files
├── model_scraper.py            # Main script to scrape models
├── dataset_scraper.py          # Main script to scrape datasets
├── utils.py                    # Shared utility functions for logging, networking, and processing
└── README.md                   # This documentation file

• model_scraper.py: Contains the logic specific to fetching and processing Hugging Face models.
• dataset_scraper.py: Contains the logic specific to fetching and processing Hugging Face datasets.
• utils.py: A helper module that provides all the core, reusable functionality, including argument parsing, checkpoint management, and the parallel processing loop.

---

🤝 Contributing

Contributions are welcome! If you have suggestions for improvements or find any issues, please feel free to open an issue or submit a pull request.