hfgit

Download Hugging Face repositories with plain git + git-lfs and install them into the Hugging Face hub cache, so that transformers, vllm, diffusers, etc. find them exactly as if hf download had fetched them.

Use it when the official hf / huggingface-cli tool is misbehaving, or when you simply prefer git's well-understood transport.

hfgit download Qwen/Qwen3-0.6B

Requirements

• git
• git-lfs (apt install git-lfs / brew install git-lfs) — a one-time git lfs install is not required; the script passes the filters it needs.
• bash 4+

Usage

hfgit download <org>/<name> [FILE ...] [options]

Option	Description
--repo-type TYPE	model (default), dataset, or space
--revision REV	Branch, tag, or full 40-char commit SHA. Default main
--include PATTERN	Only fetch LFS files matching the glob. Repeatable, or comma-separated
--exclude PATTERN	Skip LFS files matching the glob. Repeatable, or comma-separated
--local-dir DIR	Clone into DIR as a normal git checkout instead of the cache
--cache-dir DIR	Cache root. Default $HF_HUB_CACHE, then $HF_HOME/hub, then ~/.cache/huggingface/hub
--token TOKEN	Auth token. Default $HF_TOKEN, then ~/.cache/huggingface/token
--jobs N	Concurrent LFS transfers (default 8)
--endpoint URL	Hub endpoint. Default $HF_ENDPOINT or https://huggingface.co
--force	Re-download even if the snapshot already exists
-q, --quiet	Print only the final path
-h, --help	Show help

Positional FILE arguments are shorthand for --include FILE.

On success the path of the installed snapshot is printed to stdout (like hf download), and all progress goes to stderr, so it's safe to use in $(...).

Examples

# Whole model into the cache
hfgit download Qwen/Qwen3-0.6B

# Only specific files
hfgit download Qwen/Qwen3-0.6B config.json tokenizer.json

# Only safetensors, skipping any consolidated/duplicate weights
hfgit download mistralai/Mistral-7B-Instruct-v0.3 \
    --include '*.safetensors' --exclude 'consolidated*'

# Gated model (accept the license on the model page first)
hfgit download meta-llama/Llama-3.1-8B-Instruct --token hf_xxxxxxxx

# A dataset subset
hfgit download HuggingFaceFW/fineweb --repo-type dataset --include 'sample/10BT/*'

# Pin an exact commit
hfgit download Qwen/Qwen3-0.6B --revision 83b669136f67d235730c7705afe0e633ec25eabb

# Plain checkout outside the cache (keeps .git, so `git pull` works later)
hfgit download Qwen/Qwen3-0.6B --local-dir ./qwen3-0.6b

# Feed the path straight to something
python -m vllm.entrypoints.openai.api_server --model "$(hfgit download -q Qwen/Qwen3-0.6B)"

Authentication

Gated and private repos need a token with at least read scope (https://huggingface.co/settings/tokens). The script looks, in order, at --token, $HF_TOKEN, and the file ~/.cache/huggingface/token that hf auth login writes.

The token is handed to git through an ephemeral credential helper for the duration of the command. It is never written to .git/config, never placed in the clone URL, and does not appear in ps output as part of a URL.

How it works

Every Hugging Face repo is a git repository with the large files in Git LFS, and the "revision" the hub client keys on is simply the git commit SHA. So:

1. Shallow-clone the repo with GIT_LFS_SKIP_SMUDGE=1 — this fetches only the small files and ~130-byte LFS pointers, and is fast.
2. git lfs pull with the include/exclude filters to fetch the weights you actually asked for. Any LFS file that was filtered out is deleted so no application ever opens a pointer file thinking it's a model.
3. Remove .git and move the tree into <cache>/models--<org>--<name>/snapshots/<commit>/.
4. Write the commit SHA to <cache>/models--<org>--<name>/refs/<revision>.

huggingface_hub resolves a repo id by reading refs/<revision> to get the commit, then opening snapshots/<commit>/<file>. That is the entire read path, so those two directories are all that is created.

The official downloader also writes blobs/ (content-addressed storage that snapshots/ symlinks into, for dedup across revisions), trees/ (a cached file listing per commit, new in huggingface_hub 1.28, which lets it skip a HEAD request per file when online), and .no_exist/ (a negative cache of 404s). None of these are consulted when loading a model, and their absence does not affect anything except one extra HTTP round-trip per file the first time an online lookup happens.

Disk usage

The clone happens in a temp directory inside the cache (<cache>/models--…/.hfgit-tmp.XXXXXX/), never in /tmp, so the final move is an atomic rename on the same filesystem and no weights are ever copied.

During git lfs pull, git-lfs keeps one copy of each object in .git/lfs/objects/ and one in the working tree, so you transiently need about 2× the model size free. The .git directory is deleted immediately after the pull, bringing it back to 1×. In --local-dir mode, .git/lfs/objects is removed but the rest of .git is kept so the checkout stays updatable.

Re-running

• Same repo + revision already installed, no filters: exits immediately and prints the existing snapshot path. Use --force to re-fetch.
• With --include/--exclude: always clones and merges new files into the existing snapshot for that commit, so you can fetch a repo incrementally (e.g. config.json first, weights later).
• Note that refs/main is only updated by running hfgit download again; upstream changes are not detected automatically.

Caveats

• Online lookups can bypass the cache. When not in offline mode, huggingface_hub first asks the Hub for the current commit of main. If the upstream repo has been updated since your download, the commit will not match your snapshot and the library will try to download the new revision. Either set HF_HUB_OFFLINE=1, pass revision=<sha> in from_pretrained, or re-run hfgit download to fetch the update.
• Clones are always --depth 1, so repos with a long history of huge weight commits cost nothing extra.
• Only download is implemented. For cache inspection use hf cache scan / huggingface-cli scan-cache, which read the same directories and work fine with snapshots produced by this tool (they will just report "no blobs").

Verifying an install

HF_HUB_OFFLINE=1 python -c "
from transformers import AutoConfig
print(AutoConfig.from_pretrained('Qwen/Qwen3-0.6B'))
"

HF_HUB_OFFLINE=1 guarantees the result came from the local cache and not from a fresh download.