# Downloading files

## Downloading files

### Download a single file

#### ba\_hub\_download

**boincai\_hub.hf\_hub\_download**

[\<source>](https://github.com/huggingface/huggingface_hub/blob/v0.18.0.rc0/src/huggingface_hub/file_download.py#L990)

( repo\_id: strfilename: strsubfolder: typing.Optional\[str] = Nonerepo\_type: typing.Optional\[str] = Nonerevision: typing.Optional\[str] = Noneendpoint: typing.Optional\[str] = Nonelibrary\_name: typing.Optional\[str] = Nonelibrary\_version: typing.Optional\[str] = Nonecache\_dir: typing.Union\[str, pathlib.Path, NoneType] = Nonelocal\_dir: typing.Union\[str, pathlib.Path, NoneType] = Nonelocal\_dir\_use\_symlinks: typing.Union\[bool, typing.Literal\['auto']] = 'auto'user\_agent: typing.Union\[typing.Dict, str, NoneType] = Noneforce\_download: bool = Falseforce\_filename: typing.Optional\[str] = Noneproxies: typing.Optional\[typing.Dict] = Noneetag\_timeout: float = 10resume\_download: bool = Falsetoken: typing.Union\[bool, str, NoneType] = Nonelocal\_files\_only: bool = Falselegacy\_cache\_layout: bool = False )

Parameters

* **repo\_id** (`str`) — A user or an organization name and a repo name separated by a `/`.
* **filename** (`str`) — The name of the file in the repo.
* **subfolder** (`str`, *optional*) — An optional value corresponding to a folder inside the model repo.
* **repo\_type** (`str`, *optional*) — Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`.
* **revision** (`str`, *optional*) — An optional Git revision id which can be a branch name, a tag, or a commit hash.
* **endpoint** (`str`, *optional*) — Hugging Face Hub base url. Will default to <https://huggingface.co/>. Otherwise, one can set the `HF_ENDPOINT` environment variable.
* **library\_name** (`str`, *optional*) — The name of the library to which the object corresponds.
* **library\_version** (`str`, *optional*) — The version of the library.
* **cache\_dir** (`str`, `Path`, *optional*) — Path to the folder where cached files are stored.
* **local\_dir** (`str` or `Path`, *optional*) — If provided, the downloaded file will be placed under this directory, either as a symlink (default) or a regular file (see description for more details).
* **local\_dir\_use\_symlinks** (`"auto"` or `bool`, defaults to `"auto"`) — To be used with `local_dir`. If set to “auto”, the cache directory will be used and the file will be either duplicated or symlinked to the local directory depending on its size. It set to `True`, a symlink will be created, no matter the file size. If set to `False`, the file will either be duplicated from cache (if already exists) or downloaded from the Hub and not cached. See description for more details.
* **user\_agent** (`dict`, `str`, *optional*) — The user-agent info in the form of a dictionary or a string.
* **force\_download** (`bool`, *optional*, defaults to `False`) — Whether the file should be downloaded even if it already exists in the local cache.
* **proxies** (`dict`, *optional*) — Dictionary mapping protocol to the URL of the proxy passed to `requests.request`.
* **etag\_timeout** (`float`, *optional*, defaults to `10`) — When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`.
* **resume\_download** (`bool`, *optional*, defaults to `False`) — If `True`, resume a previously interrupted download.
* **token** (`str`, `bool`, *optional*) — A token to be used for the download.
  * If `True`, the token is read from the HuggingFace config folder.
  * If a string, it’s used as the authentication token.
* **local\_files\_only** (`bool`, *optional*, defaults to `False`) — If `True`, avoid downloading the file and return the path to the local cached file if it exists.
* **legacy\_cache\_layout** (`bool`, *optional*, defaults to `False`) — If `True`, uses the legacy file cache layout i.e. just call [hf\_hub\_url()](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/file_download#huggingface_hub.hf_hub_url) then `cached_download`. This is deprecated as the new cache layout is more powerful.

Download a given file if it’s not already present in the local cache.

The new cache file layout looks like this:

* The cache directory contains one subfolder per repo\_id (namespaced by repo type)
* inside each repo folder:
  * refs is a list of the latest known revision => commit\_hash pairs
  * blobs contains the actual file blobs (identified by their git-sha or sha256, depending on whether they’re LFS files or not)
  * snapshots contains one subfolder per commit, each “commit” contains the subset of the files that have been resolved at that particular commit. Each filename is a symlink to the blob at that particular commit.

If `local_dir` is provided, the file structure from the repo will be replicated in this location. You can configure how you want to move those files:

* If `local_dir_use_symlinks="auto"` (default), files are downloaded and stored in the cache directory as blob files. Small files (<5MB) are duplicated in `local_dir` while a symlink is created for bigger files. The goal is to be able to manually edit and save small files without corrupting the cache while saving disk space for binary files. The 5MB threshold can be configured with the `HF_HUB_LOCAL_DIR_AUTO_SYMLINK_THRESHOLD` environment variable.
* If `local_dir_use_symlinks=True`, files are downloaded, stored in the cache directory and symlinked in `local_dir`. This is optimal in term of disk usage but files must not be manually edited.
* If `local_dir_use_symlinks=False` and the blob files exist in the cache directory, they are duplicated in the local dir. This means disk usage is not optimized.
* Finally, if `local_dir_use_symlinks=False` and the blob files do not exist in the cache directory, then the files are downloaded and directly placed under `local_dir`. This means if you need to download them again later, they will be re-downloaded entirely.

Copied

```
[  96]  .
└── [ 160]  models--julien-c--EsperBERTo-small
    ├── [ 160]  blobs
    │   ├── [321M]  403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
    │   ├── [ 398]  7cb18dc9bafbfcf74629a4b760af1b160957a83e
    │   └── [1.4K]  d7edf6bd2a681fb0175f7735299831ee1b22b812
    ├── [  96]  refs
    │   └── [  40]  main
    └── [ 128]  snapshots
        ├── [ 128]  2439f60ef33a0d46d85da5001d52aeda5b00ce9f
        │   ├── [  52]  README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
        │   └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
        └── [ 128]  bbc77c8132af1cc5cf678da3f1ddf2de43606d48
            ├── [  52]  README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
            └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
```

Raises the following errors:

* [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `token=True` and the token cannot be found.
* [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined.
* [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid
* [RepositoryNotFoundError](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/utilities#huggingface_hub.utils.RepositoryNotFoundError) If the repository to download from cannot be found. This may be because it doesn’t exist, or because it is set to `private` and you do not have access.
* [RevisionNotFoundError](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/utilities#huggingface_hub.utils.RevisionNotFoundError) If the revision to download from cannot be found.
* [EntryNotFoundError](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/utilities#huggingface_hub.utils.EntryNotFoundError) If the file to download cannot be found.
* [LocalEntryNotFoundError](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/utilities#huggingface_hub.utils.LocalEntryNotFoundError) If network is disabled or unavailable and file is not found in cache.

#### hf\_hub\_url

**huggingface\_hub.hf\_hub\_url**

[\<source>](https://github.com/huggingface/huggingface_hub/blob/v0.18.0.rc0/src/huggingface_hub/file_download.py#L173)

( repo\_id: strfilename: strsubfolder: typing.Optional\[str] = Nonerepo\_type: typing.Optional\[str] = Nonerevision: typing.Optional\[str] = Noneendpoint: typing.Optional\[str] = None )

Parameters

* **repo\_id** (`str`) — A namespace (user or an organization) name and a repo name separated by a `/`.
* **filename** (`str`) — The name of the file in the repo.
* **subfolder** (`str`, *optional*) — An optional value corresponding to a folder inside the repo.
* **repo\_type** (`str`, *optional*) — Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`.
* **revision** (`str`, *optional*) — An optional Git revision id which can be a branch name, a tag, or a commit hash.
* **endpoint** (`str`, *optional*) — Hugging Face Hub base url. Will default to <https://huggingface.co/>. Otherwise, one can set the `HF_ENDPOINT` environment variable.

Construct the URL of a file from the given information.

The resolved address can either be a huggingface.co-hosted url, or a link to Cloudfront (a Content Delivery Network, or CDN) for large files which are more than a few MBs.

Example:

Copied

```
>>> from huggingface_hub import hf_hub_url

>>> hf_hub_url(
...     repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin"
... )
'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin'
```

Notes:

Cloudfront is replicated over the globe so downloads are way faster for the end user (and it also lowers our bandwidth costs).

Cloudfront aggressively caches files by default (default TTL is 24 hours), however this is not an issue here because we implement a git-based versioning system on huggingface.co, which means that we store the files on S3/Cloudfront in a content-addressable way (i.e., the file name is its hash). Using content-addressable filenames means cache can’t ever be stale.

In terms of client-side caching from this library, we base our caching on the objects’ entity tag (`ETag`), which is an identifier of a specific version of a resource \[1]\_. An object’s ETag is: its git-sha1 if stored in git, or its sha256 if stored in git-lfs.

References:

* \[1] <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag>

### Download a snapshot of the repo

**huggingface\_hub.snapshot\_download**

[\<source>](https://github.com/huggingface/huggingface_hub/blob/v0.18.0.rc0/src/huggingface_hub/_snapshot_download.py#L23)

( repo\_id: strrepo\_type: typing.Optional\[str] = Nonerevision: typing.Optional\[str] = Noneendpoint: typing.Optional\[str] = Nonecache\_dir: typing.Union\[str, pathlib.Path, NoneType] = Nonelocal\_dir: typing.Union\[str, pathlib.Path, NoneType] = Nonelocal\_dir\_use\_symlinks: typing.Union\[bool, typing.Literal\['auto']] = 'auto'library\_name: typing.Optional\[str] = Nonelibrary\_version: typing.Optional\[str] = Noneuser\_agent: typing.Union\[typing.Dict, str, NoneType] = Noneproxies: typing.Optional\[typing.Dict] = Noneetag\_timeout: float = 10resume\_download: bool = Falseforce\_download: bool = Falsetoken: typing.Union\[str, bool, NoneType] = Nonelocal\_files\_only: bool = Falseallow\_patterns: typing.Union\[typing.List\[str], str, NoneType] = Noneignore\_patterns: typing.Union\[typing.List\[str], str, NoneType] = Nonemax\_workers: int = 8tqdm\_class: typing.Optional\[tqdm.asyncio.tqdm\_asyncio] = None )

Parameters

* **repo\_id** (`str`) — A user or an organization name and a repo name separated by a `/`.
* **repo\_type** (`str`, *optional*) — Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`.
* **revision** (`str`, *optional*) — An optional Git revision id which can be a branch name, a tag, or a commit hash.
* **endpoint** (`str`, *optional*) — Hugging Face Hub base url. Will default to <https://huggingface.co/>. Otherwise, one can set the `HF_ENDPOINT` environment variable.
* **cache\_dir** (`str`, `Path`, *optional*) — Path to the folder where cached files are stored.
* **local\_dir** (`str` or `Path`, *optional*) — If provided, the downloaded files will be placed under this directory, either as symlinks (default) or regular files (see description for more details).
* **local\_dir\_use\_symlinks** (`"auto"` or `bool`, defaults to `"auto"`) — To be used with `local_dir`. If set to “auto”, the cache directory will be used and the file will be either duplicated or symlinked to the local directory depending on its size. It set to `True`, a symlink will be created, no matter the file size. If set to `False`, the file will either be duplicated from cache (if already exists) or downloaded from the Hub and not cached. See description for more details.
* **library\_name** (`str`, *optional*) — The name of the library to which the object corresponds.
* **library\_version** (`str`, *optional*) — The version of the library.
* **user\_agent** (`str`, `dict`, *optional*) — The user-agent info in the form of a dictionary or a string.
* **proxies** (`dict`, *optional*) — Dictionary mapping protocol to the URL of the proxy passed to `requests.request`.
* **etag\_timeout** (`float`, *optional*, defaults to `10`) — When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`.
* **resume\_download** (`bool`, *optional*, defaults to `False) -- If` True\`, resume a previously interrupted download.
* **force\_download** (`bool`, *optional*, defaults to `False`) — Whether the file should be downloaded even if it already exists in the local cache.
* **token** (`str`, `bool`, *optional*) — A token to be used for the download.
  * If `True`, the token is read from the HuggingFace config folder.
  * If a string, it’s used as the authentication token.
* **local\_files\_only** (`bool`, *optional*, defaults to `False`) — If `True`, avoid downloading the file and return the path to the local cached file if it exists.
* **allow\_patterns** (`List[str]` or `str`, *optional*) — If provided, only files matching at least one pattern are downloaded.
* **ignore\_patterns** (`List[str]` or `str`, *optional*) — If provided, files matching any of the patterns are not downloaded.
* **max\_workers** (`int`, *optional*) — Number of concurrent threads to download files (1 thread = 1 file download). Defaults to 8.
* **tqdm\_class** (`tqdm`, *optional*) — If provided, overwrites the default behavior for the progress bar. Passed argument must inherit from `tqdm.auto.tqdm` or at least mimic its behavior. Note that the `tqdm_class` is not passed to each individual download. Defaults to the custom HF progress bar that can be disabled by setting `HF_HUB_DISABLE_PROGRESS_BARS` environment variable.

Download repo files.

Download a whole snapshot of a repo’s files at the specified revision. This is useful when you want all files from a repo, because you don’t know which ones you will need a priori. All files are nested inside a folder in order to keep their actual filename relative to that folder. You can also filter which files to download using `allow_patterns` and `ignore_patterns`.

If `local_dir` is provided, the file structure from the repo will be replicated in this location. You can configure how you want to move those files:

* If `local_dir_use_symlinks="auto"` (default), files are downloaded and stored in the cache directory as blob files. Small files (<5MB) are duplicated in `local_dir` while a symlink is created for bigger files. The goal is to be able to manually edit and save small files without corrupting the cache while saving disk space for binary files. The 5MB threshold can be configured with the `HF_HUB_LOCAL_DIR_AUTO_SYMLINK_THRESHOLD` environment variable.
* If `local_dir_use_symlinks=True`, files are downloaded, stored in the cache directory and symlinked in `local_dir`. This is optimal in term of disk usage but files must not be manually edited.
* If `local_dir_use_symlinks=False` and the blob files exist in the cache directory, they are duplicated in the local dir. This means disk usage is not optimized.
* Finally, if `local_dir_use_symlinks=False` and the blob files do not exist in the cache directory, then the files are downloaded and directly placed under `local_dir`. This means if you need to download them again later, they will be re-downloaded entirely.

An alternative would be to clone the repo but this requires git and git-lfs to be installed and properly configured. It is also not possible to filter which files to download when cloning a repository using git.

Raises the following errors:

* [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `token=True` and the token cannot be found.
* [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined.
* [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid

### Get metadata about a file

#### get\_hf\_file\_metadata

**huggingface\_hub.get\_hf\_file\_metadata**

[\<source>](https://github.com/huggingface/huggingface_hub/blob/v0.18.0.rc0/src/huggingface_hub/file_download.py#L1581)

( url: strtoken: typing.Union\[bool, str, NoneType] = Noneproxies: typing.Optional\[typing.Dict] = Nonetimeout: typing.Optional\[float] = 10.0 )

Parameters

* **url** (`str`) — File url, for example returned by [hf\_hub\_url()](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/file_download#huggingface_hub.hf_hub_url).
* **token** (`str` or `bool`, *optional*) — A token to be used for the download.
  * If `True`, the token is read from the HuggingFace config folder.
  * If `False` or `None`, no token is provided.
  * If a string, it’s used as the authentication token.
* **proxies** (`dict`, *optional*) — Dictionary mapping protocol to the URL of the proxy passed to `requests.request`.
* **timeout** (`float`, *optional*, defaults to 10) — How many seconds to wait for the server to send metadata before giving up.

Fetch metadata of a file versioned on the Hub for a given url.

#### HfFileMetadata

#### class huggingface\_hub.HfFileMetadata

[\<source>](https://github.com/huggingface/huggingface_hub/blob/v0.18.0.rc0/src/huggingface_hub/file_download.py#L150)

( commit\_hash: typing.Optional\[str]etag: typing.Optional\[str]location: strsize: typing.Optional\[int] )

Parameters

* **commit\_hash** (`str`, *optional*) — The commit\_hash related to the file.
* **etag** (`str`, *optional*) — Etag of the file on the server.
* **location** (`str`) — Location where to download the file. Can be a Hub url or not (CDN).
* **size** (`size`) — Size of the file. In case of an LFS file, contains the size of the actual LFS file, not the pointer.

Data structure containing information about a file versioned on the Hub.

Returned by [get\_hf\_file\_metadata()](https://huggingface.co/docs/huggingface_hub/v0.18.0.rc0/en/package_reference/file_download#huggingface_hub.get_hf_file_metadata) based on a URL.

### Caching

The methods displayed above are designed to work with a caching system that prevents re-downloading files. The caching system was updated in v0.8.0 to become the central cache-system shared across libraries that depend on the Hub.

Read the [cache-system guide](https://huggingface.co/docs/huggingface_hub/manage-cache) for a detailed presentation of caching at at HF.