huggingface_hub

The official Python client for the Huggingface Hub.

APACHE-2.0 License

Downloads
43.3M
Stars
1.6K
Committers
197
View Code on GitHub Visit Website
Ecosystems: PyTorch, Svelte

Bot releases are visible (Hide)

huggingface_hub - v0.13.0: download files to a specific folder, documentation, duplicate spaces, and more

Published by Wauplin over 1 year ago

Download files to a specific folder

It is now possible to download files from the Hub and move them to a specific folder!

Two behaviors are possible: either create symlinks or move the files from the cache. This can be controlled with the local_dir_use_symlinks input parameter. The default -and recommended- value is "auto" which will duplicate small files to ease user experience (no symlinks when editing a file) and create symlinks for big files (save disk usage).

from huggingface_hub import snapshot_download
# or "from huggingface_hub import hf_hub_download"

# Download and cache files + duplicate small files (<5MB) to "my-folder" + add symlinks for big files
snapshot_download(repo_id, local_dir="my-folder")

# Download and cache files + add symlinks in "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=True)

# Duplicate files already existing in cache and/or download missing files directly to "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=False)
  • Download file to specific destination by @Wauplin in #1360

Documentation

Efforts to improve documentation have continued. The guides overview has been refactored to display which topics are covered (repository, upload, download, search, inference, community tab, cache, model cards, space management and integration).

Upload / Download files

The repository, upload and download guides have been revisited to showcase the different possibilities to manage a repository and upload/download files to/from it. The focus has been explicitly put on the HTTP endpoints rather than the git cli.

  • Refactor guides section + promote HTTP over GIT by @Wauplin in #1338

Integrate a library

A new guide has been added on how to integrate any ML framework with the Hub. It explains what is meant by that and how to do it. Here is the summary table to remember:

2023-03-07_16-32

  • Write Mixins/Integration guide by @Wauplin in #1362

Other

  • Add repo_type to repo_info docstring by @albertvillanova in #1347

New endpoints + QOL improvements

Duplicate a Space

It's now possible to duplicate a Space programmatically!

>>> from huggingface_hub import duplicate_space

# Duplicate a Space to your account
>>> duplicate_space("multimodalart/dreambooth-training")
RepoUrl('https://huggingface.co/spaces/nateraw/dreambooth-training',...)
  • ✨ add duplicate_space by @nateraw in #1353

delete_patterns in upload_folder

New input parameter delete_patterns for the upload_folder method. It allows to delete some remote files before pushing a folder to the Hub, in a single commit. Useful when you don't exactly know which files have already been pushed. Here is an example to upload log files while deleting existing logs on the Hub:

api.upload_folder(
    folder_path="/path/to/local/folder/logs",
    repo_id="username/trained-model",
    path_in_repo="experiment/logs/",
    allow_patterns="*.txt", # Upload all local text files
    delete_patterns="*.txt", # Delete all remote text files before
)
  • Add delete_patterns option to upload_folder by @Wauplin in #1370

List repo history

Get the repo history (i.e. all the commits) for a given revision.

# Get initial commit on a repo
>>> from huggingface_hub import list_repo_commits
>>> initial_commit = list_repo_commits("gpt2")[-1]

# Initial commit is always a system commit containing the `.gitattributes` file.
>>> initial_commit
GitCommitInfo(
    commit_id='9b865efde13a30c13e0a33e536cf3e4a5a9d71d8',
    authors=['system'],
    created_at=datetime.datetime(2019, 2, 18, 10, 36, 15, tzinfo=datetime.timezone.utc),
    title='initial commit',
    message='',
    formatted_title=None,
    formatted_message=None
)
  • Add list_repo_commits to list git history of a repo by @Wauplin in #1331

Accept token in huggingface-cli login

--token and --add-to-git-credential option have been added to login directly from the CLI using an environment variable. Useful to login in a Github CI script for example.

huggingface-cli login --token $HUGGINGFACE_TOKEN --add-to-git-credential
  • Add token and git credentials to login cli command by @silvanocerza in #1372
  • token in CLI login docs by @Wauplin (direct commit on main)

Telemetry helper

Helper for external libraries to track usage of specific features of their package. Telemetry can be globally disabled by the user using HF_HUB_DISABLE_TELEMETRY.

from huggingface_hub.utils import send_telemetry

send_telemetry("gradio/local_link", library_name="gradio", library_version="3.22.1")
  • Add utility to send telemetry by @Wauplin in #1344

Breaking change

When loading a model card with an invalid model_index in the metadata, an error is explicitly raised. Previous behavior was to trigger a warning and ignore the model_index. This was problematic as it could lead to a loss of information. Fixing this is a breaking change but impact should be limited as the server is already rejecting invalid model cards. An optional ignore_metadata_errors argument (default to False) can be used to load the card with only a warning.

  • Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Bugfixes & small improvements

Model cards, datasets cards and space cards

A few improvements in repo cards: expose RepoCard as top-level, dict-like methods for RepoCardData object (#1354), updated template and improved type annotation for metadata.

  • Updating MC headings by @EziOzoani in #1367
  • Switch datasets type in ModelCard to a list of datasets by @davanstrien in #1356
  • Expose RepoCard at top level + few qol improvements by @Wauplin in #1354
  • Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Misc

  • Fix contrib CI for timm by @Wauplin in #1346
  • 🚑 hotfix isHf test user permissions bug by @nateraw in #1357
  • Remove unused private methods by @Wauplin in #1359
  • Updates types for RepoCards fields by @davanstrien in #1358
  • some typos by @Wauplin (direct commit on main)
  • [WIP] make repr formatting narrower by @davanstrien in #903
  • Add ReprMixin to Repofile by @Wauplin (direct commit on main)
  • Fix ReprMixin for python3.7 + test by @Wauplin in #1380
  • fix create_commit on lowercased repo_id + add regression test by @Wauplin in #1376
  • FIX create_repo with exists_ok but no permission by @Wauplin in #1364
huggingface_hub - Patch release v0.12.1

Published by Wauplin over 1 year ago

Hot-fix to remove authorization header when following redirection (using cached_download). Fix was already implemented for hf_hub_download but we forgot about this one. Has only a consequence when downloading LFS files from Spaces. Problem arose since a server-side change on how files are served. See https://github.com/huggingface/huggingface_hub/pull/1345.

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.12.0...v0.12.1

huggingface_hub - v0.12.0: Spaces on steroids 🚀, extended HfApi, documentation, modelcards and more

Published by Wauplin almost 2 years ago

Spaces on steroids 🚀

Spaces support has been substantially enhanced. You can now:

  • Request hardware for your Space !
  • Configure your Space secrets
  • Get runtime information about your Space
# Assign hardware when creating the Space
api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="gradio", space_hardware="cpu-upgrade")

# Configure some secrets
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value="hf_api_***")

# Request hardware on the fly
api.request_space_hardware(repo_id=repo_id, hardware="t4-medium")

# Get Space runtime (state, hardware, sdk,...)
api.get_space_runtime(repo_id=repo_id)

Visit the docs for more details.

  • Manage space hardware and secrets programmatically by @Wauplin in #1243
  • Mock Space tests requiring GPUs by @Wauplin in #1263
  • Remove get_space_secrets endpoint by @Wauplin in #1264

And in bonus: Spaces now support Dockerfile natively !

api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="docker")

Check out the docs for more details.

New features and improvements for HfApi

List all branches and tags from a repo

It's now possible to list branches/tags from a repo, getting exact ref and target_commit.
More details in the docs.

>>> api.list_repo_refs("bigcode/the-stack", repo_type='dataset')
GitRefs(
    branches=[
        GitRefInfo(name='main', ref='refs/heads/main', target_commit='18edc1591d9ce72aa82f56c4431b3c969b210ae3'),
        GitRefInfo(name='v1.1.a1', ref='refs/heads/v1.1.a1', target_commit='f9826b862d1567f3822d3d25649b0d6d22ace714')],
    converts=[],
    tags=[
        GitRefInfo(name='v1.0', ref='refs/tags/v1.0', target_commit='c37a8cd1e382064d8aced5e05543c5f7753834da')
    ]
)
  • List branches and tags from a repo by @Wauplin in #1276

Manage your "favorite repos" list

New endpoints to like a repo, unlike it and list_liked_repos.

  • Add method to list liked repos by @Wauplin in #1254
  • Fix list_liked_repos (only public likes are returned) by @Wauplin in #1273
  • Fix like test by @Wauplin (direct commit on main)

Get the "real" repo_id when creating a repo

When using create_repo, one can provide a simple repo_name without specifying a namespace (example: "my-cool-model" instead of "Wauplin/my-cool-model"). This was annoying for as one could easily know if a namespace has been added to the provided repo_id. To facilitate this, the return value of create_repo is now an instance of RepoUrl which contains information like the endpoint, namespace, repo_id and repo_type.

  • Extend create_repo return value by @Wauplin in #1268

Create branch from any revision

By default, new branches start from main HEAD. It's now possible to specify any branch, tag or commit id to start from.

  • Create branch from given revision by @Wauplin in #1293

Modelcards

Modelcards module is getting some adjustments to better integrate with the Hub. The scope of this work is larger than "just" huggingface_hub and resulted in the launch of the HF Model Card Guidebook to help operationalize model cards in the ML community.

  • Clarify what "More Resources" is by breaking it into its parts. by @meg-huggingface in #1238
  • Adding "summary" variables by @meg-huggingface in #1248
  • datasetcard_template: I think linking to a GH user does not make sense anymore now that dataset repos are fully on the Hub by @julien-c in #1257
  • Push empty model card and dataset card to the Hub by @Wauplin in #1261
  • [Dataset | Model card] When pushing to template repos, work on actual raw contents by @julien-c in #1282
  • Update REGEX_YAML_BLOCK by @julien-c in #1285
  • Make model use more visible by @osanseviero in #1286

Documentation updates

Quite some effort has been put into the documentation in the past few weeks:

  • Revisit documentation part 1: divio-like landing page, installation page, configuration page by @Wauplin in #1233
  • Updates in docs by @Wauplin in #1235
  • Doc page: git vs http by @Wauplin in #1245
  • Minor suggestions post #1245 by @osanseviero in #1249
  • fix broken urls in docs by @Wauplin in #1253
  • [Minor doc tweak] *currently... by @julien-c in #1256
  • Update Repository docstring by @Wauplin in #1297
  • add link to create_repo by @Wauplin in #1299
  • Document .no_exist folder by @Wauplin in #1308
  • Revisit "search the hub" page (+HfApi reference page) by @Wauplin in #1303

Test suite for dependent libraries

huggingface_hub is getting more and more mature but you might still have some friction if you are maintainer of a library depending on huggingface_hub. To help detect breaking changes that would affect third-party libraries, we built a framework to run simple end-to-end tests in our CI. This is still preliminary work but the hope is make hfh ecosystem more and more robust over time. Check out our README for more details.

  • [RFC] Contrib test suite + tests for timm and sentence_transformers by @Wauplin in #1200

Faster download

Goal is to download files faster. First step has been to increase the chunk size by which the files are uploaded. Second step has been to add an optional Rust extension. This is not officially documented for now as we are internally assessing its benefits and limits. Installing and activating hf_transfer is purely optional.

  • Enabling hf_transfer use. by @Narsil in #1272
  • Increase the chunk size for faster download by @Narsil in #1267

Breaking changes

  • Repository "clone_from" feature do not create the remote repository if it doesn't exist on the Hub. Please use create_repo first before cloning it locally. The private init argument has also been removed as it was not needed anymore.
  • allow_regex and ignore_regex have been removed from snapshot_download in favor allow_patterns and ignore_patterns.
  • git-based implementation of push_to_hub_fastai has been removed in favor of the HTTP-based approach. Same for ModelHubMixin, PyTorchModelHubMixin, push_to_hub_keras and KerasModelHubMixin.
  • create_repo is now forced to use keyword-arguments. Same for metadata_eval_result.
  • Remove deprecated code in v0.12 by @Wauplin in #1196

QOL improvements

Not really some features, not really some fixes. But definitely a quality of life improvement for users 🙂

  • Progress bar when uploading LFS files by @Wauplin in #1266
  • tell user how to paste token on Windows by @fristhon in #1289
  • fixes 1290 - check for existing token during interpreter login by @lonnen in #1291
  • Store token in configurable HF_HOME (in HfFolder) by @Wauplin in #1236
  • Compute sha by chunks when uploading by @Wauplin in #1296

Internal

CI runs on Windows !

  • Add windows machine for CI by @Wauplin in #1112

Misc

  • Remove obsolete comment by @severo in #1265
  • fix: authorization header should never forwarded by @rtrompier in #1269
  • refacto Offline repository tests by @Wauplin (direct commit on main)
  • Revert "refacto Offline repository tests" by @Wauplin (direct commit on main)
  • 1278 More robust temporary directory by @Wauplin in #1279
  • Refacto Repository tests by @Wauplin in #1277
  • skip flake8 B028 by @Wauplin (direct commit on main)

Bugfixes & small improvements

  • Fix metadata_update for verified evaluations by @lewtun in #1214
  • Accept pathlib.Path in upload_file by @Wauplin in #1247
  • Skip failing tests related to DatasetFilter by @Wauplin in #1251
  • fix tasks ids test by @Wauplin (direct commit on main)
  • Fix InferenceAPI on image task by @Wauplin in #1270
  • Update "move_repo" doc url + fix Bigcode test in CI by @Wauplin in #1292
  • Fix test_comment_discussion by @Wauplin in #1295
  • Handle hf:// urls + raise ValueError if repo type is unknown by @Wauplin in #1298
  • Make ModelSearchArguments and DatasetSearchArguments more robust by @Wauplin in #1300
  • Make tests more robust (no pre-existing repo on staging Hub) by @Wauplin in #1302
  • fix: use cache for snapshots even if refs does not exist by @brycedrennan @Wauplin in #1306
  • Fix caching error when offline and no refs dir by @Wauplin in #1307
huggingface_hub - Patch release v0.11.1

Published by Wauplin almost 2 years ago

Hot-fix to fix permission issues when downloading with hf_hub_download or snapshot_download. For more details, see https://github.com/huggingface/huggingface_hub/pull/1220, https://github.com/huggingface/huggingface_hub/issues/1141 and https://github.com/huggingface/huggingface_hub/issues/1215.

Full changelog: https://github.com/huggingface/huggingface_hub/compare/v0.11.0...v0.11.1

huggingface_hub - Extended HfApi, pagination, simplified login and more

Published by Wauplin almost 2 years ago

New features and improvements for HfApi

HfApi is the central point to interact with the Hub API (manage repos, create commits,...). The goal is to propose more and more git-related features using HTTP endpoints to allow users to interact with the Hub without cloning locally a repo.

Create/delete tags and branches

from huggingface_hub import create_branch, create_tag, delete_branch, delete_tag

create_tag(repo_id, tag="v0.11", tag_message="Release v0.11")
delete_tag(repo_id, tag="something") # If you created a tag by mistake

create_branch(repo_id, branch="experiment-154")
delete_branch(repo_id, branch="experiment-1") # Clean some old branches
  • Add a create_tag method to create tags from the HTTP endpoint by @Wauplin in #1089
  • Add delete_tag method to HfApi by @Wauplin in #1128
  • Create tag twice doesn't work by @Wauplin in #1149
  • Add "create_branch" and "delete_branch" endpoints by @Wauplin #1181

Upload lots of files in a single commit

Making a very large commit was previously tedious. Files are now processed by chunks which makes it possible to upload 25k files in a single commit (and 1Gb payload limitation if uploading only non-LFS files). This should make it easier to upload large datasets.

  • Create commit by streaming a ndjson payload (allow lots of file in single commit) by @Wauplin in #1117

Delete an entire folder

from huggingface_hub import CommitOperationDelete, create_commit, delete_folder

# Delete a single folder
delete_folder(repo_id=repo_id, path_in_repo="logs/")

# Alternatively, use the low-level `create_commit`
create_commit(
    repo_id,
    operations=[
        CommitOperationDelete(path_in_repo="old_config.json") # Delete a file
        CommitOperationDelete(path_in_repo="logs/") # Delete a folder
    ],
    commit_message=...,
)
  • Delete folder with commit endpoint by @Wauplin in #1163

Support pagination when listing repos

In the future, listing models, datasets and spaces will be paginated on the Hub by default. To avoid breaking changes, huggingface_hub follows already pagination. Output type is currently a list (deprecated), will become a generator in v0.14.

  • Add support for pagination in list_models list_datasets and list_spaces by @Wauplin #1176
  • Deprecate output in list_models by @Wauplin in #1143

Misc

  • Allow create PR against non-main branch by @Wauplin in #1168
  • 1162 Reorder operations correctly in commit endpoint by @Wauplin in #1175

Login, tokens and authentication

Authentication has been revisited to make it as easy as possible for the users.

Unified login and logout methods

from huggingface_hub import login, logout

# `login` detects automatically if you are running in a notebook or a script
# Launch widgets or TUI accordingly
login()

# Now possible to login with a hardcoded token (non-blocking)
login(token="hf_***")

# If you want to bypass the auto-detection of `login`
notebook_login()  # still available
interpreter_login()  # to login from a script

# Logout programmatically
logout()

# Still possible to login from CLI
huggingface-cli login
  • Unified login/logout methods by @Wauplin in #1111

Set token only for a HfApi session

from huggingface_hub import HfApi

# Token will be sent in every request but not stored on machine
api = HfApi(token="hf_***")
  • Add token attribute to HfApi by @Wauplin in #1116

Stop using use_auth_token in favor of token, everywhere

token parameter can now be passed to every method in huggingface_hub. use_auth_token is still accepted where it previously existed but the mid-term goal (~6 months) is to deprecate and remove it.

  • Replace use_auth_token arg by token everywhere by @Wauplin in #1122

Respect git credential helper from the user

Previously, token was stored in the git credential store. Can now be in any helper configured by the user -keychain, cache,...-.

  • Refactor git credential handling in login workflow by @Wauplin in #1138

Better error handling

Helper to dump machine information

# Dump all relevant information. To be used when reporting an issue.
➜ huggingface-cli env

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 0.11.0.dev0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
...
  • 1173 Add dump env helper by @Wauplin in #1177

Misc

  • Cache not found is not an error by @singingwolfboy in #1101
  • Propagate error messages when multiple on BadRequest by @Wauplin in #1115
  • Add error message from x-error-message header if exists by @Wauplin in #1121

Modelcards

Few improvements/fixes in the modelcard module:

  • 🎨 make repocard content a property by @nateraw in #1147
  • ✅ fix content string in repocard tests by @nateraw in #1155
  • Add Hub verification token to evaluation metadata by @lewtun in #1142
  • Use default model_name in metadata_update by @lvwerra in #1157
  • Refer to modelcard creator app from doc by @Wauplin in #1184
  • Parent Model --> Finetuned from model by @meg-huggingface #1191
  • FIX overwriting metadata when both verified and unverified reported values by @Wauplin in #1186

Cache assets

New feature to provide a path in the cache where any downstream library can store assets (processed data, files from the web, extracted data, rendered images,...)

  • [RFC] Proposal for a way to cache files in downstream libraries by @Wauplin in #1088

Documentation updates

  • Fixing a typo in the doc. by @Narsil in #1113
  • Fix docstring of list_datasets by @albertvillanova in #1125
  • Add repo_type=dataset possibility to guide by @Wauplin in #1134
  • Fix PyTorch & Keras mixin doc by @lewtun in #1139
  • Update how-to-manage.mdx by @severo in #1150
  • Typo fix by @meg-huggingface in #1166
  • Adds link to model card metadata spec by @meg-huggingface in #1171
  • Removing "Related Models" & just asking for "Parent Model" by @meg-huggingface in #1178

Breaking changes

  • Cannot provide an organization to create_repo
  • identical_ok removed in upload_file
  • Breaking changes in arguments for validate_preupload_info, prepare_commit_payload, _upload_lfs_object (internal helpers for the commit API)
  • huggingface_hub.snapshot_download is not exposed as a public module anymore

Deprecations

  • Remove deprecated code from v0.9, v0.10 and v0.11 by @Wauplin in #1092
  • Rename languages to langage + remove duplicate code in tests by @Wauplin in #1169
  • Deprecate output in list_models by @Wauplin in #1143
  • Set back feature to create a repo when using clone_from by @Wauplin in #1187

Internal

  • Configure pytest to run on staging by default + flags in config by @Wauplin in #1093
  • fix search models test by @Wauplin in #1106
  • Add mypy in the CI (and fix existing type issues) by @Wauplin in #1097
  • Fix deprecation warnings for assertEquals in tests by @Wauplin in #1135
  • Skip failing test in ci by @Wauplin in #1148
  • 💚 fix mypy ci by @nateraw in #1167
  • Update pr docs actions by @mishig25 in #1170
  • Revert "Update pr docs actions" by @mishig25 #1192

Bugfixes & small improvements

  • Expose list_spaces by @osanseviero in #1132
  • respect NO_COLOR env var by @singingwolfboy in #1103
  • Fix list_models bool parameters by @Wauplin in #1152
  • FIX url encoding in hf_hub_url by @Wauplin in #1164
  • Fix cannot create pr on foreign repo by @Wauplin #1183
  • Fix HfApi.move_repo(...) and complete tests by @Wauplin in #1136
  • Commit empty files as regular and warn user by @Wauplin in #1180
  • Parse file size in get_hf_file_metadata by @Wauplin #1179
  • Fix get file size on lfs by @Wauplin #1188
  • More robust create relative symlink in cache by @Wauplin in #1109
  • Test running CI on Python 3.11 #1189
huggingface_hub - Patch release v0.10.1

Published by Wauplin about 2 years ago

Hot-fix to force utf-8 encoding in modelcards. See https://github.com/huggingface/huggingface_hub/pull/1102 and https://github.com/skops-dev/skops/pull/162#issuecomment-1263516507 for context.

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.10.0...v0.10.1

huggingface_hub - v0.10.0: Modelcards, cache management and more

Published by Wauplin about 2 years ago

Modelcards

Contribution from @nateraw to integrate the work done on Modelcards and DatasetCards (from nateraw/modelcards) directly in huggingface_hub.

>>> from huggingface_hub import ModelCard

>>> card = ModelCard.load('nateraw/vit-base-beans')
>>> card.data.to_dict()
{'language': 'en', 'license': 'apache-2.0', 'tags': ['generated_from_trainer', 'image-classification'],...}

Related commits

  • Add additional repo card utils from modelcards repo by @nateraw in #940
  • Add regression test for empty modelcard update by @Wauplin in #1060
  • Add template variables to dataset card template by @nateraw in #1068
  • Further clarifying Model Card sections by @meg-huggingface in #1052
  • Create modelcard if doesn't exist on update_metadata by @Wauplin in #1061

Related documentation

Cache management (huggingface-cli scan-cache and huggingface-cli delete-cache)

New commands in huggingface-cli to scan and delete parts of the cache. Goal is to manage the cache-system the same way for any dependent library that uses huggingface_hub. Only the new cache-system format is supported.

➜ huggingface-cli scan-cache
REPO ID                     REPO TYPE SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED REFS                LOCAL PATH
--------------------------- --------- ------------ -------- ------------- ------------- ------------------- -------------------------------------------------------------------------
glue                        dataset         116.3K       15 4 days ago    4 days ago    2.4.0, main, 1.17.0 /home/wauplin/.cache/huggingface/hub/datasets--glue
google/fleurs               dataset          64.9M        6 1 week ago    1 week ago    refs/pr/1, main     /home/wauplin/.cache/
(...)

Done in 0.0s. Scanned 6 repo(s) for a total of 3.4G.
Got 1 warning(s) while scanning. Use -vvv to print details.

Related commits

  • Feature: add an utility to scan cache by @Wauplin in #990
  • Utility to delete revisions by @Wauplin in #1035
  • 1025 add time details to scan cache by @Wauplin in #1045
  • Fix scan cache failure when cached snapshot is empty by @Wauplin in #1054
  • 1025 huggingface-cli delete-cache command by @Wauplin in #1046
  • Sort repos/revisions by age in delete-cache by @Wauplin in #1063

Related documentation

Better error handling (and http-related stuff)

HTTP calls to the Hub have been harmonized to behave the same across the library.

Major differences are:

  • Unified way to handle HTTP errors using hf_raise_for_status (more informative error message)
  • Auth token is always sent by default when a user is logged in (see documentation).
  • package versions are sent as user-agent header for telemetry (python, huggingface_hub, tensorflow, torch,...). It was already the case for hf_hub_download.

Related commits

  • Always send the cached token when user is logged in by @Wauplin in #1064
  • Add user agent to all requests with huggingface_hub version (and other) by @Wauplin in #1075
  • [Repository] Add better error message by @patrickvonplaten in #993
  • Clearer HTTP error messages in huggingface_hub by @Wauplin in #1019
  • Handle backoff on HTTP 503 error when pushing repeatedly by @Wauplin in #1038

Breaking changes

  1. For consistency, the return type of create_commit has been modified. This is a breaking change, but we hope the return type of this method was never used (quite recent and niche output type).
  • Return more information in create_commit output by @Wauplin in #1066
  1. Since repo_id is now validated using @validate_hf_hub_args (see below), a breaking change can be caused if repo_id was previously miused. A HFValidationError is now raised if repo_id is not valid.

Miscellaneous improvements

Add support for autocomplete

  • Add autocomplete + tests + type checking by @Wauplin in #1041

http-based push_to_hub_fastai

  • Add changes for push_to_hub_fastai to use the new http-based approach. by @nandwalritik in #1040

Check if a file is cached

  • try_to_load_from_cache returns cached non-existence by @sgugger in #1039

Get file metadata (commit hash, etag, location) without downloading

  • Add get_hf_file_metadata to fetch metadata from the Hub by @Wauplin in #1058

Validate arguments using @validate_hf_hub_args

  • Add validator for repo id + decorator to validate arguments in huggingface_hub by @Wauplin in #1029
  • Remove repo_id validation in hf_hub_url and hf_hub_download by @Wauplin in #1031

⚠️ This is a breaking change if repo_id was previously misused ⚠️

Related documentation:

Documentation updates

  • Fix raise syntax: remove markdown bullet point by @mishig25 in #1034
  • docs render tree correctly by @mishig25 in #1070

Deprecations

  • ENH Deprecate clone_from behavior by @merveenoyan in #952
  • 🗑 Deprecate token in read-only methods of HfApi in favor of use_auth_token by @SBrandeis in #928
  • Remove legacy helper 'install_lfs_in_userspace' by @Wauplin in #1059
  • 1055 deprecate private and repo type in repository class by @Wauplin in #1057

Bugfixes & small improvements

  • Consider empty subfolder as None in hf_hub_url and hf_hub_download by @Wauplin in #1021
  • enable http request retry under proxy by @MrZhengXin in #1022
  • Add securityStatus to ModelInfo object with default value None. by @Wauplin in #1026
  • 👽️ Add size parameter for lfsFiles when committing on the hub by @coyotte508 in #1048
  • Use /models/ path for api call to update settings by @Wauplin in #1049
  • Globally set git credential.helper to store in google colab by @Wauplin in #1053
  • FIX notebook login by @Wauplin in #1073

Windows-specific bug fixes

  • Fix default cache on windows by @thomwolf in #1069
  • Degraded but fully working cache-system when symlinks are not supported by @Wauplin in #1067
  • Check symlinks support per directory instead of globally by @Wauplin in #1077
huggingface_hub - Patch release v0.9.1

Published by Wauplin about 2 years ago

Hot-fix error message on gated repositories (https://github.com/huggingface/huggingface_hub/pull/1015).

Context: https://huggingface.co/CompVis/stable-diffusion-v1-4 has been widely shared in the last days but since it's a gated-repo, lots of users are getting confused by the Authentification error received. Error message is now more detailed.

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.9.0...v0.9.1

huggingface_hub - v0.9.0: Community API and new `push_to_hub` mixins

Published by Wauplin about 2 years ago

Community API

Huge work to programmatically interact with the community tab, thanks to @SBrandeis !
It is now possible to:

  • Manage discussions (create_discussion, create_pull_request, merge_pull_request, change_discussion_status, rename_discussion)
  • Comment on them (comment_discussion, edit_discussion_comment)
  • List them (get_repo_discussions, get_discussion_details)

See full documentation for more details.

  • ✨ Programmatic API for the community tab by @SBrandeis in #930

HTTP-based push_to_hub mixins

push_to_hub mixin and push_to_hub_keras have been refactored to leverage the http-endpoint. This means pushing to the hub will no longer require to first download the repo locally. Previous git-based version is planned to be supported until v0.12.

  • Push to hub mixins that do not leverage git by @LysandreJik in #847

Miscellaneous API improvements

  • ✨ parent_commit argument for create_commit and related functions by @SBrandeis in #916
  • Add a helpful error message when commit_message is empty in create_commit by @sgugger in #962
  • ✨ create_commit: more user-friendly errors on HTTP 400 by @SBrandeis in #963
  • ✨ Add files_metadata option to repo_info by @SBrandeis in #951
  • Add list_spaces to HfApi by @cakiki in #889

Miscellaneous helpers (advanced)

Filter which files to upload in upload_folder

  • Allowlist and denylist when uploading a folder by @Wauplin in #994

Non-existence of files in a repo is now cached

  • Cache non-existence of files or completeness of repo by @sgugger in #986

Progress bars can be globally disabled via the HF_HUB_DISABLE_PROGRESS_BARS env variable or using disable_progress_bars/enable_progress_bars helpers.

  • Add helpers to disable progress bars globally + tests by @Wauplin in #987

Use try_to_load_from_cache to check if a file is locally cached

  • Add utility to load files from cache by @sgugger in #980

Documentation updates

  • [Doc] Update "Download files from the Hub" doc by @julien-c in #948
  • Docs: Fix some missing images and broken links by @NimaBoscarino in #936
  • Replace upload_file with upload_folder in upload_folder docstring by @mariosasko in #927
  • Clarify upload docs by @stevhliu in #944

Bugfixes & small improvements

  • Handle redirections in hf_hub_download for a renamed repo by @Wauplin in #983
  • PR Make path_in_repo optional in upload folder by @Wauplin in #988
  • Use a finer exception when local_files_only=True and a file is missing in cache by @Wauplin in #985
  • use fixes JSONDecodeError by @Wauplin in #974
  • 🐛 Fix PR creation for a repo the user does not own by @SBrandeis in #922
  • login: tiny messaging tweak by @julien-c in #964
  • Display endpoint URL in whoami command by @juliensimon in #895
  • Small orphaned tweaks from #947 by @julien-c in #958
  • FIX LFS track fix for Hub Mixin by @merveenoyan in #919
  • 🐛 fix multilinguality test and example by @nateraw in #941
  • Fix custom handling of refined HTTPError by @osanseviero in #924
  • Followup to #901: Tweak repocard_types.py by @julien-c in #931
  • [Keras Mixin] - Flattening out nested configurations for better table parsing. by @ariG23498 in #914
  • [Keras Mixin] Rendering the Hyperparameter table vertically by @ariG23498 in #917

Internal

  • Disable codecov + configure pytest FutureWarnings by @Wauplin in #976
  • Enable coverage in CI by @Wauplin in #992
  • Enable flake8 on W605 by @Wauplin in #975
  • Enable flake8-bugbear + adapt existing codebase by @Wauplin in #967
  • Test that TensorFlow is not imported on startup by @lhoestq in #904
  • Pin black to 22.3.0 to benefit from a stable --preview flag by @LysandreJik in #934
  • Update dev version by @gante in #921
huggingface_hub - v0.8.1: lazy loading, git-aware cache file layout, new create_commit

Published by LysandreJik over 2 years ago

Git-aware cache file layout

v0.8.1 introduces a new way of caching files from the Hugging Face Hub, to two methods: snapshot_download and hf_hub_download.
The new approach is extensively documented in the Documenting files guide and we recommend checking it out to get a better understanding of how caching works.

  • New git-aware cache file layout by @julien-c in #801

New create_commit API

A new create_commit API allows users to upload and delete several files at once using HTTP-based methods. You can read more about it in this guide. The following convenience methods were also introduced:

  • upload_folder: Allows uploading a local directory to a repo.
  • delete_file allows deleting a single file from a repo.

upload_file now uses create_commit under the hood.

create_commit also allows creating pull requests with a create_pr=True flag.

None of the methods rely on Git locally.

  • New create_commit API by @SBrandeis in #888

Lazy loading

All modules will now be lazy-loaded. This should drastically reduce the time it takes to import huggingface_hub as it will no longer load all soft dependencies.

  • ENH lazy load modules in the root init by @adrinjalali in #874

Improvements and bugfixes

  • Add request ID to all requests by @LysandreJik in #909
  • Remove deprecations by @LysandreJik in #910
  • FIX Avoid creating repository when it exists on remote by @merveenoyan in #900
  • 🏗 Use hub-ci for tests by @SBrandeis in #898
  • Refine 404 errors by @LysandreJik in #878
  • Fix typo by @lsb in #902
  • FIX metadata_update: work on a copy of the upstream file, to not mess up the cache by @julien-c in #891
  • ENH Removed history writing in Keras model card by @merveenoyan in #876
  • CI enable codecov by @adrinjalali in #893
  • MNT deprecate imports from snapshot_download by @adrinjalali in #880
  • Pushback deprecation for v0.7 release by @LysandreJik in #882
  • FIX make import machinary private by @adrinjalali in #879
  • ENH Keras Use table instead of dictionary for hyperparameters in model card by @merveenoyan in #877
  • Invert deprecation for create_repo in #912
  • Constant was accidentally removed during deprecation transition in #913
huggingface_hub - v0.7.0: Repocard metadata

Published by LysandreJik over 2 years ago

Repocard metadata

This PR adds a metadata_update function that allows the user to update the metadata in a repository on the hub. The function accepts a dict with metadata (following the same pattern as the YAML in the README) and behaves as follows for all top level fields except model-index.

Examples:

Starting from

existing_results = [{
    'dataset': {'name': 'IMDb', 'type': 'imdb'},
    'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.995}],
     'task': {'name': 'Text Classification', 'type': 'text-classification'}
}]

1. Overwrite existing metric value in existing result

new_results = deepcopy(existing_results)
new_results[0]["metrics"][0]["value"] = 0.999
_update_metadata_model_index(existing_results, new_results, overwrite=True)

[{'dataset': {'name': 'IMDb', 'type': 'imdb'},
  'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.999}],
  'task': {'name': 'Text Classification', 'type': 'text-classification'}}]

2. Add new metric to existing result

new_results = deepcopy(existing_results)
new_results[0]["metrics"][0]["name"] = "Recall"
new_results[0]["metrics"][0]["type"] = "recall"

[{'dataset': {'name': 'IMDb', 'type': 'imdb'},
  'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.995},
              {'name': 'Recall', 'type': 'recall', 'value': 0.995}],
  'task': {'name': 'Text Classification', 'type': 'text-classification'}}]

3. Add new result

new_results = deepcopy(existing_results)
new_results[0]["dataset"] = {'name': 'IMDb-2', 'type': 'imdb_2'}

[{'dataset': {'name': 'IMDb', 'type': 'imdb'},
  'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.995}],
  'task': {'name': 'Text Classification', 'type': 'text-classification'}},
 {'dataset': ({'name': 'IMDb-2', 'type': 'imdb_2'},),
  'metrics': [{'name': 'Accuracy', 'type': 'accuracy', 'value': 0.995}],
  'task': {'name': 'Text Classification', 'type': 'text-classification'}}]
  • ENH Add update metadata to repocard by @lvwerra in #844

Improvements and bug fixes

  • Keras: Saving history in a JSON file by @merveenoyan in #861
  • space after uri by @leondz in #866
huggingface_hub - v0.6.0: fastai support, binary file support, skip LFS files when pushing to the hub

Published by LysandreJik over 2 years ago

Disclaimer: This release was initially released with advertised support for #844. It was not released in this release and will be in v0.7.

fastai support

v0.6.0 introduces downstream (download) and upstream (upload) support for the fastai libraries. It supports fastai versions above 2.4.
The integration is detailed in the following blog.

  • Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions by @omarespejel in #678

Automatic binary file tracking in Repository

Binary files are now rejected by default by the Hub. v0.6.0 introduces automatic binary file tracking through the auto_lfs_track argument of the Repository.git_add method. It also introduces the Repository.auto_track_binary_files method which can be used independently of other methods.

  • ENH Auto track binary files in Repository by @LysandreJik in #828

skip_lfs_file is now added to mixins

The parameter skip_lfs_files is now added to the different mixins. This will enable pushing files to the hub without first downloading the files above 10MB. This should drammatically reduce the time needed when updating a modelcard, a configuration file, and others.

  • ✨ add skip_lfs_files to mixins' push_to_hub by @nateraw in #858

Keras support improvement

The support for Keras model is greatly improved through several additions:

  • The save_pretrained_keras method now accepts a list of tags that will automatically be added to the repository.
  • Download statistics are now available on Keras models
  • Introducing list of tags to Keras model card by @merveenoyan in #806
  • Enable keras download stats by @merveenoyan in #860

Bugfixes and improvements

  • FIX don't raise if name/organizaiton are passed postionally by @adrinjalali in #822
  • ENH Use provided token from HUGGING_FACE_HUB_TOKEN env variable if available by @FrancescoSaverioZuppichini in #794
  • tests(hf_api): remove infectionTypes field by @McPatate in #834
  • Remove docs, tasks and inference API from huggingface_hub by @osanseviero in #833
  • FEAT Uniformize hf_api a bit and add support for Spaces by @julien-c in #792
  • Add a bug report template by @osanseviero in #832
  • clean up formatting by @stevhliu in #839
  • Release guide by @LysandreJik in #820
  • Fix keras test by @osanseviero in #855
  • DOC Add quick start guide by @stevhliu in #850
  • MNT refactor: subprocess.run -> run_subprocess by @LysandreJik in #352
  • MNT enable preview on black by @adrinjalali in #849
  • Update how to guides by @stevhliu in #840
  • Update contribution guide for merging PRs by @stevhliu in #856
  • DOC Update landing page by @stevhliu in #854
  • space after uri by @leondz in #866
huggingface_hub - v0.5.1: Patch release

Published by LysandreJik over 2 years ago

This is a patch release fixing a breaking backward compatibility issue.

Linked PR: https://github.com/huggingface/huggingface_hub/pull/822

huggingface_hub - v0.5.0: Reference documentation, Keras improvements, stabilizing the API

Published by LysandreJik over 2 years ago

Documentation

Version v0.5.0 is the first version which features an API reference. It is still a work in progress with features lacking, some images not rendering, and a documentation reorg coming up, but should already provide significantly simpler access to the huggingface_hub API.

The documentation is visible here.

Model & datasets list improvements

The list_models and list_datasets methods have been improved in several ways.

List private models

These two methods now accept the token keyword to specify your token. Specifying the token will include your private models and datasets in the returned list.

Modelcard metadata

These two methods now accept the cardData boolean argument. If set to True, the modelcard metadata will also be returned when using these two methods.

Filtering by carbon emissions

The list_models method now also accepts an emissions_trehsholds parameter to filter by carbon emissions.

Keras improvements

The Keras serialization and upload methods have been worked on to provide better support for models:

  • All parameters are now included in the saved model when using push_to_hub_keras
  • log_dir parameter for TensorBoard logs, which will automatically spawn a TensorBoard instance on the Hub.
  • Automatic model card

Contributing guide

A contributing guide is now available for the huggingface_hub repository. For any and all information related to contributing to the repository, please check it out!

Read more about it here: CONTRIBUTING.md.

Pre-commit hooks

The huggingface_hub GitHub repository has several checks to ensure that the code respects code quality standards. Opt-in pre-commit hooks have been added in order to make it simpler for contributors to leverage them.

Read more about it in the aforementionned CONTRIBUTING guide.

Renaming and transferring repositories

Repositories can now be renamed and transferred programmatically using move_repo.

Breaking changes & deprecation

⛔ The following methods have now been removed following a deprecation cycle

list_repos_objs

The list_repos_objs and the accompanying CLI utility huggingface-cli repo ls-files have been removed.
The same can be done using the model_info and dataset_info methods.

Python 3.6

Python 3.6 support is now dropped as end of life. Using Python 3.6 and installing huggingface_hub will result in version v0.4.0 being installed.

⚠️ Items below are now deprecated and will be removed in a future version

What's Changed

New Contributors

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.4.0...v0.5.0

huggingface_hub - v0.4.0: Tag listing, Namespace Objects, Model Filter

Published by LysandreJik over 2 years ago

Tag listing

This PR introduces the ability to fetch all available tags for models or datasets and returns them as a nested namespace object, for example:

>>> from huggingface_hub import HfApi

>>> api = HfApi() 
>>> tags = api.get_model_tags()
>>> print(tags)
Available Attributes:
 * benchmark
 * language_creators
 * languages
 * licenses
 * multilinguality
 * size_categories
 * task_categories
 * task_ids

>>> print(tags.benchmark)
Available Attributes:
 * raft
 * superb
 * test

Namespace objects

With a goal of adding more tab-completion to the library, this PR introduces two objects:

  • DatasetSearchArguments
  • ModelSearchArguments

These two AttributeDictionary objects contain all the valid information we can extract from a model as tab-complete parameters. We also include the author_or_organization and dataset (or model) _name as well through careful string splitting.

Model Filter

This PR introduces a new way to search the hub: the ModelFilter class.

It is a simple Enum at first to the user, allowing them to specify what they want to search for, such as:

f = ModelFilter(author="microsoft", model_name="wavlm-base-sd", framework="pytorch")

From there, they can pass in this filter to the new list_models_by_filter function in HfApi to search through it:

models = api.list_modes(filter=f)

The API may then be used for complex queries:

args = ModelSearchArguments()
f = ModelFilter(framework=[args.library.pytorch, args.library.TensorFlow], model_name="bert", tasks=[args.pipeline_tag.Summarization, args.pipeline_tag.TokenClassification])

api.list_models_from_filter(f)

Ignoring filenames in snapshot_download

This PR introduces a way to limit the files that will be fetched by the snapshot_download. This is useful when you want to download and cache an entire repository without using git, and that you want to skip files according to their filenames.

What's Changed

New Contributors

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.2.1...v0.4.0

huggingface_hub - v0.2.1: Patch release

Published by LysandreJik over 2 years ago

This is a patch release fixing an issue with the notebook login.

https://github.com/huggingface/huggingface_hub/commit/5e2da9bae95ed4c99683e9572ecc32c9e0da5e15#diff-fb1696cbcf008dd89dde5e8c1da9d4be5a8f7d809bc32f07d4453caba40df15f

huggingface_hub - v0.2.0: Access tokens, skip large files, local files only

Published by LysandreJik over 2 years ago

Access tokens

Version v0.2.0 introduces the access token compatibility with the hub. It offers the access tokens as the main login handler, with the possibility to still login with username/password when doing [Ctrl/CMD]+C on the login prompt:

image

The notebook login is adapted to work with the access tokens.

Skipping large files

The Repository class now has an additional parameter, skip_lfs_files, which allows cloning the repository while skipping the large file download.

https://github.com/huggingface/huggingface_hub/pull/472

Local files only for snapshot_download

The snapshot_download method can now take local_files_only as a parameter to enable leveraging previously downloaded files.

https://github.com/huggingface/huggingface_hub/pull/505

huggingface_hub - v0.1.2: Patch release

Published by LysandreJik almost 3 years ago

What's Changed

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.1.1...v0.1.2

huggingface_hub - v0.1.1: Patch release

Published by LysandreJik almost 3 years ago

What's Changed

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.1.0...v0.1.1

huggingface_hub - v0.1.0: Optional token, `HfApi` begone, git prune

Published by LysandreJik almost 3 years ago

What's Changed

Version v0.1.0 is the first minor release of the huggingface_hub package, which promises better stability for the incoming versions. This update comes with big quality of life improvements.

Make token optional in all HfApi methods. by @sgugger in https://github.com/huggingface/huggingface_hub/pull/379

Previously, most methods of the HfApi class required the token to be explicitly passed. This is changed in this version, where it defaults to the token stored in the cache. This results in a re-ordering of arguments, but backward compatibility is preserved in most cases. Where it is not preserved, an explicit error is thrown.

Root methods instead of HfApi by @LysandreJik in https://github.com/huggingface/huggingface_hub/pull/388

The HfApi class now exposes its methods through the hf_api file, reducing the friction to access these helpers. See the example below:

# Previously
from huggingface_hub import HfApi

api = HfApi()
user = api.whoami()

# Now
from huggingface_hub.hf_api import whoami

user = whoami()

The HfApi can still be imported and works as before for backward compatibility.

Add list_repo_files util by @sgugger in https://github.com/huggingface/huggingface_hub/pull/395

Offers a list_repo_files to ... list the repo files! Supports both model repositories and dataset repositories

Add helper to generate an eval result model-index, with proper typing by @julien-c in https://github.com/huggingface/huggingface_hub/pull/382

Offers a metadata_eval_result in order to generate a YAML block to put in model cards according to evaluation results.

Add metrics to API by @mariosasko in https://github.com/huggingface/huggingface_hub/pull/429

Adds a list_metrics method to HfApi!

Git prune by @LysandreJik in https://github.com/huggingface/huggingface_hub/pull/450

Adds a git_prune method to the Repository class. This prunes local files which are unneeded as already pushed to a remote.
It adds the argument auto_lfs_prune to git_push and the commit context-manager for simpler handling.

Bug fixes

Full Changelog: https://github.com/huggingface/huggingface_hub/compare/v0.0.19...v0.1.0

Package Rankings
Top 1.13% on Pypi.org
Top 8.01% on Spack.io
Top 5.78% on Proxy.golang.org
Top 9.61% on Conda-forge.org
Top 21.07% on Anaconda.org
Top 13.04% on Npmjs.org
Related Projects
GemGPT

Explore the power of Gemma model with GemGPT, a project leveraging AI for innovative solutions. J...

25 Mar 2024 2
mlreef

The collaboration workspace for Machine Learning

20 Jul 2020 1,442
xllm

🦖 X—LLM: Cutting Edge & Easy LLM Finetuning

10 Nov 2023 375
wandb

The AI developer platform. Use Weights & Biases to train and fine-tune models, and manage models ...

24 Mar 2017 8,945
superduper

🔮 SuperDuper: Bring AI to your database! Build, deploy and manage any AI application directly wit...

30 Aug 2022 4,548
higgsfield

Fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for ...

26 May 2018 3,264