explosion / spacy-huggingface-hub Goto Github PK

🤗 Push your spaCy pipelines to the Hugging Face Hub

License: MIT License

Python 100.00%

spacy huggingface models ml-models natural-language-processing nlp machine-learning

spacy-huggingface-hub's Introduction

spacy-huggingface-hub: Push your spaCy pipelines to the Hugging Face Hub

This package provides a CLI command for uploading any trained spaCy pipeline packaged with spacy package to the Hugging Face Hub. It auto-generates all meta information for you, uploads a pretty README (requires spaCy v3.1+) and handles version control under the hood.

🤗 About the Hugging Face Hub

The Hugging Face Hub hosts Git-based repositories which are storage spaces that can contain all your files. These repositories have multiple advantages: versioning (commit history and diffs), branches, useful metadata about their tasks, languages, metrics and more, browser-based visualizers to explore the models interactively in your browser, as well as an API to use the models in production.

🚀 Quickstart

You can install spacy-huggingface-hub from pip:

pip install spacy-huggingface-hub

To check if the command has been registered successfully:

python -m spacy huggingface-hub --help

You can upload any pipeline packaged with spacy package. Make sure to set --build wheel to output a binary .whl file. The uploader will read all metadata from the pipeline package, including the auto-generated pretty README.md and the model details available in the meta.json.

huggingface-cli login
python -m spacy package ./en_ner_fashion ./output --build wheel
cd ./output/en_ner_fashion-0.0.0/dist
python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl

The command will output two things:

Where to find your repo in the Hub! For example, https://huggingface.co/spacy/en_core_web_sm
And how to install the pipeline directly from the Hub!

pip install https://huggingface.co/spacy/en_core_web_sm/resolve/main/en_core_web_sm-any-py3-none-any.whl

Now you can share your pipelines very quickly with others. Additionally, you can also test your pipeline directly in the browser!

⚙️ Usage and API

If spaCy is already installed in the same environment, this package automatically adds the spacy huggingface-hub commands to the CLI. If you don't have spaCy installed, you can also execute the CLI directly via the package.

`push`

python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]

python -m spacy_huggingface_hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]

Argument	Type	Description
`whl_path`	str / `Path`	The path to the `.whl` file packaged with `spacy package`.
`--org`, `-o`	str	Optional name of organization to which the pipeline should be uploaded.
`--msg`, `-m`	str	Commit message to use for update. Defaults to `"Update spaCy pipeline"`.
`--verbose`, `-V`	bool	Output additional info for debugging, e.g. the full generated hub metadata.

Usage from Python

Instead of using the CLI, you can also call the push function from Python. It returns a dictionary containing the "url" of the published model and the "whl_url" of the wheel file, which you can install with pip install

from spacy_huggingface_hub import push

result = push("./en_ner_fashion-0.0.0-py3-none-any.whl")
print(result["url"])

spacy-huggingface-hub's People

Contributors

Stargazers

Watchers

Forkers

thomashacker trendingtechnology playfloor adrianeboyd psraju123 aaroncwacker mirfan899 ruthalsop2 duckheada daniel-sdn statelesshz ltcevil riturr youcefs21 diyclassics

spacy-huggingface-hub's Issues

Problem installing

I have upload a model to HF hub, but I can not installed with

!pip install https://huggingface.co/Alvaro8gb/es_BreastCancerNER/resolve/main/es_BreastCancerNER-any-py3-none-any.whl

An error appears:

ERROR: Invalid requirement: 'es-BreastCancerNER==any': Expected end or semicolon (after name and no valid version specifier) es-BreastCancerNER==any

update huggingface-hub version

Version conflicts using the latest transformers.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
transformers 4.26.0 requires huggingface-hub<1.0,>=0.11.0, but you have huggingface-hub 0.8.1 which is incompatible.
Successfully installed huggingface-hub-0.8.1

can you update the version???

HTTP Error 400 when pushing model to HuggingFace hub

Hello,

I'm not quite sure if this issue is related to #5.

When I'm trying to push a model on Hugging Face Hub organisation with spaCy CLI:

python -m spacy huggingface-hub push fr_core_ner4archives_v3_default-0.0.0-py3-none-any.whl -o ner4archives -V

This raises an HTTP 400 error:

Pushing repository to the hub...
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/spacy_huggingface_hub/push.py", line 53, in huggingface_hub_push_cli
    push(whl_path, organization, commit_msg, verbose=verbose)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/spacy_huggingface_hub/push.py", line 130, in push
    url = upload_folder(
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 2109, in upload_folder
    pr_url = self.create_commit(
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1844, in create_commit
    _raise_for_status(commit_resp)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 84, in _raise_for_status
    _raise_with_request_id(request)
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 95, in _raise_with_request_id
    raise e
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 90, in _raise_with_request_id
    request.raise_for_status()
  File "/home/lterriel/Documents/dev/almanach-projects/N4A_project/Training_pipelines/venv/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/models/lterriel/fr_core_ner4archives_v3_default/commit/main (Request ID: QTOTcNAiXjAsn45fwTKLO)

However, the new model repository is well created in Hugging Face Hub organisation but without the model (files) and its card.

packages installed:

spaCy==3.3.1
spacy-huggingface-hub==0.0.7
huggingface-hub==0.8.1

"-" in the name of the model

I have noticed that you split by "-" in extracting the name and the version of the model, I am writing this issue just to inform that using "-" in the name of the model will fail to upload.

argument for adding a model card

It would be great to be able to pre-generate, (edit), and attach a model card.

I have created a model, pushed it, edited the card, then had to push it again, and quite logically it just over-wrote the edited model card. Even without full pull functionality, it would be great to be able to attach edited cards to avoid the need to re-do this job every time manually.

uploading a private model

the default behaviour of the package seems to be to publicly upload the model, how would I go about making it private by default instead of having to manually change it on hugging face?

License problem when pushing model

Hi,
I am trying to push my spaCy model to the Huggingface Hub and I get the following error:

Traceback (most recent call last):
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/huggingface_hub/repository.py", line 412, in git_push
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'push']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/spacy_huggingface_hub/push.py", line 53, in huggingface_hub_push_cli
    push(whl_path, organization, commit_msg, local_repo_path, verbose=verbose)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/spacy_huggingface_hub/push.py", line 131, in push
    url = repo.push_to_hub(commit_message=commit_msg)
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/huggingface_hub/repository.py", line 434, in push_to_hub
    return self.git_push()
  File "/home/elena/Workspace/Spacy_project/.env3/lib/python3.8/site-packages/huggingface_hub/repository.py", line 422, in git_push
    raise EnvironmentError(exc.stderr)
OSError: remote: ----------------------------------------------------------        
remote: Sorry, your push was rejected:        
remote: - error : yaml metadata schema issue on key "license"        
remote: help: key: /license must be equal to one of the allowed values (error rule: enum), please find the license identifier that fits you at https://huggingface.co/docs/hub/model-repos#list-of-license-identifiers        
remote: ----------------------------------------------------------        
remote: Please find the documentation at:        
remote: https://huggingface.co/docs/hub/model-repos        
remote: ----------------------------------------------------------        
To https://huggingface.co/nucci/sv_pipeline
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://huggingface.co/nucci/sv_pipeline'

It seems to be a license problem, so I put the identifier corresponding to the license that I want to give my model in meta.json, but it doesn't seem to solve the problem. Any thoughts?

Versioning models

It would be awesome, if this tool could be used to "git tag" models, so that they each new model release could be associated with a version number. Do you have plans for supporting this functionality?

Some background: the tool's 0.0.6 version created a temporary directory for the git repository and this behavior allowed us to tag each new HuSpaCy model release with its own version.

Error when pushing to huggingface hub

Hello,

I have a problem using this library. When I try to run the command to upload the whl file:

python -m spacy huggingface-hub push en_acnl_electra_pipeline-0.0.1-py3-none-any.whl

The error occurs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/huggingface_hub/repository.py", line 365, in git_pull
    cwd=self.local_dir,
  File "/usr/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'pull', '--rebase']' returned non-zero exit status 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy_huggingface_hub/push.py", line 53, in huggingface_hub_push_cli
    push(whl_path, organization, commit_msg, local_repo_path, verbose=verbose)
  File "/usr/local/lib/python3.7/dist-packages/spacy_huggingface_hub/push.py", line 91, in push
    repo.git_pull(rebase=True)
  File "/usr/local/lib/python3.7/dist-packages/huggingface_hub/repository.py", line 368, in git_pull
    raise EnvironmentError(exc.stderr)
OSError: error: cannot pull with rebase: Your index contains uncommitted changes.
error: please commit or stash them.

I'm using:

============================== Info about spaCy ==============================

spaCy version 3.1.3
Location /usr/local/lib/python3.7/dist-packages/spacy
Platform Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Python version 3.7.12
Pipelines