lgloege / zenodopy Goto Github PK

View Code? Open in Web Editor NEW

15.0 2.0 6.0 45 KB

Manage Zenodo projects with Python

License: MIT License

Python 100.00%

zenodopy's Introduction

zenodopy

Project under active deveopment, not production ready

A Python 3.6+ package to manage Zenodo repositories.

Functions Implemented

.create_project(): create a new project
.upload_file(): upload file to project
.download_file(): download a file from a project
.delete_file(): permanently removes a file from a project
.get_urls_from_doi(): returns the files urls for a given doi

Installing

PyPi

pip install zenodopy==0.3.0

GitHub

pip install -e git+https://github.com/lgloege/zenodopy.git#egg=zenodopy

Using the Package

Create a Zenodo access token by first logging into your account and clicking on your username in the top right corner. Navigate to "Applications" and then "+new token" under "Personal access tokens". Keep this window open while you proceed to step 2 because the token is only displayed once. Note that Sandbox.zenodo is used for testing and zenodo for production. If you want to use both, create for each a token as desribed above.
Store the token in ~/.zenodo_token using the following command.

# zenodo token
 { echo 'ACCESS_TOKEN: your_access_token_here' } > ~/.zenodo_token

# sandbox.zenodo token
 { echo 'ACCESS_TOKEN-sandbox: your_access_token_here' } > ~/.zenodo_token

start using the zenodopy package

import zenodopy

# always start by creating a Client object
zeno = zenodopy.Client(sandbox=True)

# list project id's associated to zenodo account
zeno.list_projects

# create a project
zeno.create_project(title="test_project", upload_type="other")
# your zeno object now points to this newly created project

# create a file to upload
with open("~/test_file.txt", "w+") as f:
    f.write("Hello from zenodopy")

# upload file to zenodo
zeno.upload_file("~/test.file.txt")

# list files of project
zeno.list_files

# set project to other project id's
zeno.set_project("<id>")

# delete project
zeno._delete_project(dep_id="<id>")

Notes

This project is under active development. Here is a list of things that needs improvement:

more tests: need to test uploading and downloading files
documentation: need to setup a readthedocs
download based on DOI: right now you can only download from your own projects. Would be nice to download from
asyncronous functions: use asyncio and aiohttp to write async functions. This will speed up downloading multiple files.

zenodopy's People

Contributors

Stargazers

Watchers

Forkers

geosinolmu aaristov longavailable avivajpeyi pz-max thomaswong2022

zenodopy's Issues

Improved tests

The tests need a lot of improvements. There needs to be a way to test the functinality without making the acess token visible to everbody

No documentation

Make the project documentation available via read the docs

Modernize the code to include type hints

Consider using type hints to make the code easier to understand for new developers

Allow location where to save downloaded files

Is your feature request related to a problem? Please describe.
It would be nice to specify where to download the files

Describe the solution you'd like
Add dst_path keyword to download_file()

Additional context
Not related, but do you know why the requests.get step is VERY slow? Does it depend on Zenodo server?

Download from DOI

Is your feature request related to a problem? Please describe.
The ability to download zenodo content not only from your own project, but from any public zenodo DOI.

Describe alternatives you've considered
The zenodoget project has this implemented.

tar files recursively inside dir and upload

Hi, nice project!
I use the pattern: tar all files within some dir and then upload tar file. It would be nice to do it in a single call.

Here's the code I used for now:

https://www.tutorialspoint.com/How-to-create-a-tar-file-using-Python#:~:text=Use%20the%20tarfile%20module%20to,the%20files%20in%20it%20recursively.

import os
import tarfile
def tardir(path, tar_name):
    with tarfile.open(tar_name, "w:gz") as tar_handle:
        for root, dirs, files in os.walk(path):
            for file in files:
                tar_handle.add(os.path.join(root, file))

tardir('images', 'images.tar.gz')


import zenodopy
# always start by creating a Client object
zeno = zenodopy.Client()
zeno.set_project(dep_id)
zeno.upload_file("images.tar.gz")

would be easier just having

import zenodopy
zeno = zenodopy.Client()
zeno.set_project(dep_id)
zeno.upload_dir('images/')

Project not created, something went wrong

Hi all,

Thanks for this nice tool. I went to set this up, but couldn't debug the error.

import zenodopy

# always start by creating a Client object
zeno = zenodopy.Client()

# list project id's associated to zenodo account
zeno.list_projects

# create a project
zeno.create_project(title="test_project", upload_type="other")

yields

>>> zeno.list_projects
Project Name ---- ID
------------------------
 ----- 7921375
Capturing long-tailed individual tree diversity using an airborne multi-temporal hierarchical model ----- 7301868
weecology/DeepTreeAttention: Submission ----- 7308745
Data for the NeonTreeEvaluation Benchmark ----- 5914554
A global model of bird detection in high resolution airborne images using computer vision ----- 5033174
weecology/NeonTreeEvaluation: New naming format ----- 4770593
One year of hourly temperature readings along a elevation gradient for a tropical forest in Maquipucuna Ecuador (1300-2400m) ----- 4560522
NEON Tree Crowns Dataset ----- 3765872
DeepForest - Street Trees Dataset ----- 4047083
bw4sz/DeepMeerkat: v0.0.7 ----- 1217119
>>> 
>>> # create a project
>>> zeno.create_project(title="test_project", upload_type="other")
** Project not created, something went wrong. Check that your ACCESS_TOKEN is in ~/.zenodo_token

The token has to be correct, since list_projects correctly finds my account. To prove this, if you edit the token in anyway, it doesn't what account I'm pointing at. This is a new zenodo key, maybe it takes some time? It has all scopes.

token file has no quotes or other characters.

(DeepTreeAttention) [b.weinstein@c1107a-s35 DeepTreeAttention]$ cat ~/.zenodo_token 
ACCESS_TOKEN: ############################

Failure to upload large file

Describe the bug
Failure to upload a large file (2.3 GB)

To Reproduce

>>> import zenodopy
>>> zeno = zenodopy.Client()
>>> zeno.set_project("10006217")
>>> zeno.upload_file("/Users/fx/Downloads/crystal_300K_volume_change0.tar")
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2426)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/fx/miniforge3/envs/scripts/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/fx/miniforge3/envs/scripts/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/Users/fx/miniforge3/envs/scripts/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /api/files/56b80de7-c506-4c50-b918-8571eb6d0601/crystal_300K_volume_change0.tar (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

where the file is a 2.3 GB data file.

Expected behavior
Upload should succeed.

Desktop (please complete the following information):

OS: macOS 13
Browser: not relevant
Version: zenodopy-0.3.0