ecmwf / multiurl Goto Github PK
View Code? Open in Web Editor NEWA package to download several URL as one, as well as supporting multi-part URLs.
License: Apache License 2.0
A package to download several URL as one, as well as supporting multi-part URLs.
License: Apache License 2.0
On PyPi, this package is only published with its source files, but not as a Python wheel. Therefore, installing the package requires running the build step locally. On platforms where this is not (easily) possible, e.g. inside a Pyodide environment, this limitation precludes using multiurl
.
This package is a pure Python package which does not depend on any platform-dependent C code. Therefore, a pure Python wheel (one that ends in *py3-none-any.whl
) can be published to PyPi. This published wheel can then be downloaded to install multiurl
without running the build script locally. This is useful, e.g. to use the package inside Pyodide.
This should be as simple as switching the build step from
pip install setuptools wheel twine
python setup.py sdist
to
pip install build twine
python -m build
No response
No response
University of Helsinki
multiurl fails to download from ftp servers if the SIZE command is not understood, as is the case when the ftp server is in ASCII mode. The size information is only used to construct a progress bar, therefore this is non essential for functionality.
(This may be a misconfiguration server side, however failing to download because we can't build a progress bar is not ideal)
import multiurl
url="ftp://USERNAME:[email protected]/sl-c3s/Products/... .../dt_global_twosat_phy_l4_199301_vDT2021-M01.nc"
path = "temp.nc"
multiurl.download(url, path)
Please contact me offline for the full URL, USERNAME and PASSWORD.
0.2.1
MacOS, Ubuntu
---------------------------------------------------------------------------
error_perm Traceback (most recent call last)
Cell In[4], line 1
----> 1 multiurl.download(url, path)
File ~/miniconda3/envs/CDS/lib/python3.10/site-packages/multiurl/downloader.py:111, in download(url, target, **kwargs)
110 def download(url, target, **kwargs):
--> 111 return Downloader(url, **kwargs).download(target)
File ~/miniconda3/envs/CDS/lib/python3.10/site-packages/multiurl/base.py:119, in DownloaderBase.download(self, target)
115 download = target
117 LOG.info("Downloading %s", self.url)
--> 119 size, mode, skip, trust_size = self.estimate_size(download)
121 with self.progress_bar(
122 total=size,
123 initial=skip,
124 desc=self.title(),
125 ) as pbar:
126 with open(download, mode) as f:
File ~/miniconda3/envs/CDS/lib/python3.10/site-packages/multiurl/ftp.py:51, in FTPDownloaderBase.estimate_size(self, target)
48 self.filename = os.path.basename(o.path)
49 self.ftp = ftp
---> 51 return (ftp.size(self.filename), "wb", 0, True)
File ~/miniconda3/envs/CDS/lib/python3.10/ftplib.py:630, in FTP.size(self, filename)
628 '''Retrieve the size of a file.'''
629 # The SIZE command is defined in RFC-3659
--> 630 resp = self.sendcmd('SIZE ' + filename)
631 if resp[:3] == '213':
632 s = resp[3:].strip()
File ~/miniconda3/envs/CDS/lib/python3.10/ftplib.py:281, in FTP.sendcmd(self, cmd)
279 '''Send a command and return the response.'''
280 self.putcmd(cmd)
--> 281 return self.getresp()
File ~/miniconda3/envs/CDS/lib/python3.10/ftplib.py:254, in FTP.getresp(self)
252 raise error_temp(resp)
253 if c == '5':
--> 254 raise error_perm(resp)
255 raise error_proto(resp)
error_perm: 550 SIZE not allowed in ASCII mode
### Accompanying data
_No response_
### Organisation
ECMWF
The progress bar is always active, and there is no easy way to disable it, as the call to
tqdm contains "disable=False".
Omitting this argument seems to do exactly what I would like to have: "If set to None, disable on non-TTY".
It would be great if there would be a way to disable the progress bar in non-TTY situations.
No response
No response
No response
No response
Trigger the CI actions
Release a version
0.2.4
NA
No response
No response
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.