Comments (9)
This is a MbedTLS.jl issue.
You should open an issue on their repo.
I think it might have already been fixed as it sounds similar to something I heard was fixed recently, so do a package update if you've not done one recently.
A workaround is to define a custom fetch method that uses Base.download
should be stuff in docs about that.
I should extend that bit of the docs with an example
from datadeps.jl.
It seems there is an issue already (link) to which I added a 👍
You mean that Base.download
could be added with a try/catch somewhere? Could you explain (or point out to where I can find the info) why DataDeps uses something else (HTTPS.jl?) in the first place?
from datadeps.jl.
We originally used Base.download
, but the problem is that determining filenames for things from HTTP is actually insane, and often has to be done using header information.
If you go back digging into the issues (around #22) you can see how complicated that is using commandline tools,
and it just wasn't robust enough on Windows, or differing unixen.
It also wasn't robust against different servers that might not support the HEAD
command.
The better way to do it is as part of the GET
, to get headers and content at same time.
Anyway the simple case, for when you don't need a header as part of the filename is
function fallback_download(remotepath, localdir)
@assert(isdir(localdir))
filename = basename(remotepath) # only works for URLs with filename as last part of name
localpath = joinpath(localdir, filename)
Base.download(remotepath, localpath)
return localpath
end
register(DataDep( ...
fetch_method = fallback_download
)
from datadeps.jl.
Thanks for the explanation!
Just to be sure I understand: Is the fallback_download
function in DataDeps.jl at the moment? I.e., can I use the fetch_method = fallback_download
directly? Or do I have to include the fallback_download
function definition in my code?
from datadeps.jl.
It is not in DataDeps.jl right now.
I might add it to the docs at some point as an example of how to define your own,
from datadeps.jl.
Is there a way for to write the same type of fallback (that uses Base.download
) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)
from datadeps.jl.
Oh and for the record, I just tried it now and got an UndefVarError: update_period not defined
error. So I removed the update_period
part — that is, instead, I used this
function fallback_download(remotepath, localdir)
@assert(isdir(localdir))
filename = basename(remotepath) # only works for URLs with filename as last part of name
localpath = joinpath(localdir, filename)
Base.download(remotepath, localpath)
return localpath
end
which seems to work 😄
from datadeps.jl.
For the (my) record also, note that instead of the clickable URL that FigShare gives you, you can instead use a URL of the type http://files.figshare.com/<Project/file number>/<file name>
. Useful if you want to use
filename = basename(remotepath) # only works for URLs with filename as last part of name
from datadeps.jl.
Is there a way for to write the same type of fallback (that uses
Base.download
) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)
See around #22 but it is really unrobust.
Oh and for the record, I just tried it now and got an
UndefVarError: update_period not defined
error. So I removed theupdate_period
part
Sorry, that was a typo on my part. Your code is the correct one.
from datadeps.jl.
Related Issues (20)
- Documentation badge links do not work HOT 5
- ERROR: HTTP.ExceptionRequest.StatusError HOT 1
- DataDeps.jl behavor in CI when a file cannot be read HOT 3
- Ability to silence 7zip output HOT 3
- datadep hanging HOT 1
- Closer integration with DrWatson HOT 2
- Better errors when the host is down HOT 2
- Garbage collection functionality HOT 7
- Error when using paths containing tilde HOT 1
- DataDeps doesn't build on aarch64-apple-darwin HOT 1
- display download progress bar HOT 2
- failed to install the latest DataDeps version in Julia-1.9 dev HOT 2
- Use case: nested data deps? HOT 2
- Documentation website is outdated, `gh-pages` is not being updated HOT 1
- Flush(stdout) before checking if accept terms
- Experience with Dropbox HOT 1
- Nondeterministic error in CI saying there is no writable folder in the load path HOT 5
- Change to scratch breaking? HOT 1
- At some point after version 0.7.0, DataDeps.jl broke WordTokenizers.jl HOT 3
- Error during test HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datadeps.jl.