Giter Club home page Giter Club logo

Comments (9)

oxinabox avatar oxinabox commented on June 1, 2024

This is a MbedTLS.jl issue.
You should open an issue on their repo.

I think it might have already been fixed as it sounds similar to something I heard was fixed recently, so do a package update if you've not done one recently.

A workaround is to define a custom fetch method that uses Base.download should be stuff in docs about that.
I should extend that bit of the docs with an example

from datadeps.jl.

briochemc avatar briochemc commented on June 1, 2024

It seems there is an issue already (link) to which I added a 👍

You mean that Base.download could be added with a try/catch somewhere? Could you explain (or point out to where I can find the info) why DataDeps uses something else (HTTPS.jl?) in the first place?

from datadeps.jl.

oxinabox avatar oxinabox commented on June 1, 2024

We originally used Base.download, but the problem is that determining filenames for things from HTTP is actually insane, and often has to be done using header information.
If you go back digging into the issues (around #22) you can see how complicated that is using commandline tools,
and it just wasn't robust enough on Windows, or differing unixen.
It also wasn't robust against different servers that might not support the HEAD command.
The better way to do it is as part of the GET, to get headers and content at same time.

Anyway the simple case, for when you don't need a header as part of the filename is

function fallback_download(remotepath, localdir)
    @assert(isdir(localdir))
    filename = basename(remotepath)  # only works for URLs with filename as last part of name
    localpath = joinpath(localdir, filename)
    Base.download(remotepath, localpath)
    return localpath
end


register(DataDep( ...
     fetch_method = fallback_download
)

from datadeps.jl.

briochemc avatar briochemc commented on June 1, 2024

Thanks for the explanation!

Just to be sure I understand: Is the fallback_download function in DataDeps.jl at the moment? I.e., can I use the fetch_method = fallback_download directly? Or do I have to include the fallback_download function definition in my code?

from datadeps.jl.

oxinabox avatar oxinabox commented on June 1, 2024

It is not in DataDeps.jl right now.

I might add it to the docs at some point as an example of how to define your own,

from datadeps.jl.

briochemc avatar briochemc commented on June 1, 2024

Is there a way for to write the same type of fallback (that uses Base.download) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)

from datadeps.jl.

briochemc avatar briochemc commented on June 1, 2024

Oh and for the record, I just tried it now and got an UndefVarError: update_period not defined error. So I removed the update_period part — that is, instead, I used this

function fallback_download(remotepath, localdir)
    @assert(isdir(localdir))
    filename = basename(remotepath)  # only works for URLs with filename as last part of name
    localpath = joinpath(localdir, filename)
    Base.download(remotepath, localpath)
    return localpath
end

which seems to work 😄

from datadeps.jl.

briochemc avatar briochemc commented on June 1, 2024

For the (my) record also, note that instead of the clickable URL that FigShare gives you, you can instead use a URL of the type http://files.figshare.com/<Project/file number>/<file name>. Useful if you want to use

filename = basename(remotepath)  # only works for URLs with filename as last part of name

from datadeps.jl.

oxinabox avatar oxinabox commented on June 1, 2024

Is there a way for to write the same type of fallback (that uses Base.download) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)

See around #22 but it is really unrobust.

Oh and for the record, I just tried it now and got an UndefVarError: update_period not defined error. So I removed the update_period part

Sorry, that was a typo on my part. Your code is the correct one.

from datadeps.jl.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.