Better model distribution about argos-translate HOT 10 CLOSED

There is also IPFS.

from argos-translate.

No, the training data is very large. I don't even store it myself I just re-download from Opus every time I train a new model. If you're interested in the data go to Opus. There's also a generate_wiktionary_data.py script in the training scripts.

from argos-translate.

I'm not too worried; the discussion on S.O. is pre-Microsoft acquisition. If you want to link it, go ahead.

from argos-translate.

It might be useful to consider using GitHub Releases to distribute the models, as there are no bandwidth limitations and since CloudFront is used to deliver the assets.

from argos-translate.

Related to this once better distribution is in place we'll want to add the ability to manage model downloads from the GUI.

from argos-translate.

GitHub has (so far, crossing fingers) been pretty good at hosting binary data (up to ~2GB in size per file) without limits, although the party might end sometimes in the future.

Otherwise a central server with CDN works pretty well too. For cheap, there's https://www.digitalocean.com/products/spaces/ (although careful on the outbound transfer fees!) or https://wasabi.com/ (no egress fees)

from argos-translate.

A CDN is probably the best long term solution (in addition to providing torrents) and Google Drive is definitely not ideal. Last weekend when there was a lot of traffic from Hacker News it seems that some people weren't able to download models. I'm tempted to link directly to the githubusercontent url in the LibreTranslate Models repo but that may increase the chance that we can't continue to use GitHub.

https://stackoverflow.com/questions/38768454/repository-size-limits-for-github-com

from argos-translate.

@davidak https://www.reddit.com/r/ipfs/comments/l43u9u/seeking_open_source_contribution_to_distribute/

from argos-translate.

I wonder if the raw data is also stored anywhere.
Or do we only have the compiled model files?

from argos-translate.

We're now sharing one model pair (English-Chinese) via IPFS (available on package index) as a test. A volunteer is currently pinning for us and we can expand if people find this useful.

from argos-translate.