Comments (5)
That's interesting.
I made a little test of mirroring + changing the config file for selective updating.
A couple of things emerged :
Firstly
if a package is not in the (current) filter, it will remove the existing downloads, just as if it had been deleted in the upstream channel.
That obviously needs adjusting for partial / incremental updates to work.
Secondly
Even when you only add one package that was not previously there, the whole of the existing downloads are validated. So when I re-added "pytest" to my mirror of the linux-64bit anaconda channel, it took just seconds to download all those additional binaries, but then about 3 hours to re-validate everything ...
All that is practical stuff, but I think there is also a serious question about the usage and validity of incremental usage : The existing behaviour ensures that the result is just an up-to-date replica of (maybe part of) the upstream channel -- so the meaning of the result is totally unambiguous. But not so if you allow partial updates ...
For example, if I mirror anaconda, that is a kind of "curated" resource -- so I can expect that the package versions there are mutually compatible. But a partial update may break that -- and I can't see any automatic way of avoiding those problems, except doing a full update.
from conda-mirror.
cc: @pp-mo
from conda-mirror.
Hi @Tigeraus --
conda-mirror
does not currently support what you are asking for. Such functionality would not be particularly challenging to implement. I'd be happy to guide you through the process of contributing such a feature if you are interested.
If you are not interested in contributing such a feature that's ok too, but I cannot make any guarantees about when I would have time to add this myself.
from conda-mirror.
@ericdill Actually I don't have much time now to do this, but I'm interested in it and I will do it when I'm free. One Way, I think, that could implement this functionality is checking the md5 of all files in the repo, store them somewhere else and then just download those whose md5 values have changed and update the md5 set.
from conda-mirror.
@Tigeraus that's a good idea. My hesitation here is that repodata.json is the location where the md5 values are stored for local packages. I'd like to avoid inventing another place to store the md5 information from downloaded packages. Let me go check my conda-mirror logs and see if any package has actually ever failed the md5 check. I have not checked those logs in a while. I'll report back later today with what I find.
from conda-mirror.
Related Issues (20)
- [BUG] Problematic implementation of the mirror
- [Optimization] Shuffle package validation order before validating
- keep old packages HOT 12
- Validation errors caused by incomplete tar file
- SSL Issue when creating mirror HOT 1
- sync only new packages HOT 8
- Specifying multiple platforms HOT 3
- How to catch stalling HTTP connections? HOT 1
- New release/version in pypi/conda-forge? HOT 3
- Release 0.8.0
- Cannot use conda-mirror behind proxy HOT 1
- Allow to filter packages based on the target Python version
- Fetching 'noarch' platform.
- Respect previous downloaded packages and generated `repodata.json`.
- Fetching necessary dependencies for the whitelist. HOT 1
- NOTYPE error of 'package'
- Issues with sha256 checksums HOT 2
- Support for .conda package mirroring HOT 1
- support https://anaconda.org/rapidsai
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-mirror.