Comments (12)
Hi @MWigger. Thanks for the issue report. What version of conda-mirror are you running?
Unfortunately conda-mirror deleted one of them, since it was "to old"
I suspect what happened was that conda-mirror
deleted a package that you had locally because it was removed from the upstream repository. There is currently no mechanism that checks the age of package. There are four conditions where a package will be removed.
- Package was deleted from the upstream repository
- Package is an invalid tarball
- Package fails the md5 checksum
- Package fails the sha256 checksum
If you were mirroring the defaults channel or the conda-forge channel, I can check my own logs at that time to see if I can help debug this issue.
Can you let me know which package was removed and the date that you noticed it? If you're not comfortable posting that information to github, please feel free to send me an email at [email protected] and I can help you debug this in a 1:1 conversation.
from conda-mirror.
from conda-mirror.
Are you logging the output of conda-mirror? Whenever it removes packages it notes why. What is the upstream channel that you are mirroring?
I see these errors in my logs for the anaconda channel:
WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/osx-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...
WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/win-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...
FWIW conda-mirror will attempt to redownload packages every time you run it. You should be able to run conda-mirror again and it will attempt to download again any packages that failed validation the last time you ran it.
from conda-mirror.
What I neglected to add to the last comment is that I currently have mkl-2017.0.1-0
locally. It did fail once, but has not failed in the last 3 or so weeks.
from conda-mirror.
from conda-mirror.
Yes. Any package removal gets dumped as warning level which is -v
from conda-mirror.
from conda-mirror.
from conda-mirror.
Thanks for the updated information @MWigger .
There are two distinct pieces happening there.
The first is the expected behavior of conda-mirror removing packages locally that have been removed upstream. The intent of conda-mirror is to be a complete mimic of what is available upstream. It is unfortunate that upstream removed a package that you were depending on locally. That being said, when packages are removed from upstream there is usually a good reason for this. I have zero control over or insight into the reasons for package removal from upstream.
The EOFError
is something that I need to be catching. That's a new bug. Thanks for finding that 😀
from conda-mirror.
Well, I just saw thast for example for the cffi package there is a 1.7.0 version on the coda-forge channel: https://anaconda.org/conda-forge/cffi/files
So it might be a different issue
from conda-mirror.
I have seen similar package removals for 'failed size test'. I dug in to a couple of these occurances and what I found was that the package, let's take cffi-1.7.0-py27_0.tar.bz2
for example, had been deleted from the repo that I was mirroring and re-uploaded with one or more pieces of metadata changed. I think one time I looked, a license file had been added to the package. Since this does not affect the run time behavior of the package, bumping the build number (to cffi-1.7.0-py27_1.tar.bz2
) was likely deemed unnecessary. Indeed only people that are maintaining their own local copy of the entire upstream repository would notice.
The reason why re-uploading a package with different metadata will cause conda-mirror to fail the size test is that the new uploaded package on upstream has a different size. The repodata.json file for that upstream repo has already been updated for the packages new size so when I compare the expected size in the upstream repodata.json with the size of the package that has been mirrored locally, they do not match. Because I treat the upstream repodata.json as the source of truth, that means the local package is now "incorrect" and it get's removed. Then the package that was just removed because of its "incorrect" size will be redownloaded from upstream.
from conda-mirror.
Closed by #58
from conda-mirror.
Related Issues (20)
- [BUG] Problematic implementation of the mirror
- [Optimization] Shuffle package validation order before validating
- Validation errors caused by incomplete tar file
- Question about Incremental Syncing HOT 5
- SSL Issue when creating mirror HOT 1
- sync only new packages HOT 8
- Specifying multiple platforms HOT 3
- How to catch stalling HTTP connections? HOT 1
- New release/version in pypi/conda-forge? HOT 3
- Release 0.8.0
- Cannot use conda-mirror behind proxy HOT 1
- Allow to filter packages based on the target Python version
- Fetching 'noarch' platform.
- Respect previous downloaded packages and generated `repodata.json`.
- Fetching necessary dependencies for the whitelist. HOT 1
- NOTYPE error of 'package'
- Issues with sha256 checksums HOT 2
- Support for .conda package mirroring HOT 1
- support https://anaconda.org/rapidsai
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-mirror.