[BUG] - Environment builds locally, fails on conda-store about conda-store HOT 5 CLOSED

Hey @ocefpaf 👋🏽

Agree the third option looks like the easiest/least friction path for now

from conda-store.

@jaimergp gentle ping here please 🙏

from conda-store.

@rsignell I probably made a mistake in my tests where I got this env working with conda-lock at some point. Maybe I changed the env file. However, @jaimergp is correct, in that form conda-lock will invoke poetry and it will error out with:

conda_lock._vendor.poetry.repositories.exceptions.PackageNotFound: Package adios-db (1.1.1) not found.

That is a bug in conda-lock b/c the conda version of that package is installed and available but the poetry wrapper in conda-lock cannot pass that information to it. With that said, poetry is choking b/c adios-db is not available on PyPI, making the package that requires it uninstallable via pip. Here is what we can try:

• Open an issue in conda-lock to see if they can circumvent this problem but it is not an easy one!
• Ask adios-db devs to publish a source distribution on PyPI (this one may take a while)
• Patch the package that requires it to not require it when being installed via a package manager that reiles on PyPI (this one is probably the easiest path to solve this)

from conda-store.

Thanks for opening an issue, Rich. :)

So, the issue seems to be in creating the lockfile and not the environment itself.

conda-store uses conda-lock to create the lockfile, and trying to do this locally fails with the same message for me.

conda install -c conda-forge conda-lock

conda-lock -f <path-to-yaml-file>

Moreover, the traceback suggests this is coming from Poetry. Note that conda-lock uses poetry to resolve pip dependencies and conflicts. I see that adios-db is not available on PyPI (but is required by opendrift in the pip section), which is probably why Poetry is raising this error.

For instance, the following spec also fails to create a lockfile with the same error:

channels:
  - conda-forge
dependencies:
  - python=3.11
  - ipykernel
  - adios_db
  - pip
  - pip:
      - opendrift @ git+https://github.com/OpenDrift/opendrift@master

Speaking for conda-store, failing early here is a good thing, IMO. We want to promise reproducibility+reliability, which we can't for this env even for local builds.

I don't understand the solving mechanism to know for why conda-lock is not using the adios_db available on conda-forge, maybe @jaimergp can help answer. :)

from conda-store.

The source of confusion here might be that users might expect conda-store to use the regular conda env create -f invocation, while it actually uses conda-lock to generate a lockfile. So, in that regard, I'm inclined to say that this is an issue on conda-lock's side, not conda-store.

Should we provide a fallback to conda env create if conda-lock fails? That's a separate question.

Another issue to bring up here is, for the sake of reproducibility, how the environment pins to @master, and that's a moving target. So maybe it would be better to pin the pip requirements to specific git hashes or tags.

Also I'm not sure if conda-lock exposes the contents of the conda packages to pip. It feels like it's done in isolation, which is a departure from conda env's operations. conda env will solve and install the conda packages first, and then run pip on top of that environment. That's why pip can find adios-db but conda-lock does not. So maybe one workaround is to also include the adios-db component in pip:? Not sure about that, but worth a try.

from conda-store.