Are we approaching dependency hell? about aiidalab HOT 6 CLOSED

@yakutovicha Part of the problem, namely the one of creating reproducible builds, is solved by a tool called pipenv.

The basic idea is: instead of a requirements.txt file with fixed dependencies (which, in the end are still not perfectly fixed because of dependencies of dependencies), you create a Pipenv file, where you specify the dependency ranges you are interested in (typically you fix MAJOR.MINOR and allow changes in PATH versions).
Then, before you make a release, you let pipenv "concretize" the dependencies down to the very last package that is installed (and you commit Pipfile.lock to the repo).

It's used by invenio and I started looking into this for aiidalab a few weeks ago but stopped exactly because of the dependency conflicts that exist.

Of course, pipenv can not resolve explicit dependency conflicts that exist between aiida plugins.
I propose our policy should be:

If an aiida plugin wants to be present on the AiiDA lab, it must not present any version conflicts with the existing software stack.

I will create a branch with a basic Pipfile and, for the moment, exclude any plugins that present dependency conflicts.

from aiidalab.

I started working on this in the pipenv branch. Here I'm using pipenv simply to generate a complete requirements.txt from the Pipfile (instructions added to README).

It turns out that, miraculously, all dependency conflicts can be resolved (e.g. pymatgen automatically gets resolved to 2018.12.12).

There is one issue with aiida-fleur that I opened already more than a month ago and that prevents aiida-fleur from being added.
Once this is fixed, aiida-fleur can be added back to the mix as well.

from aiidalab.

I think pipenv is a great tool. Thanks for looking into that.

Also, I think we should use it while testing aiida plugins. For example it would be nice to create an automated script that installs aiida with all the latest versions of plugins to check incompatibilities.

Of course for the moment it is just a wish, nothing concrete. But I think the issue should be solved ASAP, otherwise some users simply won't be able to use aiida because of similar problems.

from aiidalab.

Jens just updated aiida-fleur - all packages are now available again (PR #8 updated & ready for review)

from aiidalab.

it would be nice to create an automated script that installs aiida with all the latest versions of plugins to check incompatibilities.

We should define what exactly we want to test here.
As you mention, with pip the result might depend on the order of installation.

We could start with a more basic check using pipenv and integrate it into the CI of the aiida plugin registry. I.e. for every new pull request. pipenv would try to concretize the dependencies for installing all plugins together (all that provide a pip package name).
If that fails, the test should fail.

from aiidalab.

the original issue seems resolved.
discussion about testing continues in aiidateam/aiida-registry#67

from aiidalab.