Separate python environment for every app. about aiidalab HOT 9 CLOSED

To connect to the correct environment one should use the following URL:

[apps|voila]/apps/<APP_NAME>/<NOTEBOOK>.ipynb?kernel_name=<APP_NAME>

from aiidalab.

Have you guys done a test on

• how this affects disk size (e.g. installing all currently registered apps)
• how this affects installation time for apps?

from aiidalab.

I'm also wondering how these different environments are going to access the AiiDA installation running in a different environment.

from aiidalab.

I'm also wondering how these different environments are going to access the AiiDA installation running in a different environment.

The idea is to add the path to the base conda environment to the PYTHONPATH of every app-specific environment so that AiiDA and related packages will be shared.

how this affects disk size (e.g. installing all currently registered apps)

We didn't test that yet, but we are going to do it before merging this change.

from aiidalab.

The idea is to add the path to the base conda environment to the PYTHONPATH of every app-specific environment so that AiiDA and related packages will be shared.

How does this work for installing packages?
I guess, you will also need a way to inform pip and conda in the app environment of the packages installed in that base environment you are sharing; otherwise any app that depends on AiiDA in some way (could be indirectly via some plugin) will install its own aiida version.

Would be good if there were some examples of other projects doing this, so that we can learn from them and don't need to figure out everything ourselves (and if noone does this, perhaps it's not a good idea in the first place?).

from aiidalab.

Another important issue we came across when testing #82 is the necessity to keep aiida and aiida plugins in the same environment. The problem is the following: when aiida is installed in the base environment it cannot see the plugins installed in the sub-environments. This makes the whole point of isolating app's dependencies unachievable.

from aiidalab.

I'm a bit puzzled how we managed to miss this glaring issue when we all looked at the proposal together.
The main problem is with the AiiDA daemon. Anything that doesn't require the daemon (even running calculations using engine.run) could in principle be executed from within the respective virtual environment, but the daemon in general needs access to all virtual environments simultaneously (which would mean their dependencies need to be compatible, which defeats the purpose of this sandboxing effort).

It seems possible in principle to record, upon submission of a new calculation, the virtual environment from which the calculation is being submitted, and to let the daemon runner execute inside this virtual environment when it picks up corresponding tasks (in principle - I don't know how involved this would be).
However, even if this worked, it would open other questions: is it really wise to record calculations from different python environments (with different AiiDA plugins installed) into the same database? The lesson may be that, in order for the data generated by apps from different python environments to be interoperable, the python environments actually do need to be compatible as well.

Of course, you could start individual daemons per environment (with separate profiles, ...).
At first sight, this would make management complicated, but perhaps it's not too difficult to make a user-friendly interface around it.
To me, this seems the only viable solution at the moment if you want to truly separate the environments of individual apps, but it would mean you can't easily access the data generated with one app from another and I'm not sure this is what we want.

I believe this topic will need another plenum discussion.
Mentioning @giovannipizzi for info

from aiidalab.

I agree with @ltalirz , let's have another good look at this.

One solution might be that everything aiida related (core+plugins) must be installed into the base environment and that only more lightweight dependencies like the widgets etc. are installed into the virtual environments.

from aiidalab.

There are currently no immediate plans to implement this model.

from aiidalab.