aiidateam / aiida-project Goto Github PK

Currently the --python option expects you to pass the full path to the Python binary. It would be nice if it could figure out the full absolute path to the binary that is found using PATH, so you can simply pass:

❯ aiida-project create cwf --core-version none --python python3.10
Usage: aiida-project create [OPTIONS] NAME
Try 'aiida-project create --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────╮
│ Invalid value for '--python': File 'python3.10' does not exist.                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

In fact, it should be easy to also just avoid the python duplication, i.e.:

❯ aiida-project create cwf --core-version none --python 3.10
Usage: aiida-project create [OPTIONS] NAME
Try 'aiida-project create --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────╮
│ Invalid value for '--python': File '3.10' does not exist.                                 │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

We should then also add some validation and useful error messages e.g. in case a Python version isn't installed.

The current approach still has some gaps in it (i.e. it doesn't install virtualenv by default), and in practise I still rely on the workon command of virtualenvwrapper without explicitly installing it. We should work on making this as seamless as possible in all possible machine setups.

The aiida-code-registry is great tool for maintaining a repository of computers and codes from which users can directly install them. However, the way I'm using the registry is somewhat clunky: I click through it and try to find the computer/codeI want, click on "Raw" and copy the link.

While doing this, I was considering we should perhaps have a CLI for accessing the registry. Some examples of usage below.

Updating the registry

A command for syncing the latest information on computers and codes stored in the registry locally. This data can then be used by other commands.

aiida-project registry update

Listing computer/codes

aiida-project registry list

To know which computers and codes are available, we need to list them. I think by default the list should be both computers and codes, but in a one-liner for each.

Installing computers/codes

aiida-project registry install

Obviously, that's the whole point. Ideally you can just put the name. In case of ambiguity the command should raise.

TODO

• Give the outputs some more thought.
• What other commands? E.g. upload to open a PR automatically with a new code.
• What conventions should we adopt on aiida-code-registry to make this work here? One thing that comes to mind is that how to install a code depends on the version of AiiDA.

Users often want to use Jupyter notebooks to run AiiDA calculations or perform analysis. It's possible to install jupyterlab in the Python environment where AiiDA is installed, but there are advantages to having one centralised Jupyter server running that can access all aiida-project environments:

1. Only one active server needed, using less memory and other resources.
2. Only have to configure the server once.
3. Having access to all AiiDA projects is useful in case there is interaction between the projects.

IDE's like VSCode can make this really simple as well, but often also use quite a bit of resources to make this happen. Some tools like jupytext for writing notebooks in Markdown are also not supported quite as well in VSCode.

Typically you can install a kernel of a certain environment quite easily, see:

https://ipython.readthedocs.io/en/latest/install/kernel_install.html

However, to use a separated environment for AiiDA properly, You also have to set the env key in the kernel.json, e.g.:

{
 "argv": [
  "/home/aiida/.virtualenvs/aiida-super/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "aiida-super",
 "language": "python",
 "env": {"AIIDA_PATH": "/home/aiida/envs/aiida-super"}
}

Moreover, providing a user-friendly CLI for doing these kind of kernel installs will also make the process less painful and error-prone.

It might be useful if the aiida-project manager was extendable so it could provide support for specific plugins. I was mainly thinking about aiida-quantumespresso. Here I often find myself having to set up a lot of different codes that are all very similar, save for the executable. Would be nice to be able to do e.g.:

aiida-project plugin qe install codes --binary-path /home/mbercx/src/qe-7.2/bin

For venv-based environments, we should consider using the --upgrade-deps option:

Upgrade core dependencies: pip setuptools to the latest version in PyPI

see the venv documentation.

I recently stumbled upon an issue while trying out a few different setups. Actually don't fully recall now how I got
there, but what was happening was that after creating new projects with aiida-project create and activating them with
cda, my AIIDA_PATH environment variable was still pointing to the global $HOME/.aiida, and, accordingly, new
project-specific profiles were being added to the global config file $HOME/.aiida/config.json.

After some digging around, I realized that the issue was an incompatibility between my main shell (fish) and the fact
that I was occasionally launching and using bash. To be precise (and as I have recently, painfully learned),
environment variables of the parent shell cannot be set from a Python subprocess. Thus, in aiida-project, this is
instead handled via the cda command, which, apart from changing the directory, also sources the Python activation
file, activate.fish for fish and activate for bash when using venv. Importantly, aiida-project adds the
command to set AIIDA_PATH to this activation file, e.g. for fish in $HOME/.aiida_venvs/<project>/bin/activate.fish:

set -gx AIIDA_PATH $HOME/aiida_projects/<project>
eval (env _VERDI_COMPLETE=fish_source verdi)

with cda defined as follows:

# Defined in $HOME/.config/fish/conf.d/aiida_project.fish @ line 48
function cda
    source "$aiida_venv_dir/$argv[1]/bin/activate.fish"
    cd "$aiida_project_dir/$argv[1]"
end

Long story short, in my case, cda was pointing to activate not activate.fish, while the command to set
AIIDA_PATH was in activate.

Now, this might be somewhat of a niche case, however, I would propose that we move the location of setting AIIDA_PATH
and the eval statement for tab completion to a general, hidden activation file inside the directory of the project.
This change should also be necessary if we want to add conda support, as here the Python environment is not activated
by sourcing an activate file.

Lastly, as it stands right now, the shell is attached to the global configuration of aiida-project, so we could
instead attach it to each project, or, even better, implement a check on the currently running shell, so that seamless
use of aiida-project is guaranteed even when running with different terminal instances and shells (though, not
entirely sure how one would go about implementing that).

Long term, aiida-project should be able to fully write the whole specification of the Python environment and AiiDA-related content to file, so it can be entirely (or selectively) reinstated. An important part of this is the ability to be able to dump and restore computers/codes.

Relevant here is aiidateam/aiida-core#3521, and we should make a decision on whether to store the computers/codes as AiiDA exports or YAML files.

It seems that initializing conda after install conda/mamba breaks the cda function set up in your e.g. .zshrc file:

# Created by `aiida-project init` on 05/09/23 11:05
export $(grep -v '^#' /Users/mbercx/.aiida_project.env | xargs)
cda () {
# source $aiida_venv_dir/$1/bin/activate  # commented out by conda initialize
  cd $aiida_project_dir/$1
}

We should look into what exactly conda initialize looks for, and try to avoid conflicts..

Often in a development project, I want to nuke the Python environment so I can rebuild it from scratch without losing anything else from the the project. One way would be to have an option to only destroy the environment, e.g.

aiida-project destroy --env

Or --environment, since destroy command should be used with care anyways. Alternatively, we can also have aiida-project create complain if the environment already exists, but prompt to have it overridden.

aiida-project currently requires pydantic~=1.10,>=1.10.8, which leads to dependency conflicts with the current, latest release of aiida-core (v2.5.1).

For instance, when one has a global aiida-core installation, e.g. via /usr/bin/python and tries to pip install aiida-project in editable mode to try out some local changes (naughty, I know), then the two packages aren't compatible due to the different pydantic requirements. I guess upgrading here would not be a bad idea anyway, as BaseSettings was moved to its own repository pydantic-settings. I'll see to resolve that soon.

Currently the installation of packages when creating a new project is limited to packages available on the PyPI. We should also allow users to directly clone and locally install package from remote git repositories.

Pretty basic stuff, but it would be nice to list the AiiDA projects that have been created.

OK, another one of my weird shell-related issues 😄

When trying to run aiida-project init in a bash subshell, I'm getting the following KeyError:

which is very strange, as env_file is definitely present when printing dir() of Config.config before...