Improve dependencies selection about spack-c2sm HOT 8 CLOSED

Possible solution
https://github.com/cosunae/spack-mch/tree/depinstall

from spack-c2sm.

I think this issue needs further discussion and consideration. To me it's not clear exactly what problem we are trying to solve. I think there are multiple problems involved here and it might be better to solve some of them separately. Additionally, it might be better to change our work-flows to better accommodate how spack is meant to be used.

from spack-c2sm.

The problem we are trying to solve is how to specify (where this information should be stored and pass to spack) the set of versions of each dependency of a package to the spack build command. This set is version (of the target package) and fork dependent. For example, the version required for claw will differ for different releases of COSMO and the c2sm or mch forks.

As stated in the title of the issue, the dependency of the dycore can be treated separately, since it seems we are all fine with enforcing its version to the same version of cosmo. So this information is uniquely specified from the release of cosmo.
Adding to the spec in the cosmo package.py would work:
spec += ^cosmo-dycore@vvv
The proposal discussed, was for all other dependencies (except) system packages.

Is it clear now @BenWeber42 ?

from spack-c2sm.

Link to the presentation https://docs.google.com/presentation/d/1qkMyaVkZ0wBWZKdfi7RydUtIkyJZuXVu46mhdNnSQZs/edit?usp=sharing

from spack-c2sm.

Few more references for readers, to how this problem is solved in the world of DevOps.
A common solution is to specify the dependencies in a pipeline of the CD/CI system, i.e. not in the build nor package manager.
The pipeline file can be store in many ways, most commonly either in the build plan of the CD/CI (like in the jenkins configuration) or the repository.
When the requirement is to keep track and capability to build multiple branches and past releases (like it is our case), the recommended way is to keep it in the repository (since a single pipeline file defines only a possible set of versions of dependencies)

Often the concept of the dependency in the CD/CI system is named artifact. In our case artifacts are the ones that would be managed by spack, but still we would need to specify in the pipeline the versions of these artifacts required to build our current release.
Alternatively (to artifacts), when using containers instead of bare metal builds, the dependencies will go in the docker version that contains the dependencies, on top of which

Pipelines are defined mostly in yaml (or similar, like groovy)

Azure DevOps:

https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
https://docs.microsoft.com/en-us/learn/modules/manage-build-dependencies/2-plan-build-dependencies-for-your-pipeline

AWS CodeBuild:

https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-example

Jenkins Pipeline:

https://www.jenkins.io/doc/book/pipeline/

My opinion is DevOps != Spack and spack is not designed to solve this problem, but rather your CD/CI, which should use spack just to build the packages and find the dependencies (as defined by the pipeline)

One more last thought...
My first proposal was to precisely solve it in a jenkins pipeline, using spack but not inside spack. A simple build.py script stage of the pipeline would read the yaml with the dependencies (store in the repo, associated to the hash being built) and call spack, passing the specific dependency versions.
After a discussion with Xavier and Jonas, their wish was to have this mechanism more integrated within spack. The argument was that we already introduced spack to users, so it would be desirable to find a solution within spack to expose them only to one tool.
I understand the argument, since we have a peculiar situation where we have users (not only the developers) that are building the software out of the CD/CI system. That's probably the first flaw we have, we should not support (outside of the devbuild mode) builds of releases out of the CD/CI. But yes, reality is that our users are not familiar with it.
These were the arguments to provide a depinstallcosmo cmd of spack,
https://github.com/cosunae/spack-mch/blob/depinstall/tools/spack-scripting/scripting/cmd/depinstallcosmo.py
but either way, being a spack cmd or a separated build.py that composes the spec and calls spack, the solution is similar, since it relies on how and how the versions of the dependencies are specified. In this case, the answer is in the yaml, attached to the repo and its history. And easily injected into the pipeline of the CD/CI

I hope it helps to better understand the problem...

from spack-c2sm.

An alternative: https://github.com/Stagno/spack-mch/blob/yaml-install/tools/spack-scripting/scripting/cmd/dev_build_yaml.py

The idea is to use spack's native YAML serialized spec format.
You can obtain a full concrete spec in YAML format with, for example (on Tsa):

spack spec -y cosmo@dev-build%[email protected] > spec.yaml

(which can readily be installed with spack install -f spec.yaml).
The goal is to not have platform-specific dependencies in the serialized spec to be stored in the package's repo. For example you can try removing openmpi (all entries!) from the file. You obtain a serialized abstract spec.
Such spec can be dev-built with the new simple command dev-build-yaml:

spack dev-build-yaml -f spec.yaml

It will consider the serialized spec as abstract and proceed to concretize it always, thus filling the missing (platform-specific) dependencies (in our example openmpi).
It should be trivial to also provide an install-yaml analogous.
The advantages here are that:

• no way to forget a dependency version, as spec.yaml is produced via a tool,
• we use spack's API on actual Spec objects, instead of manipulating spec strings,
• we rely on the concretizer to do the job of selecting defaults when something is missing,
• we are using the internal spack's yaml format (no external dependencies),
• the code is very simple and short (easy to maintain).

If we go by this solution we should write a simple spack-tool that does the equivalent of spack spec -y and removes the platform-specific dependencies (mpi, pgi, cuda, etc.). This tool could be integrated in the CI mechanism to automate the production of the serialized spec.yaml with each commit.

from spack-c2sm.

I think this approach is the way forward.

To summarize how we split and solve the problems:

• The problem of specifying compatibility should be done in the package.py with correct depends_on (possibly conflict) clauses
• The problem of reproducibility should be solved by using a fully concretized spec.yaml (an undocumented feature of spack is that it'll put a spec.yaml in a .spack folder in the install prefix)
• The problem of specifying defaults for versions should be solved by adding the functionality, that we can install with an abstract (not fully conretized) spec.yaml. How we deploy such an abstract spec.yaml is up to us (possibly in the repo of the package)

This is more or less what carlos originally suggested, but implemented differently and I think should work better with spack in the long-term.

I think it's important to consolidate and integrate this well. I'm not sure if we should have separate dev-build-yaml or install-yaml commands. We could consider adding this functionality directly to spack install and spack dev-build. We can override these commands in spack-mch with the config.py script. Additionally, I think we should consider adding PRs to upstream spack. They seem be open to changes to commands. @elsagermann already has experience with this when she added --test=root and --test=all options to spacks dev-build command.

from spack-c2sm.

merged with #216

from spack-c2sm.