Testing an installation about spack HOT 15 CLOSED

👍 This is planned but not yet on the roadmap. Some teams at LLNL have also asked for configure and install to be separated, so that will likely happen as well.

from spack.

👍 for a test block in each package as a post-install step invoked by spack test <package>. Here are a couple of examples from Homebrew: deal.ii, mumps, petsc.

Also many packages are missing make test runs or equivalent, which normally do not take more than a minute or so but are good to check that things are working as expected before doing production runs.

from spack.

I've definitely run into some make test runs that take significantly longer than the configure or make steps, so we may not want them enabled by default. Personally, I would love to run make test so I can sleep well at night. If we separate configure and install, then we could easily add a test() function. We could then add a -t/--test option to make install that runs test().

One thing to consider would be something like GCC, which requires a few additional dependencies to properly run the test suite. There has already been talk of separating build dependencies and runtime dependencies. This may require us to add test dependencies.

from spack.

@davydden: I would really like to add testing capability. One challenge is that Spack is designed for big machines, and the command to actually run stuff (esp. on BG/Q, XC40, etc.) can be very different from site to site. I would want to add a generic, auto-detected msub/qsub/srun command that could be used in Spack tests.

from spack.

@tgamblin I was thinking about the following HPC usage case: create a job for a single node which does nothing for a couple of hours; ssh to the node; run spack to install everything. Since it's on the node, one will be able to use mpiexec or whatever name it has locally, which is not always the case on the front-end.

from spack.

@tgamblin we would also be interested in scheduler support within spack. If it may be of interest this is the only package I found that seems to go in the direction of supporting different schedulers. Didn't have much time to play with it though...

from spack.

How do people feel about including make("test") in packages?

from spack.

Depends on how reliable the test suites are. There should probably be a --no-test argument to spack install to suppress testing so that test suites which take a long time (I'm thinking at least VTK and ParaView here, but they're not the only ones) can be skipped.

from spack.

i would not have a separate option --no-test, instead all the tests which are reasonable fast should be done unconditionally. Whereas tests which are long should be done (if possible) by something like spack test <package>.

The rationale is that --no-test does not change the variant of the package (like +openmp, +mpi, +shared, etc) so i would avoid those.

from spack.

Well, there's the problem that the default is to not keep stage directories around, so a spack test would require rebuilding the package from scratch in the "common" case.

from spack.

I think we can distinguish between install time tests (make test or similar things) and post-install tests (like the small executable that @davydden provided for openblas). For the former I agree with @mathstuf in that spack install --no-tests ... is needed, for the latter I think we should be able to execute them after the installation like @davydden proposes.

from spack.

A couple random thoughts on this:

• #1169 added a switch to run tests at install time
• #1186 might provide a modular solution to attach custom tests to packages

from spack.

i think essentially when/if #1186 is merged, what's missing is spack test <package> which would run a post-install step install_test(), if it is defined in a package.

from spack.

@eschnett Is there something more that needs to be done? Or are you fine with :

spack install --run-tests ...

if appropriate functions have been defined in the package (like in hdf5 fo instance)?

from spack.

@eschnett I am going to close this because I think it has been fixed already. Please reopen if still necessary.

from spack.