alisw / alibuild Goto Github PK

Looks like if we have somethink like this in root.sh:

...
requires:
  - Alien-runtime
  - CMake
...

and:

...
requires:
  - CMake
...

in Alien-runtime we end up with:

DEBUG:AliRoot:63655bd0: Calculating hashes.
Traceback (most recent call last):
  File "alibuild/aliBuild", line 301, in <module>
    h.update(specs[dep]["hash"])
KeyError: 'hash'

this is due to some wrong dependency ordering.

When people checkout packages locally, we expect them to be in the same case as the one specified in the package. We should have an error in case the detect that a given package has been checked out with different casing.

• By default assume that if a <repository> is checked out at the same level as alidist and it matches a package name, then --devel <repository> is implicit.
• Provide a mode in which everything is installed in the same location.

Hi,

could you add modulecmd on the docker images to allow using aliModules without cvmfs?

Cheers,
Jochen

For packages which are built in-source, the build directory is the same across architectures.

Proposal: cp/rsync SOURCEDIR to BUILDDIR and do configure, make, make install from there

There is a lot of junk which is left behind in the build areas, which probably should be deleted. There is a few gotchas that need to be discussed, but we could get rid of:

• INSTALLROOT after the build (with an eye on the fact that devel packages need to be handled specially).
• BUILD after the build (making sure we do not do it if the build fails, since we want to check what went wrong).
• TARS (if the remote is not there, otherwise they will get downloaded again).

This will require being able to track state of which packages have been disabled for a given build area.

some init.sh scripts generated by AliBuild/AliDist have a random order for variable initialization.

I noticed for instance the following wrong order in GEANT4/.../init.sh :

export G4LEDATA="$G4INSTALL_DATA/data/G4EMLOW6.41"
export G4INSTALL_DATA="$GEANT4_ROOT/share/..."

It would be great to have a list of available defaults if a wrong one is provided.

Hi all,
I've installed two different development versions of aliroot.
For one, I just use the master:
aliBuild init -w ../sw AliRoot@master,AliPhysics -z ali-master --dist IB/v5-08/next

and for the other, I need a specific branch:
aliBuild init -w ../sw AliRoot -z ali-muonhlt --dist IB/v5-08/next
cd ali-muonhlt/AliRoot
git checkout feature-muonhlt

When I compile:
in ali-master: aliBuild -z -w ../../sw --disable GEANT4,GEANT4_VMC build AliPhysics
and
in ali-muonhlt: aliBuild -z -w ../../sw --disable GEANT4,GEANT4_VMC build AliRoot

I find this in the BUILD directory:
AliRoot-latest -> d20faaa6662993d882eef0df002b58f08e8754d0
AliRoot-latest-ali-master -> d20faaa6662993d882eef0df002b58f08e8754d0
AliRoot-latest-ali-muonhlt -> d20faaa6662993d882eef0df002b58f08e8754d0

i.e., despite the AliRoot source code is different (I have the AliRoot master in ali-master and the branch feature-muonhlt in ali-muonhlt), the two different builds are assigned the same hashtag...and each time I compile one of the two, I automatically destroy the other.

Is there some options that I'm missing which would provide a clearly separate build for two clearly different development versions?

Thanks in advance,
best regards,
Diego

When building a package with the devel option, I end up with a wrong modulefile. The problem is that the version is not properly replaced but contains something like:
0-@@PKGREVISION0af2c75190cad609e350ddd415ed9dca0487db1f@@

What's interesting is that the version is correct in the modulefile for another package also using the devel mode but which was built as a dependence.

In order to enable a simpler environment loading/unloading a helper script to source in your shell rc file is being provided. This enables alienv load|unload without needing to eval the result of the script.

The most convenient way to load the helper function in the environment (say, ~/.bashrc) is to have a sort of alienv shell-helper command that returns a line that can be easily eval'd in the rc file.

Your ~/.bashrc should eventually contain:

eval `alienv shell-helper`

When building GSL for O2 I get :

/home/flpprotodev/alice/sw/BUILD/49db3fe4c312a3f2fefa8fe09f95f98e85f981ae/GSL/missing: line 81: makeinfo: command not found
WARNING: 'makeinfo' is missing on your system.
You should only need it if you modified a '.texi' file, or
any other file indirectly affecting the aspect of the manual.
You might want to install the Texinfo package:
http://www.gnu.org/software/texinfo/
The spurious makeinfo call might also be the consequence of
using a buggy 'make' (AIX, DU, IRIX), in which case you might
want to install GNU make:
http://www.gnu.org/software/make/
make[2]: *** [gsl-ref.info] Error 127
make[2]: Leaving directory /home/flpprotodev/alice/sw/BUILD/49db3fe4c312a3f2fefa8fe09f95f98e85f981ae/GSL/doc' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/home/flpprotodev/alice/sw/BUILD/49db3fe4c312a3f2fefa8fe09f95f98e85f981ae/GSL'
make: *** [all] Error 2

Either install makeinfo with alibuild or (preferably) disable documentation for GSL.

It seems we need LHAPDF 5.x for ThePEG since some PDFsets are not available in v6.x. What's the idea to have multiple versions installaed and pulled in as dependency? In a way, LHAPDF 5 and 6 are similar to pythia6 and pythia8, which simply got different package names.

Ciao Giulio,

Maybe the architecture recognition can be extended for some other common GNU/Linux distributions. In my case for example I get:

platform.dist()
('LinuxMint', '17.3', 'rosa')
lsb_release.get_lsb_information()
{'RELEASE': '17.3', 'CODENAME': 'rosa', 'ID': 'LinuxMint', 'DESCRIPTION': 'Linux Mint 17.3 Rosa'}
From which I understand that I should launch the script with "-a ubuntu1404_x86-64" since in /etc/os-release I see "VERSION="14.04.4 LTS, Trusty Tahr". It would be nice if that could be taken care by the script itself.

Cheers, ciao,
Raffaele

For some packages, the dependencies for different versions of the package change. It would be beneficial to have the possibility of specifying dependencies to be pulled in only for some versions.

In contrast to other issues, this is not really critical at the moment.

When sha1sum is not installed on Mac, aliBuild fails with this error

Unable to detect source code changes.

Maybe you could detect it and provide a more insightful message.

We should have the following policy:

On mac:

• LIBENVNAME=DYLD_LIBRARY_PATH
• SONAME=dylib

On linux:

• LIBENVNAME=LD_LIBRARY_PATH
• SONAME=so

so that recipes do not need to decide that themselves.

I tried to install aliphysics and aliroot on SL 6.6 with the following components:
Python 2.7.8
cmake version 2.8.12.1
gcc 4.8.4

The components are loaded with module load ...

For the installation process I used the following commands:
aliBuild init AliRoot,AliPhysics -z ali-master
aliDoctor 2>&1 | tee -a alidoctor.log
aliBuild -z -w ../sw build AliPhysics --jobs 10 --disable fastjet 2>&1 | tee -a build.log

The installation process fails when installing XRootD, because the used compiler is gcc 4.4.7 and not 4.8.4 anymore.
The log files and the output of aliDoctor are attached.
The same problem appears when using gcc 5.2.0 and disabling the GCC-toolchain

log.txt
alidoctor.txt
build.txt

Command issued:

alibuild/aliBuild --debug --architecture slc5 build yaml-cpp

Main package is yaml-cpp but it says it's zlib instead (in log prefixes too):

DEBUG: Using aliBuild from alibuild@abadf46eba2fed82d5694fd7161220819f4b270f recipes in alidist@0a98b600a6ceb33caaa4b4eecb79d7660cfdc55a
DEBUG: appending boost as a requirement
DEBUG: appending CMake as a requirement
DEBUG: appending zlib as a requirement
DEBUG: Main package is zlib@8822efa61f2a385e0bc83ca5819d608111b2168a
DEBUG:zlib:8822efa6: Commit hash for https://github.com/star-externals/zlib@master is 8822efa61f2a385e0bc83ca5819d608111b2168a
DEBUG:zlib:8822efa6: Commit hash for https://github.com/Kitware/[email protected] is v2.8.12
DEBUG:zlib:8822efa6: Commit hash for https://github.com/alisw/[email protected] is v1.57.0
DEBUG:zlib:8822efa6: Commit hash for https://github.com/jbeder/[email protected] is release-0.5.2
DEBUG:zlib:8822efa6: Calculating hashes.
DEBUG:zlib:8822efa6: Hash for recipe zlib is 1c385d89a718c24c35eaa49bf794f93ccd888020
DEBUG:zlib:8822efa6: Hash for recipe CMake is 52c82016da3124bc2ed1a3cf38632bf5e16bc0ef
DEBUG:zlib:8822efa6: Hash for recipe boost is 15e3b4697caa0068f288356df65dc069e02d56d2
DEBUG:zlib:8822efa6: Hash for recipe yaml-cpp is a94259c64a9fa9b64397ef72e36efb6f74475409
...

--disable Foo --disable Bar should have the same effect as --disable Foo,Bar, not --disable Bar.

Many people do not seem to have this installed. We should include PyYAML 3.09 in alibuild and use our own version.

• Discuss how to specify build_requires.

Test case: ThePEG depends on pythia.

We first build pythia on a certain machine and we decide not to publish the build results (i.e. no ::rw option).

If we then build ThePEG, this time using the ::rw option, and the build ends on the same machine, aliBuild finds the local copy of the pythia build and it does not rebuild it, therefore it does not upload the package.

Instead, inside the ThePEG dist* directories, it creates dangling links to a non-existing repository version of pythia.

Example builds exhibiting the problem: pythia, ThePEG.

In particular we should:

• Make sure that whenever we reuse a tar file from a previous build we say so rather than "Not building"

Basic behaviour of this should really be shortcut for:

• rm sw/BUILD/<package>/<version>
• rm sw/<package>/<version>/

additionally it should probably propose to prune dependencies which are not used by anything else.

After installing O2 using alibuild, I try to set the environment :

[eulisse@aidrefflp01 barth]$ alibuild/alienv enter VO_ALICE@O2::latest
BASE/1.0(6):ERROR:105: Unable to locate a modulefile for 'Boost/v1.59.0-1'
BASE/1.0(6):ERROR:105: Unable to locate a modulefile for 'Boost/v1.59.0-1'

You can see it here : eulisse@aidrefflp01:/home/eulisse/barth

E.g. get a build of root, alienv load ROOT/latest, which root-config == $ROOT_ROOT/bin/root-config.

Instead of telling users to --disable manually lists of unwanted packages we should provide them with different sensible defaults according to their use case (e.g. daq, nosim, etc.)

Dependencies with wrong case silently break DAG.

See for instance Build #988.

So, the GSL recipe was changed in alisw/alidist#78 to get the sources from Git (using the sources: parameter) as opposed to downloading the tarball (in the recipe's body).

What we can see from the following log is that:

• aliBuild correctly clones the new Git repository
• but it runs the old recipe's body from /build/workarea/sw/20150829/SPECS/GSL.sh

Here's the log:

DEBUG:AliPhysics:vAN-2015: + mkdir -p /build/workarea/sw/20150829/INSTALLROOT/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/slc5_x86-64/GSL/v1.16-1 /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8 /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/GSL /build/workarea/sw/20150829/INSTALLROOT/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/slc5_x86-64/GSL/v1.16-1
DEBUG:AliPhysics:vAN-2015: + cd /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8
DEBUG:AliPhysics:vAN-2015: + ln -snf bbde5365b7427b311b02f8ea4d3b7576ac3a81c8 /build/workarea/sw/20150829/BUILD/GSL-latest
DEBUG:AliPhysics:vAN-2015: + [[ ! git://git.savannah.gnu.org/gsl.git == '' ]]
DEBUG:AliPhysics:vAN-2015: + [[ g != \/ ]]
DEBUG:AliPhysics:vAN-2015: + [[ ! -d /build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16 ]]
DEBUG:AliPhysics:vAN-2015: + rm -rf /build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16
DEBUG:AliPhysics:vAN-2015: + git clone git://git.savannah.gnu.org/gsl.git /build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16
DEBUG:AliPhysics:vAN-2015: Cloning into '/build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16'...
DEBUG:AliPhysics:vAN-2015: + cd /build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16
DEBUG:AliPhysics:vAN-2015: + git checkout release-1-16
DEBUG:AliPhysics:vAN-2015: Note: checking out 'release-1-16'.
DEBUG:AliPhysics:vAN-2015: 
DEBUG:AliPhysics:vAN-2015: You are in 'detached HEAD' state. You can look around, make experimental
DEBUG:AliPhysics:vAN-2015: changes and commit them, and you can discard any commits you make in this
DEBUG:AliPhysics:vAN-2015: state without impacting any branches by performing another checkout.
DEBUG:AliPhysics:vAN-2015: 
DEBUG:AliPhysics:vAN-2015: If you want to create a new branch to retain commits you create, you may
DEBUG:AliPhysics:vAN-2015: do so (now or later) by using -b with the checkout command again. Example:
DEBUG:AliPhysics:vAN-2015: 
DEBUG:AliPhysics:vAN-2015:   git checkout -b new_branch_name
DEBUG:AliPhysics:vAN-2015: 
DEBUG:AliPhysics:vAN-2015: HEAD is now at 0b8a3d7... 1.16 release
DEBUG:AliPhysics:vAN-2015: + git remote set-url --push origin git://git.savannah.gnu.org/gsl.git
DEBUG:AliPhysics:vAN-2015: + mkdir -p /build/workarea/sw/20150829/SOURCES/GSL/v1.16/release-1-16
DEBUG:AliPhysics:vAN-2015: + cd /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/GSL
DEBUG:AliPhysics:vAN-2015: + CACHED_TARBALL=
DEBUG:AliPhysics:vAN-2015: + [[ '' == '' ]]
DEBUG:AliPhysics:vAN-2015: + [[ ! -f /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/log ]]
DEBUG:AliPhysics:vAN-2015: + set -o pipefail
DEBUG:AliPhysics:vAN-2015: + bash -ex /build/workarea/sw/20150829/SPECS/GSL.sh
DEBUG:AliPhysics:vAN-2015: + tee /build/workarea/sw/20150829/BUILD/bbde5365b7427b311b02f8ea4d3b7576ac3a81c8/log
DEBUG:AliPhysics:vAN-2015: + Url=ftp://ftp.gnu.org/gnu/gsl/gsl-v1.16.tar.gz
DEBUG:AliPhysics:vAN-2015: + curl -o gsl.tar.gz ftp://ftp.gnu.org/gnu/gsl/gsl-v1.16.tar.gz
DEBUG:AliPhysics:vAN-2015:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
DEBUG:AliPhysics:vAN-2015:                                  Dload  Upload   Total   Spent    Left  Speed
DEBUG:AliPhysics:vAN-2015: 
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
DEBUG:AliPhysics:vAN-2015: curl: (19) RETR response: 550
DEBUG:AliPhysics:vAN-2015: 

Right now alibuild uses the following format for debug output:

DEBUG:<package>:<hash>: <output from the build>

Where package is the final package being built. It would be great if we could add an additional bit where the current package being built is show, e.g.:

DEBUG:<package>:<hash>:<current-package>: <output from the build>

Doing this requires:

• Trivial change in alibuild.
• Change https://github.com/alisw/ali-bot/blob/master/logstash/logstash.conf to parse the new bit and put it in a new field.
• Change the elasticsearch schema to avoid having the field being analysed.
• Deploy the new elasticsearch schema.
• Restart logstash instance in marathon so that the new configuration is picked up.

Installing ROOT with alibuild and custom GCC-toolchain on SL 6.6 fails (same enviroment as #208). The following components are loaded via module load ...:
Python 2.7.8
cmake version 2.8.12.1
gcc 5.2.0

For the installation process I used the following commands:
aliBuild init AliRoot,AliPhysics -z ali-master
aliDoctor 2>&1 | tee -a alidoctor.log
aliBuild -z -w ../sw build AliPhysics --jobs 10 --disable fastjet 2>&1 | tee -a build.log

ROOT compilation fails after 76%, see attached log. The error causing the problem, probably is in line 3164 of the log file.
alidoctor.log.txt
build.log.txt
log.txt

A dependency listed in require: is pulled in correctly but the depending package does not get built. See e.g. the change required for pythia6 in alisw/alidist#278.