omnia-md / conda-recipes Goto Github PK

View Code? Open in Web Editor NEW

24.0 8.0 37.0 1.56 MB

conda build recipes for the Omnia project

Home Page: http://omnia.md

License: MIT License

Shell 29.01% Python 50.63% Batchfile 16.25% PowerShell 3.75% C++ 0.36%

conda-recipes's Introduction

Travis-CI linuxand osx builds
Appveyor-CI win builds

omnia-md/conda-recipes

The recipes here create conda packages for scientific and numerical software components associated with the omnia project. The packages built from these recipes are shared with the community on anaconda.org. These packages also depend on the conda-forge conda channel.

Installing packages from omnia

To install a package (mdtraj for example)

# Add conda-forge and omnia to your channel list, one time action
conda config --add channels omnia --add channels conda-forge
# Install the 'mdtraj' a package
conda install mdtraj

To install a package in one line:

# channels searched in order they are added here
conda install -c conda-forge -c omnia mdtraj

When setting up the configuration through conda config, the channels are added to the top of the search priority sequentially. So conda config --add channels omnia --add channels conda-forge first adds omnia to the top of the list, then adds conda-forge on top of that, giving conda-forge the highest priority.

When temporarily searching through channels with conda install, the channels are prioritized in the order they are provided. So conda install -c conda-forge -c omnia prioritizes conda-forge first, then omnia.

Installing development packages

Some Omnia projects, such as openmm, have nightly development builds packaged and pushed to the Anaconda cloud. These can be found at https://anaconda.org/omnia-dev

If you want to install the development version of a particular package, use the omnia-dev channel. For example, to install the development snapshot of openmm, use:

conda install -c omnia-dev openmm

To ensure that development versions are always installed if available, add them to your conda channels (but only if you are sure you always want the bleeding-edge development snapshots!):

conda config --add channels omnia-dev

Migration to conda-forge

The Omnia project has started migrating to conda-forge. New packages that do not depend on OpenMM should be developed on conda-forge and existing packages which do not depend on OpenMM should start migrating if possible.

Planed Migration Stages

Update build image to CentOS 6 from CentOS 5

The base Docker image for linux builds will be updated to CentOS 6 with its new glibc. The base image is the conda-forge anvil, with some custom addons to include things like the AMD SDK, TexLive, and CUDA for GPU builds. The updated version will ensure packages can work on the conda-forge platform which is CentOS 6 based.
(Current) For packages that appear in conda-forge, remove the corresponding recipes in omnia

We want to minimize the amount of work we have to do as maintainers. To that end, we will stop building things which freely appear on conda-forge and maintained by someone other than us! For reproducibility purposes, we will keep our previously compiled versions, but they will not longer be updated.
- Developers: if you want users to still exclusivley use the omnia conda channel, please see the Copying from conda-forge section below
Allow recipes that do not depend on OpenMM to migrate from omnia to conda-forge
- Packages which do not depend on OpenMM and can be run on CPUs only should start migrating over to conda-forge in preparation for the total migration.
- Packages which can compile with just the conda-forge linux-anvil should also start migrating.
- We highly encourage devs of individual packages to start migrating now.
- Once a package is on conda-forge, it should no longer depend packages from omnia!
Determine the appropriate way to build packages which require more than the conda-forge linux-anvil can provide
- The conda-forge linux-anvil does not support some things such as TeXLive with requisite LaTeX packages (required by sphinx and/or OpenMM's sphinx configuration) and the CUDA Toolkit.
- We will need to reach out to the conda-forge people to see what the best course of action is
Migrate OpenMM to conda-forge
- This requires us to identify the best way to add the CUDA Toolkit and TeXLive to the conda-forge linux-anvil docker image. We have some ideas, but no concrete solution yet. In particular, the anaconda channel provides cudatoolkit, but not all versions are available yet. conda-forge is also experimenting with builds. conda-forge also has a texlive-core package, but tlmgr package installs are not supported; it may be possible to prepare selections of TeXLive packages.
Move the remainder of packages to conda-forge
- Also ensure that all former omnia packages can be installed without the omnia conda channel
Change this repo into an archive for reproducibility.

How to migrate to `conda-forge` (for existing packages)

PLACEHOLDER

Copying from Conda Forge

It can be a bit confusing to rely on two conda channels where the order they are specified in changes which version of packages are installed. During the migration to conda-forge, developers can copy their binary tarballs from conda-forge to the omnia channel using the Anaconda Cloud API, allowing users to rely only on the omnia channel. There are a couple conditions for this though:

Packages still built in omnia will search for dependencies in conda-forge then omnia, in that order
Your package should not depend on packages which only exist in omnia
You the developer will be responsible for also copying any dependencies from conda-forge to omnia that you need and are not on the default channel

To copy packages:

If you have write access to the omnia cloud

Open an Issue
- This is important so we can track changes made to the cloud in a public space
- Note which package, version, and any dependencies you are bringing over
Get the Anaconda CLI Tool
Copy the package with the CLI
- anaconda copy conda-forge/{PACKAGE}/{VERSION} --to-owner omnia
- Replace {PACKAGE} and {VERSION} accordingly
Copy any dependencies you need in the same way
Close the issue

If you do NOT have cloud write access

Open an issue, request which package and dependencies
Request a maintainer who has cloud write access follow the steps above.

Eventually, all packages will be on conda-forge and we won't have to worry about the multiple channels any more, until then, we thank you for your patience as we go through this transition.

Supported versions

Python packages are built against latest two releases of python (3.5 and 3.6) and python 2.7. Packages which have a binary dependency on numpy are built against the latest two releases of numpy (1.10 and 1.11).

WARNING: Numpy 1.09 support will be phased out now that numpy 1.11 has been released.

Building the packages

The recipes here are automatically built using Travis-CI for linux and osx and Appveyor-CI for win.

For linux builds, we use a modified version of the conda-forge linux-anvil, to ensure that the packages are fully compatible across multiple linux distributions and versions. This build image contains the additional tools:

TeXLive 2016
The CUDA Toolkit version 8.0
The AMD APP SDK 3.0

There is an additional image which has clang 3.8.1

To build a package yourself, run conda build <package_name>, or ./conda-build-all ./* to build multiple packages across each of the supported python/numpy configurations.

Contributing a recipe (this has not been updated to reflect the conda-forge changes)

Fork this repo
Add your conda recipe for building your package packagename in a subdirectory called packagename. Feel free to use other recipes here as examples.
Open a pull request to merge your branch into this master repo.
It will automatically be tested to make sure it compiles.
We will discuss the recipe and give suggestions about how to fix any issues.
The recipe will be merged and our automated build framework will build and deploy the packages to the omnia anaconda channel under the rc label.
Test the binaries by using conda install -c omnia/label/rc packagename
When you're sure the binaries are ready for a full release, comment on the original pull request and a maintainer will move the package from the rc label to the main label.

FAQ

Q: Should I include an md5 hash in my source: section if using a Github compressed archive url:?
A: No. Github compressed archives are frequently regenerated with different compression settings, etc., so md5 hashes cannot be trusted to be invariant. (#699)

conda-recipes's People

Contributors

Stargazers

Watchers

conda-recipes's Issues

The build for pdbfixer fails on osx jenkins

Seems to be due to a missing numpy dependency:

BUILD START: pdbfixer-1.1.dev0-py27_0
Fetching package metadata: ....
Solving package specifications: .+ /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/bin/python setup.py install
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    import numpy
ImportError: No module named numpy

The following NEW packages will be INSTALLED:

    fftw3f:     3.3.3-1   
    openmm:     6.2-py27_0
    openssl:    1.0.1j-4  
    python:     2.7.9-1   
    readline:   6.2-2     
    setuptools: 7.0-py27_0
    sqlite:     3.8.4.1-0 
    tk:         8.5.15-0  
    zlib:       1.2.7-1   

Linking packages ...
 |                                                                       |   0%
[sqlite-3.8.4.1-0 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  11%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |#     |  22%
[openssl-1.0.1j-4 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  33%
[tk-8.5.15-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###    |  44%
[readline-6.2-2 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  55%
[fftw3f-3.3.3-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  66%
[python-2.7.9-1 /Users/jenkins-slave/workspace/conda-omnia-release] |### |  77%
[openmm-6.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] | |  88%
[setuptools-7.0-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] || 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Source cache directory is: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/src_cache
Found source in cache: master.zip
Extracting download
Package: pdbfixer-1.1.dev0-py27_0
source tree in: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/work/openmmtools-master
Command failed: /bin/bash -x -e /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/conda-recipes/pdbfixer/build.sh

I think I can fix this.

openmm recipe troubles on OS X 10.9 and CUDA 6.5

On OS-X, using the recipe in this repo, I get the following error:

Applying patch: u'/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/conda-recipes/openmm/plugin-dir.patch'
Package: openmm-6.1-py27_0
source tree in: /Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/conda-bld/work/openmm-6.1
number of files: 379
Traceback (most recent call last):
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 110, in main
    args_func(args, p)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 308, in args_func
    args.func(args, p)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 268, in execute
    build.build(m, verbose=not args.quiet, post=post)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 359, in build
    post_build(m, sorted(files2 - files1))
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 226, in post_build
    mk_relative(m, f)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 197, in mk_relative
    mk_relative_osx(path)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 155, in mk_relative_osx
    s = macho.install_name_change(path, osx_ch_link)
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/macho.py", line 70, in install_name_change
    for link in otool(path):
  File "/Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/lib/python2.7/site-packages/conda_build/macho.py", line 56, in otool
    assert line[0] == '\t', path
AssertionError: /Users/Shared/Jenkins/Home/workspace/openmm-conda-osx/miniconda/envs/_build/lib/plugins/libOpenMMAmoebaCUDA.dylib
Build step 'Execute shell' marked build as failure
Finished: FAILURE

Issues installing openmm 6.2 conda package on latest miniconda

I've just run into this odd error from both an existing and a fresh (see below) miniconda install. Has anyone else seen this?

[LSKI1497:~] choderaj% sh Miniconda-3.7.0-MacOSX-x86_64.sh -p /Users/choderaj/anaconda -b
PREFIX=/Users/choderaj/anaconda
installing: python-2.7.8-1 ...
installing: openssl-1.0.1h-1 ...
installing: pycosat-0.6.1-py27_0 ...
installing: pyyaml-3.11-py27_0 ...
installing: readline-6.2-2 ...
installing: requests-2.4.1-py27_0 ...
installing: sqlite-3.8.4.1-0 ...
installing: tk-8.5.15-0 ...
installing: yaml-0.1.4-1 ...
installing: zlib-1.2.7-1 ...
installing: conda-3.7.0-py27_0 ...
Python 2.7.8 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
[LSKI1497:~] choderaj% conda config --add channels https://conda.binstar.org/omnia
Skipping channels: https://conda.binstar.org/omnia, item already exists
[LSKI1497:~] choderaj% conda install openmm
Fetching package metadata: .......
Solving package specifications: .
Package plan for installation in environment /Users/choderaj/anaconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-3.7.4                |           py27_0         160 KB
    fftw3f-3.3.3               |                1         3.2 MB
    openmm-6.2                 |           py27_0         7.9 MB
    openssl-1.0.1j             |                4         2.5 MB
    python-2.7.9               |                1        11.3 MB
    requests-2.5.1             |           py27_0         586 KB
    ------------------------------------------------------------
                                           Total:        25.6 MB

The following NEW packages will be INSTALLED:

    fftw3f:   3.3.3-1     
    openmm:   6.2-py27_0  

The following packages will be UPDATED:

    conda:    3.7.0-py27_0 --> 3.7.4-py27_0
    openssl:  1.0.1h-1     --> 1.0.1j-4    
    python:   2.7.8-1      --> 2.7.9-1     
    requests: 2.4.1-py27_0 --> 2.5.1-py27_0

Proceed ([y]/n)? 

Fetching packages ...
conda-3.7.4-py 100% |################################| Time: 0:00:00 222.59 kB/s
fftw3f-3.3.3-1 100% |################################| Time: 0:00:12 263.85 kB/s
openmm-6.2-py2 100% |################################| Time: 0:00:15 531.09 kB/s
openssl-1.0.1j 100% |################################| Time: 0:00:08 303.36 kB/s
python-2.7.9-1 100% |################################| Time: 0:00:33 349.15 kB/s
requests-2.5.1 100% |################################| Time: 0:00:01 521.36 kB/s
Extracting packages ...
[      COMPLETE      ] |##################################################| 100%
Unlinking packages ...
[      COMPLETE      ] |##################################################| 100%
Linking packages ...
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
mv: illegal option -- t
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
Error: Error: post-link failed for: openmm-6.2-py27_0                     |  50%

cvxpy is not python 3.3 or 3.4 compatible

The conda-build-all script produces lots of errors when trying to build with python 3.3 or 3.4.

Is this OK, @rmcgibbo? Do any of the tools that require it need to work with python 3.3 or 3.4?

Building separate conda packages for release and master versions of Omnia components

We discussed this during the recent Omnia phone call. Just making an issue here as a placeholder.

openmm: Move examples to $PREFIX/share/openmm/examples

We currently dump the OpenMM examples into $PREFIX/examples, which for most users will just dump everything into ~/anaconda/examples/ without any indication this is from OpenMM.

We should probably move these into $PREFIX/share/openmm/examples or somesuch.

Which CUDA version will Omnia be linked against?

See this thread, which suggests we may be stuck with CUDA 6.5 on Linux if we want to support Redhat.

Move Omnia conda-recipes to SimTK?

Should we move Omnia-related conda-recipes to github.com/simtk?

Trouble with automated build-all.py on Linux vagrant VM with Jenkins

The Jenkins build using the Linux vagrant VM dies trying to build docs on OpenMM:

gmake[4]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
gmake[4]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
/home/vagrant/miniconda/conda-bld/work/openmm-6.2/docs-source/usersguide/header.rst:: WARNING: document isn't included in any toctree
gmake[5]: *** [OpenMMDeveloperGuide.pdf] Error 1
gmake[4]: *** [latexpdf] Error 2
make[3]: *** [sphinx-docs/developerguide/latex/OpenMMDeveloperGuide.pdf] Error 2
make[3]: *** Waiting for unfinished jobs....
gmake[5]: *** [OpenMMUsersGuide.pdf] Error 1
gmake[4]: *** [latexpdf] Error 2
make[3]: *** [sphinx-docs/userguide/latex/OpenMMUsersGuide.pdf] Error 2
make[2]: *** [docs-source/CMakeFiles/sphinxpdf.dir/all] Error 2
make[1]: *** [docs-source/CMakeFiles/sphinxpdf.dir/rule] Error 2
make: *** [sphinxpdf] Error 2

I'm wondering if some prereqs for building the docs are missing here.

openmm-6.2-py27_0.tar.bz2 contains some erroneous files

If you look in the openmm linux py27 package, there are some erroneous files.

$ wget https://binstar.org/omnia/openmm/6.2/download/linux-64/openmm-6.2-py27_0.tar.bz2
$ tar -tjvf openmm-6.2-py27_0.tar.bz2

In lib/python2.7/site-packages/, there's a full copy of sphinx and sphinxcontrib-bibtex. I don't think this is terrible, but it's not what we want.

Alternative build strategy

We're focusing on using build-all.py to build omnia for the moment, but I wonder if there wasn't a simpler strategy that involved creating a separate conda environment for each Python/NumPy environment and building packages in this environment.

For example, this would look something like

for CONDA_PY in ['2.7', '3.3', '3.4']:
    for CONDA_NPY in ['1.7', '1.8', '1.9']:
       conda create -n omnia python=$CONDA_PY numpy=$CONDA_NPY --yes
       source activate omnia
       conda build omnia
       # upload to binstar here...
       source deactivate
       rm -rf $HOME/anaconda/envs/omnia

binstar error

The latest jenkins linux build encountered a binstar error when pushing to the omnia channel:
https://jenkins.choderalab.org/job/conda-omnia-release-linux-vagrant/14/console

+ vagrant ssh --command 'binstar --quiet -t **** upload --force -u omnia --channel main miniconda/conda-bld/*/*.tar.bz2'
Using binstar api site https://api.binstar.org
[warning] Distribution linux-64/cvxpy-0.2.15-np18py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/ecos-1.0.5-py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/ecos-1.0.5-py34_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/ambermini-14.0.1-0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/docopt-0.6.2-py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/docopt-0.6.2-py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/docopt-0.6.2-py34_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/ecos-1.0.5-py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np18py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np18py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np18py34_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np19py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np19py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fastcluster-1.1.13-np19py34_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/fftw3f-3.3.3-1.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/jpype1-0.5.7-np18_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/jpype1-0.5.7-np19_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/latexcodec-1.0.1-py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/latexcodec-1.0.1-py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/latexcodec-1.0.1-py34_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/mdtraj-1.2.0-np18py27_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/mdtraj-1.2.0-np18py33_0.tar.bz2 already exists ... removing 
[warning] Distribution linux-64/mdtraj-1.2.0-np18py34_0.tar.bz2 already exists ... removing 
[SSLError] EOF occurred in violation of protocol (_ssl.c:581) 
Traceback (most recent call last):
  File "/home/vagrant/miniconda/bin/binstar", line 6, in <module>
    sys.exit(main())
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 94, in main
    description=__doc__, version=version)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 76, in binstar_main
    return args.main(args)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/binstar_client/commands/upload.py", line 203, in main
    callback=upload_print_callback(args))
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/binstar_client/__init__.py", line 418, in upload
    s3res = requests.post(s3url, data=data_stream, verify=self.session.verify, timeout=10 * 60 * 60, headers=headers)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/requests/api.py", line 99, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/vagrant/miniconda/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
SSLError: EOF occurred in violation of protocol (_ssl.c:581)

Anyone ever seen this before?

Consolidate instructions for continuous integration here

So right now, I have 4 copies of the CI scripts--e.g. the tools/ci and tools/conda-recipe/ directories. There is quite a bit of duplication. It might be nice to factor out some of that and put it here.

The first things to factor out are the two README.md files, which have already gotten out of sync between the repositories (MDTraj, Gaff2XML, Yank, Repex, PyMBAR). We can just have the local README.md files point to this repository for the actual instructions.

Eventually, it might be useful to factor out some of the actual scripts. For example, I have never actually changed the after_success.sh script.

Recent Yank builds are missing bin directory

https://binstar.org/omnia/yank/files

This is missing the bin directory:
https://binstar.org/omnia/yank/0.0.0/download/linux-64/yank-0.0.0-np19_0.tar.bz2

This has the bin directory
https://binstar.org/omnia/yank/0.0.0/download/linux-64/yank-0.0.0-np19py27_0.tar.bz2

Another dependency resolution bug

I found another dependency resolution bug in conda-build, which is related to conda/conda-build#291, but I think harder to fix.

The issue can be seen on L2676-L2676 of the first travis build, where you start seeing py3 to build the dependencies properly. I can reproduce this issue locally. The problem is that we're trying to build the py33 mdtraj package. It depends on scripttest, of which a py27 version has been built, but not a py33 version. So ideally, before contining with the py33 mdtraj build, it should build scripttest.

But the mechanism that conda-build uses to do this is actually quite brittle, and it fails. Basically, because of the existance of the scripttest py27 package, the SAT solver fails. This gives a different error message than when the py27 package doesn't exist. conda-build tries to explicitly parse the error message, but since it's different, it doesn't work.

Packaging tests

What do you guys think about whether and how we should package tests?

I would really like to have some way to confirm that everything is working correctly on the platforms that have been installed for OpenMM. But it may not be a great idea to throw a bunch of tests directly into the $PATH.

We currently have the OpenMM tests packaged up with the conda build:

https://github.com/pandegroup/openmm/blob/master/devtools/conda-recipe/build.sh#L51-L52

This approach works, but

There isn't a script to run through the tests
The tests take up ~14MB
The tests all end up on the $PATH

I wonder if there's another approach that might serve the same purpose but fix these issues, such as a small Python script that contains some high-level tests just as a sanity check that can be run from the $PATH.

Build for pyhmc conda package fails on jenkins osx

Might be for source code checkout reasons.

The output:

BUILD START: pyhmc-0.1.1-np18py27_0
Fetching package metadata: ....
Solving package specifications: .Cloning into bare repository '/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/git_cache/rmcgibbo_pyhmc.git'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda-build/issues

Include the output of the command 'conda info' in your report.



The following NEW packages will be INSTALLED:

    cython:     0.21.2-py27_0
    numpy:      1.8.2-py27_0 
    openssl:    1.0.1j-4     
    python:     2.7.9-1      
    readline:   6.2-2        
    setuptools: 7.0-py27_0   
    sqlite:     3.8.4.1-0    
    tk:         8.5.15-0     
    zlib:       1.2.7-1      

Linking packages ...
 |                                                                       |   0%
[readline-6.2-2 /Users/jenkins-slave/workspace/conda-omnia-release] |    |  11%
[sqlite-3.8.4.1-0 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  22%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##    |  33%
[openssl-1.0.1j-4 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  44%
[tk-8.5.15-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###    |  55%
[python-2.7.9-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  66%
[cython-0.21.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  77%
[numpy-1.8.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  88%
[setuptools-7.0-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] || 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Traceback (most recent call last):
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 110, in main
    args_func(args, p)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 312, in args_func
    args.func(args, p)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 272, in execute
    build.build(m, verbose=not args.quiet, post=post)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 310, in build
    source.provide(m.path, m.get_section('source'))
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 236, in provide
    git_source(meta, recipe_dir)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 100, in git_source
    check_call([git, 'clone', '--mirror', git_url, cache_repo_arg], cwd=recipe_dir)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/git', 'clone', '--mirror', u'[email protected]:rmcgibbo/pyhmc.git', u'/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/git_cache/rmcgibbo_pyhmc.git']' returned non-zero exit status 128

Is this needed for omnia 1.0?

Update conda-build-all mechanism to be aware of which python versions each recipe supports

Not all packages in omnia support python 3.x versions, so we will need to build some sort of infrastructure that tracks which versions of python are supported by each package and only builds these versions.

issue with clean openmm via conda

I tried to upgrade from 6.0 to 6.1 with a clean conda install of openmm, using:

conda install -c https://conda.binstar.org/omnia openmm

This is the error I get:
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/cli/main_install.py", line 167, in execute
plan.execute_actions(actions, index, verbose=not args.quiet)
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/plan.py", line 374, in execute_actions
execute_plan(plan, index, verbose)
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/plan.py", line 359, in execute_plan
link(prefix, arg)
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/plan.py", line 314, in link
install.link(pkgs_dir, prefix, dist, lt)
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/install.py", line 386, in link
update_prefix(join(prefix, f), prefix)
File "/Users/antonia/anaconda/lib/python2.7/site-packages/conda/install.py", line 167, in update_prefix
with open(path, 'rb') as fi:
IOError: [Errno 2] No such file or directory: '/Users/jenkins-slave/workspace/conda-openmm-release-osx/miniconda/envs/_build text lib/python2.7/site-packages/simtk/openmm/version.py'

Any ideas why this might be or how to fix it? Thanks for the help.

Build 'omnia' recipe for installing entire Omnia universe

Clean issues and make release

So it looks like our things are close to working, as py27 on linux is able to grab the latest omnia metapackage (https://binstar.org/omnia/omnia/files). This suggests that once we iron out the outstanding issues and clear the issue tracker of deprecated issues, we can make a 1.0 release. We should probably also make a release on the virtual-machines repo, mostly for our record keeping.

I also added the install instructions to the web page (http://www.omnia.md/install/).

glibc issue with fftw3f

I was doing a ./conda-build-all openmm in my Centos VM and I saw this:

[ 67%] Built target TestVectorize
Scanning dependencies of target TestReferenceDrudeForce
[ 67%] /home/vagrant/miniconda/envs/_build/lib/libfftw3f.so: undefined reference to `memcpy@GLIBC_2.14'
/home/vagrant/miniconda/envs/_build/lib/libfftw3f.so: undefined reference to `__log_finite@GLIBC_2.15'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I believe I'm getting my ( fftw3f: 3.3.3-1) from binstar (https://binstar.org/omnia/fftw3f/files). Any idea on the provenance of that binary? E.g. was it built from our Ubuntu 12.04 boxes? We might want to disable uploading those to binstar (or use a separate channel?), as we know those builds likely have glibc incompatabilties and should not be broadly deployed.

Unsatisfiable package specifications

I'm still running into this error when trying to build packages:

Error: Unsatisfiable package specifications.
Generating hint: 

Hint: the following combinations of packages create a conflict with the
remaining packages:
  - sphinxcontrib-bibtex
  - python 2.6*
Building conda packages on grid of python / numpy versions
This will take a while... (15 minutes or more)

See:
https://jenkins.choderalab.org/job/conda-omnia-release-osx/18/console

This occurs when trying to run python build-all.py.

Anybody have any ideas here?

Add py3.4 to the list of pythons

People are using py3.4 now, so I think it's a platform we should provide binaries for. cc: https://github.com/rmcgibbo/mdtraj/issues/539

Some packages (e.g. msmbuilder) are not being built by jenkins

Some packages---notably, msmbuilder---are not being built by jenkins.

No information about them appears in the console output:
https://jenkins.choderalab.org/job/conda-omnia-release-linux-vagrant/16/console

Not sure why this is. Debugging.

swig dependency issues

I'm still having trouble with this swig issue that @kyleabeauchamp ran into:

BUILD START: openmm-dev-dev-np19py27_0
Fetching package metadata: ....
Solving package specifications: .New string '/home/jenkins/workspace/conda-openmm-dev-linux-native/miniconda/envs/_build/share/swig/2.0.10' is longer than old string '/opt/anaconda1anaconda2anaconda3/share/swig/2.0.10'. Skipping.

Has there been any progress here?

Rebuilding linux openmm

So I noticed that we didn't have any openmm6.2 py3x packages (https://binstar.org/omnia/openmm/files) in the main channel, which was probably because those packages already existed in the omnia-beta channel from my previous manual builds. I therefore deleted the manual packages and restarted the build on jenkins to see if we can get everything built by monday

Add testing of OpenMM conda to Jenkins build script before binstar upload

Did someone delete the mdtraj-1.2.0 binaries from binstar?

Many of the mdtraj 1.2.0 binaries have gone missing from the main channel. See in particular the linux and OS-X builds, which now only have py26 versions.
https://binstar.org/omnia/mdtraj/files?version=1.2.0&channel=main

This, for example, has broken the msmbuilder travis. https://travis-ci.org/msmbuilder/msmbuilder/jobs/46279339, which was working as of 15 days ago (the last PR https://travis-ci.org/msmbuilder/msmbuilder/builds/45003144).

Having trouble with automated build-all.py on osx Jenkins

The build dies partway through trying to build msmbuilder:

conda build --output msmbuilder
Traceback (most recent call last):
  File "build-all.py", line 24, in <module>
    build('msmbuilder')
  File "build-all.py", line 14, in build
    shell('conda build %s' % name)
  File "build-all.py", line 5, in <lambda>
    shell = lambda cmd: check_output(cmd, shell=True)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx/miniconda/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'conda build msmbuilder' returned non-zero exit status 1

Unfortunately, there's not much more helpful information.

Full output is here

Any ideas?

Debugging dependency resolution

For the sake of debugging the dependency resolution and "unsatisfiable package requirements" type stuff, a really easy way to speed up the debugging is just to replace all of the build.sh scripts with a no-op, so that the packages build instaneously, but still have the same dependency structure.

$ cat mock-packages
#!/bin/bash
for recipe in "$@"; do
    if [ -f "$recipe/build.sh" ]; then
        echo "mocking $recipe"
        echo "echo 'building $recipe'" > $recipe/build.sh
    fi;
done
$ chmod +x mock-packages
$ mock-package ./*

Weirdness with jenkins builds

All sorts of weirdness is going on now.

For osx:
https://jenkins.choderalab.org/job/conda-omnia-release-osx-2/31/consoleFull

I see stuff like:

from distutils.version import StrictVersion
curr = StrictVersion(platform.mac_ver()[0])
mavericks = StrictVersion('\''10.9'\'')
if not hasattr(curr, '\''version'\''): # not on apple
   sys.exit(1)
if curr >= mavericks:
   sys.exit(0)
else:
   sys.exit(1)
'
+ /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/bin/python -c 'import platform; import sys
from distutils.version import StrictVersion
curr = StrictVersion(platform.mac_ver()[0])
mavericks = StrictVersion('\''10.9'\'')
if not hasattr(curr, '\''version'\''): # not on apple
   sys.exit(1)
if curr >= mavericks:
   sys.exit(0)
else:
   sys.exit(1)
'
+ echo 'on mavericks or higher'
on mavericks or higher

and

+ binstar --quiet -t **** upload -u omnia --channel main 'miniconda/conda-bld/*/*.tar.bz2'
Using binstar api site https://api.binstar.org
[BinstarError] file miniconda/conda-bld/*/*.tar.bz2 does not exist 
Traceback (most recent call last):
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/bin/binstar", line 6, in <module>
    sys.exit(main())
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 94, in main
    description=__doc__, version=version)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 76, in binstar_main
    return args.main(args)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/commands/upload.py", line 162, in main
    raise BinstarError('file %s does not exist' % (filename))
BinstarError: file miniconda/conda-bld/*/*.tar.bz2 does not exist

On linux, the vagrant box is no longer working:

+ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: VirtualBox VM is already running.
+ vagrant ssh --command 'rm -f ~/.condarc && true'
ssh_exchange_identification: Connection closed by remote host

Reduce repo size

We may be able to use bfg to reduce the repo size by getting rid of large old unused files in the git history.

Conflicts with existing packages

Have you seen these messages in the osx jenkins build?
https://jenkins.choderalab.org/job/conda-omnia-release-osx-2/30/console

+ binstar --quiet -t **** upload -u omnia --channel main miniconda/conda-bld/broken/repex-dev-py27_0.tar.bz2 miniconda/conda-bld/osx-64/gaff2xml-0.0.0-py33_0.tar.bz2 miniconda/conda-bld/osx-64/gaff2xml-0.0.0-py34_0.tar.bz2 miniconda/conda-bld/osx-64/pybtex-docutils-0.2.0-py27_0.tar.bz2 miniconda/conda-bld/osx-64/pybtex-docutils-0.2.0-py33_0.tar.bz2 miniconda/conda-bld/osx-64/pybtex-docutils-0.2.0-py34_0.tar.bz2 miniconda/conda-bld/osx-64/sphinxcontrib-bibtex-0.3.1-py27_0.tar.bz2 miniconda/conda-bld/osx-64/sphinxcontrib-bibtex-0.3.1-py33_0.tar.bz2 miniconda/conda-bld/osx-64/sphinxcontrib-bibtex-0.3.1-py34_0.tar.bz2
Using binstar api site https://api.binstar.org
[Conflict] file osx-64/pybtex-docutils-0.2.0-py27_0.tar.bz2 already exists for package pybtex-docutils version 0.2.0 
Traceback (most recent call last):
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/bin/binstar", line 6, in <module>
    sys.exit(main())
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 94, in main
    description=__doc__, version=version)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/scripts/cli.py", line 76, in binstar_main
    return args.main(args)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/commands/upload.py", line 203, in main
    callback=upload_print_callback(args))
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/__init__.py", line 398, in upload
    self._check_response(res)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/binstar_client/__init__.py", line 155, in _check_response
    raise ErrCls(msg, res.status_code)
Conflict: (u'file osx-64/pybtex-docutils-0.2.0-py27_0.tar.bz2 already exists for package pybtex-docutils version 0.2.0', 409)

Build step 'Execute shell' marked build as failure
Finished: FAILURE

This flags the jenkins run as a failure, but does this mean packages are not actually being uploaded when they should?

Also, isn't the new build script supposed to be not rebuilding recipes that are already present, meaning we shouldn't be seeing conflicts?

Build failure for packmol on jenkins osx

The build for packmol fails because build.sh tries to use /usr/bin/gfortran; note that this is often installed in a different location on osx (e.g. /usr/local/bin/gfortran):

BUILD START: packmol-1.0.0-0
Fetching package metadata: ....
Solving package specifications: .+ export 'CFLAGS=-I/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/include -arch x86_64'
+ CFLAGS='-I/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/include -arch x86_64'
+ export 'LDFLAGS=-L/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/lib -arch x86_64'
+ LDFLAGS='-L/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/lib -arch x86_64'
+ make
 ------------------------------------------------------ 
 Compiling packmol with /usr/bin/gfortran 
 Flags: -O3 -ffast-math  
 ------------------------------------------------------ 
make: /usr/bin/gfortran: Permission denied
make: *** [serial] Error 1

The following NEW packages will be INSTALLED:

    zlib: 1.2.7-1

Linking packages ...
 |                                                                       |   0%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |######| 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Source cache directory is: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/src_cache
Downloading source to cache: packmol.tar.gz
packmol.tar.gz   0% |                              | ETA:  --:--:--   0.00  B/s
packmol.tar.gz  15% |####                           | ETA:  0:00:00  16.93 MB/s
packmol.tar.gz  31% |#########                      | ETA:  0:00:00  24.98 MB/s
packmol.tar.gz  47% |##############                 | ETA:  0:00:00  29.71 MB/s
packmol.tar.gz  63% |###################            | ETA:  0:00:00  32.96 MB/s
packmol.tar.gz  79% |########################       | ETA:  0:00:00  35.37 MB/s
packmol.tar.gz  95% |#############################  | ETA:  0:00:00  37.12 MB/s
packmol.tar.gz 100% |###############################| ETA:  0:00:00  34.66 MB/s
packmol.tar.gz 100% |###############################| Time: 0:00:00  31.35 MB/s
Extracting download
Package: packmol-1.0.0-0
source tree in: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/work/packmol
Command failed: /bin/bash -x -e /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/conda-recipes/packmol/build.sh

Is packmol needed for omnia 1.0?

fftwf building issues on os-x

Testing build-all.py on our OS X 10.9 build machine ended up with some weird errors:

+ [[ darwin13 == \d\a\r\w\i\n* ]]
+ ./configure --prefix=/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build --enable-shared --disable-fortran --enable-threads --enable-sse2 --enable-single 'CC=gcc -arch i386 -arch x86_64' 'CXX=g++ -arch i386 -arch x86_64' 'CPP=gcc -E' 'CXXCPP=g++ -E'
+ make
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libkernel.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libkernel.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libsimd_support.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libsimd_support.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libsimd_support.a(neon.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libsimd_support.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libsimd_support.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libsimd_support.a(neon.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libapi.a(f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libapi.a(f77api.o) has no symbols
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f.a(f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f.a(neon.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f.a(f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f.a(neon.o) has no symbols
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: .libs/libfftw3f_threads.a(libfftw3f_threads_la-f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: .libs/libfftw3f_threads.a(libfftw3f_threads_la-f77api.o) has no symbols
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib'
+ make install
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(neon.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(debug.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(altivec.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(avx.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.a(neon.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: i386 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f_threads.a(libfftw3f_threads_la-f77api.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: for architecture: x86_64 file: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f_threads.a(libfftw3f_threads_la-f77api.o) has no symbols


An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda-build/issues

Include the output of the command 'conda info' in your report.


Traceback (most recent call last):
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 110, in main
    args_func(args, p)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 312, in args_func
    args.func(args, p)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 272, in execute
    build.build(m, verbose=not args.quiet, post=post)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 338, in build
    post_build(m, sorted(files2 - files1))
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 229, in post_build
    mk_relative(m, f)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 200, in mk_relative
    mk_relative_osx(path)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 180, in mk_relative_osx
    assert_relative_osx(path)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/site-packages/conda_build/post.py", line 184, in assert_relative_osx
    assert not name.startswith(config.build_prefix), path
AssertionError: /Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/envs/_build/lib/libfftw3f.3.dylib
Building python2.7 and python3.3 conda packages
This will take a while... (15 minutes or more)
conda build --output fftw3f
Traceback (most recent call last):
  File "build-all.py", line 16, in <module>
    build('fftw3f')
  File "build-all.py", line 14, in build
    shell('conda build %s' % name)
  File "build-all.py", line 5, in <lambda>
    shell = lambda cmd: check_output(cmd, shell=True)
  File "/Users/Shared/Jenkins/Home/workspace/conda-all-release-osx/miniconda/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'conda build fftw3f' returned non-zero exit status 1
Build step 'Execute shell' marked build as failure
Finished: FAILURE

Which Python versions should we support?

Currently, I think it's

python 2.7
python 3.3
python 3.4

Is 3.3 necessary?

Naming conventions for binstar release and development packages

How should we name the release and development packages for binstar?

I believe we currently use:

packagename release X.Y.Z for actual releases
packagename-dev release dev for development builds

Is it worth sticking with this?

We experimented with using packagename release dev for development builds, but I believe conda install packagename brings back the "latest" package by default by some comparison scheme of release numbers which seems to yield the dev package as the latest version. @kyleabeauchamp suggested 0.0.0 could be the development version, but that may be odd.

What should we do with travis after_success scripts that push to binstar?

I just realized we still have a number of live after_success.sh scripts run by travis-ci with lines like this:

if [[ "2.7 3.3 3.4" =~ "$python" ]]; then                                                                                                                                           
    conda install --yes binstar                                                                                                                                                     
    binstar -t $BINSTAR_TOKEN upload --force -u omnia -p openmmtools $HOME/miniconda/conda-bld/linux-64/${PACKAGENAME}-*                                                            
fi

We presumably do not want to keep these active, since they might overwrite our released packages.

What should we do instead, considering the issues we've had with pushing to a different channel overwriting packages in the main channel? Should we have a separate omnia-dev org?

Packaging docs

Should conda packages include docs as well?

Duplicated recipes?

So can @rmcgibbo recall why it's a good idea to have our recipes here, rather than the in the project repositories? I'm just curious if it's more than trouble than it's worth...

Testing omnia

I've started to build a preliminary Jenkins-based omnia test:
https://jenkins.choderalab.org/job/test-conda-omnia-release-osx/

It creates a fresh miniconda installation and cycles through python 2.7, 3.3, and 3.4, creating a conda environment for each in which it attempts to install omnia. We can add some simple import packagename tests too.

More generally, how should we actually test omnia? I think @frabjous5 was putting together some IPython notebooks that show off how to use multiple components of omnia together---can we use these as test scripts?

Investigate binstar build for building omnia packages

The binstar-build capability looks interesting. Anyone know more about this?

OpenMM package issues

I just built a Python 2.7 OpenMM conda package in the vagrant VM:
https://binstar.org/omnia/openmm

I'm running into trouble installing it even with a clean miniconda installation:

[chodera@mskcc-ln1 ~]$ conda install -c https://conda.binstar.org/omnia openmm
...
Traceback (most recent call last):
  File "/cbio/jclab/home/chodera/miniconda/bin/conda", line 5, in <module>
    sys.exit(main())
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/cli/main.py", line 201, in main
    args_func(args, p)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/cli/main.py", line 206, in args_func
    args.func(args, p)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/cli/main_install.py", line 46, in execute
    install.install(args, parser, 'install')
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/cli/install.py", line 397, in install
    plan.execute_actions(actions, index, verbose=not args.quiet)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/plan.py", line 587, in execute_actions
    execute_plan(plan, index, verbose)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/plan.py", line 569, in execute_plan
    link(prefix, arg, index=index)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/plan.py", line 525, in link
    install.link(pkgs_dir, prefix, dist, lt, index=index)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/conda/install.py", line 509, in link
    os.makedirs(dst_dir)
  File "/cbio/jclab/home/chodera/miniconda/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/cbio/jclab/home/chodera/miniconda/lib/python2.7/site-packages/PyYAML-3.11-py2.7.egg-info'

Has anyone seen this before?

Does the current openmm conda package already support OpenCL?

I notice that openmm/build.sh has the line

    CMAKE_FLAGS+=" -DOPENCL_LIBRARY=/opt/AMDAPP/lib/x86_64/libOpenCL.so"

Does this mean the conda package already compiles the OpenCL platform?

Repo is Giant

Why is the repo like 100MB to download?

Add emma to omnia metapackage

Enable travis-ci to test recipes by trying to build all of them (but not upload to binstar)

We can test all of these recipes now using a scheme similar to that @rmcgibbo and @kyleabeauchamp worked out for mdtraj, repex, yank, and openmm by having travis-ci attempt to conda build packagename; conda install packagename for each package.

Later, we can also have it build and test an omnia package.

pymbar recipe is not python 3.x compatible

This is a known issue that we may punt on.

conda-omnia-release-osx-2 config

This line in the configuration is not correct:

miniconda/bin/pip install --force-reinstall ordereddict sphinxcontrib-bibtex # needed for OpenMM