conda-tools / conda-build-all Goto Github PK

Would like to tag a new release using the current master. Based on the noarch functionality, am leaning towards calling it 1.1.0 (instead of 1.0.5). Any thoughts on the version or releasing?

cc @conda-tools/core

The master branch of conda-build introduces a lot of somewhat disturbing output into conda-build-all. Specifically it looks like this:

$ conda-build-all /tmp/tagged-recipes/recipes --upload-channels nsls2builder --matrix-conditions "numpy >=1.8" "python >=2.7,<3|>=3.4" --inspect-channels nsls2builder
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ..............
Resolving distributions from 120 recipes... 
fatal: ambiguous argument 'v0.1.0': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'v0.1.0': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
...

The offending commit this this: conda/conda-build@7ce7be2

Modifying https://github.com/conda/conda-build/blob/master/conda_build/environ.py#L76 to add a pdb.set_trace() allows me to determine that conda-build is expecting the source code for the recipe that conda-build-all is asking about to have been checked out into the work directory.

The actual exception is

Command '['git', 'log', '-n1', '--format=%H', 'v0.1.0']' returned non-zero exit status 128

Here are the values for the variables that are local to conda_build.environ.get_git_build_info() at this point in the program execution:

expected_rev: 'v0.1.0'
src_dir: '/home/edill/mc/conda-bld/work'
git_url: 'http://github.com/kennethreitz/args'
current_commit: '1f4685f3f67dcf8442b16aa1204b3ed72ce5bb2c\n'  # note that the current_commit needs to be stripped

and the contents of src_dir are the last thing that I did with conda-build,

$ ls ~/mc/conda-bld/work
build                     conda-build-all.recipe  LICENSE      README.md         setup.cfg
conda_build_all           conda_build.sh          MANIFEST.in  record.txt        setup.py
conda_build_all.egg-info  CONTRIBUTING.md         __pycache__  requirements.txt  versioneer.py

I do not see a clear solution here, or I would implement something and submit a PR and we could discuss further on the PR. That being said, I am happy to work on fixing this, but will need guidance from you. What are your thoughts? cc @pelson

Also cc @stuarteberg since you might be able to provide some guidance on the conda-build side of things.

And the versions of conda related things on my system

$ conda-build --version
conda-build 1.19.0+4.g0491732
$ conda-build-all --version
0.9.0
$ git --version
git version 2.5.0
hub version 2.2.2
$ conda --version
conda 3.19.1

[edit] add versions

I am attempting to use conda-build-all to upload to our internal anaconda server and am getting the following traceback.

Traceback (most recent call last):
  File "/tmp/root/ramdisk/mc/bin/conda-build-all", line 9, in <module>
    load_entry_point('conda-build-all==0.8.1.dev14', 'console_scripts', 'conda-build-all')()
  File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/cli.py", line 85, in main
    b.main()
  File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/builder.py", line 208, in main
    self.post_build(meta, built_dist_location, was_built)
  File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/builder.py", line 225, in post_build
    artefact_destination.make_available(meta, built_dist_location, was_built)
  File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/artefact_destination.py", line 119, in make_available
    raise NotImplementedError('cross owner copying not yet implemented.')
NotImplementedError: cross owner copying not yet implemented.

I am executing conda-build-all with

$ conda-build-all ../staged-recipes-dev/ --upload-channels nsls2-dev --inspect-channels nsls2-dev --matrix-conditions "python >=3.5"

I'm not convinced that I am trying to copy across owners which makes this error all the more puzzling. Additionally puzzling is that this is failing on different packages each time I run the aforementioned conda-build-all command.

Have you seen this before?

So I did a bit of investigation after I wrote the stuff above, but before I posted the issue. Updating to conda-build master seems to fix this problem. I will go ahead and post this and then close it immediately, mostly for posterity's sake.

I'm guessing this has something to do with the fact that this was originally a fork of your copy (which is still indexed).

Not a huge priority...

I'm running into this issue with 0.13.0 and 0.13.1, but not with 0.12.0. It sometimes fails to get all of the items in a matrix that I would expect. An example would be trying to run conda smithy rerender on this feedstock at that commit. It only provides Python with 3.5 and not any earlier versions. Though there is no version constraint or skip provided. I have provided all of the version info below to help narrow this down.

conda list -n root
anaconda-client           1.4.0                    py27_0    defaults
anaconda-verify           1.1.0                    py27_0    defaults
conda                     4.1.8                    py27_0    defaults
conda-build               1.21.5                   py27_0    defaults
ca-certificates           2016.2.28                     1    conda-forge
clyent                    1.2.1                    py27_0    conda-forge
conda-build-all           0.12.0                   py27_1    conda-forge
conda-env                 2.5.2                    py27_0    conda-forge
conda-execute             0.6.0                    py27_0    conda-forge
conda-smithy              1.0.1                    py27_0    conda-forge
curl                      7.49.1                        0    conda-forge
expat                     2.1.0                         1    conda-forge
funcsigs                  1.0.2                    py27_0    conda-forge
gitdb                     0.6.4                    py27_0    conda-forge
gitpython                 1.0.1                    py27_0    conda-forge
jinja2                    2.8                      py27_0    conda-forge
markupsafe                0.23                     py27_0    conda-forge
mock                      2.0.0                    py27_0    conda-forge
ncurses                   5.9                           7    conda-forge
openssl                   1.0.2h                        0    conda-forge
psutil                    4.3.0                    py27_0    conda-forge
pycosat                   0.6.1                    py27_0    conda-forge
pycrypto                  2.6.1                    py27_0    conda-forge
pygithub                  1.26.0                   py27_0    conda-forge
python-dateutil           2.5.2                    py27_0    conda-forge
pytz                      2016.3                   py27_0    conda-forge
pyyaml                    3.11                     py27_0    conda-forge
readline                  6.2                           0    conda-forge
requests                  2.10.0                   py27_0    conda-forge
ruamel.ordereddict        0.4.6                    py27_0    conda-forge
ruamel.yaml               0.11.11                  py27_1    conda-forge
setuptools                23.0.0                   py27_0    conda-forge
six                       1.10.0                   py27_0    conda-forge
smmap                     0.9.0                    py27_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
yaml                      0.1.6                         0    conda-forge
zlib                      1.2.8                         3    conda-forge
constructor               1.3.0                    py27_0    defaults
libconda                  4.0.0                    py27_0    defaults
pbr                       1.10.0                   py27_0    defaults
pip                       8.1.2                    py27_0    defaults
python                    2.7.12                        1    defaults
ruamel_yaml               0.11.7                   py27_0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py27_0    defaults

PR pelson/Obvious-CI#51 needs porting to conda-build-all, and tests added to https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/tests/unit/test_version_matrix.py.

We are seeing this error if a recipe depends on other recipes in the same directory. This seem to have been fixed a while ago. We are using the combo of, and

- conda install conda-build=2.0.10
- conda install -c conda-forge conda-build-all=1.0*

When trying run conda-build-all on the included recipe, I encounter a build issue when the root environment is Python 3, but not when the root environment is Python 2. More details about the recipe and environments below. Was unable to reproduce this with conda-build alone in either environment. Am sort of suspicious that this issue is actually caused by Python versions. Though it could be.

Python 3 traceback (Python 2 has no issues).

$ conda build-all recipes/wrf-python --matrix-condition "numpy >=1.11" "python >=2.7,<3|>=3.5" 
Fetching package metadata .............
Error: Invalid selector in meta.yaml line 15:
  skip: True  # [win32 or np>=112 or (win and py>=35)]

Recipe below.

# meta.yaml
{% set version = "1.0b1" %}

package:
  name: wrf-python
  version: {{ version }}

source:
  fn: wrf-python-{{ version }}.tar.gz
  url: https://github.com/NCAR/wrf-python/archive/{{ version }}.tar.gz
  sha256: 94f05a9719701959702094eafb84555a996bc4f7d75042fb07bca3c35bda9f31

build:
  number: 1
  detect_binary_files_with_prefix: true
  skip: True  # [win32 or np>=112 or (win and py>=35)]

requirements:
  build:
    - pip
    - numpy x.x
    - wrapt
    - m2w64-toolchain  # [win]
    - gcc  # [unix]
    - libgcc  # [unix]
    - python
    - vc 9  # [win and py27]
    - vc 10  # [win and py34]
    - vc 14  # [win and py>=35]
  run:
    - numpy x.x
    - python
    - wrapt 
    - m2w64-toolchain  # [win]
    - gcc  # [unix]
    - libgcc  # [unix]
    - xarray
    - vc 9  # [win and py27]
    - vc 10  # [win and py34]
    - vc 14  # [win and py>=35]

test:
  requires:
    - nose
  imports:
    - wrf 

about:
  home: https://github.com/NCAR/wrf-python
  license: "NCL Source Code"
  summary: "Diagnostic and interpolation routines for WRF-ARW data."

extra:
  recipe-maintainers:
    - bladwig1
    - marylhaley
    - david-ian-brown
    - khallock

# environment2.yml
name: root
channels: !!python/tuple
- !!python/unicode
  'nanshe'
- !!python/unicode
  'conda-forge'
- !!python/unicode
  'jakirkham'
- !!python/unicode
  'defaults'
dependencies:
- beautifulsoup4=4.5.3=py27_0
- chardet=2.3.0=py27_0
- conda-forge::anaconda-client=1.5.1=py27_0
- conda-forge::anaconda-verify=1.2.0=py27_0
- conda-forge::appnope=0.1.0=py27_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py27_1
- conda-forge::ca-certificates=2016.9.26=0
- conda-forge::certifi=2016.9.26=py27_0
- conda-forge::cffi=1.7.0=py27_0
- conda-forge::clyent=1.2.1=py27_0
- conda-forge::conda=4.2.13=py27_0
- conda-forge::conda-build=2.1.1=py27_1
- conda-forge::conda-build-all=1.0.1=py27_1
- conda-forge::conda-env=2.6.0=0
- conda-forge::conda-execute=0.8.0=py27_0
- conda-forge::constructor=1.3.4=py27_0
- conda-forge::contextlib2=0.5.4=py27_0
- conda-forge::coverage=3.7.1=py27_0
- conda-forge::curl=7.52.1=0
- conda-forge::decorator=4.0.10=py27_0
- conda-forge::enum34=1.1.6=py27_1
- conda-forge::expat=2.1.0=2
- conda-forge::filelock=2.0.6=py27_0
- conda-forge::funcsigs=1.0.2=py27_0
- conda-forge::futures=3.0.5=py27_0
- conda-forge::gitdb=0.6.4=py27_1
- conda-forge::gitpython=2.1.1=py27_0
- conda-forge::idna=2.1=py27_0
- conda-forge::ipaddress=1.0.16=py27_0
- conda-forge::ipython=5.1.0=py27_2
- conda-forge::ipython_genutils=0.1.0=py27_0
- conda-forge::jinja2=2.8=py27_1
- conda-forge::libconda=4.0.0=py27_0
- conda-forge::libffi=3.2.1=3
- conda-forge::markupsafe=0.23=py27_1
- conda-forge::mock=2.0.0=py27_0
- conda-forge::ncurses=5.9=10
- conda-forge::nose=1.3.7=py27_2
- conda-forge::openssl=1.0.2h=3
- conda-forge::pathlib2=2.1.0=py27_0
- conda-forge::pbr=1.10.0=py27_0
- conda-forge::pexpect=4.2.1=py27_0
- conda-forge::pickleshare=0.7.3=py27_0
- conda-forge::pip=9.0.1=py27_0
- conda-forge::pkginfo=1.2.1=py27_0
- conda-forge::prompt_toolkit=1.0.9=py27_0
- conda-forge::psutil=4.3.0=py27_0
- conda-forge::ptyprocess=0.5.1=py27_0
- conda-forge::pyasn1=0.1.9=py27_0
- conda-forge::pycosat=0.6.1=py27_0
- conda-forge::pycparser=2.17=py27_0
- conda-forge::pycrypto=2.6.1=py27_0
- conda-forge::pygithub=1.29=py27_0
- conda-forge::pygments=2.1.3=py27_1
- conda-forge::pyopenssl=16.0.0=py27_0
- conda-forge::python=2.7.12=1
- conda-forge::python-dateutil=2.6.0=py27_0
- conda-forge::pytz=2016.10=py27_0
- conda-forge::pyyaml=3.11=py27_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.12.4=py27_0
- conda-forge::ruamel.ordereddict=0.4.6=py27_0
- conda-forge::ruamel.yaml=0.13.7=py27_0
- conda-forge::ruamel_yaml=0.11.14=py27_0
- conda-forge::setuptools=32.3.1=py27_0
- conda-forge::simplegeneric=0.8.1=py27_0
- conda-forge::six=1.10.0=py27_1
- conda-forge::smmap=2.0.1=py27_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.1=py27_0
- conda-forge::typing=3.5.3.0=py27_0
- conda-forge::wcwidth=0.1.7=py27_0
- conda-forge::wheel=0.29.0=py27_0
- conda-forge::yaml=0.1.6=0
- conda-forge::zlib=1.2.8=3
- conda-verify=2.0.0=py27_0
- cryptography=1.7.1=py27_0
- python.app=1.2=py27_4
- pip:
  - backports.shutil-get-terminal-size==1.0.0
  - conda-build-all (/zopt/conda2/lib/python2.7/site-packages)==1.0.1
  - conda-smithy (/Users/kirkhamj/Developer/Conda/conda-forge/conda-smithy)==2.0.0
  - ipython-genutils==0.1.0
  - prompt-toolkit==1.0.9
  - ruamel-yaml==0.11.14
  - smmap2==2.0.1
prefix: /zopt/conda2

# environment3.yml
name: root
channels: !!python/tuple
- nanshe
- conda-forge
- jakirkham
- defaults
dependencies:
- beautifulsoup4=4.5.3=py35_0
- chardet=2.3.0=py35_0
- conda-forge::anaconda-client=1.5.1=py35_0
- conda-forge::appnope=0.1.0=py35_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py35_1
- conda-forge::ca-certificates=2016.9.26=0
- conda-forge::certifi=2016.9.26=py35_0
- conda-forge::clyent=1.2.1=py35_0
- conda-forge::conda=4.2.13=py35_0
- conda-forge::conda-build=2.1.1=py35_1
- conda-forge::conda-build-all=1.0.1=py35_1
- conda-forge::conda-env=2.6.0=0
- conda-forge::decorator=4.0.10=py35_0
- conda-forge::filelock=2.0.6=py35_0
- conda-forge::freetype=2.6.3=1
- conda-forge::gitdb=0.6.4=py35_1
- conda-forge::gitpython=2.1.1=py35_0
- conda-forge::ipython=5.1.0=py35_2
- conda-forge::ipython_genutils=0.1.0=py35_0
- conda-forge::jinja2=2.8=py35_1
- conda-forge::jpeg=9b=0
- conda-forge::libpng=1.6.28=0
- conda-forge::libtiff=4.0.6=7
- conda-forge::markupsafe=0.23=py35_1
- conda-forge::mock=2.0.0=py35_0
- conda-forge::ncurses=5.9=10
- conda-forge::nose=1.3.7=py35_2
- conda-forge::openssl=1.0.2h=3
- conda-forge::path.py=8.2.1=py35_0
- conda-forge::pbr=1.10.0=py35_0
- conda-forge::pexpect=4.2.1=py35_0
- conda-forge::pickleshare=0.7.3=py35_0
- conda-forge::pillow=4.0.0=py35_0
- conda-forge::pip=9.0.1=py35_0
- conda-forge::pkginfo=1.2.1=py35_0
- conda-forge::prompt_toolkit=1.0.9=py35_0
- conda-forge::ptyprocess=0.5.1=py35_0
- conda-forge::pycosat=0.6.1=py35_0
- conda-forge::pycrypto=2.6.1=py35_0
- conda-forge::pygithub=1.29=py35_0
- conda-forge::pygments=2.1.3=py35_1
- conda-forge::python=3.5.2=5
- conda-forge::python-dateutil=2.6.0=py35_0
- conda-forge::python-simplegeneric=0.8.1=py35_0
- conda-forge::pytz=2016.10=py35_0
- conda-forge::pyyaml=3.11=py35_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.12.4=py35_0
- conda-forge::ruamel.yaml=0.13.7=py35_0
- conda-forge::ruamel_yaml=0.11.14=py35_0
- conda-forge::setuptools=32.3.1=py35_0
- conda-forge::simplegeneric=0.8.1=py35_0
- conda-forge::six=1.10.0=py35_1
- conda-forge::smmap=2.0.1=py35_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.1=py35_0
- conda-forge::wcwidth=0.1.7=py35_0
- conda-forge::wheel=0.29.0=py35_0
- conda-forge::xz=5.2.2=0
- conda-forge::yaml=0.1.6=0
- conda-forge::zlib=1.2.8=3
- conda-verify=2.0.0=py35_0
- jbig=2.1=0
- python.app=1.2=py35_4
- pip:
  - backports.shutil-get-terminal-size==1.0.0
  - conda-smithy (/Users/kirkhamj/Developer/Conda/conda-forge/conda-smithy)==2.0.0
  - ipython-genutils==0.1.0
  - prompt-toolkit==1.0.9
  - ruamel-yaml==0.11.14
  - smmap2==2.0.1
prefix: /zopt/conda3

An expired token is causing the integration tests that reach out to anaconda.org to fail. See, e.g., https://travis-ci.org/SciTools/conda-build-all/jobs/131601936#L354

After some discussion with @johanneskoester and @mingwandroid , it seems that R is not being handled in conda-build-all as it is in conda-build. Namely it sounds like it should be searching for r-base and using that as part of the matrix and not r. Please feel free to correct me if I am mistaken.

xref: conda-forge/staged-recipes#2098

conda-build-all works with conda-4.3.22, but fails in 4.3.23 because NoPackagesFound is not found anymore in conda.resolve

this is with:

$ conda-build-all --version
1.0.1
$ conda-build --version
conda-build 2.0.12

when I run conda-build-all it does everything nicely, until it comes time to upload a package, and then I get:

Uploading unit_conversion to the main channel.
INFO artefact_destination:make_available(116): Uploading unit_conversion to the main channel.
Traceback (most recent call last):
File "/home/chris.barker/miniconda2/bin/conda-build-all", line 6, in
sys.exit(conda_build_all.cli.main())
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/cli.py", line 90, in main
b.main()
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/builder.py", line 269, in main
config=build_config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/builder.py", line 289, in post_build
config=config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/artefact_destination.py", line 118, in make_available
config=config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/build.py", line 43, in upload
fname = bldpkg_path(meta, config)
TypeError: bldpkg_path() takes exactly 1 argument (2 given)
Traceback (most recent call last):
File "./build-all.py", line 46, in
env=this_env)
File "/home/chris.barker/miniconda2/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda-build-all', './', '--matrix-conditions', 'python 2.7.*', '--inspect-channels', 'NOAA-ORR-ERD', '--upload-channels', 'NOAA-ORR-ERD', '--no-inspect-conda-bld-directory', '--artefact-directory', 'packages']' returned non-zero exit status 1

I see this code around the error:

    if hasattr(conda_build, 'api'):
        fname = bldpkg_path(meta, config)
    else:
        fname = bldpkg_path(meta)

so it looks like conda_build 2.0.12 has the 'api' attribute, but has the wrong signature for pldpkg_path ??

I get the same error with conda-build 2.0.11

But it works with 2.0.10 !!

-CHB

From @PeterDSteinberg on September 20, 2016 19:30

Hi. I am getting an error in conda-build on conda-forge that I am not getting locally with conda-build in more recent versions than 1.21.14.

The error is several parts.

• Warns I'm on out of date conda-build version 1.21.14
• Conda-build has no attribute "api" error:

...Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 200, in build
    conda_build.api.build(meta.meta, config=config)
AttributeError: module 'conda_build' has no attribute 'api'

• In processing of that error, another error occurs: Unsatisfiable constraint on pandas 0.18.1 and numpy 1.10|1.11, which I think should be fine as a combination of packages

conda.resolve.Unsatisfiable: The following specifications were found to be in conflict:
  - pandas ==0.18.1 -> numpy 1.10*|1.11*
  - setuptools

Full build on circle CI (scroll to the bottom)

Here's the recipe I am trying to build

Any ideas? Can this be solved with upgrading conda-build?

Copied from original issue: conda-forge/conda-forge.github.io#240

Continuing from the discussion near the tail end of issue ( conda-forge/conda-execute-feedstock#17 ).

Would like to figure out how we preserve the history from the conda-forge fork of conda-build-all. In particular the history from issue/PR discussions should they need to be revisited. One option might be to migrate the issues/PRs to this repo somehow (though not sure of a tool to do this for PRs). Another option would be to migrate that repo over here and note that it is historic. Though if there are other options, would be interested to hear them.

If --upload-channels is specified but --inspect-channels is not, all recipes will be built and at the end of the recipe building, the following message will be printed out:

Assuming the distribution we've just built and the one on lightsource2/main are the same.

It seems like it would be sensible for --inspect-channels to default to --upload-channels if upload is present but inspect is not. Thoughts?

Thanks a bunch for making this! I'd like to get rid of maintaining my separate script that does mostly the same thing as this package. There are a two features that would be nice to have here to make that possible.

1. Fail gracefully when a package fails to build
   • It's pretty common that a package might fail to build for a certain version of Python or something. Currently, this kills the whole job. IMO it would be nice to catch this and continue through the matrix even if a package fails to build, but then to set the exit status so that the calling script is notified about the failure. IIRC in omnia-md, we set the exit status to the number of failed builds, so if it's nonzero you still get a failure inside your CI system.
2. Print summary of the packages that will be built.
   • Currently, the script prints out, for each possible build matrix item, whether or not it will be built here. That's useful but it's a little overwhelming if you have a lot of packages but are using --inspect-channels, so probably very few of them will be built. IMO it would be nice, at end maybe, to print out just the ones that will be built. Or maybe to use color highlighting for the ones that will be built?

Hi folks,

I seem to have hit this NotImplementedError again that is in #21 and #18 . I spent more time digging in to this problem with the pdb this time and have determined that I am getting this error due to two things.

1. The name of the file on anaconda.org/lightsource2 has an extra "linux-64" prepending the conda package name, linux-64/linux-64-pcaspy-0.5.1-py34_2.tar.bz2, which is odd. Seen here: https://anaconda.org/lightsource2/pcaspy/files. I have been unsuccessful in determining why this pcaspy conda package has an extra linux-64- prepending the pcaspy-0.5.1-py34_2.tar.bz2 package name.
2. conda-build-all thinks that it does not have to build this package initially when it checks the lightsource2 channel with the conda.api.get_index() function, but then when it skips the build but checks to see if it should upload something, conda-build-all fails because it uses a different pathway to determine if the file is already on anaconda.org, with the anaconda_client.distribution() function call.

Would you be opposed to making these checks symmetric by using the same conda.api.get_index() call in both of these places? As it is correctly finding the oddly named package on anaconda.org/lightsource2, I am inclined to declare that is more robust than the anaconda_client.distribution() function call. I'll get to work on a PR to make these checks symmetric so that we can see what a viable solution looks like. If it is horrible, I'll just fix this problem by deleting the offending packages on the lightsource2 channel.

Here are my notes from debugging this issue (mostly so I don't have to dig in to this again!) https://gist.github.com/ericdill/4415a840ed6b191f3ac7

conda-build-all version 1.0.4 fails with conda-build 3.0.3

conda build-all recipes --matrix-max-n-minor-versions 3 --matrix-conditions "python $PYTHON_BUILD_RESTRICTIONS" "$NUMPY_BUILD_RESTRICTION" 
Fetching package metadata .........
Traceback (most recent call last):
  File "/home/yfeng1/anaconda3/install/bin/conda-build-all", line 6, in <module>
    sys.exit(conda_build_all.cli.main())
  File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/cli.py", line 90, in main
    b.main()
  File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/builder.py", line 249, in main
    if build_config.CONDA_NPY is None:
AttributeError: 'Config' object has no attribute 'CONDA_NPY'

Got this issue when the filenames compression did not match the actual compression type. Admittedly the issue goes away when that is fixed, but maybe it is just exposing an underlying issue. Hence why I have raised it. Traceback below. See build log for more details.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 200, in build
    conda_build.api.build(meta.meta, config=config)
AttributeError: module 'conda_build' has no attribute 'api'

I get:

TypeError: add_release() got an unexpected keyword argument 'description'

Looks like this recent change to anaconda-client has removed the 'description' argument:
anaconda/anaconda-client#394

It would be super useful to have an option of soft failing the build when RuntimeError is raised due to unsatisfiable dependencies. So it would allow to finish all the builds and just gives a summary of packages where it failed.

E.g here I would update a few packages, a total of 15 combinations, but the build fails in the middle as one of the version combinations cannot find all the dependencies. Some of these combinations doesn't make much sense anyway, but I didn't find a good way to disable them one-by-one in requirements.yml.

it seems to me that the most common need for to make a release is to

1. build a package for a list of supported python versions
2. convert these to all platforms
3. upload all to anaconda.org

it sounds like this project does 1), would adding 2 and 3 be difficult?

This is a Docs issue, I'm sure. But I haven't figured it out.

I have a module that uses the numpy API, so I want the conda package pinned to the numpy version.

So I added:

numpy x.x

to the meta.yaml. Now, when I run conda build, it requires a numpy version -- makes sense.

but when I try to use conda-build-all like so:

conda-build-all ./ --matrix-conditions "python 2.7.*" "numpy >=1.11"

I get:

  File "/Users/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build/metadata.py", line 478, in ms_depends
    ms = handle_config_version(ms, ver, typ)
  File "/Users/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build/metadata.py", line 320, in handle_config_version
    raise RuntimeError("'%s' requires external setting" % ms.spec)
RuntimeError: 'numpy x.x' requires external setting

Not sure if this is a conda-build or conda-build-all issue (am leaning towards the latter), but am going to start by raising it here.

If numpy is listed as a build dependency and x.x is not used, then a build with conda-build-all tries to download numpy 1.0*. See this comment. This is likely due to this line. Removal of numpy from build is an effective workaround for the problem.

The easiest "fix" (workaround) would probably be to set this to the newest version of NumPy so change to 111. Though it would be nice to know why it is happening in the first place. Namely is this intentional behavior by conda-build to do this now or is this a bug.

cc @pelson @msarahan @ocefpaf

I just encountered an issue packaging a recipe which depended on another, but only conditionally (based on py<34). I was running conda-build-all with py35, so the condition was not met, and conda-build-all built the packages in the wrong order.

In order to solve this, we are going to have to resolve the dependencies for all matrix conditions - that is an expensive requirement but may unfortunately be necessary.

As a heads up, the next release of conda-build-all will support conda-build 2.0, and it will be released as version 1.0.

Recently I ran conda_build_all, the relevant packages were built fine however following the build I received the following errors

anaconda_upload is not set.  Not uploading wheels: []
pyyaml-3.12-py27_1 ['/data/local/lcarroll/conda_bld/root/linux-64/pyyaml-3.12-py27_1.tar.bz2'] True
Traceback (most recent call last):
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/bin/conda-build-all", line 6, in <module>
    exit(main())
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/cli.py", line 93, in main
    b.main()
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/builder.py", line 299, in main
    config=build_config)
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/builder.py", line 319, in post_build
    config=config)
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/artefact_destination.py", line 65, in make_available
    shutil.copy(built_dist_path, self.directory)
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/shutil.py", line 240, in copy
    dst = os.path.join(dst, os.path.basename(src))
  File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/posixpath.py", line 144, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not list
Aborted!
CalledProcessError: Command '['bash', '/var/tmp/lcarroll/conda_recipes/configuration/scidesktop/conda_build/build.sh', 'conda-recipes']' returned non-zero exit status 1.

This seems to be related to Pull Request #93, we were not encountering these errors before this Pull Request.

Since #93 was merged built_dist_path or self.directory within artefact_destination.DirectoryDestination have become lists. Hence passing them to shutil.copy results in the error:
TypeError: expected str, bytes or os.PathLike object, not list

Hi @pelson, I get the following error when trying to work with conda-gitenv and conda-build-all if using conda 4.5.5:

(venv-gitenv) [root@localhost environments]# conda gitenv resolve ${ENV_REPO}
Traceback (most recent call last):
  File "/venvs/venv-gitenv/bin/conda-gitenv", line 7, in <module>
    from conda_gitenv.cli import main
  File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_gitenv/cli.py", line 6, in <module>
    import conda_gitenv.resolve as resolve
  File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_gitenv/resolve.py", line 22, in <module>
    import conda_build_all.version_matrix
  File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_build_all/version_matrix.py", line 6, in <module>
    from .conda_interface import (MatchSpec, Unsatisfiable, NoPackagesFound, Resolve,
  File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_build_all/conda_interface.py", line 60, in <module>
    % (CONDA_VERSION, str(CONDA_VERSION_MAJOR_MINOR)))
NotImplementedError: CONDA_VERSION: 4.5.5  CONDA_VERSION_MAJOR_MINOR: (4, 5)
(venv-gitenv) [root@localhost environments]# conda --version
conda 4.5.5

What is the quickest fix? Just switching conda versions or is it minor to patch conda-build-all?

The issue is that anaconda.org now returns package names qualified by channel names so the current logic at https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/builder.py#L166 fails.

For example, new list of keys from a call to index is: [u'astropy-ci-extras::obvious-ci-0.6.0-py35_0.tar.bz2', u'astropy-ci-extras::obvious-ci-0.6.0-py27_0.tar.bz2', ...].

I'll get a fix in later today, wanted to record the cause before I forget. It would be nice to get #39 fixed so that integration tests worked again 😄

In the recipe attached, run with conda-build-all --inspect-channels=astropy recipes, the build gets as far as trying to resolve dependencies then fails with an error RuntimeError: 'numpy x.x' requires external setting.

The full traceback is:

$ conda-build-all --inspect-channels=astropy recipes
Fetching package metadata: ......
Resolving distributions from 1 recipes...
Traceback (most recent call last):
  File "/Users/mcraig/miniconda-clean/bin/conda-build-all", line 9, in <module>
    load_entry_point('conda-build-all==0.8.0', 'console_scripts', 'conda-build-all')()
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/cli.py", line 85, in main
    b.main()
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/builder.py", line 195, in main
    all_distros = self.compute_build_distros(index, recipe_metas)
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/builder.py", line 183, in compute_build_distros
    if distro.pkg_fn() not in index:
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/resolved_distribution.py", line 82, in with_vn_mtx_setup
    return orig_result(*args, **kwargs)
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 504, in pkg_fn
    return "%s.tar.bz2" % self.dist()
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 501, in dist
    return '%s-%s-%s' % (self.name(), self.version(), self.build_id())
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 474, in build_id
    for ms in self.ms_depends():
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 447, in ms_depends
    ms = handle_config_version(ms, ver)
  File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 333, in handle_config_version
    raise RuntimeError("'%s' requires external setting" % ms.spec)
RuntimeError: 'numpy x.x' requires external setting

recipes.zip

Seems conda-build-all has had a wrench thrown in it with the recent conda-build release (1.21.7). Not really sure what is causing it, but it seems to have some issues identifying where source should be extracted. See this comment for one example. Also here is another that springs up in this build. It would be nice to figure out why source extraction is getting messed up in these cases.

cc @jjhelmus @patricksnape @msarahan

From @asmeurer on January 16, 2017 4:8

I tried to rerender sphinxcontrib-autoprogram and got this error:

$conda smithy rerender
No circle token.  Create a token at https://circleci.com/account/api and
put it in ~/.conda-smithy/circle.token
No appveyor token. Create a token at https://ci.appveyor.com/api-token and
Put one in ~/.conda-smithy/appveyor.token
No anaconda token. Create a token via
  anaconda auth --create --name conda-smithy --scopes "repos conda api"
and put it in ~/.conda-smithy/anaconda.token
Fetching package metadata .................
Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda3/bin/conda-smithy", line 6, in <module>
    sys.exit(conda_smithy.cli.main())
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 289, in main
    args.subcommand_func(args)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 200, in __call__
    configure_feedstock.main(args.feedstock_directory)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 644, in main
    render_run_docker_build(env, config, forge_dir)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 77, in render_run_docker_build
    forge_config.get('channels', {}).get('sources', tuple())
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 565, in compute_build_matrix
    mtx = special_case_version_matrix(meta, index)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_build_all/version_matrix.py", line 212, in special_case_version_matrix
    py_vn = minor_vn(index[python_pkg.fn]['version'])
AttributeError: 'Dist' object has no attribute 'fn'

Copied from original issue: conda-forge/conda-smithy#435

I'm not sure where the underlying issue is, but when I copy across channels and then try to use a package from the channel to which the package was copied I get errors like this: https://travis-ci.org/astropy/ccdproc/jobs/146212815#L472

Error: HTTPError: 404 Client Error: NOT FOUND for url: https://conda.anaconda.org/astropy/linux-64/reproject-0.3.1-np111py27_0.tar.bz2: https://conda.anaconda.org/astropy/linux-64/reproject-0.3.1-np111py27_0.tar.bz2

Should this be getting updated on conda-forge?

I kind of liked being able to grab it from there....

-CHB

Quoting @jochym ...

Several (phonopy, elastic, pyhmc) staged-build PRs got broken by the numpy/openblas problems on osx and linux for py3k. I had to block builds for py3k on these platforms. At the same time the feedstocks with similar code and setup (e.g. spglib) build fine. I'll be happy to provide any additional info to help with this.

Appears this problem does not occur with conda-build-all version 1.0.0, but it does occur with conda-build-all version 1.0.1. This points to an issue with PR ( #68 ), but it is not clear to me yet what it is. Any ideas @mwcraig ?

My use case:

I have a repo of packages that I'm maintaining.

I update one or two, and want conda-build-all to build and upload them to anaconda.org.

However, in testing, I've already make sure that the new one builds. THis means that it is in the conda-build dir.

which means that conda-build-all doesn't build it again -- good.

But it also doesn't upload it -- which I do want.

So I set: --no-inspect-conda-bld-directory, and it works.

But it seems silly to re-build all those, when all I want is to upload the already built packages...

I'm thinking there is no reason, if the --no-inspect-conda-bld-directory flag isn't set, not to look in conda-build, if it's there, don't rebuild, but do upload.

Hi Phil,

My use case here is to have a git repo full of dev recipes where the repo is set up with circleci and conda-build-all to compare the dev recipes in the aforementioned git repo with the conda packages on my dev channel on anaconda.org and only build ones with commits in master (and therefore new version numbers).

However, if I use any of the GIT_* info in a conda-recipe, conda-build-all is unable to correctly interpret the version string to compare against the built packages on anaconda.org/user_name.

The following recipe is a relevant snippet from a master branch meta.yaml:

package:
  name: metadatastore
  version: {{ environ['GIT_DESCRIBE_TAG'] }}.post{{ environ['GIT_DESCRIBE_NUMBER'] }}

source:
  git_url: https://github.com/NSLS-II/metadatastore.git
  git_rev: master
  # use a local url and leave rev blank to have conda build current head, currently
  # requires master branch of conda-build to work
  # git_rev: ?

build:
  string: {{ environ.get('GIT_BUILD_STR', '') }}_py{{ py }}
  number: 0
--snip--

which conda-build-all interprets as

Resolved dependencies, will be built in the following order: 
    metadatastore-.post-_py27 (will be built: True)
    metadatastore-.post-_py34 (will be built: True)
    metadatastore-.post-_py35 (will be built: True)

Is it within the scope of conda-build-all to be able to recognize a meta.yaml has GIT_* info inside it and attempt to figure out what the build string should be? As this is a feature that would make my life loads easier, I am happy to work on adding this to conda-build-all. I would imagine that it would have to go something like:

• check recipe for presence of GIT_* env vars
• clone git repo from source.git_url
• check out source.git_rev
• git describe to get the exact version string
• Interpret the git describe in the context of the meta.yaml
• Check to see if that version exists on anaconda.org/channel-of-interest and then build or don't depending on if the version exists already on anaconda.org

This is non-trivial and is certainly not guaranteed to work in all situations, but would potentially let conda-build-all be used for continuous delivery.

#93 added conda-build 3 API compatibility, but did not add support for actually building recipes that use the features of conda-build 3.

This is a meta-issue for items that crop up:

• AssertionError: modifying metadata after finalization (sorry, don't have the traceback to hand, but it comes from resolved_distribution getattr (line 45).
• Computed binary filename from conda-build-all is different to what conda-build reports. This is because of the hash that gets generated is missing in conda-build-all.

Fixing these things is a pretty deep exercise, and if this is going to get fixed, conda-build-all will cease support for conda-build <3.

From @jakirkham on July 25, 2016 4:4

Some packages don't have a source section. Recently I have been having some issues with a package that has no source. However, have had some difficulties producing it offline with conda-build alone. So it may be a conda-build-all issue in the end. Here is a log demonstrating this failure. A traceback is shown below. It seems that maybe there are some cases where the work directory is not removed when it is expected to have been.

no source - creating empty work folder
Traceback (most recent call last):
  File "/opt/conda/bin/conda-build-all", line 9, in <module>
    load_entry_point('conda-build-all==0.13.1', 'console_scripts', 'conda-build-all')()
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/cli.py", line 85, in main
    b.main()
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 230, in main
    built_dist_location = self.build(meta)
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 187, in build
    return bldpkg_path(build.build(meta.meta))
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/build.py", line 32, in build
    build_module.build(meta, post=None, need_source_download=True, **kwd)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/build.py", line 548, in build
    os.makedirs(source.WORK_DIR)
  File "/opt/conda/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/opt/conda/conda-bld/work'

cc @msarahan @pelson

Copied from original issue: conda/conda-build#1145

I'm trying to build a set of packages, with::

    conda-build-all ./ --matrix-conditions "python 2.7.*" \
                       --inspect-channels NOAA-ORR-ERD \
                       --upload-channels NOAA-ORR-ERD

I have not yet set a BINSTAR_TOKEN. I get:

Traceback (most recent call last):
  File "/home/chris.barker/miniconda/bin/conda-build-all", line 9, in <module>
    load_entry_point('conda-build-all==0.8.0', 'console_scripts', 'conda-build-all')()
  File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/cli.py", line 85, in main
    b.main()
  File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 208, in main
    self.post_build(meta, built_dist_location, was_built)
  File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 225, in post_build
    artefact_destination.make_available(meta, built_dist_location, was_built)
  File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/artefact_destination.py", line 119, in make_available
    raise NotImplementedError('cross owner copying not yet implemented.')
NotImplementedError: cross owner copying not yet implemented.

I'm assuming this is because I don't have the TOKEN set, but know knows? It would be nice to get a better message in any case...

The keys in a conda index have changed to include the source, therefore the unit-tests are no longer representative of the actual conda integration.

https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/tests/unit/dummy_index.py#L51

Latest conda-build has a rendering step that broke conda-build-all (@msarahan can explain it better than me 😬).

For now I added two workarounds that need to be removed once we fix this:

• https://github.com/conda-forge/conda-forge-build-setup-feedstock/blob/master/recipe/run_conda_forge_build_setup_win#L10
• https://github.com/conda-forge/staged-recipes/blob/master/.CI/create_feedstocks#L13

Given a recipe that includes environ.GIT_DESCRIBE_TAG as a jinja variable in the meta.yaml, trying to use conda-build-all to build py27, py34 and py35 successfully builds py27, but then fails on py34 with the following error:

Error: Failed to render jinja template in /tmp/nb_conda_kernels-recipe/recipe/meta.yaml:
'dict object' has no attribute 'GIT_DESCRIBE_TAG'

I find this rather surprising and cannot reproduce it by invoking conda-build directly. Here's a script that you can run on a unix system that will download and install a fresh copy of miniconda into /tmp. It will then update --all from conda-forge and install conda-build-all from conda-forge into that temp environment. Finally it will git clone a git repo that contains the lightweight recipe (jupyter extension) that I'm trying to build with conda-build-all (and hopefully it will fail in the same way it is failing for me!)

repo: https://github.com/ericdill/nb_conda_kernels-recipe
script: https://github.com/ericdill/nb_conda_kernels-recipe/blob/master/run-build.sh

Hi Phil,

I'm sure there is a better repo to ask this question against, but what is your recommended scopes argument for generating a binstar token?

It would be nice to be able to run::

conda-build-all ./ --matrix-conditions "some stuff" --dry-run

and see what would get built, etc, without it actually starting the build process....

Combined with #8's idea for showing what really will get built, this could be handy

not that hitting ctrl+C is that hard....

Basically issue ( conda-forge/conda-smithy#191 ) being punted upstream. @alexbw , can you please advise us one how we set the versioning for this?

While we have discussed this several places before, wasn't seeing this tracked anywhere. So thought it best to open an issue here. Guessing we will need to handle VC correctly here before we can address it in conda-smithy.

Currently in order to get a package to build with multiple VC runtimes, we need to add python as a build dependency and track a feature. However, this has proved impractical and in some cases limiting. Instead discussion over the past several months has concluded the best way forward is to have a separate vc package for tracking the compiler that python and other packages track. Though ultimately this must be addressed with some kind of matrix. While we can find somewhat effective ways to do this manually, conda-build-all has reigned supreme at helping convert an implicit matrix to an explicit matrix (via leveraging by conda-smithy). Hence I'm raising this here to ask that vc be handled in the matrix configuration used by staged-recipes and can picked up with conda-smithy for use in the feedstocks. There are already some considerations about how this might play out.

xref: conda-forge/glpk-feedstock#10
xref: conda-forge/tk-feedstock#2
xref: conda-forge/curl-feedstock#11
xref: conda-forge/conda-smithy#245

cc @jjhelmus @msarahan

Appears that conda-build version 2.0.9 and conda version 4.1 are incompatible. As this appears to be a temporary issue, would recommend that an exclusion against conda-build version 2.0.9. If this is combined with a version bump, it will hopefully avoid issues with conda ignoring the pinning. A similar strategy seems to be working with conda-smithy so far.

cc @bjlittle