conda-tools / conda-build-all Goto Github PK
View Code? Open in Web Editor NEWLicense: BSD 3-Clause "New" or "Revised" License
License: BSD 3-Clause "New" or "Revised" License
Hi @pelson, I get the following error when trying to work with conda-gitenv
and conda-build-all
if using conda 4.5.5:
(venv-gitenv) [root@localhost environments]# conda gitenv resolve ${ENV_REPO}
Traceback (most recent call last):
File "/venvs/venv-gitenv/bin/conda-gitenv", line 7, in <module>
from conda_gitenv.cli import main
File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_gitenv/cli.py", line 6, in <module>
import conda_gitenv.resolve as resolve
File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_gitenv/resolve.py", line 22, in <module>
import conda_build_all.version_matrix
File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_build_all/version_matrix.py", line 6, in <module>
from .conda_interface import (MatchSpec, Unsatisfiable, NoPackagesFound, Resolve,
File "/venvs/venv-gitenv/lib/python3.6/site-packages/conda_build_all/conda_interface.py", line 60, in <module>
% (CONDA_VERSION, str(CONDA_VERSION_MAJOR_MINOR)))
NotImplementedError: CONDA_VERSION: 4.5.5 CONDA_VERSION_MAJOR_MINOR: (4, 5)
(venv-gitenv) [root@localhost environments]# conda --version
conda 4.5.5
What is the quickest fix? Just switching conda versions or is it minor to patch conda-build-all
?
I'm trying to build a set of packages, with::
conda-build-all ./ --matrix-conditions "python 2.7.*" \
--inspect-channels NOAA-ORR-ERD \
--upload-channels NOAA-ORR-ERD
I have not yet set a BINSTAR_TOKEN. I get:
Traceback (most recent call last):
File "/home/chris.barker/miniconda/bin/conda-build-all", line 9, in <module>
load_entry_point('conda-build-all==0.8.0', 'console_scripts', 'conda-build-all')()
File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/cli.py", line 85, in main
b.main()
File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 208, in main
self.post_build(meta, built_dist_location, was_built)
File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/builder.py", line 225, in post_build
artefact_destination.make_available(meta, built_dist_location, was_built)
File "/home/chris.barker/miniconda/lib/python2.7/site-packages/conda_build_all/artefact_destination.py", line 119, in make_available
raise NotImplementedError('cross owner copying not yet implemented.')
NotImplementedError: cross owner copying not yet implemented.
I'm assuming this is because I don't have the TOKEN set, but know knows? It would be nice to get a better message in any case...
I just encountered an issue packaging a recipe which depended on another, but only conditionally (based on py<34). I was running conda-build-all with py35, so the condition was not met, and conda-build-all built the packages in the wrong order.
In order to solve this, we are going to have to resolve the dependencies for all matrix conditions - that is an expensive requirement but may unfortunately be necessary.
The issue is that anaconda.org now returns package names qualified by channel names so the current logic at https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/builder.py#L166 fails.
For example, new list of keys from a call to index
is: [u'astropy-ci-extras::obvious-ci-0.6.0-py35_0.tar.bz2', u'astropy-ci-extras::obvious-ci-0.6.0-py27_0.tar.bz2', ...]
.
I'll get a fix in later today, wanted to record the cause before I forget. It would be nice to get #39 fixed so that integration tests worked again ๐
PR pelson/Obvious-CI#51 needs porting to conda-build-all, and tests added to https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/tests/unit/test_version_matrix.py.
I'm guessing this has something to do with the fact that this was originally a fork of your copy (which is still indexed).
Not a huge priority...
Should this be getting updated on conda-forge?
I kind of liked being able to grab it from there....
-CHB
Would like to tag a new release using the current master
. Based on the noarch
functionality, am leaning towards calling it 1.1.0
(instead of 1.0.5
). Any thoughts on the version or releasing?
cc @conda-tools/core
From @jakirkham on July 25, 2016 4:4
Some packages don't have a source section. Recently I have been having some issues with a package that has no source. However, have had some difficulties producing it offline with conda-build
alone. So it may be a conda-build-all
issue in the end. Here is a log demonstrating this failure. A traceback is shown below. It seems that maybe there are some cases where the work
directory is not removed when it is expected to have been.
no source - creating empty work folder
Traceback (most recent call last):
File "/opt/conda/bin/conda-build-all", line 9, in <module>
load_entry_point('conda-build-all==0.13.1', 'console_scripts', 'conda-build-all')()
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/cli.py", line 85, in main
b.main()
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 230, in main
built_dist_location = self.build(meta)
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 187, in build
return bldpkg_path(build.build(meta.meta))
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/build.py", line 32, in build
build_module.build(meta, post=None, need_source_download=True, **kwd)
File "/opt/conda/lib/python3.5/site-packages/conda_build/build.py", line 548, in build
os.makedirs(source.WORK_DIR)
File "/opt/conda/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/opt/conda/conda-bld/work'
Copied from original issue: conda/conda-build#1145
The master branch of conda-build introduces a lot of somewhat disturbing output into conda-build-all. Specifically it looks like this:
$ conda-build-all /tmp/tagged-recipes/recipes --upload-channels nsls2builder --matrix-conditions "numpy >=1.8" "python >=2.7,<3|>=3.4" --inspect-channels nsls2builder
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ..............
Resolving distributions from 120 recipes...
fatal: ambiguous argument 'v0.1.0': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'v0.1.0': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
...
The offending commit this this: conda/conda-build@7ce7be2
Modifying https://github.com/conda/conda-build/blob/master/conda_build/environ.py#L76 to add a pdb.set_trace() allows me to determine that conda-build
is expecting the source code for the recipe that conda-build-all
is asking about to have been checked out into the work directory.
The actual exception is
Command '['git', 'log', '-n1', '--format=%H', 'v0.1.0']' returned non-zero exit status 128
Here are the values for the variables that are local to conda_build.environ.get_git_build_info()
at this point in the program execution:
expected_rev: 'v0.1.0'
src_dir: '/home/edill/mc/conda-bld/work'
git_url: 'http://github.com/kennethreitz/args'
current_commit: '1f4685f3f67dcf8442b16aa1204b3ed72ce5bb2c\n' # note that the current_commit needs to be stripped
and the contents of src_dir
are the last thing that I did with conda-build,
$ ls ~/mc/conda-bld/work
build conda-build-all.recipe LICENSE README.md setup.cfg
conda_build_all conda_build.sh MANIFEST.in record.txt setup.py
conda_build_all.egg-info CONTRIBUTING.md __pycache__ requirements.txt versioneer.py
I do not see a clear solution here, or I would implement something and submit a PR and we could discuss further on the PR. That being said, I am happy to work on fixing this, but will need guidance from you. What are your thoughts? cc @pelson
Also cc @stuarteberg since you might be able to provide some guidance on the conda-build side of things.
And the versions of conda related things on my system
$ conda-build --version
conda-build 1.19.0+4.g0491732
$ conda-build-all --version
0.9.0
$ git --version
git version 2.5.0
hub version 2.2.2
$ conda --version
conda 3.19.1
[edit] add versions
We are seeing this error if a recipe depends on other recipes in the same directory. This seem to have been fixed a while ago. We are using the combo of, and
- conda install conda-build=2.0.10
- conda install -c conda-forge conda-build-all=1.0*
conda-build-all
works with conda-4.3.22, but fails in 4.3.23
because NoPackagesFound
is not found anymore in conda.resolve
From @PeterDSteinberg on September 20, 2016 19:30
Hi. I am getting an error in conda-build on conda-forge that I am not getting locally with conda-build in more recent versions than 1.21.14.
The error is several parts.
...Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 200, in build
conda_build.api.build(meta.meta, config=config)
AttributeError: module 'conda_build' has no attribute 'api'
conda.resolve.Unsatisfiable: The following specifications were found to be in conflict:
- pandas ==0.18.1 -> numpy 1.10*|1.11*
- setuptools
Full build on circle CI (scroll to the bottom)
Here's the recipe I am trying to build
Any ideas? Can this be solved with upgrading conda-build?
Copied from original issue: conda-forge/conda-forge.github.io#240
Hi Phil,
I'm sure there is a better repo to ask this question against, but what is your recommended scopes argument for generating a binstar token?
Several (phonopy, elastic, pyhmc) staged-build PRs got broken by the numpy/openblas problems on osx and linux for py3k. I had to block builds for py3k on these platforms. At the same time the feedstocks with similar code and setup (e.g. spglib) build fine. I'll be happy to provide any additional info to help with this.
Appears this problem does not occur with conda-build-all
version 1.0.0
, but it does occur with conda-build-all
version 1.0.1
. This points to an issue with PR ( #68 ), but it is not clear to me yet what it is. Any ideas @mwcraig ?
Hi Phil,
My use case here is to have a git repo full of dev recipes where the repo is set up with circleci and conda-build-all to compare the dev recipes in the aforementioned git repo with the conda packages on my dev channel on anaconda.org and only build ones with commits in master (and therefore new version numbers).
However, if I use any of the GIT_* info in a conda-recipe, conda-build-all
is unable to correctly interpret the version string to compare against the built packages on anaconda.org/user_name.
The following recipe is a relevant snippet from a master branch meta.yaml:
package:
name: metadatastore
version: {{ environ['GIT_DESCRIBE_TAG'] }}.post{{ environ['GIT_DESCRIBE_NUMBER'] }}
source:
git_url: https://github.com/NSLS-II/metadatastore.git
git_rev: master
# use a local url and leave rev blank to have conda build current head, currently
# requires master branch of conda-build to work
# git_rev: ?
build:
string: {{ environ.get('GIT_BUILD_STR', '') }}_py{{ py }}
number: 0
--snip--
which conda-build-all
interprets as
Resolved dependencies, will be built in the following order:
metadatastore-.post-_py27 (will be built: True)
metadatastore-.post-_py34 (will be built: True)
metadatastore-.post-_py35 (will be built: True)
Is it within the scope of conda-build-all
to be able to recognize a meta.yaml has GIT_* info inside it and attempt to figure out what the build string should be? As this is a feature that would make my life loads easier, I am happy to work on adding this to conda-build-all
. I would imagine that it would have to go something like:
source.git_url
source.git_rev
git describe
to get the exact version stringThis is non-trivial and is certainly not guaranteed to work in all situations, but would potentially let conda-build-all be used for continuous delivery.
It would be nice to be able to run::
conda-build-all ./ --matrix-conditions "some stuff" --dry-run
and see what would get built, etc, without it actually starting the build process....
Combined with #8's idea for showing what really will get built, this could be handy
not that hitting ctrl+C is that hard....
From @asmeurer on January 16, 2017 4:8
I tried to rerender sphinxcontrib-autoprogram and got this error:
$conda smithy rerender
No circle token. Create a token at https://circleci.com/account/api and
put it in ~/.conda-smithy/circle.token
No appveyor token. Create a token at https://ci.appveyor.com/api-token and
Put one in ~/.conda-smithy/appveyor.token
No anaconda token. Create a token via
anaconda auth --create --name conda-smithy --scopes "repos conda api"
and put it in ~/.conda-smithy/anaconda.token
Fetching package metadata .................
Traceback (most recent call last):
File "/Users/aaronmeurer/anaconda3/bin/conda-smithy", line 6, in <module>
sys.exit(conda_smithy.cli.main())
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 289, in main
args.subcommand_func(args)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 200, in __call__
configure_feedstock.main(args.feedstock_directory)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 644, in main
render_run_docker_build(env, config, forge_dir)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 77, in render_run_docker_build
forge_config.get('channels', {}).get('sources', tuple())
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 565, in compute_build_matrix
mtx = special_case_version_matrix(meta, index)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_build_all/version_matrix.py", line 212, in special_case_version_matrix
py_vn = minor_vn(index[python_pkg.fn]['version'])
AttributeError: 'Dist' object has no attribute 'fn'
Copied from original issue: conda-forge/conda-smithy#435
This is a Docs issue, I'm sure. But I haven't figured it out.
I have a module that uses the numpy API, so I want the conda package pinned to the numpy version.
So I added:
numpy x.x
to the meta.yaml. Now, when I run conda build, it requires a numpy version -- makes sense.
but when I try to use conda-build-all like so:
conda-build-all ./ --matrix-conditions "python 2.7.*" "numpy >=1.11"
I get:
File "/Users/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build/metadata.py", line 478, in ms_depends
ms = handle_config_version(ms, ver, typ)
File "/Users/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build/metadata.py", line 320, in handle_config_version
raise RuntimeError("'%s' requires external setting" % ms.spec)
RuntimeError: 'numpy x.x' requires external setting
Continuing from the discussion near the tail end of issue ( conda-forge/conda-execute-feedstock#17 ).
Would like to figure out how we preserve the history from the conda-forge
fork of conda-build-all
. In particular the history from issue/PR discussions should they need to be revisited. One option might be to migrate the issues/PRs to this repo somehow (though not sure of a tool to do this for PRs). Another option would be to migrate that repo over here and note that it is historic. Though if there are other options, would be interested to hear them.
Given a recipe that includes environ.GIT_DESCRIBE_TAG
as a jinja variable in the meta.yaml, trying to use conda-build-all to build py27, py34 and py35 successfully builds py27, but then fails on py34 with the following error:
Error: Failed to render jinja template in /tmp/nb_conda_kernels-recipe/recipe/meta.yaml:
'dict object' has no attribute 'GIT_DESCRIBE_TAG'
I find this rather surprising and cannot reproduce it by invoking conda-build
directly. Here's a script that you can run on a unix system that will download and install a fresh copy of miniconda into /tmp. It will then update --all from conda-forge and install conda-build-all from conda-forge into that temp environment. Finally it will git clone a git repo that contains the lightweight recipe (jupyter extension) that I'm trying to build with conda-build-all
(and hopefully it will fail in the same way it is failing for me!)
repo: https://github.com/ericdill/nb_conda_kernels-recipe
script: https://github.com/ericdill/nb_conda_kernels-recipe/blob/master/run-build.sh
I get:
TypeError: add_release() got an unexpected keyword argument 'description'
Looks like this recent change to anaconda-client has removed the 'description' argument:
anaconda/anaconda-client#394
Not sure if this is a conda-build
or conda-build-all
issue (am leaning towards the latter), but am going to start by raising it here.
If numpy
is listed as a build dependency and x.x
is not used, then a build with conda-build-all
tries to download numpy 1.0*
. See this comment. This is likely due to this line. Removal of numpy
from build
is an effective workaround for the problem.
The easiest "fix" (workaround) would probably be to set this to the newest version of NumPy so change to 111
. Though it would be nice to know why it is happening in the first place. Namely is this intentional behavior by conda-build
to do this now or is this a bug.
I am attempting to use conda-build-all
to upload to our internal anaconda server and am getting the following traceback.
Traceback (most recent call last):
File "/tmp/root/ramdisk/mc/bin/conda-build-all", line 9, in <module>
load_entry_point('conda-build-all==0.8.1.dev14', 'console_scripts', 'conda-build-all')()
File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/cli.py", line 85, in main
b.main()
File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/builder.py", line 208, in main
self.post_build(meta, built_dist_location, was_built)
File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/builder.py", line 225, in post_build
artefact_destination.make_available(meta, built_dist_location, was_built)
File "/tmp/root/ramdisk/mc/lib/python3.5/site-packages/conda_build_all-0.8.1.dev14-py3.5.egg/conda_build_all/artefact_destination.py", line 119, in make_available
raise NotImplementedError('cross owner copying not yet implemented.')
NotImplementedError: cross owner copying not yet implemented.
I am executing conda-build-all with
$ conda-build-all ../staged-recipes-dev/ --upload-channels nsls2-dev --inspect-channels nsls2-dev --matrix-conditions "python >=3.5"
I'm not convinced that I am trying to copy across owners which makes this error all the more puzzling. Additionally puzzling is that this is failing on different packages each time I run the aforementioned conda-build-all command.
Have you seen this before?
So I did a bit of investigation after I wrote the stuff above, but before I posted the issue. Updating to conda-build master seems to fix this problem. I will go ahead and post this and then close it immediately, mostly for posterity's sake.
Seems conda-build-all
has had a wrench thrown in it with the recent conda-build
release (1.21.7
). Not really sure what is causing it, but it seems to have some issues identifying where source should be extracted. See this comment for one example. Also here is another that springs up in this build. It would be nice to figure out why source extraction is getting messed up in these cases.
In the recipe attached, run with conda-build-all --inspect-channels=astropy recipes
, the build gets as far as trying to resolve dependencies then fails with an error RuntimeError: 'numpy x.x' requires external setting
.
The full traceback is:
$ conda-build-all --inspect-channels=astropy recipes
Fetching package metadata: ......
Resolving distributions from 1 recipes...
Traceback (most recent call last):
File "/Users/mcraig/miniconda-clean/bin/conda-build-all", line 9, in <module>
load_entry_point('conda-build-all==0.8.0', 'console_scripts', 'conda-build-all')()
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/cli.py", line 85, in main
b.main()
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/builder.py", line 195, in main
all_distros = self.compute_build_distros(index, recipe_metas)
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/builder.py", line 183, in compute_build_distros
if distro.pkg_fn() not in index:
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build_all/resolved_distribution.py", line 82, in with_vn_mtx_setup
return orig_result(*args, **kwargs)
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 504, in pkg_fn
return "%s.tar.bz2" % self.dist()
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 501, in dist
return '%s-%s-%s' % (self.name(), self.version(), self.build_id())
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 474, in build_id
for ms in self.ms_depends():
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 447, in ms_depends
ms = handle_config_version(ms, ver)
File "/Users/mcraig/miniconda-clean/lib/python2.7/site-packages/conda_build/metadata.py", line 333, in handle_config_version
raise RuntimeError("'%s' requires external setting" % ms.spec)
RuntimeError: 'numpy x.x' requires external setting
it seems to me that the most common need for to make a release is to
it sounds like this project does 1), would adding 2 and 3 be difficult?
Thanks a bunch for making this! I'd like to get rid of maintaining my separate script that does mostly the same thing as this package. There are a two features that would be nice to have here to make that possible.
--inspect-channels
, so probably very few of them will be built. IMO it would be nice, at end maybe, to print out just the ones that will be built. Or maybe to use color highlighting for the ones that will be built?I'm not sure where the underlying issue is, but when I copy across channels and then try to use a package from the channel to which the package was copied I get errors like this: https://travis-ci.org/astropy/ccdproc/jobs/146212815#L472
Error: HTTPError: 404 Client Error: NOT FOUND for url: https://conda.anaconda.org/astropy/linux-64/reproject-0.3.1-np111py27_0.tar.bz2: https://conda.anaconda.org/astropy/linux-64/reproject-0.3.1-np111py27_0.tar.bz2
this is with:
$ conda-build-all --version
1.0.1
$ conda-build --version
conda-build 2.0.12
when I run conda-build-all it does everything nicely, until it comes time to upload a package, and then I get:
Uploading unit_conversion to the main channel.
INFO artefact_destination:make_available(116): Uploading unit_conversion to the main channel.
Traceback (most recent call last):
File "/home/chris.barker/miniconda2/bin/conda-build-all", line 6, in
sys.exit(conda_build_all.cli.main())
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/cli.py", line 90, in main
b.main()
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/builder.py", line 269, in main
config=build_config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/builder.py", line 289, in post_build
config=config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/artefact_destination.py", line 118, in make_available
config=config)
File "/home/chris.barker/miniconda2/lib/python2.7/site-packages/conda_build_all/build.py", line 43, in upload
fname = bldpkg_path(meta, config)
TypeError: bldpkg_path() takes exactly 1 argument (2 given)
Traceback (most recent call last):
File "./build-all.py", line 46, in
env=this_env)
File "/home/chris.barker/miniconda2/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda-build-all', './', '--matrix-conditions', 'python 2.7.*', '--inspect-channels', 'NOAA-ORR-ERD', '--upload-channels', 'NOAA-ORR-ERD', '--no-inspect-conda-bld-directory', '--artefact-directory', 'packages']' returned non-zero exit status 1
I see this code around the error:
if hasattr(conda_build, 'api'):
fname = bldpkg_path(meta, config)
else:
fname = bldpkg_path(meta)
so it looks like conda_build 2.0.12 has the 'api' attribute, but has the wrong signature for pldpkg_path ??
I get the same error with conda-build 2.0.11
But it works with 2.0.10 !!
-CHB
It would be super useful to have an option of soft failing the build when RuntimeError is raised due to unsatisfiable dependencies. So it would allow to finish all the builds and just gives a summary of packages where it failed.
E.g here I would update a few packages, a total of 15 combinations, but the build fails in the middle as one of the version combinations cannot find all the dependencies. Some of these combinations doesn't make much sense anyway, but I didn't find a good way to disable them one-by-one in requirements.yml.
As a heads up, the next release of conda-build-all will support conda-build 2.0, and it will be released as version 1.0.
After some discussion with @johanneskoester and @mingwandroid , it seems that R is not being handled in conda-build-all
as it is in conda-build
. Namely it sounds like it should be searching for r-base
and using that as part of the matrix and not r
. Please feel free to correct me if I am mistaken.
I'm running into this issue with 0.13.0 and 0.13.1, but not with 0.12.0. It sometimes fails to get all of the items in a matrix that I would expect. An example would be trying to run conda smithy rerender
on this feedstock at that commit. It only provides Python with 3.5 and not any earlier versions. Though there is no version constraint or skip provided. I have provided all of the version info below to help narrow this down.
conda list -n root
anaconda-client 1.4.0 py27_0 defaults
anaconda-verify 1.1.0 py27_0 defaults
conda 4.1.8 py27_0 defaults
conda-build 1.21.5 py27_0 defaults
ca-certificates 2016.2.28 1 conda-forge
clyent 1.2.1 py27_0 conda-forge
conda-build-all 0.12.0 py27_1 conda-forge
conda-env 2.5.2 py27_0 conda-forge
conda-execute 0.6.0 py27_0 conda-forge
conda-smithy 1.0.1 py27_0 conda-forge
curl 7.49.1 0 conda-forge
expat 2.1.0 1 conda-forge
funcsigs 1.0.2 py27_0 conda-forge
gitdb 0.6.4 py27_0 conda-forge
gitpython 1.0.1 py27_0 conda-forge
jinja2 2.8 py27_0 conda-forge
markupsafe 0.23 py27_0 conda-forge
mock 2.0.0 py27_0 conda-forge
ncurses 5.9 7 conda-forge
openssl 1.0.2h 0 conda-forge
psutil 4.3.0 py27_0 conda-forge
pycosat 0.6.1 py27_0 conda-forge
pycrypto 2.6.1 py27_0 conda-forge
pygithub 1.26.0 py27_0 conda-forge
python-dateutil 2.5.2 py27_0 conda-forge
pytz 2016.3 py27_0 conda-forge
pyyaml 3.11 py27_0 conda-forge
readline 6.2 0 conda-forge
requests 2.10.0 py27_0 conda-forge
ruamel.ordereddict 0.4.6 py27_0 conda-forge
ruamel.yaml 0.11.11 py27_1 conda-forge
setuptools 23.0.0 py27_0 conda-forge
six 1.10.0 py27_0 conda-forge
smmap 0.9.0 py27_0 conda-forge
sqlite 3.13.0 1 conda-forge
yaml 0.1.6 0 conda-forge
zlib 1.2.8 3 conda-forge
constructor 1.3.0 py27_0 defaults
libconda 4.0.0 py27_0 defaults
pbr 1.10.0 py27_0 defaults
pip 8.1.2 py27_0 defaults
python 2.7.12 1 defaults
ruamel_yaml 0.11.7 py27_0 defaults
tk 8.5.18 0 defaults
wheel 0.29.0 py27_0 defaults
Got this issue when the filenames compression did not match the actual compression type. Admittedly the issue goes away when that is fixed, but maybe it is just exposing an underlying issue. Hence why I have raised it. Traceback below. See build log for more details.
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 200, in build
conda_build.api.build(meta.meta, config=config)
AttributeError: module 'conda_build' has no attribute 'api'
When trying run conda-build-all
on the included recipe, I encounter a build issue when the root
environment is Python 3, but not when the root
environment is Python 2. More details about the recipe and environments below. Was unable to reproduce this with conda-build
alone in either environment. Am sort of suspicious that this issue is actually caused by Python versions. Though it could be.
Python 3 traceback (Python 2 has no issues).
$ conda build-all recipes/wrf-python --matrix-condition "numpy >=1.11" "python >=2.7,<3|>=3.5"
Fetching package metadata .............
Error: Invalid selector in meta.yaml line 15:
skip: True # [win32 or np>=112 or (win and py>=35)]
Recipe below.
# meta.yaml
{% set version = "1.0b1" %}
package:
name: wrf-python
version: {{ version }}
source:
fn: wrf-python-{{ version }}.tar.gz
url: https://github.com/NCAR/wrf-python/archive/{{ version }}.tar.gz
sha256: 94f05a9719701959702094eafb84555a996bc4f7d75042fb07bca3c35bda9f31
build:
number: 1
detect_binary_files_with_prefix: true
skip: True # [win32 or np>=112 or (win and py>=35)]
requirements:
build:
- pip
- numpy x.x
- wrapt
- m2w64-toolchain # [win]
- gcc # [unix]
- libgcc # [unix]
- python
- vc 9 # [win and py27]
- vc 10 # [win and py34]
- vc 14 # [win and py>=35]
run:
- numpy x.x
- python
- wrapt
- m2w64-toolchain # [win]
- gcc # [unix]
- libgcc # [unix]
- xarray
- vc 9 # [win and py27]
- vc 10 # [win and py34]
- vc 14 # [win and py>=35]
test:
requires:
- nose
imports:
- wrf
about:
home: https://github.com/NCAR/wrf-python
license: "NCL Source Code"
summary: "Diagnostic and interpolation routines for WRF-ARW data."
extra:
recipe-maintainers:
- bladwig1
- marylhaley
- david-ian-brown
- khallock
# environment2.yml
name: root
channels: !!python/tuple
- !!python/unicode
'nanshe'
- !!python/unicode
'conda-forge'
- !!python/unicode
'jakirkham'
- !!python/unicode
'defaults'
dependencies:
- beautifulsoup4=4.5.3=py27_0
- chardet=2.3.0=py27_0
- conda-forge::anaconda-client=1.5.1=py27_0
- conda-forge::anaconda-verify=1.2.0=py27_0
- conda-forge::appnope=0.1.0=py27_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py27_1
- conda-forge::ca-certificates=2016.9.26=0
- conda-forge::certifi=2016.9.26=py27_0
- conda-forge::cffi=1.7.0=py27_0
- conda-forge::clyent=1.2.1=py27_0
- conda-forge::conda=4.2.13=py27_0
- conda-forge::conda-build=2.1.1=py27_1
- conda-forge::conda-build-all=1.0.1=py27_1
- conda-forge::conda-env=2.6.0=0
- conda-forge::conda-execute=0.8.0=py27_0
- conda-forge::constructor=1.3.4=py27_0
- conda-forge::contextlib2=0.5.4=py27_0
- conda-forge::coverage=3.7.1=py27_0
- conda-forge::curl=7.52.1=0
- conda-forge::decorator=4.0.10=py27_0
- conda-forge::enum34=1.1.6=py27_1
- conda-forge::expat=2.1.0=2
- conda-forge::filelock=2.0.6=py27_0
- conda-forge::funcsigs=1.0.2=py27_0
- conda-forge::futures=3.0.5=py27_0
- conda-forge::gitdb=0.6.4=py27_1
- conda-forge::gitpython=2.1.1=py27_0
- conda-forge::idna=2.1=py27_0
- conda-forge::ipaddress=1.0.16=py27_0
- conda-forge::ipython=5.1.0=py27_2
- conda-forge::ipython_genutils=0.1.0=py27_0
- conda-forge::jinja2=2.8=py27_1
- conda-forge::libconda=4.0.0=py27_0
- conda-forge::libffi=3.2.1=3
- conda-forge::markupsafe=0.23=py27_1
- conda-forge::mock=2.0.0=py27_0
- conda-forge::ncurses=5.9=10
- conda-forge::nose=1.3.7=py27_2
- conda-forge::openssl=1.0.2h=3
- conda-forge::pathlib2=2.1.0=py27_0
- conda-forge::pbr=1.10.0=py27_0
- conda-forge::pexpect=4.2.1=py27_0
- conda-forge::pickleshare=0.7.3=py27_0
- conda-forge::pip=9.0.1=py27_0
- conda-forge::pkginfo=1.2.1=py27_0
- conda-forge::prompt_toolkit=1.0.9=py27_0
- conda-forge::psutil=4.3.0=py27_0
- conda-forge::ptyprocess=0.5.1=py27_0
- conda-forge::pyasn1=0.1.9=py27_0
- conda-forge::pycosat=0.6.1=py27_0
- conda-forge::pycparser=2.17=py27_0
- conda-forge::pycrypto=2.6.1=py27_0
- conda-forge::pygithub=1.29=py27_0
- conda-forge::pygments=2.1.3=py27_1
- conda-forge::pyopenssl=16.0.0=py27_0
- conda-forge::python=2.7.12=1
- conda-forge::python-dateutil=2.6.0=py27_0
- conda-forge::pytz=2016.10=py27_0
- conda-forge::pyyaml=3.11=py27_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.12.4=py27_0
- conda-forge::ruamel.ordereddict=0.4.6=py27_0
- conda-forge::ruamel.yaml=0.13.7=py27_0
- conda-forge::ruamel_yaml=0.11.14=py27_0
- conda-forge::setuptools=32.3.1=py27_0
- conda-forge::simplegeneric=0.8.1=py27_0
- conda-forge::six=1.10.0=py27_1
- conda-forge::smmap=2.0.1=py27_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.1=py27_0
- conda-forge::typing=3.5.3.0=py27_0
- conda-forge::wcwidth=0.1.7=py27_0
- conda-forge::wheel=0.29.0=py27_0
- conda-forge::yaml=0.1.6=0
- conda-forge::zlib=1.2.8=3
- conda-verify=2.0.0=py27_0
- cryptography=1.7.1=py27_0
- python.app=1.2=py27_4
- pip:
- backports.shutil-get-terminal-size==1.0.0
- conda-build-all (/zopt/conda2/lib/python2.7/site-packages)==1.0.1
- conda-smithy (/Users/kirkhamj/Developer/Conda/conda-forge/conda-smithy)==2.0.0
- ipython-genutils==0.1.0
- prompt-toolkit==1.0.9
- ruamel-yaml==0.11.14
- smmap2==2.0.1
prefix: /zopt/conda2
# environment3.yml
name: root
channels: !!python/tuple
- nanshe
- conda-forge
- jakirkham
- defaults
dependencies:
- beautifulsoup4=4.5.3=py35_0
- chardet=2.3.0=py35_0
- conda-forge::anaconda-client=1.5.1=py35_0
- conda-forge::appnope=0.1.0=py35_0
- conda-forge::backports.shutil_get_terminal_size=1.0.0=py35_1
- conda-forge::ca-certificates=2016.9.26=0
- conda-forge::certifi=2016.9.26=py35_0
- conda-forge::clyent=1.2.1=py35_0
- conda-forge::conda=4.2.13=py35_0
- conda-forge::conda-build=2.1.1=py35_1
- conda-forge::conda-build-all=1.0.1=py35_1
- conda-forge::conda-env=2.6.0=0
- conda-forge::decorator=4.0.10=py35_0
- conda-forge::filelock=2.0.6=py35_0
- conda-forge::freetype=2.6.3=1
- conda-forge::gitdb=0.6.4=py35_1
- conda-forge::gitpython=2.1.1=py35_0
- conda-forge::ipython=5.1.0=py35_2
- conda-forge::ipython_genutils=0.1.0=py35_0
- conda-forge::jinja2=2.8=py35_1
- conda-forge::jpeg=9b=0
- conda-forge::libpng=1.6.28=0
- conda-forge::libtiff=4.0.6=7
- conda-forge::markupsafe=0.23=py35_1
- conda-forge::mock=2.0.0=py35_0
- conda-forge::ncurses=5.9=10
- conda-forge::nose=1.3.7=py35_2
- conda-forge::openssl=1.0.2h=3
- conda-forge::path.py=8.2.1=py35_0
- conda-forge::pbr=1.10.0=py35_0
- conda-forge::pexpect=4.2.1=py35_0
- conda-forge::pickleshare=0.7.3=py35_0
- conda-forge::pillow=4.0.0=py35_0
- conda-forge::pip=9.0.1=py35_0
- conda-forge::pkginfo=1.2.1=py35_0
- conda-forge::prompt_toolkit=1.0.9=py35_0
- conda-forge::ptyprocess=0.5.1=py35_0
- conda-forge::pycosat=0.6.1=py35_0
- conda-forge::pycrypto=2.6.1=py35_0
- conda-forge::pygithub=1.29=py35_0
- conda-forge::pygments=2.1.3=py35_1
- conda-forge::python=3.5.2=5
- conda-forge::python-dateutil=2.6.0=py35_0
- conda-forge::python-simplegeneric=0.8.1=py35_0
- conda-forge::pytz=2016.10=py35_0
- conda-forge::pyyaml=3.11=py35_0
- conda-forge::readline=6.2=0
- conda-forge::requests=2.12.4=py35_0
- conda-forge::ruamel.yaml=0.13.7=py35_0
- conda-forge::ruamel_yaml=0.11.14=py35_0
- conda-forge::setuptools=32.3.1=py35_0
- conda-forge::simplegeneric=0.8.1=py35_0
- conda-forge::six=1.10.0=py35_1
- conda-forge::smmap=2.0.1=py35_0
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.1=py35_0
- conda-forge::wcwidth=0.1.7=py35_0
- conda-forge::wheel=0.29.0=py35_0
- conda-forge::xz=5.2.2=0
- conda-forge::yaml=0.1.6=0
- conda-forge::zlib=1.2.8=3
- conda-verify=2.0.0=py35_0
- jbig=2.1=0
- python.app=1.2=py35_4
- pip:
- backports.shutil-get-terminal-size==1.0.0
- conda-smithy (/Users/kirkhamj/Developer/Conda/conda-forge/conda-smithy)==2.0.0
- ipython-genutils==0.1.0
- prompt-toolkit==1.0.9
- ruamel-yaml==0.11.14
- smmap2==2.0.1
prefix: /zopt/conda3
While we have discussed this several places before, wasn't seeing this tracked anywhere. So thought it best to open an issue here. Guessing we will need to handle VC correctly here before we can address it in conda-smithy
.
Currently in order to get a package to build with multiple VC runtimes, we need to add python
as a build
dependency and track a feature. However, this has proved impractical and in some cases limiting. Instead discussion over the past several months has concluded the best way forward is to have a separate vc
package for tracking the compiler that python
and other packages track. Though ultimately this must be addressed with some kind of matrix. While we can find somewhat effective ways to do this manually, conda-build-all
has reigned supreme at helping convert an implicit matrix to an explicit matrix (via leveraging by conda-smithy
). Hence I'm raising this here to ask that vc
be handled in the matrix configuration used by staged-recipes
and can picked up with conda-smithy
for use in the feedstocks. There are already some considerations about how this might play out.
xref: conda-forge/glpk-feedstock#10
xref: conda-forge/tk-feedstock#2
xref: conda-forge/curl-feedstock#11
xref: conda-forge/conda-smithy#245
Recently I ran conda_build_all, the relevant packages were built fine however following the build I received the following errors
anaconda_upload is not set. Not uploading wheels: []
pyyaml-3.12-py27_1 ['/data/local/lcarroll/conda_bld/root/linux-64/pyyaml-3.12-py27_1.tar.bz2'] True
Traceback (most recent call last):
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/bin/conda-build-all", line 6, in <module>
exit(main())
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/cli.py", line 93, in main
b.main()
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/builder.py", line 299, in main
config=build_config)
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/builder.py", line 319, in post_build
config=config)
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/site-packages/conda_build_all/artefact_destination.py", line 65, in make_available
shutil.copy(built_dist_path, self.directory)
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/shutil.py", line 240, in copy
dst = os.path.join(dst, os.path.basename(src))
File "/var/tmp/lcarroll/conda_recipes/tmp_envs/6f743906216fccdf7c10/lib/python3.6/posixpath.py", line 144, in basename
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not list
Aborted!
CalledProcessError: Command '['bash', '/var/tmp/lcarroll/conda_recipes/configuration/scidesktop/conda_build/build.sh', 'conda-recipes']' returned non-zero exit status 1.
This seems to be related to Pull Request #93, we were not encountering these errors before this Pull Request.
Since #93 was merged built_dist_path
or self.directory
within artefact_destination.DirectoryDestination
have become lists. Hence passing them to shutil.copy
results in the error:
TypeError: expected str, bytes or os.PathLike object, not list
Latest conda-build has a rendering step that broke conda-build-all
(@msarahan can explain it better than me ๐ฌ).
For now I added two workarounds that need to be removed once we fix this:
Basically issue ( conda-forge/conda-smithy#191 ) being punted upstream. @alexbw , can you please advise us one how we set the versioning for this?
Hi folks,
I seem to have hit this NotImplementedError again that is in #21 and #18 . I spent more time digging in to this problem with the pdb this time and have determined that I am getting this error due to two things.
linux-64/linux-64-pcaspy-0.5.1-py34_2.tar.bz2
, which is odd. Seen here: https://anaconda.org/lightsource2/pcaspy/files. I have been unsuccessful in determining why this pcaspy conda package has an extra linux-64-
prepending the pcaspy-0.5.1-py34_2.tar.bz2
package name.conda-build-all
thinks that it does not have to build this package initially when it checks the lightsource2 channel with the conda.api.get_index()
function, but then when it skips the build but checks to see if it should upload something, conda-build-all fails because it uses a different pathway to determine if the file is already on anaconda.org, with the anaconda_client.distribution()
function call.Would you be opposed to making these checks symmetric by using the same conda.api.get_index() call in both of these places? As it is correctly finding the oddly named package on anaconda.org/lightsource2, I am inclined to declare that is more robust than the anaconda_client.distribution() function call. I'll get to work on a PR to make these checks symmetric so that we can see what a viable solution looks like. If it is horrible, I'll just fix this problem by deleting the offending packages on the lightsource2 channel.
Here are my notes from debugging this issue (mostly so I don't have to dig in to this again!) https://gist.github.com/ericdill/4415a840ed6b191f3ac7
My use case:
I have a repo of packages that I'm maintaining.
I update one or two, and want conda-build-all to build and upload them to anaconda.org.
However, in testing, I've already make sure that the new one builds. THis means that it is in the conda-build dir.
which means that conda-build-all doesn't build it again -- good.
But it also doesn't upload it -- which I do want.
So I set: --no-inspect-conda-bld-directory, and it works.
But it seems silly to re-build all those, when all I want is to upload the already built packages...
I'm thinking there is no reason, if the --no-inspect-conda-bld-directory flag isn't set, not to look in conda-build, if it's there, don't rebuild, but do upload.
If --upload-channels is specified but --inspect-channels is not, all recipes will be built and at the end of the recipe building, the following message will be printed out:
Assuming the distribution we've just built and the one on lightsource2/main are the same.
It seems like it would be sensible for --inspect-channels
to default to --upload-channels
if upload is present but inspect is not. Thoughts?
Appears that conda-build
version 2.0.9
and conda
version 4.1
are incompatible. As this appears to be a temporary issue, would recommend that an exclusion against conda-build
version 2.0.9
. If this is combined with a version bump, it will hopefully avoid issues with conda
ignoring the pinning. A similar strategy seems to be working with conda-smithy
so far.
cc @bjlittle
conda-build-all version 1.0.4 fails with conda-build 3.0.3
conda build-all recipes --matrix-max-n-minor-versions 3 --matrix-conditions "python $PYTHON_BUILD_RESTRICTIONS" "$NUMPY_BUILD_RESTRICTION"
Fetching package metadata .........
Traceback (most recent call last):
File "/home/yfeng1/anaconda3/install/bin/conda-build-all", line 6, in <module>
sys.exit(conda_build_all.cli.main())
File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/cli.py", line 90, in main
b.main()
File "/home/yfeng1/anaconda3/install/lib/python3.5/site-packages/conda_build_all/builder.py", line 249, in main
if build_config.CONDA_NPY is None:
AttributeError: 'Config' object has no attribute 'CONDA_NPY'
No circle token. Create a token at https://circleci.com/account/api and
put it in ~/.conda-smithy/circle.token
No appveyor token. Create a token at https://ci.appveyor.com/api-token and
Put one in ~/.conda-smithy/appveyor.token
No anaconda token. Create a token via
anaconda auth --create --name conda-smithy --scopes "repos conda api"
and put it in ~/.conda-smithy/anaconda.token
Fetching package metadata .................
Traceback (most recent call last):
File "/Users/aaronmeurer/anaconda3/bin/conda-smithy", line 6, in <module>
sys.exit(conda_smithy.cli.main())
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 289, in main
args.subcommand_func(args)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/cli.py", line 200, in __call__
configure_feedstock.main(args.feedstock_directory)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 644, in main
render_run_docker_build(env, config, forge_dir)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 77, in render_run_docker_build
forge_config.get('channels', {}).get('sources', tuple())
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_smithy/configure_feedstock.py", line 565, in compute_build_matrix
mtx = special_case_version_matrix(meta, index)
File "/Users/aaronmeurer/anaconda3/lib/python3.5/site-packages/conda_build_all/version_matrix.py", line 212, in special_case_version_matrix
py_vn = minor_vn(index[python_pkg.fn]['version'])
KeyError: 'python-1.0.1-0.tar.bz2'
The keys in a conda index have changed to include the source, therefore the unit-tests are no longer representative of the actual conda integration.
An expired token is causing the integration tests that reach out to anaconda.org to fail. See, e.g., https://travis-ci.org/SciTools/conda-build-all/jobs/131601936#L354
#93 added conda-build 3 API compatibility, but did not add support for actually building recipes that use the features of conda-build 3.
This is a meta-issue for items that crop up:
AssertionError: modifying metadata after finalization
(sorry, don't have the traceback to hand, but it comes from resolved_distribution getattr (line 45).Fixing these things is a pretty deep exercise, and if this is going to get fixed, conda-build-all will cease support for conda-build <3.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.