Comments (11)
Context: conda/conda-build#761 and conda/conda-build#754 (comment)
from conda-build-all.
Thanks for reporting @ericdill. Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata (am I reading that right?). It is safe to say that it is not tenable for conda-build-all to support requiring the source to determine whether a recipe can be built (imagine having to checkout 120 sources, only to find you don't need to build any of them). So it seems to me we need to silence the messages (as already has to happen with the resolve logs from conda etc.), and try to identify cases where these variables are being used.
To be completely explicit about the implications of this, it will not be possible to build a conda recipe of the form:
{% set data = load_setuptools() %}
package:
name: foobar
version: {{data.get('version')}}
that conda-build-all itself uses: https://raw.githubusercontent.com/SciTools/conda-build-all/master/conda-build-all.recipe/meta.yaml.
I think that is a reasonable limitation - essentially, conda-build-all is supremely useful if you have a load of recipes for which you know the version you wish to build, but isn't good at building recipes which require runtime determination of versions (e.g. an untagged source such as master).
from conda-build-all.
@pelson So conda lint
should check that load_setuptools()
is not used in a recipe in a conda forge repo?
from conda-build-all.
To be completely explicit about the implications of this, it will not be possible to build a conda recipe of the form:
Is this because load_setuptools()
requires the source to be checked out?
Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata
That is my understanding as well.
but isn't good at building recipes which require runtime determination of versions (e.g. an untagged source such as master).
On the second draft of my response, I convinced myself that it is out of the scope of conda-build-all to build dev recipes. If you are trying to build a bunch of dev recipes, you almost certainly have all of those packages available in tagged recipes. As such, you don't need conda-build-all to resolve the build order for you since the tagged deps are already available. Therefore, the only real advantage you get from using conda-build-all to build dev recipes is a time savings, when it would be just as easy (or perhaps easier?) to simply write a short script to check the output of conda-build /recipe-path --output --python $PYVER --numpy $NPVER
and see if that package already exists on some target anaconda channel. If is does, great, if it doesn't build it and upload it.
from conda-build-all.
Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata (am I reading that right?).
Not quite -- only if your meta.yaml
uses GIT_
variables.
If meta.yaml
contains a git_url
, then we attempt to load values into the GIT_
environ variables. That's done by simply reading the contents of the currently-checked-out git repo in conda-bld/work
. But if the git repo hasn't been checked out yet (or if conda-bld/work
isn't the repo we expected), then we can't provide GIT_
variables at all.
So, your problem is that you see a bunch of spam on stderr, even for recipes that don't use GIT_
variables. I think this could be avoided if we simply combine stderr and stdout when we call git. Specifically, we need to add stderr=STDOUT
to all of the check_output()
lines in get_git_build_info()
.
If you try that, does it fix your problem?
from conda-build-all.
@stuarteberg That sure does it. Thank you for the well-targeted suggestion 😀 Shall I submit a PR into conda-build?
$ git diff master
diff --git a/conda_build/environ.py b/conda_build/environ.py
index 6489797..496a6b0 100644
--- a/conda_build/environ.py
+++ b/conda_build/environ.py
@@ -4,6 +4,7 @@ import os
import sys
from os.path import join, normpath, isabs
import subprocess
+from subprocess import STDOUT
import multiprocessing
import warnings
@@ -48,20 +49,20 @@ def get_git_build_info(src_dir, git_url, expected_rev):
env['GIT_DIR'] = git_dir
try:
# Verify current commit matches expected commit
- current_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H"], env=env)
+ current_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H"], env=env, stderr=STDOUT)
current_commit = current_commit.decode('utf-8')
- expected_tag_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H", expected_rev], env=env)
+ expected_tag_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H", expected_rev], env=env, stderr=STDOUT)
expected_tag_commit = expected_tag_commit.decode('utf-8')
# Verify correct remote url.
# (Need to find the git cache directory, and check the remote from there.)
- cache_details = subprocess.check_output(["git", "remote", "-v"], env=env)
+ cache_details = subprocess.check_output(["git", "remote", "-v"], env=env, stderr=STDOUT)
cache_details = cache_details.decode('utf-8')
cache_dir = cache_details.split('\n')[0].split()[1]
assert "conda-bld/git_cache" in cache_dir
env['GIT_DIR'] = cache_dir
- remote_details = subprocess.check_output(["git", "remote", "-v"], env=env)
+ remote_details = subprocess.check_output(["git", "remote", "-v"], env=env, stderr=STDOUT)
remote_details = remote_details.decode('utf-8')
remote_url = remote_details.split('\n')[0].split()[1]
if '://' not in remote_url:
from conda-build-all.
Shall I submit a PR into conda-build?
Sounds good to me!
from conda-build-all.
Are we able to close this issue now @ericdill?
from conda-build-all.
Ah, yes. Almost good to close!
I think there is one more question from @JanSchulz that may have not been answered:
@pelson So conda lint should check that load_setuptools() is not used in a recipe in a conda forge repo?
from conda-build-all.
So conda lint should check that load_setuptools() is not used in a recipe in a conda forge repo?
Not necessarily. Linting cannot verify that a recipe is good (the only way of doing that is to run it), it is simply there to check against a subset of common issues.
from conda-build-all.
Thanks @ericdill.
from conda-build-all.
Related Issues (20)
- Update conda-build to 2.0.1? HOT 3
- Support CONDA_LUA in matrix HOT 3
- Exclude conda-build version 2.0.9 HOT 2
- Bug with conda-build 2.0.11 and above. HOT 2
- Handling R/CONDA_R correctly HOT 1
- Problem with numpy/openblas in staged-recipes HOT 4
- AttributeError: 'Dist' object has no attribute 'fn' from rerender HOT 22
- Selector resolution problem / Python 2,3 compatibility issue? HOT 2
- KeyError: 'python-1.0.1-0.tar.bz2' from rerender HOT 11
- Packages missing in current linux-64 channels HOT 2
- Adding option of excepting failing builds due to missing dependencies.
- Exception raised with anaconda-client 1.6.3
- feaure to : build-all, convert all, upload all? HOT 1
- Migrate history from conda-forge HOT 2
- Fails with conda-build 3.0.3 HOT 10
- New release HOT 2
- conda-build-all not compatible with conda-4.3.23 HOT 9
- DirectoryDestination.make_available Passes List rather than String HOT 8
- Compatibility with conda-build=3 HOT 1
- Unsupported conda version: 4.5.5 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conda-build-all.