conda-build master introduces a large volume of spam to conda-build-all about conda-build-all HOT 11 CLOSED

Context: conda/conda-build#761 and conda/conda-build#754 (comment)

from conda-build-all.

Thanks for reporting @ericdill. Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata (am I reading that right?). It is safe to say that it is not tenable for conda-build-all to support requiring the source to determine whether a recipe can be built (imagine having to checkout 120 sources, only to find you don't need to build any of them). So it seems to me we need to silence the messages (as already has to happen with the resolve logs from conda etc.), and try to identify cases where these variables are being used.

To be completely explicit about the implications of this, it will not be possible to build a conda recipe of the form:

{% set data = load_setuptools() %}

package:
    name: foobar
    version: {{data.get('version')}}

that conda-build-all itself uses: https://raw.githubusercontent.com/SciTools/conda-build-all/master/conda-build-all.recipe/meta.yaml.

I think that is a reasonable limitation - essentially, conda-build-all is supremely useful if you have a load of recipes for which you know the version you wish to build, but isn't good at building recipes which require runtime determination of versions (e.g. an untagged source such as master).

from conda-build-all.

@pelson So conda lint should check that load_setuptools() is not used in a recipe in a conda forge repo?

from conda-build-all.

To be completely explicit about the implications of this, it will not be possible to build a conda recipe of the form:

Is this because load_setuptools() requires the source to be checked out?

Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata

That is my understanding as well.

but isn't good at building recipes which require runtime determination of versions (e.g. an untagged source such as master).

On the second draft of my response, I convinced myself that it is out of the scope of conda-build-all to build dev recipes. If you are trying to build a bunch of dev recipes, you almost certainly have all of those packages available in tagged recipes. As such, you don't need conda-build-all to resolve the build order for you since the tagged deps are already available. Therefore, the only real advantage you get from using conda-build-all to build dev recipes is a time savings, when it would be just as easy (or perhaps easier?) to simply write a short script to check the output of conda-build /recipe-path --output --python $PYVER --numpy $NPVER and see if that package already exists on some target anaconda channel. If is does, great, if it doesn't build it and upload it.

from conda-build-all.

Looks like conda-build has added an assumption on the source being checked-out in order to correctly inspect the recipe metadata (am I reading that right?).

Not quite -- only if your meta.yaml uses GIT_ variables.

If meta.yaml contains a git_url, then we attempt to load values into the GIT_ environ variables. That's done by simply reading the contents of the currently-checked-out git repo in conda-bld/work. But if the git repo hasn't been checked out yet (or if conda-bld/work isn't the repo we expected), then we can't provide GIT_ variables at all.

So, your problem is that you see a bunch of spam on stderr, even for recipes that don't use GIT_ variables. I think this could be avoided if we simply combine stderr and stdout when we call git. Specifically, we need to add stderr=STDOUT to all of the check_output() lines in get_git_build_info().

If you try that, does it fix your problem?

from conda-build-all.

@stuarteberg That sure does it. Thank you for the well-targeted suggestion 😀 Shall I submit a PR into conda-build?

$ git diff master
diff --git a/conda_build/environ.py b/conda_build/environ.py
index 6489797..496a6b0 100644
--- a/conda_build/environ.py
+++ b/conda_build/environ.py
@@ -4,6 +4,7 @@ import os
 import sys
 from os.path import join, normpath, isabs
 import subprocess
+from subprocess import STDOUT
 import multiprocessing
 import warnings

@@ -48,20 +49,20 @@ def get_git_build_info(src_dir, git_url, expected_rev):
     env['GIT_DIR'] = git_dir
     try:
         # Verify current commit matches expected commit
-        current_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H"], env=env)
+        current_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H"], env=env, stderr=STDOUT)
         current_commit = current_commit.decode('utf-8')
-        expected_tag_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H", expected_rev], env=env)
+        expected_tag_commit = subprocess.check_output(["git", "log", "-n1", "--format=%H", expected_rev], env=env, stderr=STDOUT)
         expected_tag_commit = expected_tag_commit.decode('utf-8')

         # Verify correct remote url.
         # (Need to find the git cache directory, and check the remote from there.)
-        cache_details = subprocess.check_output(["git", "remote", "-v"], env=env)
+        cache_details = subprocess.check_output(["git", "remote", "-v"], env=env, stderr=STDOUT)
         cache_details = cache_details.decode('utf-8')
         cache_dir = cache_details.split('\n')[0].split()[1]
         assert "conda-bld/git_cache" in cache_dir

         env['GIT_DIR'] = cache_dir
-        remote_details = subprocess.check_output(["git", "remote", "-v"], env=env)
+        remote_details = subprocess.check_output(["git", "remote", "-v"], env=env, stderr=STDOUT)
         remote_details = remote_details.decode('utf-8')
         remote_url = remote_details.split('\n')[0].split()[1]
         if '://' not in remote_url:

from conda-build-all.

Shall I submit a PR into conda-build?

Sounds good to me!

from conda-build-all.

Are we able to close this issue now @ericdill?

from conda-build-all.

Ah, yes. Almost good to close!

I think there is one more question from @JanSchulz that may have not been answered:

@pelson So conda lint should check that load_setuptools() is not used in a recipe in a conda forge repo?

from conda-build-all.

So conda lint should check that load_setuptools() is not used in a recipe in a conda forge repo?

Not necessarily. Linting cannot verify that a recipe is good (the only way of doing that is to run it), it is simply there to check against a subset of common issues.

from conda-build-all.

Thanks @ericdill.

from conda-build-all.