Alternatively, mamba repoquery may provide more information:
# Search all versions available on your platform:
mamba repoquery search conda-forge-ci-setup --channel conda-forge
# List packages depending on `conda-forge-ci-setup`:
mamba repoquery whoneeds conda-forge-ci-setup --channel conda-forge
# List dependencies of `conda-forge-ci-setup`:
mamba repoquery depends conda-forge-ci-setup --channel conda-forge
About conda-forge
conda-forge is a community-led conda channel of installable packages.
In order to provide high-quality builds, the process has been automated into the
conda-forge GitHub organization. The conda-forge organization contains one repository
for each of the installable packages. Such a repository is known as a feedstock.
A feedstock is made up of a conda recipe (the instructions on what and how to build
the package) and the necessary configurations for automatic building using freely
available continuous integration services. Thanks to the awesome service provided by
Azure, GitHub,
CircleCI, AppVeyor,
Drone, and TravisCI
it is possible to build and upload installable packages to the
conda-forgeanaconda.org
channel for Linux, Windows and OSX respectively.
To manage the continuous integration and simplify feedstock maintenance
conda-smithy has been developed.
Using the conda-forge.yml within this repository, it is possible to re-render all of
this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.
feedstock - the conda recipe (raw material), supporting scripts and CI configuration.
conda-smithy - the tool which helps orchestrate the feedstock.
Its primary use is in the construction of the CI .yml files
and simplify the management of many feedstocks.
conda-forge - the place where the feedstock and smithy live and work to
produce the finished article (built conda distributions)
Updating conda-forge-ci-setup-feedstock
If you would like to improve the conda-forge-ci-setup recipe or build a new
package version, please fork this repository and submit a PR. Upon submission,
your changes will be run on the appropriate platforms to give the reviewer an
opportunity to confirm that the changes result in a successful build. Once
merged, the recipe will be re-built and uploaded automatically to the
conda-forge channel, whereupon the built conda packages will be available for
everybody to install and use from the conda-forge channel.
Note that all branches in the conda-forge/conda-forge-ci-setup-feedstock are
immediately built and any created packages are uploaded, so PRs should be based
on branches in forks and branches in the main repository should only be used to
build distinct package versions.
In order to produce a uniquely identifiable distribution:
If the version of a package is not being increased, please add or increase
the build/number.
If the version of a package is being increased, please remember to return
the build/number
back to 0.
Over in conda-forge/fftw-feedstock#81 it appears that the version right before we added the boa run constraint is getting pulled. IDK why but it is causing errors for cross-compiled builds.
Would be good to update the check here to handle downloading the macOS 10.9 SDK generally if it is missing (as opposed to checking CircleCI specifically). Should make this work more generally on different providers with different offerings.
It appears that uploading packages again (if there version and build number are the same) errors. This marks the CI build as a failure, which is erroneous. Would be better if it simply skipped the upload in these cases and passed.
+ upload_or_check_non_existence /home/conda/recipe_root conda-forge --channel=main -m /home/conda/feedstock_root/.ci_support/linux_python3.5.yaml
Adding in variants from internal_defaults
Adding in variants from /home/conda/feedstock_root/.ci_support/linux_python3.5.yaml
Attempting to finalize metadata for numba
Solving environment: ...working... done
Solving environment: ...working... done
Processing numba
Traceback (most recent call last):
File "/opt/conda/bin/upload_or_check_non_existence", line 129, in <module>
main()
File "/opt/conda/bin/upload_or_check_non_existence", line 119, in main
upload(cli, fname, owner, channel)
File "/opt/conda/bin/upload_or_check_non_existence", line 66, in upload
env=os.environ)
File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', '/tmp/tmpws6dl6tj/binstar.token', 'upload', '/home/conda/feedstock_root/build_artifacts/linux-64/numba-0.36.1-py35hdc42b47_0.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1.
PR ( anaconda/anaconda-client#241 ) deprecated get_binstar and switched to get_server_api. So it has been deprecated for a long time. However we are still using get_binstar in the upload script, which recently caused us problems as noted in issue ( anaconda/anaconda-client#463 ). While get_binstar got added back, we should switch over to get_server_api to avoid issues in the future.
Solution to issue cannot be found in the documentation.
I checked the documentation.
Issue
Hi,
There is an issue with dependency of this package leading to this error:
conda create -n conda-forge-ci-setup-env conda-forge-ci-setup
conda activate conda-forge-ci-setup-env
python -c "from conda_forge_ci_setup import build_utils"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/peter/miniconda3/envs/conda-forge-ci-setup-env/lib/python3.11/site-packages/conda_forge_ci_setup/build_utils.py", line 15, in <module>
from conda_forge_ci_setup.upload_or_check_non_existence import retry_upload_or_check
File "/Users/peter/miniconda3/envs/conda-forge-ci-setup-env/lib/python3.11/site-packages/conda_forge_ci_setup/upload_or_check_non_existence.py", line 12, in <module>
from binstar_client.utils import get_server_api
File "/Users/peter/miniconda3/envs/conda-forge-ci-setup-env/lib/python3.11/site-packages/binstar_client/__init__.py", line 26, in <module>
from .requests_ext import NullAuth
File "/Users/peter/miniconda3/envs/conda-forge-ci-setup-env/lib/python3.11/site-packages/binstar_client/requests_ext.py", line 11, in <module>
from urllib3.filepost import choose_boundary, iter_fields
ImportError: cannot import name 'iter_fields' from 'urllib3.filepost' (/Users/peter/miniconda3/envs/conda-forge-ci-setup-env/lib/python3.11/site-packages/urllib3/filepost.py)
This is more of an FYI as this is not really a direct issue with this package, but you might consider pinning urllib3 <2. I think the most upstream issue I can find is anaconda/anaconda-client#654.
Hope this helps, even if just an early warning.
Installed packages
conda-forge-ci-setup
Environment info
active environment : conda-forge-ci-setup-env
active env location : /Users/peter/miniconda3/envs/conda-forge-ci-setup-env
shell level : 2
user config file : /Users/peter/.condarc
populated config files : /Users/peter/.condarc
conda version : 23.1.0
conda-build version : not installed
python version : 3.10.10.final.0
virtual packages : __archspec=1=arm64
__osx=13.2.1=0
__unix=0=0
base environment : /Users/peter/miniconda3 (writable)
conda av data dir : /Users/peter/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
package cache : /Users/peter/miniconda3/pkgs
/Users/peter/.conda/pkgs
envs directories : /Users/peter/miniconda3/envs
/Users/peter/.conda/envs
platform : osx-arm64
user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.10 Darwin/22.3.0 OSX/13.2.1 solver/libmamba conda-libmamba-solver/22.8.1 libmambapy/1.4.0
UID:GID : 501:20
netrc file : None
offline mode : False
Would be good to take a look and see if there is anything we would need to do to our upload script to upload wheels in addition to conda packages to anaconda.org.
Snippets from an actual run (don't have a link to Azure because it's washed out, sorry):
Could not solve for environment specs
The following packages are incompatible
\u251c\u2500 cuda-cudart-dev is installable and it requires
\u2502 \u2514\u2500 cuda-cudart 12.0.107 h63175ca_6, which requires
\u2502 \u2514\u2500 cuda-cudart_win-64 12.0.107 h63175ca_6, which requires
\u2502 \u2514\u2500 cuda-version >=12.0,<12.1.0a0 , which requires
\u2502 \u2514\u2500 cudatoolkit 12.0|12.0.* , which can be installed;
\u2514\u2500 cudnn >=8.0.*,<9 is not installable because there are no viable options
\u251c\u2500 cudnn 8.0.5.39 would require
\u2502 \u2514\u2500 cudatoolkit 10.1|10.1.* , which conflicts with any installable versions previously reported;
\u251c\u2500 cudnn [8.0.5.39|8.1.0.77|8.2.0.53|8.2.1.32] would require
\u2502 \u2514\u2500 cudatoolkit [10.2.* |10.2|10.2.* ], which conflicts with any installable versions previously reported;
\u251c\u2500 cudnn 8.0.5.39 would require
\u2502 \u2514\u2500 cudatoolkit 11.0|11.0.* , which conflicts with any installable versions previously reported;
\u251c\u2500 cudnn [8.1.0.77|8.2.0.53|...|8.4.1.50] would require
\u2502 \u2514\u2500 cudatoolkit 11.* , which conflicts with any installable versions previously reported;
\u251c\u2500 cudnn [8.3.2.44|8.4.0.27] would require
\u2502 \u2514\u2500 cudatoolkit >=11.2,<12 , which conflicts with any installable versions previously reported;
\u2514\u2500 cudnn 8.8.0.121 would require
\u2514\u2500 cuda-version >=11.0,<12.0a0 , which conflicts with any installable versions previously reported.
Something seems to be off when rendering the output.
Currently the fast finish script doesn't handle fast finishing jobs on Azure. We should update it to handle that case. This will require using some API to see what jobs are running for a PR and seeing where are job ranks in that list (canceling if it is not the newest entry).
$ upload_package ./ ./recipe ./.ci_support/${CONFIG}.yaml
Adding in variants from internal_defaults
Adding in variants from ./.ci_support/osx_c_compilertoolchain_c.yaml
Attempting to finalize metadata for make
Solving environment: ...working... done
Solving environment: ...working... done
[ERROR] File "/Users/travis/miniconda3/conda-bld/osx-64/make-4.2.1-h470a237_1002.tar.bz2" does not exist
Traceback (most recent call last):
File "/Users/travis/miniconda3/bin/upload_package", line 11, in <module>
sys.exit(upload_package())
File "/Users/travis/miniconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/travis/miniconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/travis/miniconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/travis/miniconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/travis/miniconda3/lib/python3.6/site-packages/conda_forge_ci_setup/build_utils.py", line 81, in upload_package
upload_or_check(recipe_root, owner, channel, [config_file])
File "/Users/travis/miniconda3/lib/python3.6/site-packages/conda_forge_ci_setup/upload_or_check_non_existence.py", line 123, in upload_or_check
upload(token_fn, path, owner, channel)
File "/Users/travis/miniconda3/lib/python3.6/site-packages/conda_forge_ci_setup/upload_or_check_non_existence.py", line 65, in upload
env=os.environ)
File "/Users/travis/miniconda3/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', '/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/tmpx90qiwh6/binstar.token', 'upload', '/Users/travis/miniconda3/conda-bld/osx-64/make-4.2.1-h470a237_1002.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1.
It appears that when conda build runs downstream tests it leaves the downstream packages in the build artifacts directory. This causes our code to attempt to validate and upload them, which leads to jobs that fail when they should otherwise pass. We can compute the names of the outputs from the meta.yaml (but not their versions) and filter things we find by globing.
Cc @isuruf this is why your binutils pr appears to be failing.
We have been using conda.config.subdir for some time to detect the platform and thus determine where things should be uploaded. This no longer exists in conda 4.4+. How would we like to address this?
However, AFAICT from https://cmake.org/cmake/help/latest/manual/cmake-env-variables.7.html, that variable isn't one that cmake will automatically pick up from the environment, so one currently needs to include something like -DCMAKE_CROSSCOMPILING_EMULATOR:STRING="${CMAKE_CROSSCOMPILING_EMULATOR}" in the build script to pass it along manually.
I'm not sure where this would be applied, but is it reasonable to add CMAKE_CROSSCOMPILING_EMULATOR to the CMAKE_ARGS variable currently configured by the C compiler activate scripts? Or maybe there's a better way.
There are some things in branch2.0 that we will inevitably want in master. This serves as a reminder to do this at some point. Can even be done incrementally as needed.
Solution to issue cannot be found in the documentation.
I checked the documentation.
Issue
The duplicate PR (conda-forge/gammapy-feedstock#30) has been closed very soon it was opened, yet none of the CI runs got canceled, and instead, they were let run and finish. Would be great to auto-cancel them if possible.
The inclusion of CUDA in the build matrix to make sure things are tested has caused some weird solver features where packages at the same build number and version are swapped back and forth.
This is annoying and there are a couple of solutions.
Prioritize the cuda None builds via build number
Test each CUDA version in a different python version (suggestion by @hmaarrfk)
Traceback shows this is triggered by a urllib2 call in ff_ci_pr_build.py.
The Appveyor build is working for Python 3.7 but not for Python 3.6
If there's a better place to open this, please let me know, thanks!
Full error message below.
Checking to see if this PR build is outdated.
9Traceback (most recent call last):
10 File "C:\projects\climlab-feedstock\ff_ci_pr_build.py", line 201, in <module>
11 sys.exit(main())
12 File "C:\projects\climlab-feedstock\ff_ci_pr_build.py", line 192, in main
13 exit_code = int(appveyor_check_latest_pr_build(repo, pr, bld) is False)
14 File "C:\projects\climlab-feedstock\ff_ci_pr_build.py", line 117, in appveyor_check_latest_pr_build
15 data = request_json(url.format(repo=repo, total_builds=total_builds), headers=headers)
16 File "C:\projects\climlab-feedstock\ff_ci_pr_build.py", line 46, in request_json
17 with contextlib.closing(urlopen(request)) as response:
18 File "C:\Python27\lib\urllib2.py", line 154, in urlopen
19 return opener.open(url, data, timeout)
20 File "C:\Python27\lib\urllib2.py", line 429, in open
21 response = self._open(req, data)
22 File "C:\Python27\lib\urllib2.py", line 447, in _open
23 '_open', req)
24 File "C:\Python27\lib\urllib2.py", line 407, in _call_chain
25 result = func(*args)
26 File "C:\Python27\lib\urllib2.py", line 1241, in https_open
27 context=self._context)
28 File "C:\Python27\lib\urllib2.py", line 1198, in do_open
29 raise URLError(err)
30urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)>
31Command exited with code 1
When conda cannot solve an environment, it will try to build a conflict report detailing the possible causes. This takes a while and is often not very helpful. Users are mostly recommended to try build the same package with mambabuild to get a better report.
Suggestion:
Turns out there's a config key (check bottom of page) to disable these reports (or hints as they call it): unsatisfiable_hints. If set to false, conda will not compute them. I guess this is honored by conda-build too.
Instead of printing the conflicts report, we can print instructions on how to enable mambabuild for debugging.
Do you think this would be a sane default behaviour for CF? It can save a lot of CI resources down the line.
However if we run into an issue where the uploading itself is broken and we would like to debug it ( like issue conda-forge/kubo-feedstock#5 ), currently we lack the ability to configure a verbose upload. Would be good to have some way to configure this
Hi @conda-forge/conda-forge-ci-setup! This is the friendly automated conda-forge-webservice!
It appears that one or more of your feedstock's outputs did not copy from the
staging channel (cf-staging) to the production channel (conda-forge). :(
This failure can happen for a lot of reasons, including an outdated feedstock
token. Below we have put some information about the failure to help you debug it.
Rerendering the feedstock will usually fix these problems.
If you have any issues or questions, you can find us on Element in the
community channel or you can bump us right here.
output validation (is this output allowed for your feedstock?):
As part of the python312 migration uploaded the python3.12 RC to a seperate channel conda-forge/label/python_rc.
The migration yaml then adds a new python version value to a recipes feedstock and zips it with channel_sources. Thereby we can add a new channel_source that just gets used by the python312 build.
setup_conda_rc is not yet able to handle this and will always just add channels from the first channel_sources value to the condarc. This results in the conda-forge/label/python_rc channel being missing in the condarc and mamba failing to solve.
As @isurufpointed out just adding all channels from all values to the condarc will not work due to strict repo priority
OSError: [WinError 1455] The paging file is too small for this operation to complete
in conda-forge/numpy-feedstock#237 & many other pull requests (for numpy), I decided to have a quick look at what's behind that.
Stumbling over this discussion (regarding GH Actions), someone noted:
I’ve investigated this issue and the problem is that Windows decides to create a single page file on the D:\ drive, which has only 14 GB, and by default it allocates only 1/8th of the volume size, or about 1.75 GB. This is a very small page file by modern standards and many applications will fail.
How about adding an option (or step) to the ci-setup that allows setting the page file size? It seems to be possible for GHAs - someone published an action for it (it's MIT licensed so we could even reuse some of that code).
Conda-forge currently builds against the OSX SDK 10.9. This causes issues for example with Qt + Pyside2, which requires both Qt and python to be built with OSX SDK 10.14 or 10.15.
I understand that Conda-forge builds against SDK 10.9 to be backwards compatible to that version of OSX. However, this is also possible with building against the newest SDK (10.15), and the setting the deployment target to a lower OSX version, like 10.9.
Here again the details:
Where did you read this? That's not how how SDKs work. You build with 10.12 and it will run on 10.14, not the other way around.
In the example of the link I posted, they build with base SDK 10.6 and run on 10.4 and newer. If the code doesn't use any of the newer API's (not existing in 10.6) this works out of the box. If the code wants to use new 10.5 or 10.6 API's on supported operating systems, the code must first check the OS version before using the API's. Which I guess Qt does, when they say on their official docs that compiling against the 10.14 SDK and then running on 10.12 works fine...
So Apple recommends always using the newest SDK, and then setting the Deployment target to the lowest OSX version you want to support...
Currently CPU_COUNT is set for the previously existing CIs (i.e Travis, Circle, and AppVeyor). However to the best of my knowledge CPU_COUNT is likely not set correctly for Azure. Would be good to update the values set in these scripts to configure Azure correctly.
Since this constitutes a real library now (with a setup.py and everything) we should most likely put the code in this repo into its own repo. This will allow us to write tests and have them run against CI without tying ourselves into knots.
As noted in this comment and again in this comment, it seems that sometimes we lack permissions to do a little bit of cleanup on AppVeyor (possibly due to resource contention), which causes the build to fail after uploading packages. Since this appears to be occurring after upload it isn't actually messing the uploads, but it does give one pause. Restarting seems to make it go away, but it does show up too often for comfort and it does make users nervous. Maybe sleeping for a bit before cleaning up would avoid this issue. There may be better fixes though.
We run into this issue with the freecad-feedstock. Git is used to download the sources to retrive some version information. The build fails on windows with:
error: remote-curl: usage: git remote-curl <remote> [<url>]
Warning: failed to download source. If building, will try again after downloading recipe dependencies.
Any ideas how this can be related to the conda-forge-ci-setup version?
This issue was resolved by pinning conda-forge-ci-setup to 3.0.3 in the past. But now there seems to be a dependency conflict and this workaround doesn't work any more.
the traceback:
Traceback (most recent call last):
File "C:\Miniconda\Scripts\conda-mambabuild-script.py", line 9, in <module>
sys.exit(main())
File "C:\Miniconda\lib\site-packages\boa\cli\mambabuild.py", line 164, in main
call_conda_build(action, config)
File "C:\Miniconda\lib\site-packages\boa\cli\mambabuild.py", line 137, in call_conda_build
result = api.build(
File "C:\Miniconda\lib\site-packages\conda_build\api.py", line 186, in build
return build_tree(
File "C:\Miniconda\lib\site-packages\conda_build\build.py", line 3068, in build_tree
packages_from_this = build(metadata, stats,
File "C:\Miniconda\lib\site-packages\conda_build\build.py", line 2118, in build
try_download(m, no_download_source=False, raise_error=True)
File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 663, in try_download
raise RuntimeError("Failed to download or patch source. Please see build log for info.")