Giter Club home page Giter Club logo

rattler-build's Introduction

License Build Status Project Chat Pixi Badge

rattler-build: a fast conda-package builder

The rattler-build tooling and library creates cross-platform relocatable binaries / packages from a simple recipe format. The recipe format is heavily inspired by conda-build and boa, and the output of a regular rattler-build run is a package that can be installed using mamba, rattler or conda.

rattler-build does not have any dependencies on conda-build or Python and works as a standalone binary.

Installation

You can grab a prerelease version of rattler-build from the Github Releases.

It is (of course) also available from conda-forge:

pixi global install rattler-build
# or with micromamba
micromamba install rattler-build -c conda-forge

Distro Packages

Packaging status

Homebrew

You can install rattler-build via Homebrew:

brew install rattler-build

Arch Linux

rattler-build is available on Arch Linux in the extra repository:

pacman -S rattler-build

Alpine Linux

rattler-build is available for Alpine Edge. It can be installed via apk after enabling the testing repository.

apk add rattler-build

Dependencies

Currently rattler-build needs some dependencies on the host system which are executed as subprocess. We plan to reduce the number of external dependencies over time by writing what we need in Rust to make rattler-build fully self-contained.

  • tar to unpack tarballs downloaded from the internet in a variety of formats. .gz, .bz2 and .xz are widely used and one might have to install the compression packages as well (e.g. gzip, bzip2, ...)
  • patch to patch source code after downloading
  • install_name_tool is necessary on macOS to rewrite the rpath of shared libraries and executables to make it relative
  • patchelf is required on Linux to rewrite the rpath and runpath of shared libraries and executables
  • git to checkout Git repositories (not implemented yet, but will require git in the future)
  • msvc on Windows because we cannot ship the MSVC compiler on conda-forge (needs to be installed on the host machine)

On Windows, to obtain these dependencies from conda-forge, one can install m2-patch, m2-bzip2, m2-gzip, m2-tar.

Documentation

We have extensive documentation for rattler-build. You can find the book here.

GitHub Action

There is a GitHub Action for rattler-build. It can be used to install rattler-build in CI/CD workflows and run a build command. Please check out the GitHub Action documentation for more information.

Usage

rattler-build comes with two commands: build and test.

The build command takes a --recipe recipe.yaml as input and produces a package as output. The test subcommand can be used to test existing packages (tests are shipped with the package).

There is also a terminal user interface (TUI) that can help with building multiple packages and easily viewing logs.

The recipe format

Note You can find all examples below in the examples folder and run them with rattler-build.

A simple example recipe for the xtensor header-only C++ library:

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

context:
  name: xtensor
  version: 0.24.6

package:
  name: ${{ name|lower }}
  version: ${{ version }}

source:
  url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz
  sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655

build:
  number: 0
  script:
    - if: win
      then: |
        cmake -G "Ninja" -D BUILD_TESTS=OFF -D CMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% %SRC_DIR%
        ninja install
      else: |
        cmake ${CMAKE_ARGS} -G "Ninja" -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX $SRC_DIR -DCMAKE_INSTALL_LIBDIR=lib
        ninja install

requirements:
  build:
    - ${{ compiler('cxx') }}
    - cmake
    - ninja
  host:
    - xtl >=0.7,<0.8
  run:
    - xtl >=0.7,<0.8
  run_constraints:
    - xsimd >=8.0.3,<10

tests:
  - script:
    - if: unix or emscripten
      then:
        - test -d ${PREFIX}/include/xtensor
        - test -f ${PREFIX}/include/xtensor/xarray.hpp
        - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake
        - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake
    - if: win
      then:
        - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1)
        - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfig.cmake (exit 1)
        - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfigVersion.cmake (exit 1)

about:
  homepage: https://github.com/xtensor-stack/xtensor
  license: BSD-3-Clause
  license_file: LICENSE
  summary: The C++ tensor algebra library
  description: Multi dimensional arrays with broadcasting and lazy computing
  documentation: https://xtensor.readthedocs.io
  repository: https://github.com/xtensor-stack/xtensor

extra:
  recipe-maintainers:
    - some-maintainer
A recipe for the `rich` Python package (using `noarch`)
context:
  version: "13.4.2"

package:
  name: "rich"
  version: ${{ version }}

source:
  - url: https://pypi.io/packages/source/r/rich/rich-${{ version }}.tar.gz
    sha256: d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898

build:
  # Thanks to `noarch: python` this package works on all platforms
  noarch: python
  script:
    - python -m pip install . -vv --no-deps --no-build-isolation

requirements:
  host:
    - pip
    - poetry-core >=1.0.0
    - python 3.10
  run:
    # sync with normalized deps from poetry-generated setup.py
    - markdown-it-py >=2.2.0
    - pygments >=2.13.0,<3.0.0
    - python 3.10
    - typing_extensions >=4.0.0,<5.0.0

tests:
  - python:
      imports:
        - rich
      pip_check: true

about:
  homepage: https://github.com/Textualize/rich
  license: MIT
  license_file: LICENSE
  summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
  description: |
    Rich is a Python library for rich text and beautiful formatting in the terminal.

    The Rich API makes it easy to add color and style to terminal output. Rich
    can also render pretty tables, progress bars, markdown, syntax highlighted
    source code, tracebacks, and more — out of the box.
  documentation: https://rich.readthedocs.io
  repository: https://github.com/Textualize/rich
A recipe for the `curl` library
context:
  version: "8.0.1"

package:
  name: curl
  version: ${{ version }}

source:
  url: http://curl.haxx.se/download/curl-${{ version }}.tar.bz2
  sha256: 9b6b1e96b748d04b968786b6bdf407aa5c75ab53a3d37c1c8c81cdb736555ccf

build:
  number: 0

requirements:
  build:
    - ${{ compiler('c') }}
    - if: win
      then:
        - cmake
        - ninja
    - if: unix
      then:
        - make
        - perl
        - pkg-config
        - libtool
  host:
    - if: linux
      then:
        - openssl

about:
  homepage: http://curl.haxx.se/
  license: MIT/X derivate (http://curl.haxx.se/docs/copyright.html)
  license_file: COPYING
  summary: tool and library for transferring data with URL syntax
  description: |
    Curl is an open source command line tool and library for transferring data
    with URL syntax. It is used in command lines or scripts to transfer data.
  documentation: https://curl.haxx.se/docs/
  repository: https://github.com/curl/curl

For this recipe, two additional script files (build.sh and build.bat) are needed.

build.sh

#!/bin/bash

# Get an updated config.sub and config.guess
cp $BUILD_PREFIX/share/libtool/build-aux/config.* .

if [[ $target_platform =~ linux.* ]]; then
    USESSL="--with-openssl=${PREFIX}"
else
    USESSL="--with-secure-transport"
fi;

./configure \
    --prefix=${PREFIX} \
    --host=${HOST} \
    ${USESSL} \
    --with-ca-bundle=${PREFIX}/ssl/cacert.pem \
    --disable-static --enable-shared

make -j${CPU_COUNT} ${VERBOSE_AT}
make install

# Includes man pages and other miscellaneous.
rm -rf "${PREFIX}/share"

build.bat

mkdir build

cmake -GNinja ^
      -DCMAKE_BUILD_TYPE=Release ^
      -DBUILD_SHARED_LIBS=ON ^
      -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% ^
      -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% ^
      -DCURL_USE_SCHANNEL=ON ^
      -DCURL_USE_LIBSSH2=OFF ^
      -DUSE_ZLIB=ON ^
      -DENABLE_UNICODE=ON ^
      %SRC_DIR%

IF %ERRORLEVEL% NEQ 0 exit 1

ninja install --verbose

rattler-build's People

Contributors

0xbe7a avatar arpitpandey29 avatar baszalmstra avatar beeankha avatar beenje avatar benjaminlowry avatar benmoss avatar daenny avatar dependabot[bot] avatar derthorsten avatar grayjack avatar hadim avatar jarrettsjohnson avatar kassoulait avatar matthiasbeyer avatar mdornacher avatar nichmor avatar orhun avatar pavelzw avatar raybellwaves avatar ruben-arts avatar striezel avatar swarnimarun avatar wolfv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rattler-build's Issues

Hash added to package name with rattler-build but not mambabuild

I have created a small project for testing rattler-build, using the approach we use on many of our projects.

In this project, I run conda mambabuild and rattler-build build and get different results for the names of the resulting files. Mambabuild returns a file without a hash, e.g. rattler_build_test-0.0.2-0.tar.bz2, while rattler-build returns a file with the has included e.g. rattler_build_test-0.0.1-pyh4616a5c_0.tar.bz2

Additional small noarch: python fixes

I think we don't ignore .egg and .exe files yet. From conda-build:

    # Ignore egg-info and pyc files.
    if f.endswith(('.egg-info', '.pyc', '.pyo')):
        os.unlink(path)

    elif f.endswith('.exe') and (isfile(os.path.join(prefix, f[:-4] + '-script.py')) or
                               basename(f[:-4]) in d['python-scripts']):
        os.unlink(path)  # this is an entry point with a matching xx-script.py

Metapackages shouldn't require a `build.sh` script

I've got a metapackage (a package which just installs other packages - i.e. has no source). When I try to build this with ratter-build it wants to run a build.sh script (which doesn't make sense for a metapackage)

❯ rattler-build build --output-dir='./.build/dist' --recipe='./boa/recipe.yaml'
 INFO rattler_build: Starting the build process
 INFO rattler_build: No target platform specified, using current platform
Found variants:
╭─────────────────┬──────────╮
│ Variant         ┆ Version  │
╞═════════════════╪══════════╡
│ target_platform ┆ linux-64 │
╰─────────────────┴──────────╯

 INFO rattler_build::build: Reading recipe file: "/home/user/code/github/energy-quants/quantdev/boa/build.sh"
ERROR rattler_build: Error: No such file or directory (os error 2)


❯ rattler-build build --output-dir='./.build/dist' --recipe='./boa'
 INFO rattler_build: Starting the build process
 INFO rattler_build: No target platform specified, using current platform
Found variants:
╭─────────────────┬──────────╮
│ Variant         ┆ Version  │
╞═════════════════╪══════════╡
│ target_platform ┆ linux-64 │
╰─────────────────┴──────────╯

 INFO rattler_build::build: Reading recipe file: "/home/user/code/github/energy-quants/quantdev/boa/build.sh"
ERROR rattler_build: Error: No such file or directory (os error 2)


❯ ls -l ./boa
total 40
-rw-r--r-- 1 user user 37942 Jul 25 10:18 recipe.yaml


❯ rattler-build --version
rattler-build 0.2.0

Support `.conda` packages

AFAICS there is no option to build .conda packages?

In boa it's --pkg-format 2 but I can't see an equivalent in ratter-build?

Do we need `ignore_version` for hash?

From conda-build:

# A hash will be added if all of these are true for any dependency:
#
# 1. package is an explicit dependency in build, host, or run deps
# 2. package has a matching entry in conda_build_config.yaml which is a pin to a specific
#    version, not a lower bound
# 3. that package is not ignored by ignore_version
#
# The hash is computed based on the pinning value, NOT the build
#    dependency build string. This means hashes won't change as often,
#    but it also means that if run_exports is overly permissive,
#    software may break more often.
#
# A hash will also ALWAYS be added when a compiler package is a build
#    or host dependency. Reasoning for that is that the compiler
#    package represents compiler flags and other things that can and do
#    dramatically change compatibility. It is much more risky to drop
#    this info (by dropping the hash) than it is for other software.

# used variables - anything with a value in conda_build_config.yaml that applies to this
#    recipe.  Includes compiler if compiler jinja2 function is used.

This is apparently used in some variant configuration files on conda-forge:

https://github.com/search?q=org%3Aconda-forge%20ignore_version&type=code

SHA256 not matching for rich example

Steps to reproduce:

git clone https://github.com/prefix-dev/rattler-build.git
cd rattler-build/examples/rich
rattler-build build --recipe recipe.yaml

I can't work out why running the rich example (https://github.com/prefix-dev/rattler-build/blob/main/examples/rich/recipe.yaml) I get

ERROR rattler_build::source: SHA256 values of downloaded file not matching!
Downloaded = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, should be d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898
ERROR rattler_build::source: Checksum validation failed!
ERROR rattler_build: Error: Download could not be validated with checksum!

The sha should match (https://pypi.org/project/rich/#copy-hash-modal-2010ddfe-5e36-4f05-9a00-6ce880fc7e80) e.g. d65... Not sure where e3b... is coming from.

I made a slight modification to the file to match the latest feedstock (https://github.com/conda-forge/rich-feedstock/blob/main/recipe/meta.yaml)

context:
  name: "rich"
  min_python: ">=3.7.0"
  version: "13.4.2"

package:
  name: "{{ name|lower }}"
  version: "{{ version }}"

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
  sha256: d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898

build:
  # Thanks to `noarch: python` this package works on all platforms
  noarch: python
  script: python -m pip install . -vv --no-deps --no-build-isolation

requirements:
  host:
    - pip
    - poetry-core >=1.0.0
    - python {{ min_python }}
  run:
    # sync with normalized deps from poetry-generated setup.py
    - markdown-it-py >=2.2.0
    - pygments >=2.13.0,<3.0.0
    - python {{ min_python }}
    - typing_extensions >=4.0.0,<5.0.0

test:
  imports:
    - rich
  commands:
    - pip check
  requires:
    - pip

about:
  home: https://github.com/Textualize/rich
  license: MIT
  license_family: MIT
  license_file: LICENSE
  summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
  description: |
    Rich is a Python library for rich text and beautiful formatting in the terminal.

    The Rich API makes it easy to add color and style to terminal output. Rich
    can also render pretty tables, progress bars, markdown, syntax highlighted
    source code, tracebacks, and more — out of the box.
  doc_url: https://rich.readthedocs.io
  dev_url: https://github.com/Textualize/rich

and I run the command rattler-build build --recipe recipe.yaml. The full output is

❯ rattler-build build --recipe recipe.yaml
INFO rattler_build: Starting the build process
Found variants:
╭─────────────────┬─────────╮
│ Variant ┆ Version │
╞═════════════════╪═════════╡
│ target_platform ┆ noarch │
╰─────────────────┴─────────╯

INFO rattler_build::source: Fetching source from URL: https://pypi.io/packages/source//rich/rich-13.4.2.tar.gz
ERROR rattler_build::source: SHA256 values of downloaded file not matching!
Downloaded = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, should be d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898
ERROR rattler_build::source: Checksum validation failed!
ERROR rattler_build: Error: Download could not be validated with checksum!

Is `{{ name[0] }}` not supported?

I tried modifying the rich recipe from the example to use:

@@ -1,12 +1,13 @@
 context:
+  name: "rich"
   version: "13.3.3"

 package:
-  name: rich
+  name: "{{ name|lower }}"
   version: "{{ version }}"

 source:
-  - url: https://pypi.io/packages/source/r/rich/rich-{{ version }}.tar.gz
+  - url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/rich-{{ version }}.tar.gz
     sha256: dc84400a9d842b3a9c5ff74addd8eb798d155f36c1c91303888e0a66850d2a15

Running rattler-build fails:

rattler-build build --recipe-file /build/recipe.yaml.new
 INFO rattler_build: Starting the build process
Found variants:
╭─────────────────┬─────────╮
│ Variant         ┆ Version │
╞═════════════════╪═════════╡
│ target_platform ┆ noarch  │
╰─────────────────┴─────────╯

 INFO rattler_build::source: Fetching source from URL: https://pypi.io/packages/source//rich/rich-13.3.3.tar.gz
ERROR rattler_build::source: SHA256 values of downloaded file not matching!
Downloaded = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, should be dc84400a9d842b3a9c5ff74addd8eb798d155f36c1c91303888e0a66850d2a15
ERROR rattler_build::source: Checksum validation failed!
Error: Download could not be validated with checksum!

We can see it's trying to download https://pypi.io/packages/source//rich/rich-13.3.3.tar.gz instead of https://pypi.io/packages/source/r/rich/rich-13.3.3.tar.gz

{{ name[0] }} is converted to an empty string instead of r. Is that expected?

grayskull is generating recipes with this URL, so it would be nice if it was working.
Similar URL is shown in the docs: https://prefix-dev.github.io/rattler-build/recipe_file.html#examples

No python files included in test package

I created a small test package for experimenting with rattler-build using the directory structure we use on our internal projects. With this package conda mambabuild is able to create a working package, while rattler-build doesn't include the necessary python files. Below are the lists of files in the tar.bz files.

conda mambabuild

tar -tvf output/noarch/rattler_build_test-0.0.2-0.tar.bz2 
-rw-rw-r-- 0/0               2 2023-07-26 09:33 info/hash_input.json
-rw-rw-r-- 0/0              65 2023-07-26 09:33 info/test/run_test.py
-rw-rw-r-- 0/0              75 2023-07-26 09:33 info/link.json
-rw-rw-r-- 0/0             248 2023-07-26 09:33 info/index.json
-rw-rw-r-- 0/0             459 2023-07-26 09:33 info/recipe/conda_build_config.yaml
-rw-rw-r-- 0/0             515 2023-07-26 09:33 info/recipe/meta.yaml.template
-rw-rw-r-- 0/0             557 2023-07-26 09:33 info/files
-rw-rw-r-- 0/0             629 2023-07-26 09:27 info/recipe/recipe.yaml
-rw-rw-r-- 0/0            1690 2023-07-26 09:33 info/recipe/meta.yaml
-rw-rw-r-- 0/0            2322 2023-07-26 09:33 info/paths.json
-rw-rw-r-- 0/0            4677 2023-07-26 09:33 info/about.json
-rw-rw-r-- 0/0               0 2023-07-26 09:33 info/git
-rw-rw-r-- 0/0              19 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/top_level.txt
-rw-rw-r-- 0/0               5 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/INSTALLER
-rw-rw-r-- 0/0              92 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/WHEEL
-rw-rw-r-- 0/0            1030 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/RECORD
-rw-rw-r-- 0/0            1328 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/METADATA
-rw-rw-r-- 0/0            1523 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/LICENSE
-rw-rw-r-- 0/0               0 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/REQUESTED
-rw-rw-r-- 0/0             122 2023-07-26 09:33 site-packages/rattler_build_test-0.0.2.dist-info/direct_url.json
-rw-rw-r-- 0/0              43 2023-07-26 09:33 site-packages/rattler_build_test/run.py
-rw-rw-r-- 0/0               0 2023-07-26 09:33 site-packages/rattler_build_test/__init__.py

rattler build

tar -tvf output/noarch/rattler_build_test-0.0.1-pyh4616a5c_0.tar.bz2 
-rw-rw-r-- 0/0             206 2023-07-26 09:34 info/about.json
-rw-rw-r-- 0/0             252 2023-07-26 09:34 info/index.json
-rw-rw-r-- 0/0            1523 2023-07-26 09:34 info/licenses/LICENSE
-rw-rw-r-- 0/0              75 2023-07-26 09:34 info/link.json
-rw-rw-r-- 0/0              39 2023-07-26 09:34 info/paths.json
-rw-rw-r-- 0/0              27 2023-07-26 09:34 info/test/run_test.py

Support local archive as source

It would be nice to support local sdist as source.

Using:

source:
  - url: file:///build/rich/rich-{{ version }}.tar.gz
    sha256: fb9d6c0a0f643c99eed3875b5377a184132ba9be4d61516a55273d3554d75a39

raises:

INFO rattler_build::source: Fetching source from URL: file:///build/rich/rich-13.5.2.tar.gz
ERROR rattler_build: Error: Failed to download source from url: builder error for url (file:///build/rich/rich-13.5.2.tar.gz): URL scheme is not allowed

`fetch_repo_data` error

I'm seeing an error from fetch_repo_data:

ERROR fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_repodata_gateway::fetch: error=repodata not found
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for *.conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for *.conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json
 WARN fetch_repo_data{cache_path=/home/user/.cache/rattler/cache/repodata}: rattler_networking::authentication_storage::storage: Error retrieving credentials for *.conda.anaconda.org: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files, using fallback storage at ~/.rattler/rattler_auth_store.json

Is this an actual error? The build seems to complete fine despite this.

Reproducibility: control "tools" as well

We currently use some system tools:

  • install_name_tool on macOS for changing @rpath after building
  • patchelf on Linux to change RPATH after building

These tools could potentially influence how reproducible the final binaries are. Ideally we get rid of these third-party tools and implement the functionality natively in Rust but that is no small feat.
In the meantime it might be good to install them into their respective virtual environments and use them with a very specific version / lockfile and store that alongside the rendered recipe.

Rework relinking for macOS

Contrary to the claims, the dylib are not actually relinked to @rpath/... because the check for the prefix is currently wrong (should compare with encoded_prefix and not prefix).

So the full path to the linked library stays in the binary. This still works because at install time they are rewritten to the installation prefix.

However, I think we should do the following:

  • remove any rpath that starts with encoded_prefix
  • add one new rpath that is host_prefix/lib
  • make all dylibs that have an absolute path in the host prefix relative to the new rpath

Error: Failed to resolve dependencies No such file or directory (os error 2)

Hi!

I'm excited to try this new project :-)

I ran the following to test one recipe from the examples directory but it failed:

$ docker run --rm -it  mambaorg/micromamba bash
(base) mambauser@a74f99838c28:/tmp$ cd
(base) mambauser@a74f99838c28:~$ micromamba install -n base -c conda-forge curl rattler-build
(base) mambauser@a74f99838c28:~$ curl -LO https://raw.githubusercontent.com/prefix-dev/rattler-build/main/examples/rich/recipe.yaml
(base) mambauser@a74f99838c28:~$ rattler-build build --recipe-file recipe.yaml
 INFO rattler_build: Starting the build process
Found variants:
╭─────────────────┬─────────╮
│ Variant         ┆ Version │
╞═════════════════╪═════════╡
│ target_platform ┆ noarch  │
╰─────────────────┴─────────╯

 INFO rattler_build::source: Fetching source from URL: https://pypi.io/packages/source/r/rich/rich-13.3.3.tar.gz
 INFO rattler_build::source: Validated SHA256 values of the downloaded file!
 INFO rattler_build::source: Extracted to "/home/mambauser/rich_1686157757599/work"

Resolving for environment specs:
 - pip
 - poetry-core >=1.0.0
 - python ==3.10


⠁ /home/mambauser/outp [00:00:00] [────────────────────────────────────────]       0B @ -
⠁ /home/mambauser/outp [00:00:00] [────────────────────────────────────────]       0B @ -
❌ /home/mambauser/outp [00:00:00] Error
✔ /home/mambauser/outp [00:00:00] Using cache
✔ conda-forge/linux-64 [00:00:06] Done
✔ conda-forge/noarch   [00:00:03] Done                                                                                                                                       Error: Failed to resolve dependencies No such file or directory (os error 2)

Caused by:
    No such file or directory (os error 2)

Am I doing something wrong?

Implement list - dict selection and make it work

The following syntax doesn't seem to work yet:

python:
  - sel(linux): "3.8"
    sel(win): "3.10"

While

python:
  - sel(linux): "3.8"
  - sel(win): "3.10"

works as expected. We could decide if we want to support the former syntax or not.

No such file error on first build

When running the rich example I get Error: Failed to resolve dependencies No such file or directory (os error 2).

I'm not sure where this is coming from.

❯ rattler-build build --recipe recipe.yaml
 INFO rattler_build: Starting the build process
Found variants:
╭─────────────────┬─────────╮
│ Variant         ┆ Version │
╞═════════════════╪═════════╡
│ target_platform ┆ noarch  │
╰─────────────────┴─────────╯

 INFO rattler_build::source: Fetching source from URL: https://pypi.io/packages/source/r/rich/rich-13.4.2.tar.gz
 INFO rattler_build::source: Validated SHA256 values of the downloaded file!
 INFO rattler_build::source: Found valid source cache file.
 INFO rattler_build::source: Extracted to "/var/folders/rf/26llfhwd68x7cftb1z3h000w0000gp/T/rattler-build_rich_1688657182562/work"

Resolving for environment specs:
 - pip
 - poetry-core >=1.0.0
 - python ==3.10


⠁ /Users/ray/Documents [00:00:00] [────────────────────────────────────────]       0B @ -
⠁ /Users/ray/Documents [00:00:00] [────────────────────────────────────────]       0B @ -
❌ /Users/ray/Documents [00:00:00] Error
✔ /Users/ray/Documents [00:00:00] Using cache
⠚ conda-forge/osx-64   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠚ conda-forge/noarch   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠂ conda-forge/osx-64   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠂ conda-forge/noarch   [00:00:00] [━━╾─────────────────────────────────────] 672.00 KiB @ 997.03 KiB/s
✔ conda-forge/osx-64   [00:00:01] Using cache
✔ conda-forge/noarch   [00:00:01] Done                                                                                                                                                                                 

ERROR rattler_build: Error: Failed to resolve dependencies No such file or directory (os error 2)

I tried a different output dir to see if that helps

❯ rattler-build build --recipe recipe.yaml --output-dir /tmp/conda
 INFO rattler_build: Starting the build process
Found variants:
╭─────────────────┬─────────╮
│ Variant         ┆ Version │
╞═════════════════╪═════════╡
│ target_platform ┆ noarch  │
╰─────────────────┴─────────╯

 INFO rattler_build::source: Fetching source from URL: https://pypi.io/packages/source/r/rich/rich-13.4.2.tar.gz
 INFO rattler_build::source: Validated SHA256 values of the downloaded file!
 INFO rattler_build::source: Extracted to "/var/folders/rf/26llfhwd68x7cftb1z3h000w0000gp/T/rattler-build_rich_1688657969579/work"

Resolving for environment specs:
 - pip
 - poetry-core >=1.0.0
 - python ==3.10


⠁ /private/tmp/conda/o [00:00:00] [────────────────────────────────────────]       0B @ -
⠁ /private/tmp/conda/n [00:00:00] [────────────────────────────────────────]       0B @ -
❌ /private/tmp/conda/o [00:00:00] Error
✔ /private/tmp/conda/n [00:00:00] Using cache
⠒ conda-forge/osx-64   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠒ conda-forge/noarch   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠲ conda-forge/osx-64   [00:00:00] [────────────────────────────────────────]       0B @ 0B/s
⠲ conda-forge/noarch   [00:00:00] Deserializing..
✔ conda-forge/osx-64   [00:00:01] Using cache
✔ conda-forge/noarch   [00:00:00] Using cache                                                                                                                                                                          

ERROR rattler_build: Error: Failed to resolve dependencies No such file or directory (os error 2)

Respect channel alias from condarc

If I'm using a tool like artifactory, I will likely have a condarc like:

channel_alias: artifactory.paine.nyc/artifactory/api/conda/
channels:
- mychannel
- conda-forge-mirror

I can pass channels into rattler build with the -c flag, but it would be useful if I can also specify the default channel alias. If I pass in e.g. -c artifactory.paine.nyc/artifactory/api/conda/conda-forge-mirror, it doesn't seem to resolve things properly.

Support injecting the `environ` variable in recipes

...which boa works for.

❯ rattler-build build --recipe-file ./build/conda/recipe.yaml
 INFO rattler_build: Starting the build process
 INFO rattler_build: No target platform specified, using current platform
Found variants:
╭─────────────────┬──────────╮
│ Variant         ┆ Version  │
╞═════════════════╪══════════╡
│ target_platform ┆ linux-64 │
╰─────────────────┴──────────╯

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: UndefinedError, name: "<string>", line: 1 }', src/render/recipe.rs:123:74
stack backtrace:
   0:     0x55f878b5213a - std::backtrace_rs::backtrace::libunwind::trace::ha9053a9a07ca49cb
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55f878b5213a - std::backtrace_rs::backtrace::trace_unsynchronized::h9c2852a457ad564e
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55f878b5213a - std::sys_common::backtrace::_print_fmt::h457936fbfaa0070f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55f878b5213a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h5779d7bf7f70cb0c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55f878b7d8de - core::fmt::write::h5a4baaff1bcd3eb5
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/fmt/mod.rs:1232:17
   5:     0x55f878b4d795 - std::io::Write::write_fmt::h4bc1f301cb9e9cce
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/io/mod.rs:1684:15
   6:     0x55f878b51f05 - std::sys_common::backtrace::_print::h5fcdc36060f177e8
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x55f878b51f05 - std::sys_common::backtrace::print::h54ca9458b876c8bf
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x55f878b536bf - std::panicking::default_hook::{{closure}}::hbe471161c7664ed6
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:271:22
   9:     0x55f878b533fb - std::panicking::default_hook::ha3500da57aa4ac4f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:290:9
  10:     0x55f878b53c68 - std::panicking::rust_panic_with_hook::h50c09d000dc561d2
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:692:13
  11:     0x55f878b53b69 - std::panicking::begin_panic_handler::{{closure}}::h9e2b2176e00e0d9c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:583:13
  12:     0x55f878b525a6 - std::sys_common::backtrace::__rust_end_short_backtrace::h5739b8e512c09d02
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:150:18
  13:     0x55f878b53872 - rust_begin_unwind
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
  14:     0x55f87818ccf3 - core::panicking::panic_fmt::hf33a1475b4dc5c3e
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
  15:     0x55f87818d1a3 - core::result::unwrap_failed::hdff5465d74574b44
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/result.rs:1750:5
  16:     0x55f8782f4863 - rattler_build::render::recipe::render_recipe::h1091a334c82e2ed9
  17:     0x55f87836e6a3 - rattler_build::run_build_from_args::{{closure}}::hcfc39f2258319605
  18:     0x55f8783712b6 - rattler_build::main::{{closure}}::h51a545dcc6054cd2
  19:     0x55f878351386 - tokio::runtime::context::BlockingRegionGuard::block_on::h98038fbba076f2c9
  20:     0x55f8782124fb - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::ha18e7cb6fb9e086f
  21:     0x55f8782beed0 - tokio::runtime::runtime::Runtime::block_on::hf87ebd0dace1c556
  22:     0x55f87831f6df - rattler_build::main::h9434a6185ad6cc85
  23:     0x55f878218453 - std::sys_common::backtrace::__rust_begin_short_backtrace::h61a6a709095f2fb1
  24:     0x55f8781d3e3d - std::rt::lang_start::{{closure}}::h744d5772ab20b6e7
  25:     0x55f878b4663c - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hd6efcd3bec896f2c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ops/function.rs:287:13
  26:     0x55f878b4663c - std::panicking::try::do_call::hce04e543bb1f4cbb
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  27:     0x55f878b4663c - std::panicking::try::h3342dd4e1f680968
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  28:     0x55f878b4663c - std::panic::catch_unwind::h148ce1e59ac0cee7
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  29:     0x55f878b4663c - std::rt::lang_start_internal::{{closure}}::h25f9dda2057a67fe
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:48
  30:     0x55f878b4663c - std::panicking::try::do_call::h7caaaeaf9401650b
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  31:     0x55f878b4663c - std::panicking::try::he7d15285746cbbc2
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  32:     0x55f878b4663c - std::panic::catch_unwind::h89fb4f50c0301fe0
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  33:     0x55f878b4663c - std::rt::lang_start_internal::h078acd489417d3c1
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:20
  34:     0x55f87831f7c5 - main
  35:     0x7f0c17b4bd90 - <unknown>
  36:     0x7f0c17b4be40 - __libc_start_main
  37:     0x55f87818d335 - <unknown>
context:
  name: dwgm-forecast
  version: "{{ environ['SETUPTOOLS_SCM_PRETEND_VERSION'] }}"

package:
  name: "{{ name }}"
  version: "{{ version }}"

build:
  number: 0

outputs:
  - package:
      name: "{{ name }}"
      version: "{{ version }}"

    build:
      noarch: python
      script_env:
        PIP_ROOT_USER_ACTION: "ignore"
      script: echo "test" $PREFIX/test.txt

    requirements:
      host:
        - python >=3.10
        - hatchling
        - hatch-vcs
        - pip >=22.3

      run:
        - python >=3.10
        - click
        - numpy
        - pandas
        - scikit-learn
        - scikit-learn-intelex
        - sqlalchemy
        - statsmodels

about:
  summary: 'Bug report'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.