Giter Club home page Giter Club logo

conda-env-update's Introduction

Paketo Buildpack for Conda Env Update Cloud Native

The Conda Env Update Buildpack runs commands to update the conda environment. It installs the conda environment into a layer which makes it available for subsequent buildpacks and in the final running container.

The buildpack is published for consumption at gcr.io/paketo-buildpacks/conda-env-update and paketobuildpacks/conda-env-update.

Behavior

This buildpack participates when there is an environment.yml or package-list.txt file in the app directory.

The buildpack will do the following:

  • At build time:
    • Requires that conda has already been installed in the build container
    • Updates the conda environment and stores it in a layer
      • If a package-list.txt is in the app dir, a new environment is created from it
      • Otherwise, the environment.yml file is used to update the environment
    • Reuses the cached conda environment layer from a previous build if and only if a package-list.txt is in the app dir and it has not changed since the previous build
  • At run time:
    • Does nothing

Integration

The Conda Env Update Buildpack provides conda-environment as a dependency. Downstream buildpacks can require the conda-environment dependency by generating a Build Plan TOML file that looks like the following:

[[requires]]
# The name of the Conda Env Update dependency is "conda-environment". This value is considered
# part of the public API for the buildpack and will not change without a plan
# for deprecation.
name = "conda-environment"

# The Conda Env Update buildpack supports some non-required metadata options.
[requires.metadata]

# Setting the build flag to true will ensure that the conda environment
# layer is available for subsequent buildpacks during their build phase.
# If you are writing a buildpack that needs the conda environment
# during its build process, this flag should be set to true.
build = true

# Setting the launch flag to true will ensure that the conda environment is
# available to the running application. If you are writing an application
# that needs to use the conda environment at runtime, this flag should be set to true.
launch = true

SBOM

This buildpack can generate a Software Bill of Materials (SBOM) for the dependencies of an application.

However, this feature only works if an application vendors its dependencies in the vendor directory. This is due to a limitation in the upstream SBOM generation library (Syft).

Applications that declare their dependencies via a package-list.txt file but do not vendor them will result in an empty SBOM. Check out the Paketo SBOM documentation for more information about how to access the SBOM.

Usage

To package this buildpack for consumption:

$ ./scripts/package.sh --version <version-number>

This will create a buildpackage.cnb file under the build directory which you can use to build your app as follows: pack build <app-name> -p <path-to-app> -b build/buildpackage.cnb -b <other-buildpacks..>

To run the unit and integration tests for this buildpack:

$ ./scripts/unit.sh && ./scripts/integration.sh

conda-env-update's People

Watchers

 avatar  avatar  avatar  avatar  avatar

conda-env-update's Issues

Better documentation around exporting packages into environment.yml

Could we get some better documentation in this README (and eventual Paketo docs) around how users can setup an environment.yml file when the conda packages have been installed? In my case, I setup a conda virtual env, installed my packages via, conda install, and exported into a environment.yml file via conda env export > environment.yml. With this, the build process were failing with a confusing error:

 Failed to run CONDA_PKGS_DIRS=/layers/paketo-community_conda-env-update/conda-env-cache conda env update --prefix /layers/paketo-community_conda-env-update/conda-env --file /workspace/environment.yml
        Collecting package metadata (repodata.json): ...working... done
        Solving environment: ...working... failed

        ResolvePackageNotFound:
          - openssl==1.1.1k=h9ed2024_0
          - markupsafe==2.0.1=py39h9ed2024_0
          - zlib==1.2.11=h1de35cc_3
          - importlib-metadata==3.10.0=py39hecd8cb5_0
          - ca-certificates==2021.4.13=hecd8cb5_1
          - python==3.9.5=h88f2d9e_3
          - certifi==2020.12.5=py39hecd8cb5_0
          - setuptools==52.0.0=py39hecd8cb5_0
          - tk==8.6.10=hb0a8c7a_0
          - libffi==3.3=hb1e8313_2
          - xz==5.2.5=h1de35cc_0
          - sqlite==3.35.4=hce871da_0
          - readline==8.1=h9ed2024_0
          - ncurses==6.2=h0a44026_1
          - libcxx==10.0.0=1
          - pip==21.1.1=py39hecd8cb5_0

ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle: failed with status code: 145

Turns out that this is all an architecture thing since I was running conda install on my Macbook. Running conda env export --from-history > environment.yml resolved this, so would be great if we could document this.

Confirm to RFC RFC0043: Reproducible builds

To confirm to RFC0043 this buildpack should ensure that builds are reproducible. Specifically it should not include a built_at metadata field. In the tests that leverage this field to assert layer reuse, we should instead compare layer SHA values across rebuilds.

See also the tracking issue: paketo-buildpacks/rfcs#165.

"Failed to cache layer 'paketo-community/conda-env-update:conda-env-cache' because it has no contents"

When I build a vendored app using conda I see the following log output at the end of the build:

Warning: Failed to cache layer 'paketo-community/conda-env-update:conda-env-cache' because it has no contents

This is a little confusing because on a re-build of the same app I can see that the build process did re-use the cached conda env update layer, which is expected.

Full logs:

pack build kv-vendor-conda-app --buildpack paketo-community/python
base: Pulling from paketobuildpacks/builder
d0c52a206d00: Pulling fs layer
1879a9e18f37: Pulling fs layer
1320020c14ba: Pulling fs layer
f4e4561ab79a: Waiting
192d971c2090: Waiting
a786b7416026: Waiting
584fa9e4bb0f: Pulling fs layer
357fefdf9bc9: Pulling fs layer
a697cc8a1d5c: Waiting
641fd92af6e1: Waiting
a1f3f619215b: Waiting
009acd529e4b: Waiting
f038bb970d1c: Waiting
09474fe777cd: Waiting
bcbee467462d: Waiting
d1a08c221a9d: Waiting
98b7526ef57a: Waiting
e2c2a5497ab8: Waiting
e8ddca67c9c1: Waiting
2a8d533e0f64: Waiting
56d7bcca752e: Waiting
96b66c250c1d: Waiting
fa74baf6cfb4: Waiting
a088cd2b7e92: Waiting
c207725752d4: Waiting
6171f4b495c3: Waiting
8b78e691a530: Waiting
862834ce0af4: Waiting
1496aef483f6: Waiting
77c9938e5b75: Waiting
60d7b1898b3a: Waiting
1421d4b9c318: Pulling fs layer
78c1b79015a6: Pulling fs layer
4c998684cb61: Pulling fs layer
282602fc640a: Pulling fs layer
28628c1e526c: Pull complete
b1e7533bc019: Pull complete
df3a0a0a56d8: Pull complete
88d16ca07b74: Pull complete
d1132d93a2b9: Pull complete
0dbd1fa6ff8c: Pull complete
4c14aa3f4c28: Pull complete
3d413b1b9b8a: Pull complete
28b5dceccc55: Pull complete
b567de06cb59: Pull complete
a71d88676efc: Pull complete
33d5ad1d909e: Pull complete
4e64ce695e87: Pull complete
f144b959f858: Pull complete
fc222dead982: Pull complete
6b8dd1778060: Pull complete
4ca22103a370: Pull complete
d9312c9a3c52: Pull complete
b9f9948fdb8f: Pull complete
77dfed658907: Pull complete
db863a500cf3: Pull complete
4a8f9381f1c4: Pull complete
69844fb96b46: Pull complete
9911db742a6c: Pull complete
8151954a9698: Pull complete
9406565c4195: Pull complete
858dfe120bf2: Pull complete
be5310d8b8c2: Pull complete
130f6ca3220b: Pull complete
fa494bae78c2: Pull complete
3618926392ec: Pull complete
92cd0822736b: Pull complete
c9c50a1f0605: Pull complete
a0816326ac7c: Pull complete
116ef6a12a78: Pull complete
b9bf72658c9f: Pull complete
b60ea4865b7e: Pull complete
72ad9888618d: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a
Status: Downloaded newer image for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
d0c52a206d00: Already exists
1879a9e18f37: Already exists
1320020c14ba: Already exists
da3765496a46: Pull complete
f06fe2450b17: Pull complete
50e06ee72084: Pull complete
Digest: sha256:b3b13c57527ce4c98c0e41fd96f3178bf398d70ffb7d1168990480197f681960
Status: Downloaded newer image for paketobuildpacks/run:base-cnb
docker.io/paketocommunity/python@sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4: Pulling from paketocommunity/python
Digest: sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4
Status: Image is up to date for paketocommunity/python@sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4
===> DETECTING
paketo-community/miniconda        0.2.0
paketo-community/conda-env-update 0.2.0
paketo-community/python-start     0.6.0
paketo-buildpacks/procfile        4.1.0
===> ANALYZING
Previous image with name "kv-vendor-conda-app" not found
===> RESTORING
===> BUILDING
Paketo Miniconda Buildpack 0.2.0
  Executing build process
    Installing Miniconda 4.9.2
      Completed in 6.568s

Paketo Conda Env Update Buildpack 0.2.0
  Executing build process
    Running conda create --file /workspace/package-list.txt --prefix /layers/paketo-community_conda-env-update/conda-env --yes --quiet --channel /workspace/vendor --override-channels --offline
      Completed in 4.025s

Paketo Python Start Buildpack 0.6.0
  Assigning launch process
    web: python

Paketo Procfile Buildpack 4.1.0
  https://github.com/paketo-buildpacks/procfile
  Process types:
    web: python -m flask run --host=0.0.0.0
===> EXPORTING
Adding layer 'paketo-community/conda-env-update:conda-env'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving kv-vendor-conda-app...
*** Images (5572b715c17a):
      kv-vendor-conda-app
Adding cache layer 'paketo-community/miniconda:conda'
Warning: Failed to cache layer 'paketo-community/conda-env-update:conda-env-cache' because it has no contents
Successfully built image kv-vendor-conda-app
(base) kashyap@Kashyaps-MacBook-Pro vendored % pack build kv-vendor-conda-app --buildpack paketo-community/python
base: Pulling from paketobuildpacks/builder
Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:b3b13c57527ce4c98c0e41fd96f3178bf398d70ffb7d1168990480197f681960
Status: Image is up to date for paketobuildpacks/run:base-cnb
docker.io/paketocommunity/python@sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4: Pulling from paketocommunity/python
Digest: sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4
Status: Image is up to date for paketocommunity/python@sha256:494b194a0a6d44d8e2116298038e5bb2f0a0892f4c80a372de97d211a0706dc4
===> DETECTING
paketo-community/miniconda        0.2.0
paketo-community/conda-env-update 0.2.0
paketo-community/python-start     0.6.0
paketo-buildpacks/procfile        4.1.0
===> ANALYZING
Restoring metadata for "paketo-community/miniconda:conda" from cache
Restoring metadata for "paketo-community/conda-env-update:conda-env" from app image
===> RESTORING
Restoring data for "paketo-community/miniconda:conda" from cache
===> BUILDING
Paketo Miniconda Buildpack 0.2.0
  Reusing cached layer /layers/paketo-community_miniconda/conda

Paketo Conda Env Update Buildpack 0.2.0
  Reusing cached layer /layers/paketo-community_conda-env-update/conda-env

Paketo Python Start Buildpack 0.6.0
  Assigning launch process
    web: python

Paketo Procfile Buildpack 4.1.0
  https://github.com/paketo-buildpacks/procfile
  Process types:
    web: python -m flask run --host=0.0.0.0
===> EXPORTING
Reusing layer 'paketo-community/conda-env-update:conda-env'
Adding 1/1 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving kv-vendor-conda-app...
*** Images (73a340bd6fc4):
      kv-vendor-conda-app
Reusing cache layer 'paketo-community/miniconda:conda'
Warning: Failed to cache layer 'paketo-community/conda-env-update:conda-env-cache' because it has no contents
Successfully built image kv-vendor-conda-app

Upgrade to packit v2

Please upgrade to the latest packit v2 release to enable new features and extended support.

Restructure: Create conda-env-update buildpack

Background

See RFC 0001 that outlines the proposed restructure of python family of buildpacks. In order to conform to the RFC, a new conda-env-update buildpack should be created that is responsible for updating/creating a conda environment based on the environment file in app source code.

Proposal

  • Create a new repository named conda-env-update at github.com/paketo-community/conda-env-update
  • Transfer this issue to conda-env-update.
  • Create a buildpack using packit that runs the logic of "conda env update" or "conda env create" by requiring conda.
  • Buildpack API should be as follows:
    • provides: conda-environment
    • requires: conda during build
  • Write an RFC describing the behavior of this buildpack.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.