paketo-community / cargo Goto Github PK

When using the Cargo CNB, you should have BOM entries for crates installed by Cargo.

Binaries built with Cargo are placed under the /workspace/bin directory. This is not presently on the $PATH so when you want to run a binary that was installed, you need to use the full path to it, i.e. /workspace/bin/webserver. It would be a little more convenient if one could just execute webserver.

This impacts users with Procfile's primarily. Since the Procfile would need to reference the binary to execute.

Currently works:

web: /workspace/bin/webserver

Desired goal:

web: webserver

Cargo stores downloaded into $CARGO_HOME, which defaults to $HOME/.cargo. That is not going to be useful in CNBS, so we should be setting $CARGO_HOME to a layer that is set with build + cache flags. That way downloaded resources are restored for subsequent runs of the buildpack & if something needs to be recompiled, Cargo will not need to hit the Internet to download resources again.

The entire folder does not need to be cached though. This actually retains some duplicate information. According to this doc link we can trim down the size a bit by storing only these folder:

• bin/
• registry/index/
• registry/cache/
• git/db/

If one is developing and building code locally, there will be a target/ directory for the project and this directory is often multiple GB.

If one were to pack build from the project directory, which is typically what you'd do, then the pack cli will copy the entire current directory, including the large target folder, and into the build container. The application files, despite being not needed in the final image will be included in the final image (based on the way the buildpack spec works).

We should be more intelligent about this.

1. If a target directory is present, we should delete it. This will prevent it from making its way into the launch layer.
2. We should have a configuration option to delete other files that get pushed also, since app source code isn't needed in the launch layer either.

My coworker and I ran into an unexpected issue today, a dependency in our stack had a different version of the crate then the main library but this was fine when built locally because the lock file resolved to a version that both libraries could use.

However, we found out that unlike cargo build, cargo install doesn't use the lock file by default, causing two different versions of the library to installed during build.

We fixed it by adding --locked to the build args, however it seems like that's the behaviour you would always want anyway? Since the detect requires you to have a lock file, seems surprising that we wouldn't use it.

Right now we have a default set of arguments used for cargo install.

https://github.com/paketo-community/cargo-install/blob/main/cargo/cli_runner.go#L48-L53

Ex: cargo install --color=never --path=. --root=<destination layer>

This works fine for simple projects but doesn't work when you have multiple workspaces.

This issue is to support customizing these arguments so one can easily include the arguments required to build a project, such as

• --path=./todo (to build a single project in a workspace).
• --bins to build all binaries
• --bin=foo to build the foo binary
• -v for verbose output
• --frozen, --locked or --offline for customizing if Cargo will access the network

Users should not be able to override --color=never and --root=<destination layer>. These are required for the buildpack to work properly.

We should add additional default arguments to cargo install of --target x86_64-unknown-linux-musl when running on the Tiny stack. This target is required for the build to work on Tiny (well, the build works without this but the generated binary doesn't run).

You can work around this now by setting -e BP_CARGO_INSTALL_ARGS='--target x86_64-unknown-linux-musl'. This will just automatically set that argument for the Tiny stack.

Using the following commands:

kn func create -l rust func-demo
cd func-demo
kn func build --registry=quay.io/CONTAINER_ORG --push

The build errors with the following output:

...
Rust Cargo Build Pack 0.9.0
  https://github.com/paketo-community/cargo
  Build Configuration:
    $BP_CARGO_INSTALL_ARGS        --locked                              additional arguments to pass to Cargo install
    $BP_CARGO_INSTALL_TOOLS                                             additional tools to be add with Cargo install
    $BP_CARGO_INSTALL_TOOLS_ARGS                                        additional arguments to pass to Cargo install for tools
    $BP_CARGO_TINI_DISABLED       false                                 Skip installing tini
    $BP_CARGO_WORKSPACE_MEMBERS                                         the subset of workspace members for Cargo to install
    $BP_DISABLE_SBOM              false                                 Skip running SBOM scan
    $BP_EXCLUDE_FILES                                                   colon separated list of glob patterns, matched source files are removed
    $BP_INCLUDE_FILES             static/*:templates/*:public/*:html/*  colon separated list of glob patterns, matched source files are included
  Tini 0.19.0: Contributing to layer
    Downloading from https://github.com/krallin/tini/releases/download/v0.19.0/tini-amd64
    Verifying checksum
    Copying to /layers/paketo-community_cargo/tini
    Creating cached target directory /workspace/target
  Rust Application: Contributing to layer
    File modification times not restored
    File modification times not restored
    File modification times not restored
    cargo install --locked --color=never --root=/layers/paketo-community_cargo/Cargo --path=.
        Installing function v0.1.0 (/workspace)
      error: failed to get `actix-http` as a dependency of package `function v0.1.0 (/workspace)`
      
      Caused by:
        failed to load source for dependency `actix-http`
      
      Caused by:
        Unable to update registry `crates-io`
      
      Caused by:
        usage of sparse registries requires `-Z sparse-registry`

This line looks suspicious but I have not investigated further

Unable to update registry `crates-io`

The stack id for the tiny stack on Jammy has changed.

1. Review if this is still necessary. If the library for gnu support is now present in Jammy, we may be able to just remove this additional logic and use gnu by default.
2. If it is still required, then check the new stack id as well as the old ID in comparisons.

Also, add support for the static stack. This stack is built for running static binaries. It should always trigger a static binary build.

This also requires investigating tini. The tini binary we are shipping looks to dynamic link against glibc. If possible we should switch to using a statically compiled tini, then it'll just work on all the stacks. If that's not possible, the buildpack should disable tini in the static stack since it won't have libc.

Right now the buildpack will look at the Cargo.lock file, take its hash, and store that for future runs. The next time it runs, the buildpack will pull this hash & compare it to the hash of the Cargo.lock file that was pushed. The assumption is that if the two are different, then we need to rebuild. That seems likely to be true, but I'm not sure if that is always true and I think if it will hold true it will depend on the cargo install arguments used. Given that we'll be allowing users to set their own custom arguments in #19, this might become a problem.

At the same time, it's not clear that this is really doing anything advantageous. If we are saving state between builds, which we try to do (the target/ directory is placed into its own layer marked with build + cache), cargo install should be able to run and make its own determination if something needs to be rebuilt. This should still be fast, but should also be accurate.

After #19 is merged, we need to investigate if for small/medium/large projects taking out this hash-based caching impacts the build speed & if so, how much. If the impact is minimal, as expected, then we can take out this caching.

Right now the cargo install buildpack uses cargo install with the --path= to install a particular member in a workspace. This works for simple cases, but if you have a workspace with many members, you might want/need to build a subset of the projects or all of them. This isn't currently supported.

Running cargo install --path=foo --path=bar isn't supported, nor is there a --all flag like with cargo build. It only accepts one --path. See rust-lang/cargo#4101.

To resolve this, we need could switch to using cargo build instead of cargo install. This command has controls around building a full workspace or specific packages. The main issue with cargo build is that cargo install nicely places the binaries that are built into the destination layer for us. If we use cargo build there is an --out-dir flag that looks promising, but it's only in nightly Rust at the moment (so it may be a future fix, but doesn't work now). That means the buildpack would be responsible for copying the output to the destination layer manually. The problem is that there is not an easy way to get the targets.

What you'd end up needing to do is run cargo metadata --format-version=1 and parsing the output. This will give you the list of packages in the workspace and then for each package you can find the available targets. The list of installable stuff would be any binary or example target. You could then pull out the binary that's built for each target based on the target name & copy that into the layer. It's not clear what the buildpack would do if a partial build was done, some logic would need to be applied to copy only what was built.

Another option would be to use cargo metadata --format-version=1 to extract the list of packages in the workspace. That list also has the file paths for each package. You could then iterate through each package and do a cargo install --path=<parsed package> to install each. This would let cargo install perform the actual installation for you. The tricky part with this one would be how does the buildpack allow you to pass through flags to cargo install? Is there one set that's applied to all the packages or is there a way to apply per package flags? You'd also have the issue of how to perform a partial build, where you only want a subset of packages. There would need to be a buildpack environment variable that allows the user to indicate this.

The naming convention in Paketo is to name the buildpack with its function (e.g. bundle-install, npm-install, yarn-install). We should change the name of this repository to align with that naming convention.

Right now, cargo install is used to build and install the application binaries. Running cargo install and cargo build are not quite the same though. There are some subtle differences like build will use the lock file by default, build also handles workspaces differently, and build allows you to build tests.

The main trick will be running cargo build and then finding the right binaries to "install". You get that for free with cargo install. You can find the binaries by reading the cargo metadata, which we're already doing for some things.

Open questions:

1. Do we need to support both cargo build and cargo install? or should we replace cargo install with cargo build?
2. If we support both, how do you signal to the buildpack which one should be used? Possibly an env variable like BP_CARGO_COMMAND.
3. What should we default to? build or install?

Cross compilation is currently not possible.
There would be two ways to do it AFAIK.

1. Install a linker for the target platform and use that one.
2. Use cargo cross for compilation instead of cargo, maybe with a feature flag.

Both could be solved by #21.

Preference would be to have cross installed when using a target platform other than the host one installed.

Tried to go with the first approach, but that failed as there is no linker for arm64 installed.

pack build image_name \
-b docker.io/paketocommunity/rust \
-B paketobuildpacks/builder:tiny \
-e BP_RUST_TOOLCHAIN=nightly \
-e BP_RUST_PROFILE=default \
-e BP_RUST_TARGET=aarch64-unknown-linux-musl \
-e BP_CARGO_INSTALL_ARGS='--target aarch64-unknown-linux-musl' \
-e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-musl-gcc

pack build image_name -b docker.io/paketocommunity/rust -B paketobuildpacks/builder:tiny -e BP_RUST_TOOLCHAIN=nightly  -e BP_RUST_PROFILE=default -e BP_RUST_TARGET=aarch64-unknown-linux-musl -e BP_CARGO_INSTALL_ARGS='--target aarch64-unknown-linux-musl' -e CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-musl-gcc
tiny: Pulling from paketobuildpacks/builder
Digest: sha256:dedb220941776b89554291527e3ca7a350fdaf13290d9e7a86c1ea66c3406d90
Status: Image is up to date for paketobuildpacks/builder:tiny
tiny-cnb: Pulling from paketobuildpacks/run
Digest: sha256:4d6d3e6cab614672bd98e6e5a664597f69136fcb575f1ed95279242d444a65a0
Status: Image is up to date for paketobuildpacks/run:tiny-cnb
latest: Pulling from paketocommunity/rust
Digest: sha256:7aa9da459048919daed07ed4617c8ccfb5ad2fd5da2426915439cbe00a78352a
Status: Image is up to date for paketocommunity/rust:latest
===> ANALYZING
===> DETECTING
3 of 4 buildpacks participating
paketo-community/rustup    1.2.0
paketo-community/rust-dist 1.5.0
paketo-community/cargo     0.3.0
===> RESTORING
Restoring metadata for "paketo-community/cargo:tini" from app image
===> BUILDING

Paketo Rustup Buildpack 1.2.0
  https://github.com/paketo-community/rustup
  Build Configuration:
    $BP_RUSTUP_ENABLED       true                        use rustup to install Rust
    $BP_RUSTUP_INIT_LIBC     gnu                         libc implementation: gnu or musl
    $BP_RUSTUP_INIT_VERSION  1                           the rustup version
    $BP_RUST_PROFILE         default                     the Rust profile to install
    $BP_RUST_TARGET          aarch64-unknown-linux-musl  an additional Rust target to install
    $BP_RUST_TOOLCHAIN       nightly                     the Rust toolchain or version number to install
  Rustup (GNU libc) 1.24.3: Contributing to layer
    Downloading from https://static.rust-lang.org/rustup/archive/1.24.3/x86_64-unknown-linux-gnu/rustup-init
    Verifying checksum
    Copying to /layers/paketo-community_rustup/rustup-gnu/bin
  Cargo: Contributing to layer
  Rustup: Contributing to layer
    Installing Rustup
  Rust: Contributing to layer
    Installing Rust
      info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
      info: latest update on 2022-02-26, rust version 1.61.0-nightly (d3ad51b48 2022-02-25)
      info: downloading component 'cargo'
      info: downloading component 'clippy'
      info: downloading component 'rust-docs'
      info: downloading component 'rust-std'
      info: downloading component 'rustc'
      info: downloading component 'rustfmt'
      info: installing component 'cargo'
      info: installing component 'clippy'
      info: installing component 'rust-docs'
      info: installing component 'rust-std'
      info: installing component 'rustc'
      info: installing component 'rustfmt'
      
        nightly-x86_64-unknown-linux-gnu installed - rustc 1.61.0-nightly (d3ad51b48 2022-02-25)
      
      info: default toolchain set to 'nightly-x86_64-unknown-linux-gnu'
      info: checking for self-updates
      info: downloading component 'rust-std' for 'aarch64-unknown-linux-musl'
      info: installing component 'rust-std' for 'aarch64-unknown-linux-musl'
Warning: the following SBoM files will be ignored for buildpack api version < 0.7 [/layers/paketo-community_rustup/rustup-gnu.sbom.syft.json]

Rust Distribution Buildpack 1.5.0
  https://github.com/paketo-community/rust-dist

Rust Cargo Build Pack 0.3.0
  https://github.com/paketo-community/cargo
  Build Configuration:
    $BP_CARGO_EXCLUDE_FOLDERS    static, templates, public, html      folders that should not be deleted and should persist to the generated image
    $BP_CARGO_INSTALL_ARGS       --target aarch64-unknown-linux-musl  additional arguments to pass to Cargo install
    $BP_CARGO_TINI_DISABLED      false                                Skip installing tini
    $BP_CARGO_WORKSPACE_MEMBERS                                       the subset of workspace members for Cargo to install
  Tini 0.19.0: Reusing cached layer
    Creating cached target directory /workspace/target
  Rust Application: Contributing to layer
    File modification times not restored
    File modification times not restored
    File modification times not restored
    cargo install --target aarch64-unknown-linux-musl --color=never --root=/layers/paketo-community_cargo/Cargo --path=.
        Installing image_name v0.1.0 (/workspace)
          Updating crates.io index
         Compiling image_name v0.1.0 (/workspace)
      error: linker `aarch64-linux-musl-gcc` not found
        |
        = note: No such file or directory (os error 2)
      
      error: failed to compile `image_name v0.1.0 (/workspace)`, intermediate artifacts can be found at `/workspace/target`
      
      Caused by:
        could not compile `image_name` due to previous error
unable to invoke layer creator
unable to contribute application layer
unable to install single
unable to build
exit status 101
ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle: failed with status code: 51

You can run cargo install to install any number of crates with useful tools, such as the diesel_cli or other commands that you might want to run at build time or perhaps before your app starts at launch. This is separate from the cargo install that is run to install packages provided by source code.

There should be two environment variables that can configure this behavior: BP_CARGO_INSTALL and BPL_CARGO_INSTALL. The former will install tools that should be available during the build (the cargo cnb & all cnbs after) and the latter will install tools that are available at launch (they will control the layer into which the dependency is installed).

It is kind of a pain in demo/sample apps to need to make and include a Procfile. It's a small pain, but for new users, it's one more thing to need to do.

For simple apps, we can probably auto-detect the command to run. Logic like this might work:

1. Look for a Procfile, if one is present, do nothing
2. Look for a Cargo Workspace, if present, look for a default member/default run setting, if present use
3. Look to see what's installed, typically is going to have the same name as the package name in Cargo.toml. If a binary is built, then set that to run.

Again, it won't work for all cases, but it might help with small apps, new users, demos, etc... Veterans can continue to use Procfile and this logic will play nicely with that.

When building an app with Rust 1.77.0+, the build fails:

        panic: runtime error: index out of range [2] with length 1
        
        goroutine 1 [running]:
        github.com/paketo-community/cargo/runner.CargoRunner.WorkspaceMembers({{0xc00002003b, 0x25}, {0x0, 0x0}, {0xc0000b6cc0, 0x8}, {0x7fe480, 0xa100c0}, {{{0x0, 0x0}, ...}, ...}, ...}, ...)
        	/home/runner/work/cargo/cargo/runner/runners.go:228 +0x585
        github.com/paketo-community/cargo/cargo.Cargo.Contribute.func1()
        	/home/runner/work/cargo/cargo/cargo/cargo.go:233 +0x63e
        github.com/paketo-buildpacks/libpak.(*LayerContributor).Contribute(0xc00018ebb0, {{0x0, 0x0, 0x0}, 0x0, {0x76df70, 0x5}, {0xc0005485a0, 0x24}, 0xc000506cc0, ...}, ...)
        	/home/runner/go/pkg/mod/github.com/paketo-buildpacks/[email protected]/layer.go:95 +0x802
        github.com/paketo-community/cargo/cargo.Cargo.Contribute({0x0, {0xc0000f6198, 0x11}, {{{{...}, {...}}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...}, ...}, ...)
        	/home/runner/work/cargo/cargo/cargo/cargo.go:206 +0xe8
        github.com/buildpacks/libcnb.Build({0x7fe4a0, 0xc0000a0780}, {0x0, 0x0, 0xc0000aa880?})
        	/home/runner/go/pkg/mod/github.com/buildpacks/[email protected]/build.go:280 +0x1d47
        github.com/buildpacks/libcnb.Main({0x7fe4c0, 0xa100c0}, {0x7fe4a0, 0xc0000a0780}, {0x0, 0x0, 0x0})
        	/home/runner/go/pkg/mod/github.com/buildpacks/[email protected]/main.go:47 +0x2bd
        main.main()
        	/home/runner/work/cargo/cargo/cmd/main/main.go:28 +0xd1
        ERROR: failed to build: exit status 2

This happens because the underlying metadata format has changed. Prior to Rust 1.77.0, the format looked like:

  "workspace_members": [
    "function 0.1.0 (path+file:///Users/dmikusa/Downloads/fn-rs)"
  ],

and with Rust 1.77.0+, it looks like:

  "workspace_members": [
    "path+file:///Users/dmikusa/Downloads/fn-rs#[email protected]"
  ],

The buildpack doesn't yet understand this format and fails. A Code change will be required to support the new format.

In the meantime, you can add a rust-toolchain.toml file to your project and pin the rust version to 1.76.0 or older. The rustup buildpack will honor that file. You can do the same by using the rust-dist buildpack also. In that case, just use v1.25.0 or v1.24.0 which have Rust 1.76.0 and 1.75.0 respectively.

Ex:

pack build app-rs -b docker.io/paketocommunity/rust-dist:1.24.0 -b paketo-community/rust -e BP_RUSTUP_ENABLED=0

upx is a tool for compressing binary executables. It has some excellent compression ratios and very low runtime overhead. It would be helpful with Rust binaries. We should add support.

The cargo buildpack should follow the pattern established here and optionally run upx to compress output binaries.

The upx buildpack also needs to be added into the Rust composite buildpack/builder. paketo-community/rust#293

We should add integration tests to verify the behavior of cargo install.

Detect fails with an error if no Cargo.lock or Cargo.toml and exit with a code 21.

...
===> DETECTING
[detector] ======== Output: paketo-community/[email protected] ========
[detector] Missing [Cargo.toml: true, Cargo.lock: true], both required
[detector] err:  paketo-community/[email protected] (1)
[detector] ERROR: No buildpack groups passed detection.
[detector] ERROR: failed to detect: buildpack(s) failed with err
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 21

If I remove packeto-comunity/rust from the order in the builder.toml, it exit with a code 20.

===> DETECTING
[detector] ERROR: No buildpack groups passed detection.
[detector] ERROR: Please check that you are running against the correct path.
[detector] ERROR: failed to detect: no buildpacks participating
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 20

I am not sure why, is it required to wrap the error with packit.Fail to mute the error during detection ?

https://github.com/paketo-buildpacks/go-mod-vendor/blob/main/detect.go#L27

Use packit.Run instead of having two separate executables.

To reproduce:

$ cargo new --lib testing
$ cargo new other 
$ cat <<EOF >> Cargo.toml
[workspace]
members = [
  "other"
]
EOF


$ echo "testing = { path = "../testing"}" >> other/Cargo.toml

$ pack build testing

The result is

[builder]     cargo install --locked --color=never --root=/layers/paketo-community_cargo/Cargo --path=/workspace/other
[builder]         Installing other v0.1.0 (/workspace/other)
[builder]          Compiling testing v0.1.0 (/workspace/testing)
[builder]          Compiling other v0.1.0 (/workspace/other)
[builder]           Finished release [optimized] target(s) in 0.51s
[builder]         Installing /layers/paketo-community_cargo/Cargo/bin/other
[builder]          Installed package `other v0.1.0 (/workspace/other)` (executable `other`)
[builder]     cargo install --locked --color=never --root=/layers/paketo-community_cargo/Cargo --path=/workspace/testing
[builder]       error: no packages found with binaries or examples
[builder] unable to invoke layer creator
[builder] unable to contribute application layer
[builder] unable to install member
[builder] unable to build
[builder] exit status 101
[builder] ERROR: failed to build: exit status 1

It seems for some reason the logic detects that the lib crate is in the same directory and makes it part of the workspace?

Add metadata to document the buildpack configuration settings like from #19.

Example: https://github.com/paketo-buildpacks/apache-tomcat/blob/main/buildpack.toml#L29-L32

Right now when we run cargo install, we are passing in a set of environment variables. That set is composed of the existing set of environment variables plus the ones the buildpack adds. This means that a user could set environment variables that might conflict with what the buildpack is doing. For example, setting CARGO_TARGET_DIR or CARGO_ROOT.

We should have a list of reserved env variables & disallow setting those (strip them from the env used & log a warning message). This should help to make the buildpack a little safer.

We should change out default branch to be "main". This will align us with the rest of the Paketo project.

Try buildpack for my cargo project from this builder and just failed.

lan@lan:~/repo/git/czechwd$ pack build test -B paketocommunity/cargo:latest
latest: Pulling from paketocommunity/cargo
8c0fd3710fbf: Pull complete 
Digest: sha256:e5e6e38be403d305d996cd45230b7e1f7bd17ea7d121f7a62e6f4cec788e4dff
Status: Downloaded newer image for paketocommunity/cargo:latest
ERROR: failed to build: invalid builder paketocommunity/cargo:latest: builder index.docker.io/paketocommunity/cargo:latest missing label io.buildpacks.builder.metadata -- try recreating builder

pack version : 0.23.0+git-0db2c77.build-3056

Maybe i missed something, remind me if i missed some doc, thanks!