canonical / rockcraft Goto Github PK

View Code? Open in Web Editor NEW

28.0 11.0 37.0 1.63 MB

Tool to create OCI Images using the language from Snapcraft and Charmcraft.

License: GNU General Public License v3.0

Makefile 1.02% Python 97.97% Shell 0.96% C 0.02% Go 0.02% JavaScript 0.01%

rockcraft's Introduction

Rockcraft

Purpose

Tool to create OCI Images using the language from Snapcraft and Charmcraft.

Installing

Install Rockcraft from the Snap Store

Documentation

Documentation on the usage of the tool and tutorial can be found on https://documentation.ubuntu.com/rockcraft/en/stable/

Testing

In addition to unit tests in tests/unit, which can be run with make test-units, a number of integrated tests in tests/spread can be run with Spread. See the general notes and take note of these rockcraft-specific instructions:

Initialize/update git submodules to fetch Spread-related helper scripts:
```
$ git submodule init
$ git submodule update
```
Spread needs a rockcraft snap in order to run. Create one with snapcraft via:
```
$ snapcraft --use-lxd
$ cp <generated snap> tests/
```

Run Spread:

$ spread tests/spread
# Or run a specific test
$ spread tests/spread/tutorial/basic
# Use "-v" for verbose, and "-debug" to get a shell if the test fails
$ spread -v -debug tests/spread/tutorial/basic

rockcraft's People

Contributors

Stargazers

Watchers

rockcraft's Issues

Overlay contents might not be preserved between parts

It looks like the overlay dir is not being "preserved" between the overlay-script of different parts. This could be important if, for example, two distinct parts want to add users to the ROCK (while we don't have official user support).
Here's an example in a rock with an ubuntu base:

  p1:
    plugin: nil
    overlay-script: |
      rm ${CRAFT_OVERLAY}/lib32

  p2:
    after: [p1]
    plugin: nil
    overlay-script: |
      mkdir ${CRAFT_OVERLAY}/lib32

So one part removes the base lib32 symlink, and then the other part creates a lib32 dir. I tried to do that in #163 but it failed with "lib32 already exists", but only in ubuntu-18.04. So this needs some investigating.

add docs for Chisel

document what Chisel is under the "Reference" section

Make mypy linter more strict

Looks like the default mypy configuration is too lenient because there are a bunch of un-annotated function parameters. I think we should at least add disallow-untyped-defs for the code in the rockcraft package.

support architecture-specific builds

The goal is to support the syntax for a Rockstar to specify a list of “build” and “target” architectures, which would, in turn, result in one ROCK artefact per combination.

All of these ROCK artefacts combined would, later on, compose the multi-arch ROCK (each artefact is an OCI manifest).

Spec

# takes a map of labels. Each label produces one ROCK artefact
# dict[str, dict[str, obj]] (req)
platforms:
  # labels are free text, used as a default for build-on/build-for when needed
  # labels are included in the final ROCK artefact name, eg. my-rock_1.0_<label>.rock.
  # There must be at least one label within platforms
  amd64:
    # will build-on amd64, and build-for amd64
  foo:
    # same as the previous label "amd64" above, just a different ROCK artefact
    
    # the build environment architecture where Rockcraft CAN build the ROCK artefact
    # list (req if `build-for` is present): compatible with $GOARCH, 
    build-on: [amd64]
    # the target architecture(s) the ROCK will be built for.
    # both `base` and `build-base` must respect the provided architecture
    # list (opt): one of $GOARCH
    build-for: amd64
  arm:
    # `build-for` defaults to the valid architecture defined by the label "arm"
    build-on: [amd64, arm, arm64] 
  i386: 
    # ERROR: `build-on` must be present 
    build-for: i386   
  another-label:
    # build-for i386
    build-on: [amd64]
    build-for: i386:
  amd64:
    # ERROR: the label-defined arch differs from the one defined by `build-for`
    build-on: [amd64]
    build-for: arm64

`rockcraft` crashes at the `update apt cache` step

Overview

Rockcraft fails at the update apt cache step when running on github's runners.

Description

We build and publish our rocks using github actions. The action to build the rock crashes with the error message craft-providers error: Failed to update apt cache.

Github runner version: ubuntu-latest (22.04.1)
Rockcraft version: 0+git.11f000f

Reference

https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009

Log:

Run rockcraft pack --verbose
Starting Rockcraft 0.0.1.dev1
Logging execution to '/home/runner/.local/state/rockcraft/log/rockcraft-20230117-16061[4](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:5).0[5](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:6)1518.log'
Launching instance...
craft-providers error: Failed to update apt cache.
* Command that failed: 'lxc --project rockcraft exec local:rockcraft-lego-1072[6](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:7)[7](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:8)5 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 apt-get update'
* Command exit code: 100
* Command output: b'Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease\nIgn:2 http://security.ubuntu.com/ubuntu jammy-security InRelease\nIgn:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\nIgn:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\nIgn:1 http://archive.ubuntu.com/ubuntu jammy InRelease\nIgn:2 http://security.ubuntu.com/ubuntu jammy-security InRelease\nIgn:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\nIgn:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\nIgn:1 http://archive.ubuntu.com/ubuntu jammy InRelease\nIgn:2 http://security.ubuntu.com/ubuntu jammy-security InRelease\nIgn:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\nIgn:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\nErr:1 http://archive.ubuntu.com/ubuntu jammy InRelease\n  Cannot initiate the connection to archive.ubuntu.com:[8](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:9)0 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::1[9](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:10)). - connect ([10](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:11)1: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out\nErr:2 http://security.ubuntu.com/ubuntu jammy-security InRelease\n  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out\nErr:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable)\nErr:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable)\nReading package lists...\n'
* Command standard error output: b'E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable)\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable)\nE: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease  Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out\nE: Some index files failed to download. They have been ignored, or old ones used instead.\n'
Full execution log: '/home/runner/.local/state/rockcraft/log/rockcraft-20230[11](https://github.com/canonical/lego-rock/actions/runs/3941060880/jobs/6742971009#step:5:12)7-160614.051518.log'
Error: Process completed with exit code 1.

add how-to guide for "building docs"

Suggested by @evildmp , it would be nice to have an "how-to" guide for letting people how to build the docs locally. It isn't totally related to Rockcraft, but it would be helpful for developers+contributors.

refactor unit tests to simplify the use of fixtures

It would be simpler to use mocker instead in most cases.

For example, in test_oci.py:

@pytest.fixture
def mock_run(mocker):
    yield mocker.patch("rockcraft.oci._process_run")

could be deleted and instead a mocker could be used in whatever test methods need this fixture.

Review SPDX package

Looks like the spdx-license-list package is gone from PyPI. We use it to validate that the chosen license is valid.
#168 replaces that package with spdx-lookup as a stop-gap to "unbreak" main, but as @lengau points out that one has a single release and an outdated license db. We should review this situation and come up with something more sustainable.

Add testing framework?

It could be handy to have a testing framework in the spirit of cst, that is auto-executed after a pack is complete, similar to health-check in multipass blueprints.

fix: missing base information in the .rock metadata file

rockcraft/rockcraft/project.py

Line 124 in 1209dd9

def generate_project_metadata(self, generation_time: str) -> Tuple[dict, dict]:

we're missing both the base-digest (and respective annotation) and base

Support packages and slices as part of the same stage-packages

Hi!

I've been trying to pack a rock having both packages and slices defined among the stage-packages but I'm getting the following error:

Cannot mix packages and slices in stage-packages in field 'parts.indico'

name: Indico
summary: Indico rocks
description: Indico OCI images for the Indico charm
version: "1.0"
base: bare
build-base: ubuntu:22.04
license: Apache-2.0
platforms:
  amd64:
parts:
  indico:
    stage-packages:
      - build-essential
      - ca-certificates_data
    source: plugins
    plugin: python
    python-packages:
      - indico==3.2.0

IMHO it would be nice to support both indistinctly.

Thank you

Clean slate between rebuilds

Sometimes, the rockcraft pack command will fail with the following error,

Failed to copy image: Command '['umoci', 'init', '--layout', '/root/images/bare']' returned non-zero exit status 1. (⨯ image layout creation: path already exists: /root/images/bare)                                                                            
2022-05-19 12:28:58.701 Traceback (most recent call last):                                                                                                                                                                                                                               
2022-05-19 12:28:58.702   File "/snap/rockcraft/x1/lib/python3.8/site-packages/rockcraft/oci.py", line 239, in _process_run                                                                                                                                                              
2022-05-19 12:28:58.705     subprocess.run(                                                                                                                                                                                                                                              
2022-05-19 12:28:58.705   File "/snap/rockcraft/x1/usr/lib/python3.8/subprocess.py", line 516, in run                                                                                                                                                                                    
2022-05-19 12:28:58.705     raise CalledProcessError(retcode, process.args,                                                                                                                                                                                                              
2022-05-19 12:28:58.705 subprocess.CalledProcessError: Command '['umoci', 'init', '--layout', '/root/images/bare']' returned non-zero exit status 1.                                                                                                                                     
2022-05-19 12:28:58.707 Full execution log: '/root/.cache/rockcraft/log/rockcraft-20220519-122857.121178.log'

a way to reproduce that seem to be,

build a part with multiple stage packages
remove a package from the list
try to rebuild the part

bug: `init` is advertised as a lifecycle command

rockcraft/rockcraft/cli.py

Line 42 in f72beac

commands.InitCommand,

init is not longer a lifecycle command since 908f04c

Provide more meaningful output when rockcraft.yaml is invalid

Hi!

I've been trying to run rockcraft pack with a valid yaml wrongly indented:

name: Indico
summary: Indico rocks
description: Indico OCI images for the Indico charm
version: "1.0"
base: bare
build-base: ubuntu:22.04
license: Apache-2.0
platforms:
  amd64:
parts:
  indico:
    source: plugins
    plugin: python
      python-packages:
        - indico==3.2.0

and I'm getting the following output:

Logging execution to '/home/arturo/.local/state/rockcraft/log/rockcraft-20230206-173725.594029.log'                                                                               
rockcraft internal error: ScannerError(None, None, 'mapping values are not allowed here', <yaml.error.Mark object at 0x7fddfe017610>)                                             
Full execution log: '/home/arturo/.local/state/rockcraft/log/rockcraft-20230206-173725.594029.log'

I would have expected the output to show the affected line or element.

Thank you

allow user configuration of the Pebble entrypoint binary

Users can define the Pebble services to be executed, but they should also be able to configure certain parameters about the Pebble software which is being installed in the ROCK.

The most obvious parameter is the Pebble version, i.e., users should be able to have a say on the version of Pebble that is being used as the ROCK entrypoint.

See #59 (comment) for more details

edit: Additionally, Rockcraft should know how to handle (check/support/deny) the user definitions of custom Pebble layers. Example: the user defines the PEBBLE_LAYERS_PATH and inner layer files within a custom part, conflicting with the rockcraft-defined base Pebble layer from the services field (see related discussion in #205 (comment))

The `go` build-snap doesn't install the requested go version

Description

Using the go build-snap doesn't allow for using the specified go version. The go version installed is always 1.19.

Example

rockcraft.yaml:

  golang:
    plugin: nil
    build-snaps:
      - go/1.13/stable
    override-build: |
      set -x
      go version
      craftctl default
      go version

Execution output:

guillaume@thinkpad:~/PycharmProjects/magma-controller-rock$ rockcraft pack -v
Starting Rockcraft 0.0.1.dev1                                                                                                                                                                                                                          
Logging execution to '/home/guillaume/.cache/rockcraft/log/rockcraft-20221013-170003.209576.log'                                                                                                                                                       
Launching instance...                                                                                                                                                                                                                                  
Starting Rockcraft 0.0.1.dev1                                                                                                                                                                                                                          
Logging execution to '/tmp/rockcraft.log'                                                                                                                                                                                                              
Retrieving base ubuntu:18.04                                                                                                                                                                                                                           
Retrieved base ubuntu:18.04 for amd64                                                                                                                                                                                                                  
Extracting ubuntu:18.04                                                                                                                                                                                                                                
Extracted ubuntu:18.04                                                                                                                                                                                                                                 
Initializing parts lifecycle                                                                                                                                                                                                                           
Executing parts lifecycle                                                                                                                                                                                                                              
Executing parts lifecycle: pull golang                                                                                                                                                                                                                 
Executing action                                                                                                                                                                                                                                       
Executed: pull golang                                                                                                                                                                                                                                  
Executing parts lifecycle: skip pull pebble (already ran)                                                                                                                                                                                              
Executing action                                                                                                                                                                                                                                       
Executed: skip pull pebble (already ran)                                                                                                                                                                                                               
Executing parts lifecycle: overlay golang                                                                                                                                                                                                              
Executing action                                                                                                                                                                                                                                       
Executed: overlay golang                                                                                                                                                                                                                               
Executing parts lifecycle: reapply pebble (previous layer changed)                                                                                                                                                                                     
Executing action                                                                                                                                                                                                                                       
Executed: reapply pebble (previous layer changed)                                                                                                                                                                                                      
Executing parts lifecycle: build golang                                                                                                                                                                                                                
Executing action                                                                                                                                                                                                                                       
:: + set -x                                                                                                                                                                                                                                            
:: + go version                                                                                                                                                                                                                                        
:: go version go1.19 linux/amd64                                                                                                                                                                                                                       
:: + craftctl default                                                                                                                                                                                                                                  
:: + go version                                                                                                                                                                                                                                        
:: go version go1.19 linux/amd64

Workaround

Add snap commands as an override-build that will switch the go version

snap switch go --channel=1.12/stable
snap refresh go

Take "stack" of layers into account when adding a new layer

With #154 and #163, rockcraft now takes the contents of the base image into account when packing the payload. In order to support multiple layers we should also take into account any "intermediate" layers created when packing the ROCK itself. The big spread test has an example on how this isn't supported now: a part creates /.rock as a symlink but then our regular .rockmetadata layer breaks it.

Leftover files affecting consecutive runs of "rockcraft pack"

I have the following directory structure:

$ tree .
.
├── files
│   ├── a.txt
│   └── b.txt
└── rockcraft.yaml

And in rockcraft.yaml I want to organize those two text files, but I make a typo and point both to etc/newfiles/a.txt:

name: leftover
summary: Permissions
description: Permissions
license: Apache-2.0

version: "0.0.1"
base: ubuntu:22.04

parts:

  default-config:
    plugin: dump
    source: files
    organize:
      a.txt: etc/newfiles/a.txt
      b.txt: etc/newfiles/a.txt
    stage:
      - etc/newfiles/a.txt
      - etc/newfiles/b.txt

rockcraft pack then expectedly fails:

$ rockcraft -v pack
(...)
:: + cp --archive --link --no-dereference . /root/parts/default-config/install                                                                                        
Failed to organize part 'default-config': trying to organize file 'b.txt' to 'etc/newfiles/a.txt', but 'etc/newfiles/a.txt' already exists.                           
Traceback (most recent call last):                                                                                                                                    
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/rockcraft/parts.py", line 92, in run                                                                           
    aex.execute(action, stdout=stream, stderr=stream)                                                                                                                 
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/executor.py", line 301, in execute                                                        
    self._executor.execute(actions, stdout=stdout, stderr=stderr)                                                                                                     
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/executor.py", line 126, in execute                                                        
    self._run_action(act, stdout=stdout, stderr=stderr)                                                                                                               
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/executor.py", line 189, in _run_action                                                    
    handler.run_action(action, stdout=stdout, stderr=stderr)                                                                                                          
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/part_handler.py", line 160, in run_action                                                 
    state = handler(step_info, stdout=stdout, stderr=stderr)                                                                                                          
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/part_handler.py", line 320, in _run_build                                                 
    self._organize(overwrite=update)                                                                                                                                  
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/part_handler.py", line 833, in _organize                                                  
    organize_files(                                                                                                                                                   
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/organize.py", line 82, in organize_files                                                  
    raise errors.FileOrganizeError(                                                                                                                                   
craft_parts.errors.FileOrganizeError: Failed to organize part 'default-config': trying to organize file 'b.txt' to 'etc/newfiles/a.txt', but 'etc/newfiles/a.txt' already exists.

However, if I fix the organize section and re-run rockcraft -v pack, I get another error:

:: + cp --archive --link --no-dereference . /root/parts/default-config/install                                                                                        
:: cp: cannot create hard link '/root/parts/default-config/install/./b.txt' to './b.txt': File exists                                                                 
Failed to run the build script for part 'default-config'.                                                                                                             
Traceback (most recent call last):                                                                                                                                    
  File "/snap/rockcraft/x1/lib/python3.8/site-packages/craft_parts/executor/step_handler.py", line 134, in _builtin_build                                             
    subprocess.run(                                                                                                                                                   
  File "/snap/rockcraft/x1/usr/lib/python3.8/subprocess.py", line 516, in run                                                                                         
    raise CalledProcessError(retcode, process.args,                                                                                                                   
subprocess.CalledProcessError: Command '['/root/parts/default-config/run/build.sh']' returned non-zero exit status 1.

If I make a "big" change like renaming the rock from leftover to leftover2, the packing finishes successfully. Thanks!
(cc @cmatsuoka)

Using the 'Dive' tool to explore the ROCK/OCI content is broken

Referring to the https://github.com/wagoodman/dive tool to explore the content of an image, once imported in the local docker registry.

I noticed two different situations,

As discussed before, sometimes Dive will simply fail to open the image. This seems related to the use of whiteout files and a wrong interpretation of the OCI spec on their end.
But even when Dive will open the image and explore it, it is not possible to collapse the directories (hitting Space or Ctrl+Space). This only happened to me with images built using Rockcraft.

Check ROCK name, summary, description

We currently have no checks except for the string check

https://github.com/canonical/rockcraft/blob/main/rockcraft/project.py#L134

Priming files into /lib breaks system binaries in the final image

For the following rockcraft.yaml, the files/ directory contains just an empty file.txt:

name: broken-image
summary: summary
description: description
version: "0.0.1"
license: Apache-2.0

base: ubuntu:22.04

platforms:
  amd64:

parts:

  p1:
    plugin: dump
    source: files
    organize:
      file.txt: /lib/file.txt

After building and loading the ROCK into docker with skopeo, the binaries inside the container are broken:

$ docker run -it broken-image bash
exec /usr/bin/bash: no such file or directory

From a little testing, I think the issue is the interaction between the base image layer and the new layer created for the parts. In ubuntu:22.04, /lib is a symlink to /usr/lib. The layer created for the parts' files has a /lib/file.txt and this is breaking the symlink; the "final" filesystem used in the containers has a regular /lib dir that is not a symlink, and only contains file.txt. Thus the binaries that link against /lib are now broken because the dependencies cannot be resolved anymore.

This example is simple but the bug probably affects many plugins; in particular the python plugin breaks because it writes the site-packages in /lib.

docs for `rockcraft clean`

    thanks for the PR.

could you please create the docs for it? and for future PRs, it is easier to stay on track if we don't merge without the corresponding docs (otherwise some poor soul will have to, later on, spend hours on digging and writing existing features :p )

Originally posted by @cjdcordeiro in #93 (comment)

overlay-script for user creation fails on ubuntu:22.04 base

I've been experimenting (again!) and noticed that my trick for creating a user in the resulting image doesn't work on 22.04.

For an example, please see this PR and resulting CI failure.

I've done some troubleshooting, but can't understand why it's looking at apt at all in this case. Reproducible on another rockcraft.yaml, too.

Thanks! :)

Rockcraft wipes working directory

When running rockcraft with the organize option, the working directory gets completely wiped.

Steps to reproduce:

git clone [email protected]:jardon/zookeeper-rock.git
git checkout wipe-workdir
rockcraft -v

Here is the verbose output:

Starting Rockcraft 0.0.1.dev1                                                                                                                                                           
Logging execution to '/home/jardon/.cache/rockcraft/log/rockcraft-20221021-081009.128623.log'                                                                                           
Launching instance...                                                                                                                                                                   
Starting Rockcraft 0.0.1.dev1                                                                                                                                                           
Logging execution to '/tmp/rockcraft.log'                                                                                                                                               
Retrieving base ubuntu:20.04                                                                                                                                                            
Retrieved base ubuntu:20.04 for amd64                                                                                                                                                   
Extracting ubuntu:20.04                                                                                                                                                                 
Extracted ubuntu:20.04                                                                                                                                                                  
Initializing parts lifecycle                                                                                                                                                            
Executing parts lifecycle                                                                                                                                                               
Executing parts lifecycle: skip pull pebble (already ran)                                                                                                                               
Executing action                                                                                                                                                                        
Executed: skip pull pebble (already ran)                                                                                                                                                
Executing parts lifecycle: skip pull zookeeper (already ran)                                                                                                                            
Executing action                                                                                                                                                                        
Executed: skip pull zookeeper (already ran)                                                                                                                                             
Executing parts lifecycle: skip overlay pebble (already ran)                                                                                                                            
Executing action                                                                                                                                                                        
Executed: skip overlay pebble (already ran)                                                                                                                                             
Executing parts lifecycle: skip overlay zookeeper (already ran)                                                                                                                         
Executing action                                                                                                                                                                        
Executed: skip overlay zookeeper (already ran)                                                                                                                                          
Executing parts lifecycle: skip build pebble (already ran)                                                                                                                              
Executing action                                                                                                                                                                        
Executed: skip build pebble (already ran)                                                                                                                                               
Executing parts lifecycle: build zookeeper                                                                                                                                              
Executing action                                                                                                                                                                        
Removing suid/guid from /root/parts/zookeeper/install/bin/su                                                                                                                            
Removing suid/guid from /root/parts/zookeeper/install/usr/bin/newgrp                                                                                                                    
/root/parts/zookeeper/install/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts will be a dangling symlink                                                                      
Removing suid/guid from /root/parts/zookeeper/install/sbin/pam_extrausers_chkpwd                                                                                                        
Removing suid/guid from /root/parts/zookeeper/install/sbin/unix_chkpwd                                                                                                                  
:: + cp conf/zoo_sample.cfg conf/zoo.cfg                                                                                                                                                
:: + cp -r ./LICENSE.txt ./NOTICE.txt ./README.md ./README_packaging.md ./bin ./conf ./docs ./lib /root/parts/zookeeper/install                                                         
/root/project: Device or resource busy                                                                                                                                                  
Full execution log: '/tmp/rockcraft.log'                                                                                                                                                
craft-providers error: Failed to execute pack in instance.                                                                                                                              
Full execution log: '/home/jardon/.cache/rockcraft/log/rockcraft-20221021-081009.128623.log'

Here is the log

Attempting to pack a rock with multiple architectures defined fails

When attempting to build a rock with multiple architectures configured:

platforms:
    amd64:
    arm64:

rockcraft pack fails with an error:

$ rockcraft pack
Error for platform entry 'arm64': this machine's architecture (x86_64) is not compatible with the ROCK's target architecture. Can only build a ROCK for arm64 if the host is compatible with ['aarch64'].

It would be great to either build the available architecture or use something like qemu emulation to build on alternate architectures

Use Chisel `main` branch for Spread tests

We're temporarily using a development branch in docs/how-to/code/create-slice/task.yaml, due to an existing bug.

Once canonical/chisel#36 is merged, we can switch back to Chisel's main branch

add unit tests for lifecycle.py

rockcraft/lifecycle.py is missing a unit test. Despite it not having a lot of business logic to test, it is still worth having a unit test for it.

File permissions/ownership not preserved

Hey 👋

I've been building a ROCK for Traefik. I want to include a config file and have it owned by a specific user. Take the following excerpt as an example (full file here):

  default-config:
    plugin: dump
    source: files
    organize:
      traefik.yaml: etc/traefik/traefik.yaml
      001-default.yaml: var/lib/pebble/default/layers/001-default.yaml
    stage:
      - etc/traefik/traefik.yaml
      - var/lib/pebble/default/layers/001-default.yaml
    override-prime: |
      craftctl default
      ls -l etc/
      chown -R 1000:1000 etc/traefik
      ls -l etc/

  non-root-user:
    plugin: nil
    after: [default-config]
    overlay-script: |
      # Create a user in the $CRAFT_OVERLAY chroot
      useradd -R $CRAFT_OVERLAY -M -r -u 1000 -U traefik

I've added the ls -l as an example - the first shows the following:

# ...
2022-07-12 07:50:57.423 :: -rw-r----- 3 root shadow 523 Jul 12 07:48 shadow
2022-07-12 07:50:57.423 :: -rw-r----- 3 root shadow 501 May 31 15:43 shadow-
2022-07-12 07:50:57.423 :: drwxr-xr-x 1 root root    24 Jul 12 07:50 traefik

The second then shows that the chown is working as expected:

# ...
2022-07-12 07:50:57.426 :: -rw-r----- 3 root shadow 523 Jul 12 07:48 shadow
2022-07-12 07:50:57.426 :: -rw-r----- 3 root shadow 501 May 31 15:43 shadow-
2022-07-12 07:50:57.426 :: drwxr-xr-x 1 1000   1000  24 Jul 12 07:50 traefik

However, when the image is built, if I get a shell and list the directory:

❯ docker run --rm --entrypoint pebble --entrypoint /bin/bash -it traefik:2.8.1
root@ea2f942c09f4:/# ls -l /etc | grep trae
drw------- 1 root root      24 Jul 12 07:49 traefik
root@ea2f942c09f4:/#

I also attempted to chown the directory in the non-root-user stage after I'd created the user (within the chroot), but it can't see the /etc/traefik directory created in the previous stage from within the chroot.

Allow setting of a working directory for the resulting ROCK in rockcraft.yaml

Feature Request
Lots of software require the user to be in a given directory before the startup command can be called. As of Today, there is no way to set the working directory of a given rock, the default working directory being "/". This forces users to create an entrypoint (or cmd) that cd's in a given directory before calling the actual command). It would be great if a working-directory field was added to rockcraft.yaml that would set the working directory to the provided path.

Here's an example of how the rockcraft.yaml could look:

name: magma-orc8r-nms-magmalte
version: "1.6.0"
base: ubuntu:18.04
summary: Magma Orchestrator NMS Magmalte
description: Magma Orcestrator NMS Magmalte
license: Apache-2.0
platforms:
  amd64:

cmd: ["/bin/yarn", "run", "start:prod"]
working-directory: "/usr/src/packages/magmalte/"

Expected behaviour

The entrypoint command is executed from the provided working-directory
At run time, if you exec a command in the container, it is executed from the directory provided in the rockcraft.yaml file.
Ideally: This working directory is available through environment variables inside of parts (ex. $WORKING_DIRECTORY).

The python plugin doesn't install python

Description

The python plugin doesn't install python itself in the final image.

One would expect that with a rockcraft.yaml file like this one below, the container would contain python3 in it and the required python packages installed.

name: python-test
version: "1.6.1"
build-base: ubuntu:20.04
base: ubuntu:20.04
summary: python-test
description: python-test
license: Apache-2.0
platforms:
  amd64:

parts:
  python:
    plugin: python
    source: .
    python-packages:
      - PyYAML
      - jinja2

This issue is present with both 20.04 and 22.04 (I haven't tested with 18.04) and it seems to come from the fact that we are using the python3 software installed from a snap in the build process and that is not available at runtime.

Additional information

Content of the /bin directory:

root@569d22eaf8c3:~# ls -lrt /bin/
total 13760
lrwxrwxrwx 1 root root        4 Apr 18  2022 rbash -> bash
-rwxr-xr-x 1 root root 10567027 Nov  6 10:45 pebble
-rwxr-xr-x 1 root root  2242971 Nov  6 10:45 chrorder
lrwxrwxrwx 1 root root        7 Nov  6 10:46 python -> python3
-rw-r--r-- 1 root root     2395 Nov  6 10:46 activate.fish
-rw-r--r-- 1 root root     1243 Nov  6 10:46 activate.csh
-rw-r--r-- 1 root root     2191 Nov  6 10:46 activate
-rw-r--r-- 1 root root     8834 Nov  6 10:46 Activate.ps1
-rwxr-xr-x 1 root root      234 Nov  6 10:47 easy_install
-rwxr-xr-x 1 root root      946 Nov  6 10:47 which
-rwxr-xr-x 1 root root    14360 Nov  6 10:47 tempfile
-rwxr-xr-x 1 root root    27144 Nov  6 10:47 run-parts
-rwxr-xr-x 1 root root      225 Nov  6 10:47 pip3.8
-rwxr-xr-x 1 root root      225 Nov  6 10:47 pip3
-rwxr-xr-x 1 root root      225 Nov  6 10:47 pip
-rwxr-xr-x 1 root root      234 Nov  6 10:47 easy_install-3.8
-rwxr-xr-x 1 root root  1183448 Nov  6 10:47 bash
lrwxrwxrwx 1 root root       18 Nov  6 10:47 python3 -> /usr/bin/python3.8
root@569d22eaf8c3:~#

Content of the /usr/bin directory:

oot@569d22eaf8c3:~# ls -lrt /usr/bin/
total 23008
-rwxr-xr-x 1 root root    121288 Dec 22  2018  sed
-rwxr-xr-x 1 root root     51296 Apr  8  2019  sdiff
-rwxr-xr-x 1 root root     67800 Apr  8  2019  diff3
-rwxr-xr-x 1 root root    219480 Apr  8  2019  diff
-rwxr-xr-x 1 root root     51296 Apr  8  2019  cmp
-rwxr-xr-x 1 root root      4430 Jun 21  2019  deb-systemd-invoke
-rwxr-xr-x 1 root root     21328 Jun 21  2019  deb-systemd-helper
lrwxrwxrwx 1 root root         4 Jul 18  2019  sh -> dash
-rwxr-xr-x 1 root root    129816 Jul 18  2019  dash
-rwxr-xr-x 1 root root      1827 Aug  3  2019  debconf-show
-rwxr-xr-x 1 root root      2935 Aug  3  2019  debconf-set-selections
-rwxr-xr-x 1 root root       647 Aug  3  2019  debconf-escape
-rwxr-xr-x 1 root root      1719 Aug  3  2019  debconf-copydb
-rwxr-xr-x 1 root root       608 Aug  3  2019  debconf-communicate
-rwxr-xr-x 1 root root     11541 Aug  3  2019  debconf-apt-progress
-rwxr-xr-x 1 root root      2859 Aug  3  2019  debconf
-rwxr-xr-x 1 root root      4877 Sep  4  2019  bzexe
-rwxr-xr-x 1 root root     39256 Sep  5  2019  yes
-rwxr-xr-x 1 root root     39256 Sep  5  2019  whoami
-rwxr-xr-x 1 root root     59768 Sep  5  2019  who
-rwxr-xr-x 1 root root     47456 Sep  5  2019  wc
-rwxr-xr-x 1 root root    142144 Sep  5  2019  vdir
-rwxr-xr-x 1 root root     39256 Sep  5  2019  users
-rwxr-xr-x 1 root root     39256 Sep  5  2019  unlink
-rwxr-xr-x 1 root root     51576 Sep  5  2019  uniq
-rwxr-xr-x 1 root root     43384 Sep  5  2019  unexpand
-rwxr-xr-x 1 root root     39288 Sep  5  2019  uname
-rwxr-xr-x 1 root root     39256 Sep  5  2019  tty
-rwxr-xr-x 1 root root     43352 Sep  5  2019  tsort
-rwxr-xr-x 1 root root     43352 Sep  5  2019  truncate
-rwxr-xr-x 1 root root     39256 Sep  5  2019  true
-rwxr-xr-x 1 root root     51544 Sep  5  2019  tr
-rwxr-xr-x 1 root root    100728 Sep  5  2019  touch
-rwxr-xr-x 1 root root     43800 Sep  5  2019  timeout
-rwxr-xr-x 1 root root     55640 Sep  5  2019  test
-rwxr-xr-x 1 root root     43384 Sep  5  2019  tee
-rwxr-xr-x 1 root root     72088 Sep  5  2019  tail
-rwxr-xr-x 1 root root     43352 Sep  5  2019  tac
-rwxr-xr-x 1 root root     39256 Sep  5  2019  sync
-rwxr-xr-x 1 root root     47456 Sep  5  2019  sum
-rwxr-xr-x 1 root root     84344 Sep  5  2019  stty
-rwxr-xr-x 1 root root     51544 Sep  5  2019  stdbuf
-rwxr-xr-x 1 root root     88440 Sep  5  2019  stat
-rwxr-xr-x 1 root root     60184 Sep  5  2019  split
-rwxr-xr-x 1 root root    117376 Sep  5  2019  sort
-rwxr-xr-x 1 root root     39256 Sep  5  2019  sleep
-rwxr-xr-x 1 root root     59736 Sep  5  2019  shuf
-rwxr-xr-x 1 root root     63864 Sep  5  2019  shred
-rwxr-xr-x 1 root root     67960 Sep  5  2019  sha512sum
-rwxr-xr-x 1 root root     67960 Sep  5  2019  sha384sum
-rwxr-xr-x 1 root root     59768 Sep  5  2019  sha256sum
-rwxr-xr-x 1 root root     59768 Sep  5  2019  sha224sum
-rwxr-xr-x 1 root root     51576 Sep  5  2019  sha1sum
-rwxr-xr-x 1 root root     51544 Sep  5  2019  seq
-rwxr-xr-x 1 root root     39256 Sep  5  2019  runcon
-rwxr-xr-x 1 root root     47448 Sep  5  2019  rmdir
-rwxr-xr-x 1 root root     72056 Sep  5  2019  rm
-rwxr-xr-x 1 root root     51576 Sep  5  2019  realpath
-rwxr-xr-x 1 root root     51544 Sep  5  2019  readlink
-rwxr-xr-x 1 root root     43352 Sep  5  2019  pwd
-rwxr-xr-x 1 root root     80280 Sep  5  2019  ptx
-rwxr-xr-x 1 root root     59736 Sep  5  2019  printf
-rwxr-xr-x 1 root root     39256 Sep  5  2019  printenv
-rwxr-xr-x 1 root root     76216 Sep  5  2019  pr
-rwxr-xr-x 1 root root     43384 Sep  5  2019  pinky
-rwxr-xr-x 1 root root     39256 Sep  5  2019  pathchk
-rwxr-xr-x 1 root root     43384 Sep  5  2019  paste
-rwxr-xr-x 1 root root     72056 Sep  5  2019  od
-rwxr-xr-x 1 root root     67992 Sep  5  2019  numfmt
-rwxr-xr-x 1 root root     43352 Sep  5  2019  nproc
-rwxr-xr-x 1 root root     43352 Sep  5  2019  nohup
-rwxr-xr-x 1 root root     43448 Sep  5  2019  nl
-rwxr-xr-x 1 root root     43352 Sep  5  2019  nice
-rwxr-xr-x 1 root root    149888 Sep  5  2019  mv
-rwxr-xr-x 1 root root     47448 Sep  5  2019  mktemp
-rwxr-xr-x 1 root root     72024 Sep  5  2019  mknod
-rwxr-xr-x 1 root root     67928 Sep  5  2019  mkfifo
-rwxr-xr-x 1 root root     88408 Sep  5  2019  mkdir
lrwxrwxrwx 1 root root         6 Sep  5  2019  md5sum.textutils -> md5sum
-rwxr-xr-x 1 root root     47480 Sep  5  2019  md5sum
-rwxr-xr-x 1 root root    142144 Sep  5  2019  ls
-rwxr-xr-x 1 root root     39256 Sep  5  2019  logname
-rwxr-xr-x 1 root root     76160 Sep  5  2019  ln
-rwxr-xr-x 1 root root     39256 Sep  5  2019  link
-rwxr-xr-x 1 root root     55672 Sep  5  2019  join
-rwxr-xr-x 1 root root    158112 Sep  5  2019  install
-rwxr-xr-x 1 root root     47480 Sep  5  2019  id
-rwxr-xr-x 1 root root     39256 Sep  5  2019  hostid
-rwxr-xr-x 1 root root     47480 Sep  5  2019  head
-rwxr-xr-x 1 root root     39256 Sep  5  2019  groups
-rwxr-xr-x 1 root root     43352 Sep  5  2019  fold
-rwxr-xr-x 1 root root     47448 Sep  5  2019  fmt
-rwxr-xr-x 1 root root     39256 Sep  5  2019  false
-rwxr-xr-x 1 root root     80248 Sep  5  2019  factor
-rwxr-xr-x 1 root root     55640 Sep  5  2019  expr
-rwxr-xr-x 1 root root     43384 Sep  5  2019  expand
-rwxr-xr-x 1 root root     43352 Sep  5  2019  env
-rwxr-xr-x 1 root root     39256 Sep  5  2019  echo
-rwxr-xr-x 1 root root    108920 Sep  5  2019  du
-rwxr-xr-x 1 root root     39256 Sep  5  2019  dirname
-rwxr-xr-x 1 root root     47456 Sep  5  2019  dircolors
-rwxr-xr-x 1 root root    142144 Sep  5  2019  dir
-rwxr-xr-x 1 root root     93128 Sep  5  2019  df
-rwxr-xr-x 1 root root     80256 Sep  5  2019  dd
-rwxr-xr-x 1 root root    108920 Sep  5  2019  date
-rwxr-xr-x 1 root root     47480 Sep  5  2019  cut
-rwxr-xr-x 1 root root     55672 Sep  5  2019  csplit
-rwxr-xr-x 1 root root    153976 Sep  5  2019  cp
-rwxr-xr-x 1 root root     43384 Sep  5  2019  comm
-rwxr-xr-x 1 root root     39256 Sep  5  2019  cksum
-rwxr-xr-x 1 root root     72024 Sep  5  2019  chown
-rwxr-xr-x 1 root root     63864 Sep  5  2019  chmod
-rwxr-xr-x 1 root root     72024 Sep  5  2019  chgrp
-rwxr-xr-x 1 root root     72024 Sep  5  2019  chcon
-rwxr-xr-x 1 root root     43416 Sep  5  2019  cat
-rwxr-xr-x 1 root root     39256 Sep  5  2019  basename
-rwxr-xr-x 1 root root     43352 Sep  5  2019  base64
-rwxr-xr-x 1 root root     43352 Sep  5  2019  base32
-rwxr-xr-x 1 root root     59768 Sep  5  2019  b2sum
-rwxr-xr-x 1 root root     39288 Sep  5  2019  arch
-rwxr-xr-x 1 root root     59736 Sep  5  2019 '['
-rwxr-xr-x 1 root root      1297 Sep  5  2019  bzmore
lrwxrwxrwx 1 root root         6 Sep  5  2019  bzless -> bzmore
-rwxr-xr-x 1 root root     18584 Sep  5  2019  bzip2recover
-rwxr-xr-x 3 root root     39144 Sep  5  2019  bzip2
-rwxr-xr-x 1 root root      3775 Sep  5  2019  bzgrep
lrwxrwxrwx 1 root root         6 Sep  5  2019  bzfgrep -> bzgrep
lrwxrwxrwx 1 root root         6 Sep  5  2019  bzegrep -> bzgrep
-rwxr-xr-x 1 root root      2227 Sep  5  2019  bzdiff
lrwxrwxrwx 1 root root         6 Sep  5  2019  bzcmp -> bzdiff
-rwxr-xr-x 3 root root     39144 Sep  5  2019  bzcat
-rwxr-xr-x 3 root root     39144 Sep  5  2019  bunzip2
lrwxrwxrwx 1 root root         8 Nov  7  2019  ypdomainname -> hostname
lrwxrwxrwx 1 root root         8 Nov  7  2019  nisdomainname -> hostname
-rwxr-xr-x 1 root root     26856 Nov  7  2019  hostname
lrwxrwxrwx 1 root root         8 Nov  7  2019  domainname -> hostname
lrwxrwxrwx 1 root root         8 Nov  7  2019  dnsdomainname -> hostname
-rwxr-xr-x 1 root root       452 Nov 17  2019  sensible-pager
-rwxr-xr-x 1 root root      1169 Nov 17  2019  sensible-editor
-rwxr-xr-x 1 root root      1230 Nov 17  2019  sensible-browser
-rwxr-xr-x 1 root root      2442 Nov 17  2019  select-editor
-rwxr-xr-x 1 root root       946 Dec  7  2019  which
-rwxr-xr-x 1 root root     14360 Dec  7  2019  tempfile
-rwxr-xr-x 1 root root     27144 Dec  7  2019  run-parts
-rwxr-xr-x 1 root root        30 Jan 16  2020  rgrep
-rwxr-xr-x 1 root root    199136 Jan 29  2020  grep
-rwxr-xr-x 1 root root        28 Jan 29  2020  fgrep
-rwxr-xr-x 1 root root        28 Jan 29  2020  egrep
lrwxrwxrwx 1 root root        14 Feb 13  2020  pidof -> /sbin/killall5
-rwxr-xr-x 1 root root     76152 Feb 18  2020  xargs
-rwxr-xr-x 1 root root    320160 Feb 18  2020  find
-rwxr-xr-x 1 root root     31040 Feb 26  2020  tset
-rwxr-xr-x 1 root root     26968 Feb 26  2020  tput
-rwxr-xr-x 1 root root     22840 Feb 26  2020  toe
-rwxr-xr-x 1 root root     92584 Feb 26  2020  tic
-rwxr-xr-x 1 root root     18744 Feb 26  2020  tabs
lrwxrwxrwx 1 root root         4 Feb 26  2020  reset -> tset
lrwxrwxrwx 1 root root         3 Feb 26  2020  infotocap -> tic
-rwxr-xr-x 1 root root     63880 Feb 26  2020  infocmp
-rwxr-xr-x 1 root root     14656 Feb 26  2020  clear
lrwxrwxrwx 1 root root         3 Feb 26  2020  captoinfo -> tic
-rwxr-xr-x 1 root root     27208 Sep  9  2021  watch
-rwxr-xr-x 1 root root     22760 Sep  9  2021  w.procps
-rwxr-xr-x 1 root root     39168 Sep  9  2021  vmstat
-rwxr-xr-x 1 root root     14568 Sep  9  2021  uptime
-rwxr-xr-x 1 root root    129072 Sep  9  2021  top
-rwxr-xr-x 1 root root     22776 Sep  9  2021  tload
lrwxrwxrwx 1 root root         5 Sep  9  2021  snice -> skill
-rwxr-xr-x 1 root root     22768 Sep  9  2021  slabtop
-rwxr-xr-x 1 root root     30952 Sep  9  2021  skill
-rwxr-xr-x 1 root root     14568 Sep  9  2021  pwdx
-rwxr-xr-x 1 root root    137688 Sep  9  2021  ps
-rwxr-xr-x 1 root root     35064 Sep  9  2021  pmap
lrwxrwxrwx 1 root root         5 Sep  9  2021  pkill -> pgrep
-rwxr-xr-x 1 root root     30968 Sep  9  2021  pgrep
-rwxr-xr-x 1 root root     30952 Sep  9  2021  kill
-rwxr-xr-x 1 root root     26864 Sep  9  2021  free
lrwxrwxrwx 1 root root         7 Feb  7  2022  x86_64 -> setarch
-rwxr-xr-x 1 root root     35496 Feb  7  2022  whereis
-rwxr-xr-x 1 root root     35048 Feb  7  2022  wdctl
-rwxr-sr-x 1 root tty      35048 Feb  7  2022  wall
-rwxr-xr-x 1 root root     30952 Feb  7  2022  utmpdump
-rwxr-xr-x 1 root root     43448 Feb  7  2022  unshare
-rwsr-xr-x 1 root root     39144 Feb  7  2022  umount
-rwxr-xr-x 1 root root     35048 Feb  7  2022  taskset
-rwsr-xr-x 1 root root     67816 Feb  7  2022  su
-rwxr-xr-x 1 root root     47336 Feb  7  2022  setterm
-rwxr-xr-x 1 root root     14568 Feb  7  2022  setsid
-rwxr-xr-x 1 root root     47344 Feb  7  2022  setpriv
-rwxr-xr-x 1 root root     27136 Feb  7  2022  setarch
-rwxr-xr-x 1 root root     30952 Feb  7  2022  scriptreplay
-rwxr-xr-x 1 root root     55528 Feb  7  2022  script
-rwxr-xr-x 1 root root     14568 Feb  7  2022  rev
-rwxr-xr-x 1 root root     63720 Feb  7  2022  resizepart
-rwxr-xr-x 1 root root     14568 Feb  7  2022  renice
-rwxr-xr-x 1 root root     22760 Feb  7  2022  rename.ul
-rwxr-xr-x 1 root root     39672 Feb  7  2022  prlimit
-rwxr-xr-x 1 root root    121072 Feb  7  2022  partx
-rwxr-xr-x 1 root root     35256 Feb  7  2022  nsenter
-rwxr-xr-x 1 root root     35048 Feb  7  2022  namei
-rwxr-xr-x 1 root root     14568 Feb  7  2022  mountpoint
-rwsr-xr-x 1 root root     55528 Feb  7  2022  mount
-rwxr-xr-x 1 root root     43160 Feb  7  2022  more
-rwxr-xr-x 1 root root     14568 Feb  7  2022  mesg
-rwxr-xr-x 1 root root     35120 Feb  7  2022  mcookie
-rwxr-xr-x 1 root root     51440 Feb  7  2022  lsns
-rwxr-xr-x 1 root root     67816 Feb  7  2022  lsmem
-rwxr-xr-x 1 root root     67816 Feb  7  2022  lslogins
-rwxr-xr-x 1 root root     39480 Feb  7  2022  lslocks
-rwxr-xr-x 1 root root     96488 Feb  7  2022  lsipc
-rwxr-xr-x 1 root root    100584 Feb  7  2022  lscpu
-rwxr-xr-x 1 root root    133352 Feb  7  2022  lsblk
-rwxr-xr-x 1 root root     52040 Feb  7  2022  logger
lrwxrwxrwx 1 root root         7 Feb  7  2022  linux64 -> setarch
lrwxrwxrwx 1 root root         7 Feb  7  2022  linux32 -> setarch
lrwxrwxrwx 1 root root         4 Feb  7  2022  lastb -> last
-rwxr-xr-x 1 root root     47336 Feb  7  2022  last
-rwxr-xr-x 1 root root     67816 Feb  7  2022  ipcs
-rwxr-xr-x 1 root root     35048 Feb  7  2022  ipcrm
-rwxr-xr-x 1 root root     35120 Feb  7  2022  ipcmk
-rwxr-xr-x 1 root root     30952 Feb  7  2022  ionice
lrwxrwxrwx 1 root root         7 Feb  7  2022  i386 -> setarch
-rwxr-xr-x 1 root root     22760 Feb  7  2022  getopt
-rwxr-xr-x 1 root root     35128 Feb  7  2022  flock
-rwxr-xr-x 1 root root     73128 Feb  7  2022  findmnt
-rwxr-xr-x 1 root root     35096 Feb  7  2022  fincore
-rwxr-xr-x 1 root root     35048 Feb  7  2022  fallocate
-rwxr-xr-x 1 root root     84440 Feb  7  2022  dmesg
-rwxr-xr-x 1 root root     30952 Feb  7  2022  delpart
-rwxr-xr-x 1 root root     39144 Feb  7  2022  chrt
-rwxr-xr-x 1 root root     51432 Feb  7  2022  choom
-rwxr-xr-x 1 root root     30952 Feb  7  2022  addpart
lrwxrwxrwx 1 root root         6 Mar 14  2022  sg -> newgrp
-rwsr-xr-x 1 root root     68208 Mar 14  2022  passwd
-rwsr-xr-x 1 root root     44784 Mar 14  2022  newgrp
-rwxr-xr-x 1 root root     57104 Mar 14  2022  login
-rwxr-xr-x 1 root root     32416 Mar 14  2022  lastlog
-rwsr-xr-x 1 root root     88464 Mar 14  2022  gpasswd
-rwxr-xr-x 1 root root     23136 Mar 14  2022  faillog
-rwxr-sr-x 1 root shadow   31312 Mar 14  2022  expiry
-rwsr-xr-x 1 root root     53040 Mar 14  2022  chsh
-rwsr-xr-x 1 root root     85064 Mar 14  2022  chfn
-rwxr-sr-x 1 root shadow   84512 Mar 14  2022  chage
-rwxr-xr-x 1 root root    448112 Mar 15  2022  tar
-rwxr-xr-x 1 root root     26840 Apr  7  2022  zdump
-rwxr-xr-x 1 root root     15378 Apr  7  2022  tzselect
-rwxr-xr-x 1 root root     22904 Apr  7  2022  pldd
-rwxr-xr-x 1 root root    334808 Apr  7  2022  localedef
-rwxr-xr-x 1 root root     58944 Apr  7  2022  locale
-rwxr-xr-x 1 root root      5427 Apr  7  2022  ldd
-rwxr-xr-x 1 root root     68000 Apr  7  2022  iconv
-rwxr-xr-x 1 root root     39576 Apr  7  2022  getent
-rwxr-xr-x 1 root root     35112 Apr  7  2022  getconf
-rwxr-xr-x 1 root root      3330 Apr  7  2022  catchsegv
-rwxr-xr-x 1 root root      4577 Apr  8  2022  znew
-rwxr-xr-x 1 root root      1842 Apr  8  2022  zmore
-rwxr-xr-x 1 root root      2206 Apr  8  2022  zless
-rwxr-xr-x 1 root root      8103 Apr  8  2022  zgrep
-rwxr-xr-x 1 root root      2081 Apr  8  2022  zforce
-rwxr-xr-x 1 root root        29 Apr  8  2022  zfgrep
-rwxr-xr-x 1 root root        29 Apr  8  2022  zegrep
-rwxr-xr-x 1 root root      5898 Apr  8  2022  zdiff
-rwxr-xr-x 1 root root      1678 Apr  8  2022  zcmp
-rwxr-xr-x 1 root root      1984 Apr  8  2022  zcat
-rwxr-xr-x 2 root root      2346 Apr  8  2022  uncompress
-rwxr-xr-x 1 root root     97496 Apr  8  2022  gzip
-rwxr-xr-x 1 root root      6447 Apr  8  2022  gzexe
-rwxr-xr-x 2 root root      2346 Apr  8  2022  gunzip
lrwxrwxrwx 1 root root         4 Apr 18  2022  rbash -> bash
-rwxr-xr-x 1 root root   1183448 Apr 18  2022  bash
-rwxr-xr-x 1 root root     63960 May 24 13:08  apt-mark
-rwxr-xr-x 1 root root     27931 May 24 13:08  apt-key
-rwxr-xr-x 1 root root     47576 May 24 13:08  apt-get
-rwxr-xr-x 1 root root     27016 May 24 13:08  apt-config
-rwxr-xr-x 1 root root     31192 May 24 13:08  apt-cdrom
-rwxr-xr-x 1 root root     88536 May 24 13:08  apt-cache
-rwxr-xr-x 1 root root     18824 May 24 13:08  apt
-rwxr-xr-x 1 root root     55712 May 25 11:14  update-alternatives
-rwxr-xr-x 1 root root     84472 May 25 11:14  dpkg-trigger
-rwxr-xr-x 1 root root     63760 May 25 11:14  dpkg-statoverride
-rwxr-xr-x 1 root root    125424 May 25 11:14  dpkg-split
-rwxr-xr-x 1 root root    166488 May 25 11:14  dpkg-query
-rwxr-xr-x 1 root root     20514 May 25 11:14  dpkg-maintscript-helper
-rwxr-xr-x 1 root root    154208 May 25 11:14  dpkg-divert
-rwxr-xr-x 1 root root    178728 May 25 11:14  dpkg-deb
-rwxr-xr-x 1 root root    309944 May 25 11:14  dpkg
-rwxr-xr-x 1 root root     14656 Jun  2 00:59  lsattr
-rwxr-xr-x 1 root root     14656 Jun  2 00:59  chattr
-rwxr-xr-x 1 root root    457880 Jul  4 16:20  gpgv
-rwxr-xr-x 2 root root   3478464 Oct  5 10:27  perl5.30.0
-rwxr-xr-x 2 root root   3478464 Oct  5 10:27  perl
lrwxrwxrwx 1 root root        22 Oct 19 16:47  nawk -> /etc/alternatives/nawk
lrwxrwxrwx 1 root root        21 Oct 19 16:47  awk -> /etc/alternatives/awk
lrwxrwxrwx 1 root root        19 Oct 19 16:47  w -> /etc/alternatives/w
lrwxrwxrwx 1 root root        23 Oct 19 16:47  pager -> /etc/alternatives/pager
-rwxr-xr-x 1 root root       320 Oct 19 16:49  man
-rwxr-xr-x 1 root root     10481 Nov  6 10:47  savelog
-rwxr-xr-x 1 root root    162552 Nov  6 10:47  mawk
-rwxr-xr-x 1 root root     14496 Nov  6 10:47  locale-check
-rwxr-xr-x 1 root root     14536 Nov  6 10:47  ischroot
-rwxr-xr-x 1 root root     14568 Nov  6 10:47  clear_console
-rwxr-xr-x 1 root root      6794 Nov  6 10:47  bashbug

Cache retrieved base during build

I have hit my pull rate limit for docker hub because of rockcraft. Every time I run a rockcraft pack the base is fetched from docker and contributes against my limit. Can this image be cached and only removed during a rockcraft clean or a base change?

get rid of `:type` in docstrings

With function annotations, the :type in docstrings becomes redundant. Let's clean that up

Provide a simple, declarative way to provision users in the resulting OCI image

Right now, one has to do something like the following:

parts:
  non-root-user:
    plugin: nil
    overlay-script: |
      chroot $CRAFT_OVERLAY
      useradd -M -r my-user

Such a feature would likely need to provide some sort of language for describing UID, system user or not, home directory if required, shell, etc. An MVP might just be the ability to create system users with a given UID, since that is a very common use case.

grant integration access to RTD.com

After the migration to RTD, the new RTD project requires permissions to be integrated with this repository (so it can build the docs on commit):

https://docs.readthedocs.io/en/latest/integrations.html#github

redundant whiteouts cause a path error

Problem

When building an image where new packages are added (either via stage-packages or overlay-packages), the corresponding OCI layer will flag some of the underlying part's files to be whiteout.

Not only this is redundant (because the introduction and deletion of the package files happen within the same part/layer), but it also causes a path error to some tools (like dive).

In short, whiteout should only be used to flag files from underlying layers for deletion, but not files from the same layer.

How to reproduce

rockcraft.yaml:

name: hello
version: "1.0"
base: ubuntu:20.04

parts:
    hello:
        plugin: nil
        stage-packages:
            - hello

Then:

$ rockcraft pack
...
$ skopeo copy oci-archive:hello_1.0.rock docker-daemon:test-image:latest
$ dive test-image:latest
Image Source: docker://test-image:latest
Fetching image... (this can take a while for large images)
Analyzing image...
Building cache...
  path error at layer index Index(0:1): unable to remove '/usr/bin/hello.dpkg-new': path does not exist: /usr/bin/hello.dpkg-new
  path error at layer index Index(0:1): unable to remove '/usr/share/doc/hello.dpkg-new': path does not exist: /usr/share/doc/hello.dpkg-new
  path error at layer index Index(0:1): unable to remove '/usr/share/doc/hello/changelog.Debian.gz.dpkg-new': path does not exist: /usr/share/doc/hello/changelog.Debian.gz.dpkg-new
  path error at layer index Index(0:1): unable to remove '/usr/share/doc/hello/copyright.dpkg-new': path does not exist: /usr/share/doc/hello/copyright.dpkg-new
  path error at layer index Index(0:1): unable to remove '/usr/share/info/hello.info.gz.dpkg-new': path does not exist: /usr/share/info/hello.info.gz.dpkg-new
  path error at layer index Index(0:1): unable to remove '/var/cache/apt/archives/hello_2.10-2ubuntu2_amd64.deb': path does not exist: /var/cache/apt/archives/hello_2.10-2ubuntu2_amd64.deb
  path error at layer index Index(0:1): unable to remove '/var/lib/apt/extended_states.1NJeQR': path does not exist: /var/lib/apt/extended_states.1NJeQR
  path error at layer index Index(0:1): unable to remove '/var/lib/apt/extended_states.3b7RY9': path does not exist: /var/lib/apt/extended_states.3b7RY9
  path error at layer index Index(0:1): unable to remove '/var/lib/dpkg/status-new': path does not exist: /var/lib/dpkg/status-new
  path error at layer index Index(0:1): unable to remove '/var/lib/dpkg/tmp.ci': path does not exist: /var/lib/dpkg/tmp.ci
  path error at layer index Index(0:1): unable to remove '/var/lib/dpkg/info/hello.list-new': path does not exist: /var/lib/dpkg/info/hello.list-new
file tree has path errors

Origin of the problem

The new layer created by the rockcraft part has all of the whiteout files listed in the dive error above. This can be confirmed by:

$ skopeo copy oci-archive:hello_1.0.rock oci:test-image
$ ls -l test-image/blobs/sha256/
total 33824
-rw-r--r-- 1 cris cris      508 Apr 25 10:12 8a023a80af56db32c245ed2fb0a542eb44ceabc24695abcf26a9d9a8095689a3
-rw-r--r-- 1 cris cris  6033508 Apr 25 10:12 8a9350d62b7b6564badccbd007a354a6eab0ddf7aeb1006b5692e179d2e244e8
-rw-r--r-- 1 cris cris 28565998 Apr 25 10:12 8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba
-rw-r--r-- 1 cris cris      734 Apr 25 10:12 e41e5ca3f81a4dadba93d0fd71f2ee8c3ef693620933b65b0d325fb490162370
$
$ tar tf test-image/blobs/sha256/8a9350d62b7b6564badccbd007a354a6eab0ddf7aeb1006b5692e179d2e244e8 | grep wh
./var/cache/apt/archives/.wh.hello_2.10-2ubuntu2_amd64.deb
./var/lib/dpkg/.wh.status-new
./var/lib/dpkg/.wh.tmp.ci
./var/lib/dpkg/info/.wh.hello.list-new
./var/lib/apt/.wh.extended_states.3b7RY9
./var/lib/apt/.wh.extended_states.1NJeQR
./usr/bin/.wh.hello.dpkg-new
./usr/share/doc/.wh.hello.dpkg-new
./usr/share/doc/hello/.wh..wh..opq
./usr/share/doc/hello/.wh.changelog.Debian.gz.dpkg-new
./usr/share/doc/hello/.wh.copyright.dpkg-new
./usr/share/info/.wh.hello.info.gz.dpkg-new

These whiteout files do not exist in layer 8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba, and thus the whiteout is redundant.

Packing the "hello world" example fails: 'apt-utils' has no installation candidate

Hi! Rockcraft is exciting and I'd like to experiment with it.
How can I set it up for testing purposes?
The hello world example fails:

Failed to install packages.                                                                                                                                                                                
* Command that failed: 'lxc --project rockcraft exec local:rockcraft-hello-480 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 apt-get install -y apt-utils curl'
* Command exit code: 100
* Command output: b'Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage apt-utils is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\nHowever the following packages replace it:\n  apt\n\n'
* Command standard error output: b"E: Package 'apt-utils' has no installation candidate\nE: Unable to locate package curl\n"                                                                          
Full execution log: '/home/ubuntu/.cache/rockcraft/log/rockcraft-20220622-174218.572084.log'

Rockcraft version: latest/edge: 0+git.0e8a4f5 2022-05-23 (94) 73MB classic

add tutorial on "Install packages slices into a ROCK"

Make sure to include:

the installation of a full package vs the installation of a slice of that package, noting the difference in the syntax.

You can mention Chisel and point to #89 for knowing what it is and how it works

Allow customization/removal of automatic "go generate" step

The GoPlugin current always calls go generate ./... when building the go code. However, there have been two instances so far (chisel and pebble) where that call is incorrect - it generates code that then fails to compile. The workaround currently is to override the entire build step, so we should come up with a way to customize/toggle this.

`base: bare` + `overlay_packages` = unhelpful error message

Issue

Defining a rock with base: bare and overlay_packages does not produce a useful error message.

Error

Overlay environment execution error: [Errno 2] No such file or directory: 'apt-get'

Reproducer

name: hello-world
version: latest
build-base: ubuntu:20.04
base: bare
summary: Hello World
description: A Hello World rock
license: Apache-2.0
cmd: [/usr/bin/hello, -t]

platforms:
  amd64:
  
parts:
  hello:
    plugin: nil
    overlay-packages:
      - hello

Output

Starting Rockcraft 0.0.1.dev1
Logging execution to '/home/developer/.cache/rockcraft/log/rockcraft-20221103-112142.847011.log'

Launching instance...
Starting Rockcraft 0.0.1.dev1
Logging execution to '/tmp/rockcraft.log'
Retrieving base bare
Retrieved base bare for amd64
Extracting bare:latest
Extracted bare:latest
Initializing parts lifecycle
Executing parts lifecycle
Overlay environment execution error: [Errno 2] No such file or directory: 'apt-get'
Full execution log: '/tmp/rockcraft.log'
craft-providers error: Failed to execute pack in instance.
Full execution log: '/home/developer/.cache/rockcraft/log/rockcraft-20221103-112142.847011.log'

move "Structure of a slice definitions file" to Chisel reference

This is more of a reference than a guide, and thus should belong someplace else.

Whenever a Chisel reference is created, this section should be moved there.

make the doc's technical content dynamic

At the moment, we have a tutorial https://github.com/canonical/rockcraft/blob/main/docs/tutorials.rst with a technical example of a rockcraft.yaml file which is not only being maintained manually but not tested anywhere in the CI pipelines.

Examples and tutorials should be tested.

Proposal

Make the technical contents of the documentation dynamic, by fetching the respective blocks of code from the project's test folder. Example: docs/tutorials.rst should use tests/spread/tutorials/basic/, where the actual tutorial YAML and tasks are described, and tested by Spread during the CI.

add link to CoC in the docs

below "Project and Community"

Instance output not being logged

Inner instance messages are displayed when using --verbosity=<verbose|debug>, but not recorded in the main log:

2022-11-09 10:08:21.869 Starting Rockcraft 0.0.1.dev1
2022-11-09 10:08:21.871 Using default provider 'lxd' on linux system
2022-11-09 10:08:22.030 Executing on host: lxc --project default profile show local:default
2022-11-09 10:08:22.126 Using hostname 'rockcraft-gunicorn-k8s-operator-2212028'
2022-11-09 10:08:22.126 Launching instance...
2022-11-09 10:08:22.126 Executing on host: lxc remote list --format=yaml
2022-11-09 10:08:22.211 Remote 'craft-com.ubuntu.cloud-buildd' already exists.
2022-11-09 10:08:22.211 Executing on host: lxc project list local: --format=yaml
2022-11-09 10:08:22.318 Set LXD instance name to 'rockcraft-gunicorn-k8s-operator-2212028'
2022-11-09 10:08:22.318 Executing on host: lxc --project rockcraft list local: --format=yaml
2022-11-09 10:08:22.431 Executing on host: lxc --project rockcraft list local: --format=yaml
2022-11-09 10:08:22.542 Executing on host: lxc --project rockcraft start local:rockcraft-gunicorn-k8s-operator-2212028
2022-11-09 10:08:23.053 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 cat /etc/os-release
2022-11-09 10:08:23.176 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf
2022-11-09 10:08:23.325 Executing on host: lxc --project rockcraft file pull local:rockcraft-gunicorn-k8s-operator-2212028/etc/craft-instance.conf /home/claudio/tmppj2zbhr4.tmp-craft/tmpur4p5v38
2022-11-09 10:08:23.478 Instance is compatible with compatibility tag 'rockcraft-buildd-base-v0.0'
2022-11-09 10:08:23.478 Waiting for environment to be ready...
2022-11-09 10:08:23.478 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-11-09 10:08:23.922 Waiting for networking to be ready...
2022-11-09 10:08:23.922 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io
2022-11-09 10:08:24.095 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 snap unset system proxy.http
2022-11-09 10:08:24.772 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 snap unset system proxy.https
2022-11-09 10:08:25.084 Installing snap 'rockcraft' with channel=None and classic=True
2022-11-09 10:08:25.084 Installing snap 'rockcraft' from host (classic=True)
2022-11-09 10:08:25.095 Executing in container: lxc --project rockcraft exec local:rockcraft-gunicorn-k8s-operator-2212028 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf
2022-11-09 10:08:25.275 Executing on host: lxc --project rockcraft file pull local:rockcraft-gunicorn-k8s-operator-2212028/etc/craft-instance.conf /home/claudio/tmplzqj1fg_.tmp-craft/tmpl726wv7m
2022-11-09 10:08:25.380 Revisions found: host='418', target='418'
2022-11-09 10:08:25.380 Skipping snap injection: target is already up-to-date with revision on host
2022-11-09 10:08:25.380 Emitter: Pausing control of the terminal
2022-11-09 10:08:32.779 Emitter: Resuming control of the terminal
2022-11-09 10:08:36.427 craft-providers error: Failed to execute pack in instance.
2022-11-09 10:08:36.427 Full execution log: '/home/claudio/.cache/rockcraft/log/rockcraft-20221109-100821.868350.log'

Location of managed rockcraft instance's log file is shown

When rockcraft starts, the location of the log file is shown twice, from the the outer rockcraft and managed rockcraft processes:

Outer process: Logging execution to '/home/developer/.cache/rockcraft/log/rockcraft-20221103-105727.504666.log'
Inner/managed process: Logging execution to '/tmp/rockcraft.log'

This also occurs when rockcraft fails, the location of the two log files appear:

Full execution log: '/tmp/rockcraft.log' from inside the instance
Full execution log: '/home/developer/.cache/rockcraft/log/rockcraft-20221103-105727.504666.log' from outside the instance

Only the outer instance's log location should be displayed.

The collection of log files from the managed rockcraft instance is recent feature (I added it here and here).

I recall that snapcraft or charmcraft had a similar issue recently (I haven't searched for the bug or PR), so the solution from one of those projects can be leveraged.

rockcraft --verbose
Starting Rockcraft 0.0.1.dev1
Logging execution to '/home/developer/.cache/rockcraft/log/rockcraft-20221103-105727.504666.log'
Launching instance...
Starting Rockcraft 0.0.1.dev1
Logging execution to '/tmp/rockcraft.log'
Retrieving base ubuntu:20.04
Retrieved base ubuntu:20.04 for amd64
Extracting ubuntu:20.04
Extracted ubuntu:20.04
Initializing parts lifecycle
Executing parts lifecycle
Executing parts lifecycle: skip pull hello (already ran)
Executing action
Executed: skip pull hello (already ran)
Executing parts lifecycle: skip pull pebble (already ran)
Executing action
Executed: skip pull pebble (already ran)
Executing parts lifecycle: skip overlay hello (already ran)
Executing action
Executed: skip overlay hello (already ran)
Executing parts lifecycle: skip overlay pebble (already ran)
Executing action
Executed: skip overlay pebble (already ran)
Executing parts lifecycle: rebuild hello ('override-build' property changed)
Executing action
:: + exit 1
'override-build' in part 'hello' failed with code 1.
Review the scriptlet and make sure it's correct.
Full execution log: '/tmp/rockcraft.log'
craft-providers error: Failed to execute pack in instance.
Full execution log: '/home/developer/.cache/rockcraft/log/rockcraft-20221103-105727.504666.log'

Using the `go` plugin clobbers `/bin`

Using the go plugin recently I discovered that the /bin directory ended up clobbered in the resulting OCI image, containing just the binary I had built explicitly, and chroder

To repro:

name: traefik
version: "2.6.3"
base: ubuntu:20.04

parts:
  pebble:
    plugin: go
    source: https://github.com/canonical/pebble.git
    source-commit: 32ead042e94714625a138e61f7223c68cd6326f4

Then:

$ rockcraft pack
$ skopeo --insecure-policy copy oci-archive:traefik_2.6.3.rock docker-daemon:traefik:2.6.3
$ docker run --rm --entrypoint /usr/bin/ls -it traefik:2.6.3 -la /bin
total 11952
drwxr-xr-x 2 root root     4096 Apr  7 19:20 .
drwxr-xr-x 1 root root     4096 Apr  7 19:21 ..
-rwxr-xr-x 1 root root  2160377 Apr  7 19:20 chrorder
-rwxr-xr-x 1 root root 10065217 Apr  7 19:20 pebble

tests: spread tests fail intermittently

Overview

The spread tests are failing intermittently due to a docker error, docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

Details

@tigarmo added the retry command from snapd-testing-tools, but we're still seeing the failure.

We have two more ideas:

Increase the retry timeout
Run snap logs docker if docker never starts

Logs

source

2022-09-29T15:30:03.8353470Z 2022-09-29 15:30:03 Preparing google:ubuntu-20.04-64 (sep291529-710396)...
2022-09-29T15:30:50.7936227Z 2022-09-29 15:30:50 Error preparing google:ubuntu-18.04-64 (sep291529-710434) : 
2022-09-29T15:30:50.7937212Z -----
...
2022-09-29T15:30:50.8202707Z + snap install docker
2022-09-29T15:30:50.8203825Z docker 20.10.14 from Canonical** installed
2022-09-29T15:30:50.8204725Z + retry -n 10 --wait 2 sh -c 'docker run --rm hello-world'
2022-09-29T15:30:50.8205417Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8206155Z See 'docker run --help'.
2022-09-29T15:30:50.8206755Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8207558Z See 'docker run --help'.
2022-09-29T15:30:50.8208252Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8208953Z See 'docker run --help'.
2022-09-29T15:30:50.8209589Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8210394Z See 'docker run --help'.
2022-09-29T15:30:50.8211104Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8211805Z See 'docker run --help'.
2022-09-29T15:30:50.8212509Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8213310Z See 'docker run --help'.
2022-09-29T15:30:50.8213902Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8214654Z See 'docker run --help'.
2022-09-29T15:30:50.8215310Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8215947Z See 'docker run --help'.
2022-09-29T15:30:50.8216654Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8217364Z See 'docker run --help'.
2022-09-29T15:30:50.8217867Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8218561Z See 'docker run --help'.
2022-09-29T15:30:50.8219376Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8220001Z retry: next attempt in 2.0 second(s) (attempt 1 of 10)
2022-09-29T15:30:50.8220713Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8221349Z retry: next attempt in 2.0 second(s) (attempt 2 of 10)
2022-09-29T15:30:50.8222215Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8222833Z retry: next attempt in 2.0 second(s) (attempt 3 of 10)
2022-09-29T15:30:50.8223556Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8224459Z retry: next attempt in 2.0 second(s) (attempt 4 of 10)
2022-09-29T15:30:50.8225376Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8225965Z retry: next attempt in 2.0 second(s) (attempt 5 of 10)
2022-09-29T15:30:50.8227152Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8227788Z retry: next attempt in 2.0 second(s) (attempt 6 of 10)
2022-09-29T15:30:50.8228467Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8228917Z retry: next attempt in 2.0 second(s) (attempt 7 of 10)
2022-09-29T15:30:50.8229547Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8230011Z retry: next attempt in 2.0 second(s) (attempt 8 of 10)
2022-09-29T15:30:50.8230616Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8231159Z retry: next attempt in 2.0 second(s) (attempt 9 of 10)
2022-09-29T15:30:50.8231896Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8232682Z retry: command sh -c docker run --rm hello-world keeps failing after 10 attempts
2022-09-29T15:30:50.8233455Z -----
2022-09-29T15:30:50.8233874Z .
2022-09-29T15:30:50.8234466Z 2022-09-29 15:30:50 Discarding google:ubuntu-18.04-64 (sep291529-710434)...

`rockcraft pack` uses all available memory

OVERVIEW
For a given charm, running rockcraft pack yields to the process using all the system available memory and crashing.

STEPS TO REPRODUCE

Clone the repository
Checkout branch lte2
Run rockcraft pack (This command should crash because of an error in the rockcraft.yaml)
Run rockcraft pack again (This command should crash because of the memory qty being used)

EXECUTION OUTPUT

guillaume@thinkpad:~/PycharmProjects/magma-controller-rock$ rockcraft pack -v
Starting Rockcraft 0.0.1.dev1                                                                                                                                                                                                                          
Logging execution to '/home/guillaume/.cache/rockcraft/log/rockcraft-20221012-202543.853675.log'                                                                                                                                                       
Launching instance...                                                                                                                                                                                                                                  
Starting Rockcraft 0.0.1.dev1                                                                                                                                                                                                                          
Logging execution to '/tmp/rockcraft.log'                                                                                                                                                                                                              
Retrieving base ubuntu:18.04                                                                                                                                                                                                                           
Retrieved base ubuntu:18.04 for amd64                                                                                                                                                                                                                  
Extracting ubuntu:18.04                                                                                                                                                                                                                                
Extracted ubuntu:18.04                                                                                                                                                                                                                                 
Initializing parts lifecycle | (51.5s)                                                                                                                                                                                                                 craft-providers error: Failed to execute pack in instance.                                                                                                                                                                                             
Full execution log: '/home/guillaume/.cache/rockcraft/log/rockcraft-20221012-202543.853675.log'

LOGS

2022-10-12 20:25:43.854 Starting Rockcraft 0.0.1.dev1
2022-10-12 20:25:43.859 Using default provider 'lxd' on linux system
2022-10-12 20:25:43.923 Executing on host: lxc --project default profile show local:default
2022-10-12 20:25:43.969 Using hostname 'rockcraft-magma-orc8r-controller-5129182'
2022-10-12 20:25:43.969 Launching instance...
2022-10-12 20:25:43.969 Executing on host: lxc remote list --format=yaml
2022-10-12 20:25:44.013 Remote 'craft-com.ubuntu.cloud-buildd' already exists.
2022-10-12 20:25:44.013 Executing on host: lxc project list local: --format=yaml
2022-10-12 20:25:44.068 Set LXD instance name to 'rockcraft-magma-orc8r-controller-5129182'
2022-10-12 20:25:44.068 Executing on host: lxc --project rockcraft list local: --format=yaml
2022-10-12 20:25:44.135 Executing on host: lxc --project rockcraft list local: --format=yaml
2022-10-12 20:25:44.208 Executing on host: lxc --project rockcraft start local:rockcraft-magma-orc8r-controller-5129182
2022-10-12 20:25:44.619 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 cat /etc/os-release
2022-10-12 20:25:44.688 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf
2022-10-12 20:25:44.802 Executing on host: lxc --project rockcraft file pull local:rockcraft-magma-orc8r-controller-5129182/etc/craft-instance.conf /home/guillaume/tmp8d3rtujm.tmp-craft/tmp9qajpyea
2022-10-12 20:25:44.867 Instance is compatible with compatibility tag 'rockcraft-buildd-base-v0.0'
2022-10-12 20:25:44.867 Waiting for environment to be ready...
2022-10-12 20:25:44.867 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:45.011 systemctl is-system-running status: starting
2022-10-12 20:25:45.261 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:45.403 systemctl is-system-running status: starting
2022-10-12 20:25:45.653 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:45.751 systemctl is-system-running status: starting
2022-10-12 20:25:46.002 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:46.094 systemctl is-system-running status: starting
2022-10-12 20:25:46.345 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:46.441 systemctl is-system-running status: starting
2022-10-12 20:25:46.693 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 systemctl is-system-running
2022-10-12 20:25:46.811 Waiting for networking to be ready...
2022-10-12 20:25:46.811 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io
2022-10-12 20:25:46.921 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 snap unset system proxy.http
2022-10-12 20:25:47.174 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 snap unset system proxy.https
2022-10-12 20:25:47.416 Installing snap 'rockcraft' with channel=None and classic=True
2022-10-12 20:25:47.416 Installing snap 'rockcraft' from host (classic=True)
2022-10-12 20:25:47.433 Executing in container: lxc --project rockcraft exec local:rockcraft-magma-orc8r-controller-5129182 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin ROCKCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf
2022-10-12 20:25:47.517 Executing on host: lxc --project rockcraft file pull local:rockcraft-magma-orc8r-controller-5129182/etc/craft-instance.conf /home/guillaume/tmp78za9emn.tmp-craft/tmpytjlfxix
2022-10-12 20:25:47.565 Revisions found: host='383', target='383'
2022-10-12 20:25:47.565 Skipping snap injection: target is already up-to-date with revision on host
2022-10-12 20:25:47.565 Emitter: Pausing control of the terminal
2022-10-12 20:27:01.925 Emitter: Resuming control of the terminal
2022-10-12 20:27:04.488 craft-providers error: Failed to execute pack in instance.
2022-10-12 20:27:04.488 Full execution log: '/home/guillaume/.cache/rockcraft/log/rockcraft-20221012-202543.853675.log'

Stage-packages fails at unpack_stage_packages when installing some packages

name: test
version: 1.0
base: ubuntu:20.04

parts:
  part1:
    plugin: nil
    stage-packages:
    - ca-certificates

this example rockcraft.yaml fails with the following error while executing rockcraft pack (in the unpack_stage_packages function):

ValueError: '/root/parts/part1/install/etc/ssl/openssl.cnf' does not start with '/root/parts/part1/install/usr/lib/ssl'

See the full log rockcraft-20220422-181812.655447.log.

chisel is hardcoded to ubuntu-22.04 release

in https://github.com/canonical/craft-parts/blob/ba20b6f183fa5f3d6bd9a07737988afd1a901e39/craft_parts/packages/deb.py#L701 Chisel is hardcoded to run with 22.04 only.

The right behavior should be to inherit the release from the build environment (i.e. if build-base is ubuntu-22.10, then chisel must run with --release ubuntu-22.10)

Add support for `craftctl get version`

It would be useful to be able to grab the version of the ROCK being built as part of the various scripted sections in rockcraft.yaml. Something like:

version: "2.4.5"

# ...

parts:
  some-part:
    plugin: nil
    override-pull: |
      wget -O /tmp/product.tar.gz "https://releases.megacoolvendor.com/product/$(craftctl get version)_linux.tar.gz"

canonical / rockcraft Goto Github PK

rockcraft's Introduction

Rockcraft

Purpose

Installing

Documentation

Testing

rockcraft's People

Contributors

Stargazers

Watchers

Forkers

rockcraft's Issues

Spec

Overview

Description

Reference

Description

Additional information

Problem

How to reproduce

Origin of the problem

Issue

Error

Reproducer

Output

Proposal

Overview

Details

Logs

Recommend Projects

Recommend Topics

Recommend Org