What needs to get done

1. craft-application should use <appname>_PARALLEL_BUILD_COUNT.
2. If it is not set, use CRAFT_PARALLEL_BUILD_COUNT
3. If it is not set, take the lesser of <appname>_MAX_PARALLEL_BUILD_COUNT and the total number of CPUs on the system.
   • If <appname>_MAX_PARALLEL_BUILD_COUNT is not set, then use CRAFT_MAX_PARALLEL_BUILD_COUNT
4. If neither MAX envvars are set and the number of CPUs on the system cannot be determined, do not provide a build count to craft-parts (which will use its default of 1).

Why it needs to get done

1. To allow setting the number of parallel build jobs
2. To align with the design proposed in canonical/snapcraft#4537 and canonical/craft-parts#631

What needs to get done

Define and document:

1. When (and if) a Project gets created
2. How project mutation works (difficult case: LifecycleService writing adopt-info. See: https://github.com/canonical/craft-application/pull/217/files#r1496586665)
3. How commands and services should interact with Projects

Some questions to consider:

• Should a Project be frozen? (Pros/cons)
• What should/should not have access to a project?
• How do we handle commands where project access is optional? (e.g. remote-build)

Why it needs to get done

This is loosely defined currently and has worked fine so far, but we're reaching edge cases that make us need a

Bug Description

When a cache directory already exists, pyxdg throws an exception that we don't catch

To Reproduce

See: https://matrix.to/#/!LhFxJIPEcCacgdMghH:ubuntu.com/$kp5AKrB1uWyHgrnl5nFXH-plPE2PXgMpu_YCZLj56pw

part yaml

No response

Relevant log output

2024-01-02 11:27:02.019 Starting charmcraft
2024-01-02 11:27:02.022 Loading project file '/home/runner/work/istio-operators/istio-operators/charms/istio-gateway/charmcraft.yaml'
2024-01-02 11:27:02.023 Setting target machine to x86_64
2024-01-02 11:27:02.026 Running charmcraft:ubuntu-20.04-amd64 in amd64 instance...
2024-01-02 11:27:02.027 Preparing managed instance 'charmcraft-istio-gateway-on-amd64-for-amd64-779228'
2024-01-02 11:27:02.027 Using hostname 'charmcraft-istio-gateway-on-amd64-for-amd64-779228'
2024-01-02 11:27:02.027 Using default provider
2024-01-02 11:27:02.027 Launching managed ubuntu 20.04 instance...
2024-01-02 11:27:02.027 Executing on host: lxc remote list --format=yaml
2024-01-02 11:27:02.063 Executing on host: lxc remote add craft-com.ubuntu.cloud-buildd https://cloud-images.ubuntu.com/buildd/releases --protocol=simplestreams
2024-01-02 11:27:02.098 Remote 'craft-com.ubuntu.cloud-buildd' was successfully added.
2024-01-02 11:27:02.099 Executing on host: lxc project list local: --format=yaml
2024-01-02 11:27:02.139 Executing on host: lxc --project charmcraft project create local:charmcraft
2024-01-02 11:27:02.180 Executing on host: lxc --project default profile show local:default
2024-01-02 11:27:02.233 Executing on host: lxc --project charmcraft profile edit local:default
2024-01-02 11:27:02.285 Set LXD instance name to 'charmcraft-istio-gateway-on-amd64-for-amd64-779228'
2024-01-02 11:27:02.286 Checking for instance 'charmcraft-istio-gateway-on-amd64-for-amd64-779228' in project 'charmcraft' in remote 'local'
2024-01-02 11:27:02.286 Executing on host: lxc --project charmcraft list local: --format=yaml
2024-01-02 11:27:02.331 Instance 'charmcraft-istio-gateway-on-amd64-for-amd64-779228' does not exist.
2024-01-02 11:27:02.331 Set LXD instance name to 'base-instance-charmcraft-buildd-base-v4--b6028d8e72f849bc5897'
2024-01-02 11:27:02.331 Checking for base instance 'base-instance-charmcraft-buildd-base-v4--b6028d8e72f849bc5897' in project 'charmcraft' in remote 'local'
2024-01-02 11:27:02.332 Executing on host: lxc --project charmcraft list local: --format=yaml
2024-01-02 11:27:02.376 Base instance 'base-instance-charmcraft-buildd-base-v4--b6028d8e72f849bc5897' does not exist.
2024-01-02 11:27:02.376 Creating new instance from remote
2024-01-02 11:27:02.376 Creating new base instance from remote
2024-01-02 11:27:02.376 Creating new base instance from image 'core20' from remote 'craft-com.ubuntu.cloud-buildd'
2024-01-02 11:27:02.376 Executing on host: lxc --project charmcraft info local:
2024-01-02 11:27:02.451 Executing on host: lxc --project charmcraft launch craft-com.ubuntu.cloud-buildd:core20 local:base-instance-charmcraft-buildd-base-v4--b6028d8e72f849bc5897 --config 'raw.idmap=both 1001 0' --config security.syscalls.intercept.mknod=true --config user.craft_providers.status=STARTING --config user.craft_providers.timer=2024-01-02T11:27:02.451795+00:00 --config user.craft_providers.pid=16830
2024-01-02 11:27:02.019 Starting charmcraft
2024-01-02 11:27:02.020 charmcraft internal error: FileExistsError(17, 'File exists')
2024-01-02 11:27:02.022 Traceback (most recent call last):
2024-01-02 11:27:02.022   File "/snap/charmcraft/2180/lib/python3.10/site-packages/craft_application/application.py", line 375, in run
2024-01-02 11:27:02.022     self._configure_services(platform, build_for)
2024-01-02 11:27:02.022   File "/snap/charmcraft/2180/lib/python3.10/site-packages/charmcraft/application/main.py", line 107, in _configure_services
2024-01-02 11:27:02.022     cache_dir=self.cache_dir,
2024-01-02 11:27:02.022   File "/snap/charmcraft/2180/lib/python3.10/site-packages/craft_application/application.py", line 185, in cache_dir
2024-01-02 11:27:02.022     return save_cache_path(self.app.name)  # type: ignore[no-any-return]
2024-01-02 11:27:02.022   File "/snap/charmcraft/2180/lib/python3.10/site-packages/xdg/BaseDirectory.py", line 84, in save_cache_path
2024-01-02 11:27:02.022     os.makedirs(path)
2024-01-02 11:27:02.022   File "/snap/charmcraft/2180/bin/../usr/lib/python3.10/os.py", line 225, in makedirs
2024-01-02 11:27:02.022     mkdir(name, mode)
2024-01-02 11:27:02.022 FileExistsError: [Errno 17] File exists: '/home/runner/snap/charmcraft/common/cache/charmcraft'
2024-01-02 11:27:02.022 Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20240102-112702.018439.log'

Bug Description

AppMetadata expects version to always come from importlib.metadata, but that doesn't work if the application hasn't been pip install'd.

A better form of this would be to :

1. Check <app_name>.__version__ for a version string
2. Check importlib if that doesn't work
3. Fall back to a default string (e.g. dev)

To Reproduce

from craft_application import AppMetadata

AppMetadata(name="something that isn't installed!")

Relevant log output

https://pastebin.ubuntu.com/p/9pBDb2b5zP/

What needs to get done

We need some developer documentation for how to use craft-application

Why it needs to get done

Right now moving an app to craft-application requires a lot of looking at source code

What needs to get done

Provide an easy way for the app developer to override the default command groups. (e.g. either by making the default command group getters Application class methods or by offering a way to provide override command groups on __init__)

See: canonical/charmcraft#1340 (comment)

Why it needs to get done

Right now the lifecycle and other command groups are created by default using craft-application command groups. If you need to override those groups (like Charmcraft does), it's possible but not particularly pretty.

Bug Description

Project is loaded through _configure_services even when the command does not need it

To Reproduce

Create a command and run it without the yaml representing the project

part yaml

N/A

Relevant log output

Traceback (most recent call last):
  File "/snap/appcraft/x38/bin/appcraft", line 8, in <module>
    sys.exit(main())
  File "/snap/appcraft/x38/lib/python3.10/site-packages/appcraft/cli.py", line 33, in main
    return app.run()
  File "/snap/appcraft/x38/lib/python3.10/site-packages/craft_application/application.py", line 227, in run
    self._configure_services()
  File "/snap/appcraft/x38/lib/python3.10/site-packages/appcraft/application.py", line 33, in _configure_services
    super()._configure_services()
  File "/snap/appcraft/x38/lib/python3.10/site-packages/craft_application/application.py", line 127, in _configure_services
    self.services.project = self.project
  File "/snap/appcraft/x38/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/snap/appcraft/x38/lib/python3.10/site-packages/craft_application/application.py", line 139, in project
    return self.app.ProjectClass.from_yaml_file(project_file)
  File "/snap/appcraft/x38/lib/python3.10/site-packages/craft_application/models/base.py", line 74, in from_yaml_file
    with path.open() as file:
  File "/snap/appcraft/x38/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/home/sergiusens/appcraft.yaml'

What needs to get done

It could be handy if a recurring repo could be mentioned only once, for example:

definitions:  
  kubeflow-source: &kubeflow-source  
    source: https://github.com/kubeflow/kubeflow  
    source-type: git  
    source-tag: v1.7-branch  # upstream branch  
    source-depth: 1

parts:  
  backend:  
    plugin: nil  
    <<: *kubeflow-source

Currently this fails with:

Bad rockcraft.yaml content:
- extra field 'definitions' not permitted in top-level configuration

Why it needs to get done

If the same repo+tag are used multiple times, it could be easier to maintain and easier to read.

What needs to get done

Fail early with a good error message if the application is running a lifecycle command in destructive mode and the build plan is incompatible with the running environment (for example, running in Arch Linux)

Why it needs to get done

The current behavior is to... pretend it worked? It might actually "work" depending on the project, but at best the built artifact will contain nonsense.

Once craft-providers has a release supporting 3.12: canonical/craft-providers#317

What needs to get done

canonical/rockcraft@2b711d6 (#412)

craft-application should pass through those environment variables (and maybe a few more) automatically.

Why it needs to get done

Every app is doing this on its own.

What needs to get done

Revert #176

Why it needs to get done

The latest changes to LXD have temporarily rendered it unusable for all non-Ubuntu systems. We'll need to wait for it to be fixed to re-add these tests.

What needs to get done

Right now each AppCommand class generates a full command for its managed run. Instead, the Application should take care of the application command and global arguments and the Command should generate just the command-specific parameters.

Why it needs to get done

Various applications will have different global arguments and we shouldn't need to modify the Commands for those.

Bug Description

When using craft-provider build instances, the command " clean" should remove the build instance. Instead, it currently launches into the instance and then cleans the lifecycle inside it.

To Reproduce

Run " clean".

part yaml

No response

Relevant log output

none

What needs to get done

Add a basic linter service that provides an extensible linter framework

Why it needs to get done

Both snapcraft and charmcraft implement linters in pretty similar ways. We should provide a linter service to take care of most of that so all craft applications can use it.

What needs to get done

Make this something we can extend:

Why it needs to get done

Snapcraft is overriding it :-/

canonical/snapcraft#4444 (comment)

What needs to get done

Add an overrideable method to Application to resolve the path to the project YAML file.

Why it needs to get done

snapcraft.yaml can be in several different locations, and we shouldn't have to override the project property for that.

What needs to get done

The lifecycle service should get the parallel build size like snapcraft does:

https://github.com/snapcore/snapcraft/blob/main/snapcraft/utils.py#L211

Why it needs to get done

feature parity

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Found renovate config warnings
• WARN: Use matchDepPrefixes instead of matchPackagePrefixes
• WARN: Use excludeDepPrefixes instead of excludePackagePrefixes

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency tox-gh to v1.3.2
• chore(deps): update dependency pygit2 to >=1.15.1,<1.16.0
• chore(deps): update dependency requests to <2.33
• chore(deps): update dependency setuptools to v70.3.0
• chore(deps): update development dependencies (non-major) (dev/coverage, dev/pytest, dev/requests, lint/codespell, types/mypy, types/pyright)
• chore(deps): update pydantic etc. (major) (pydantic, pydantic-yaml)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

What needs to get done

• find snapcraft.yaml that can exist in ., snap/, etc
  • these are relative to (and inside) the project path
  • raise an error if multiple project files exist - check with @sergiusens if this should be an error or warning
• asset directory should be a member of the SnapcraftApplication class

Why it needs to get done

To support snapcraft rebasing onto craft-application.

What needs to get done

Allow the dictionary sent to dispatcher.load_command in Application.run to be overridden without overriding the whole method.

See: canonical/charmcraft#1340 (comment)

Why it needs to get done

Currently we have to override the whole method, which is suboptimal.

What needs to get done

Add --destructive-mode option for in-host builds.

Why it needs to get done

For feature-parity with the existing applications

Bug Description

If you don't have pygit2 installed in your environment, starting up a craft-application app will crash.

To Reproduce

1. Clone rockcraft's craft-application feature branch
2. Set up your virtual environment without pygit2
3. Run tests

It's expected that remote build related tests will fail, but other tests should not.

part yaml

No response

Relevant log output

n/a

Bug Description

The project has summary and description as optional keys, these should be mandatory for all Craft Applications. The packed output risks not having a summary and description

To Reproduce

Create a yaml to load with project without a summary and description

part yaml

name: foo
version: bar
# summary: foobar
# description: foobarbaz

Relevant log output

n/a

What needs to get done

Print out the app version here: https://github.com/canonical/craft-application/blob/main/craft_application/application.py#L308

Why it needs to get done

easier debugging

The TODO item here is to create the relevant tasks.

Every time we raise a CraftError of any sort, it should have a docs_url attached to it. This means several things:

1. We need documentation about each error case, why it could occur, and what we expect to do about it.
2. We need to find the raise statements and attach the relevant documentation.

A future goal is to provide a way to rewrite the docs_url link for the specific application.

What needs to get done

craft-application current has the command-line options for --shell and --shell-after but the switches aren't actually used. Also we're missing --debug (both the option and the implementation).

Why it needs to get done

For feature-parity with the existing applications.

What needs to get done

.

Why it needs to get done

.

What needs to get done

Add some basic spread tests running with a fake "testcraft" application

Why it needs to get done

The main goal is to bridge the gap between integration tests in craft-application (which are useful but can never reproduce the "production" environment of an app) and the spread tests of one of our applications (which are the ultimate source of "right" and "wrong" but are very expensive to run).

Bug Description

When running an invalid command, we get a useful message:

$ rockcraft oakfdpao
Usage: rockcraft [options] command [args]...
Try 'rockcraft -h' for help.

Error: no such command 'oakfdpao

However, when using a command line with a valid command but an invalid parameter, we get a message as if this were an internal error in the application:

$ rockcraft pack --oakfdpao
rockcraft internal error: ArgumentParsingError("Usage: rockcraft [options] command [args]...\nTry 'rockcraft pack -h' for help.\n\nError: unrecognized arguments: --oakfdpao\n")           
Full execution log: '/home/tiago/.local/state/rockcraft/log/rockcraft-20231002-190658.059266.log'

To Reproduce

Run <appname> pack --invalid-param

part yaml

n/a

Relevant log output

rockcraft internal error: ArgumentParsingError("Usage: rockcraft [options] command [args]...\nTry 'rockcraft pack -h' for help.\n\nError: unrecognized arguments: --oakfdpao\n")           
Full execution log: '/home/tiago/.local/state/rockcraft/log/rockcraft-20231002-190658.059266.log'

What needs to get done

Implement support for build-time, host-provided secrets.

Needs canonical/craft-cli#184

Why it needs to get done

This is necessary to support things like password-accessed Git repositories, enable Ubuntu Pro, etc.

What needs to get done

Create a fake testcraft app to test the application and add spread tests using it.

Why it needs to get done

Testcraft can become a good place to test new features of craft-application, etc.

Bug Description

The original design is for parts, stage and prime to live inside the managed environment.

To Reproduce

Create an app using Craft Application

part yaml

N/A

Relevant log output

N/A

Bug Description

Calling Application.add_command_group() to add a CommandGroup whose name matches one the default groups causes the groups to stay duplicated, which affects the help output.

To Reproduce

In this example, rockcraft has an init command in the Other group:

$ rockcraft --help
Usage:
    rockcraft [help] <command>

Summary:    A tool to create OCI images

Global options:
       -h, --help:  Show this help message and exit
    -v, --verbose:  Show debug information and be more verbose
      -q, --quiet:  Only show warnings and errors, not progress
      --verbosity:  Set the verbosity level to 'quiet', 'brief',
                    'verbose', 'debug' or 'trace'
    -V, --version:  Show the application version and exit

Starter commands:
          version:  Show the application version and exit

Commands can be classified as follows:
       Extensions:  expand-extensions, list-extensions
        Lifecycle:  build, clean, pack, prime, pull, stage
            Other:  version
            Other:  init

For more information about a command, run 'rockcraft help <command>'.
For a summary of all commands, run 'rockcraft help --all'.

See that there are two rows for "Other:"

part yaml

n/a

Relevant log output

n/a

See this comment thread.

The modifications made to the YAML loader may be more than is needed with the latest pyyaml, but I'm not confident enough to say so. If the questioned except clause is needed, we should have a test for it that has it raise that error, as right now the loader fails with the same issue before it gets to the dict constructor.

Bug Description

The two fields summary and description should be mandatory.

To Reproduce

n/a

part yaml

n/a

Relevant log output

n/a

What needs to get done

Create a spec + prototype for supporting build-time secrets in the applications. The idea is to follow spread's lead: For example, a declaration like:

parts:
    part1:
        source: https://user:$(HOST: GIT_PASS)@git.com
        source-type: git
        ....

will cause the application to evaluate the GIT_PASS environment variable on the host, replacing its value in the source line.

Why it needs to get done

Users need a way to handle passwords, api keys, etc without recording them in the application's project files.

Bug Description

Some problems related to logging errors:

1. "Internal" errors are not being printed with tracebacks in the logfile
2. Craft-parts lifecycle errors are being reported as "Unknown", even though we know they are a PartsError.

To Reproduce

Get an error, check the log

part yaml

parts:
  my-part:
    plugin: nil
    override-build: |
      exit 1

Relevant log output

...
2023-10-24 16:03:25.181 add action my-part:Step.STAGE(ActionType.RUN)
2023-10-24 16:03:25.181 process my-part:Step.PRIME
2023-10-24 16:03:25.181 add action my-part:Step.PRIME(ActionType.RUN)
2023-10-24 16:03:25.181 testcraft internal error: RuntimeError("Parts processing internal error: I'm a regular RuntimeError")
2023-10-24 16:03:25.181 Full execution log: '/home/tiago/.local/state/testcraft/log/testcraft-20231024-160325.177049.log'

Bug Description

Although running testcraft --help works, testcraft version --help results in application crash.

To Reproduce

Create a simple test application using craft-application, then invoke it using --help on a command.

$ snarfcraft version --help
Internal error while loading snarfcraft: TypeError("'NoneType' object is not subscriptable")                                                                                          

part yaml

N/A

Relevant log output

Traceback (most recent call last):
  File "/home/claudio/.venv/snarfcraft/bin/snarfcraft", line 8, in <module>
    sys.exit(cli.main())
  File "/home/claudio/starcraft/application-test/snarfcraft/cli.py", line 32, in main
    return app.run()
  File "/home/claudio/.venv/snarfcraft/lib/python3.10/site-packages/craft_application/application.py", line 253, in run
    dispatcher = self._get_dispatcher()
  File "/home/claudio/.venv/snarfcraft/lib/python3.10/site-packages/craft_application/application.py", line 219, in _get_dispatcher
    global_args = dispatcher.pre_parse_args(sys.argv[1:])
  File "/home/claudio/.venv/snarfcraft/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 431, in pre_parse_args
    help_text = self._get_requested_help(filtered_sysargs)
  File "/home/claudio/.venv/snarfcraft/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 315, in _get_requested_help
    command = cmd_class(None)
  File "/home/claudio/.venv/snarfcraft/lib/python3.10/site-packages/craft_application/commands/base.py", line 41, in __init__
    self._app: application.AppMetadata = config["app"]
TypeError: 'NoneType' object is not subscriptable

What needs to get done

Every existing application has some version of support for multiple builds based on a "build matrix". This matrix needs craft-application support because it ends up informing the entire rest of the process. For example:

• The build base might depend of the current build configuration;
• The instances created by craft-providers need the current build configuration (specially in their names);
• The parts lifecycle need the current build config to inform the target/build-for variables and such.

Why it needs to get done

For feature-parity with the existing applications.

Bug Description

Currently a run that uses a build instance is not fetching its logs after the instance is finished. As a result, the "outer" log ends at the point where the host application executes the instance's one.

To Reproduce

Run " pack" and check the logs afterwards.

part yaml

No response

Relevant log output

none

What needs to get done

Move the business logic for this:

into its own utility function that can either output to a stream or to a string.

Why it needs to get done

We need this in a few apps, might as well have it consolidated.

What needs to get done

Provide an easy way to get relevant paths, at least in managed mode.

Why it needs to get done

We're doing this with pathlib.Path("/root/project") in a few places and we should make it easier to keep consistent.

What needs to get done

Pydantic 2's ErrorDict includes an input key that holds the value that failed to validate. Once we update from Pydantic 1 we should use this new field in our validation failure messages.
Ref: #156

Why it needs to get done

To provide clearer messages!

What needs to get done

Remove get_build_plan from a Project since we're using a BuildPlanner

Why it needs to get done

Cleanup

Bug Description

Errors generated by the craft-libraries are being silently discarded when sending the error to the emitter. The result is a single " internal error: " in the cli and log

To Reproduce

Run any project that fails to build in craft-parts (or through a programming error).

part yaml

No response

Relevant log output

none

Bug Description

The current code that loops over the build plan to launch managed runs just no-ops if the plan is empty (like if all build infos were filtered out). I can't think of a case where that shouldn't be an error instead, specially since we've decided that trying to build an empty plan in destructive mode is an error.

To Reproduce

Create a project with a build plan that will get totally filtered out (eg only builds on arm64 and you're on amd64)

part yaml

.

Relevant log output

.

Bug Description

Craft-application uses the same base instance name. LXD requires every instance to have a unique name, even if they are in different projects.

To make a unique name, the base instance name should include the name of app.

This was an easy error to produce because the compatibility_tag parameter is optional. See canonical/craft-providers#454 for details.

To Reproduce

Run 2 different craft-applications (i.e. charmcraft feature branch and rockcraft)

part yaml

No response

Relevant log output

Error: Failed start validation for device "eth0": Instance DNS name "base-instance-buildd-base-v3-craft-comub-523148486bd0c6cc722c" already used on network

What needs to get done

Take an integer value (in AppMetadata?) that provides a per-application addendum to the craft-providers compatibility tag, defaulting to zero.

Set the compatibility_tag keyword argument for the provider base to include that value.

Why it needs to get done

Charmcraft has to override ProviderService and get_base because it adds custom compatibility tags: https://github.com/canonical/charmcraft/pull/1330/files/84ffcdd7a0731a17fcae83a889ed7e2143f7cfb6#diff-a32511469d8ae4dd52bf0ce115517e3cc980f46568ae386be734af9f2978b7c7

This shouldn't be necessary.

Bug Description

When running a craft tool in --verbose (or higher, probably), the lifecycle messages like Pulling mypart get duplicated.

To Reproduce

Run any lifecycle-using command with verbose verbosity, e.g. appcraft build -v.

part yaml

n/a

Relevant log output

Initialising lifecycle                                                                                                                          
Installing build-packages                                                                                                                       
Pulling my-part                                                                                                                                 
Pulling my-part                                                                                                                                 
Pulling pebble                                                                                                                                  
Pulling pebble                                                                                                                                  
Overlaying my-part                                                                                                                              
Overlaying my-part                                                                                                                              
Overlaying pebble                                                                                                                               
Overlaying pebble                                                                                                                               
Building my-part                                                                                                                                
Building my-part