edgelesssys / constellation Goto Github PK

Issue description

When creating a Constellation on host systems that are not Linux or not amd64, constellation config generate would generate a broken microservice version:

Problems validating config file:
    image specifies an invalid version: configured version (v2.7.0) does not adhere to SemVer syntax
    microserviceVersion specifies an invalid version: configured version (v) does not adhere to SemVer syntax
Fix the invalid entries or generate a new configuration using `constellation config generate`
Error: invalid configuration

constellation-conf.yaml:

version: v2 # Schema version of this configuration file.
image: v2.7.0 # Machine image version used to create Constellation nodes.
[...]
microserviceVersion: v # Microservice version to be installed into the cluster. Defaults to the version of the CLI.
[...]

To reproduce

Steps to reproduce the behavior:

1. Download the Constellation CLI in version 2.7.0 for a platform that is not linux/amd64

   wget https://github.com/edgelesssys/constellation/releases/download/v2.7.0/constellation-darwin-arm64
   chmod +x constellation-darwin-arm64
   

2. Generate a config file for any cloud provider: constellation config generate gcp
3. Look a the microserviceVersion field in the generated config file.
4. You can spot the issue by running constellation version, where the version field only shows the kind of build (Enterprise) but no actual version information

   Version:	 (Enterprise build; see documentation for license agreement)
   GitCommit:	d56b0ef75f5a5019f608f4907d1cbcea4b749799
   GitTreeState:	clean
   BuildDate:	1970-01-01T00:00:00Z
   GoVersion:	go1.20.2 X:nocoverageredesign
   Compiler:	bazel/gc
   Platform:	linux/arm64
   

Environment

• constellation version: v2.7.0
• Host platform that is not linux/amd64

Expected behavior

The config file should be generated correctly and it should be possible to create a Constellation with it.

Mitigation

Use the Constellation cli on linux/amd64.

Issue description

You may see this warning when initializing a cluster. This neither affects security nor functionality. Please ignore the warning.

Background: the CLI v2.6.0 includes a wrong value for the optional measurement 5. This will be corrected with the next release.

Environment

• constellation version: v2.6.0

Expected behavior

No warning should be printed on init.

Additional info / screenshot

Use case

would be great to have the possibility to use discussion in this repository.
At this time, there is only the topic @msanft has converted from issue to discussion
#2785 => #2786
but it is not possible to start a new discussion

Describe your solution

enable users to start discussions.
In this repository, at https://github.com/edgelesssys/constellation/discussions
the button is missing (disabled):

When enabled, it should look like this:

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Issue description

The cert-manager installation via helm times out during constellation init.
This happens in about 1/3 of initializations.

To mitigate this issue, terminate, re-create and re-initialize the cluster:

constellation terminate
# Either re-use or remove the "constellation-mastersecret.json"
constellation create
constellation init

To reproduce

Steps to reproduce the behavior:

$ constellation create --control-plane-nodes 3 --worker-nodes 2
The following Constellation cluster will be created:
3 control-plane nodes of type n2d-standard-4 will be created.
2 worker nodes of type n2d-standard-4 will be created.
Do you want to create this cluster? [y/n]: y
Your Constellation cluster was created successfully.
$ constellation init
Using community license.
For details, see https://docs.edgeless.systems/constellation/overview/license
Your Constellation master secret was successfully written to ./constellation-mastersecret.json
Warning: Encountered untrusted PCR value at index 5
Cluster initialization failed. This error is not recoverable.
Terminate your cluster and try again.
Error: init call: rpc error: code = Internal desc = initializing cluster: installing cert-manager: helm install: context deadline exceeded

Environment

• constellation version: v2.6.0

Expected behavior

The installation of the cert-manager should not time out and the initialization should succeed.

Additional info / screenshot

Feature

Deploy the cluster-managing and Constellation's own microservices via helm charts.
Facilitates orchestrational tasks like updating these services.

• Autoscaler
• Operators
• Cloud Controller Manager
• Coud Node Manager
• GCP CSI
• Azure CSI
• Key Management Service
• Operator Lifecycle Manager
• Join Service
• Verification Service

Hello,

I just came across this project, and I'm quite interested in testing it out, but reading the docs I often see Azure or GCP as a reference and that I have to do things where there IAM service before getting started. As I don't do any business with these companies out of various reasons, I would like to test Constellation outside these public clouds, e.g. at Hetzner or at home or basically anywhere else.

Kindly asking for feedback.

Issue description

When running upgrade apply the follow error message is shown: Error: upgrading NodeVersion: expected NodeVersion to contain /CommunityGalleries/ConstellationCVM-728bd310-e898-4450-a1ed-21cf2fb0d735/Images/main-nightly/Versions/2023.0406.182050, got /communityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/images/constellation/versions/2.6.0

To reproduce

Steps to reproduce the behavior:
Any config that will trigger a Kubernetes upgrade, but no image upgrade will print the above error. An example could be:

1. Setup a Constellation cluster with Constellation CLI v2.6.0 and Kubernetes version v1.25.7
2. Change configured Kubernetes version to v1.26.2
3. Change configured image version to /CommunityGalleries/ConstellationCVM-728bd310-e898-4450-a1ed-21cf2fb0d735/Images/main-nightly/Versions/2023.0406.182050
4. Run upgrade apply

Expected behavior

This error is not shown.

Mitigation

The error message itself is harmless. It does not necessarily indicate that the upgrade failed. To check if the upgrade succeeded run constellation status and compare the reported values with your local config.
The error is fixed in #1630.

Issue description

When running certain commands that results in an error, such as using the wrong credentials in constellation-conf.yaml or a CSP credentials with not enough privileges.

To reproduce

Steps to reproduce the behavior:

1. Run constellation config generate
2. Add your credentials in constellation-conf.yaml file (maybe falsify your credentials to trigger an error when exeucting other commands after)
3. Run constellation create --control-plane-nodes 1 --worker-nodes 1 --name test -y to try to create a cluster
4. Notice how there is an error in the STDOUT but not in STDERR. (You may need to monitor both STDOUT and STDERR beforehand to properly replicate the bug)
5. This is a sample of the error that could be shown:

An error occurred: PUT https://management.azure.com/subscriptions/6b6a1f27-55c1-4b1d-969b-60a3c9eebe64/resourceGroups/azure-customertwo-az-sandbox-84v4y/providers/Microsoft.Insights/components/constellation-insights-nm4rV
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
  "error": {
    "code": "AuthorizationFailed",
    "message": "The client '<ID>' with object id '<ID>' does not have authorization to perform action 'Microsoft.Insights/components/write' over scope '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Insights/components/constellation-insights-nm4rV' or the scope is invalid. If access was recently granted, please refresh your credentials."
  }
}
--------------------------------------------------------------------------------

Attempting to roll back.

Environment

• constellation version:
• constellation-conf.yaml
  • (make sure to remove sensitive information, e.g., yq e 'del(.provider.*.project)' constellation-conf.yaml)
• VM type used to run Constellation.

Expected behavior

As a developer, I expect a failure to output both on STDOUT and STDERR so I can integrate and handle failures more easily

Issue description

constellation mini up times out during creation.

To reproduce

Steps to reproduce the behavior:

1. Download the CLI
2. Execute constellation mini up --debug

2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:121       Checked arch and os
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:126       Checked that /dev/kvm exists
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:134       Checked CPU cores - there are 8
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:152       Scanned for available memory
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:160       Checked available memory, you have 31GB available
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:170       Checked for free space available, you have 25GB available
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:177       Preparing configuration
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:202       Configuration path is ""
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:224       Prepared configuration
2023-04-06T11:36:01Z    DEBUG   cmd/miniup.go:231       Creating mini cluster
An error occurred: exit status 1

Error: couldn't retrieve IP address of domain id: 29962dc6-14e7-4524-89d9-320f90ce2c3b. Please check following: 
1) is the domain running proplerly? 
2) has the network interface an IP address? 
3) Networking issues on your libvirt setup? 
 4) is DHCP enabled on this Domain's network? 
5) if you use bridge network, the domain should have the pkg qemu-agent installed 
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup 
 timeout while waiting for state to become 'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)

  with module.worker.libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 15, in resource "libvirt_domain" "instance_group":
  15: resource "libvirt_domain" "instance_group" {


Attempting to roll back.
Rollback succeeded.
Error: creating cluster: exit status 1

Error: couldn't retrieve IP address of domain id: 29962dc6-14e7-4524-89d9-320f90ce2c3b. Please check following: 
1) is the domain running proplerly? 
2) has the network interface an IP address? 
3) Networking issues on your libvirt setup? 
 4) is DHCP enabled on this Domain's network? 
5) if you use bridge network, the domain should have the pkg qemu-agent installed 
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup 
 timeout while waiting for state to become 'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)

  with module.worker.libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 15, in resource "libvirt_domain" "instance_group":
  15: resource "libvirt_domain" "instance_group" {

Environment

• constellation version: 2.7.0-pre.0.20230405160928-1c03b066a61b
• constellation-conf.yaml

version: v2 # Schema version of this configuration file.
image: v2.6.0 # Machine image version used to create Constellation nodes.
name: mini # Name of the cluster.
stateDiskSizeGB: 8 # Size (in GB) of a node's disk to store the non-volatile state.
kubernetesVersion: v1.25.8 # Kubernetes version to be installed into the cluster.
microserviceVersion: v2.7.0-pre.0.20230405160928-1c03b066a61b # Microservice version to be installed into the cluster. Defaults to the version of the CLI.
debugCluster: false # DON'T USE IN PRODUCTION: enable debug mode and use debug images. For usage, see: https://github.com/edgelesssys/constellation/blob/main/debugd/README.md
# Supported cloud providers and their specific configurations.
provider:
    # Configuration for QEMU as provider.
    qemu:
        imageFormat: raw # Format of the image to use for the VMs. Should be either qcow2 or raw.
        vcpus: 2 # vCPU count for the VMs.
        memory: 2048 # Amount of memory per instance (MiB).
        metadataAPIServer: ghcr.io/edgelesssys/constellation/qemu-metadata-api:v2.7.0-pre.0.20230330151913-6a2c9792e0ce@sha256:8283f9606366beaf05142aeef09a905085bc7cde071f43b43290a7f087994264 # Container image to use for the QEMU metadata server.
        libvirtSocket: "" # Libvirt connection URI. Leave empty to start a libvirt instance in Docker.
        libvirtContainerImage: ghcr.io/edgelesssys/constellation/libvirt:v2.7.0-pre.0.20230330151913-6a2c9792e0ce@sha256:56d218cc501d471d25f6a6da940db48a008a040bc26dea1fe8a61edf9ca7ce73 # Container image to use for launching a containerized libvirt daemon. Only relevant if `libvirtSocket = ""`.
        nvram: production # NVRAM template to be used for secure boot. Can be sentinel value "production", "testing" or a path to a custom NVRAM template
        firmware: "" # Path to the OVMF firmware. Leave empty for auto selection.
        # Measurement used to enable measured boot.
        measurements:
            4:
                expected: dfd62a251a234d2eccdbb4659e4990c330f3e4a4456f7274c769ad43c174c7c4
                warnOnly: false
            8:
                expected: "0000000000000000000000000000000000000000000000000000000000000000"
                warnOnly: false
            9:
                expected: a13d79910d9d98480c0d5c3f197d383d5d26895e350225e219bf286a7402e780
                warnOnly: false
            11:
                expected: "0000000000000000000000000000000000000000000000000000000000000000"
                warnOnly: false
            12:
                expected: c2b1e24a8ff734ea9546622da8ba438c6799be6b6cb4e6593d9411e9ea084c0c
                warnOnly: false
            13:
                expected: "0000000000000000000000000000000000000000000000000000000000000000"
                warnOnly: false
            15:
                expected: "0000000000000000000000000000000000000000000000000000000000000000"
                warnOnly: false

• VM type used to run Constellation.
  Miniconstellation was started on an Azure VM of type Standard_D8as_v5.

Expected behavior

The miniconstellation should be correctly created and initialized.

Additional info

We currently think that it has to do with using an AMD VM as the host. But note, that we could only reproduce this in a cloud environment. Using local AMD machines worked fine.

Mitigation

Switch the Azure VM's type to Standard_D8s_v5.

Description

There might be points in the end user documentation that are unclear. Install Constellation in an environment of your choice and deploy one of the examples. Open a PR for any errors you can find.

Use case

According to the doc, Constellation OS images are not available in region eu-west-3. Could you please make it available?

Describe your solution

Make Constellation images available in region eu-west-3

Additional context

Need to deploy in this region

Since discussions seems not to be in use in this repository, I'm just posting this topic here. If not of interest or well known plz move to discussion or close.
In this article, there is an overview of different features of SEV and the status/progress in mainline Linux kernel:
https://www.phoronix.com/news/AMD-EOY-2023-SEV-SNP-Continues

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Package lookup failures

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• deps: update dependency aspect_bazel_lib to v2.7.8
• deps: update rhysd/actionlint to v1.7.1 (com_github_rhysd_actionlint_darwin_amd64, com_github_rhysd_actionlint_darwin_arm64, com_github_rhysd_actionlint_linux_amd64, com_github_rhysd_actionlint_linux_arm64)
• deps: update GitHub action dependencies (mikepenz/action-junit-report, softprops/action-gh-release)
• deps: update Go dependencies (cloud.google.com/go/kms, github.com/Azure/azure-sdk-for-go/sdk/azidentity, github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v5, github.com/aws/aws-sdk-go, github.com/siderolabs/talos/pkg/machinery)
• deps: update Terraform dependencies (aws, azuread, azurerm, cloudinit, google, google-beta, libvirt, random, stackit)
• deps: update bufbuild/buf to v1.34.0 (com_github_bufbuild_buf_darwin_amd64, com_github_bufbuild_buf_darwin_arm64, com_github_bufbuild_buf_linux_amd64, com_github_bufbuild_buf_linux_arm64)
• deps: update dependency bazel_skylib to v1.7.1
• deps: update dependency com_github_medik8s_node_maintainance_operator to v0.17.0
• deps: update dependency containernetworking/plugins to v1.5.1
• deps: update dependency kubernetes-sigs/cri-tools to v1.30.0
• deps: update dependency rules_python to v0.33.2
• deps: update golangci/golangci-lint to v1.59.1 (com_github_golangci_golangci_lint_darwin_amd64, com_github_golangci_golangci_lint_darwin_arm64, com_github_golangci_golangci_lint_linux_amd64, com_github_golangci_golangci_lint_linux_arm64)
• deps: update public.ecr.aws/eks/aws-load-balancer-controller Docker tag to v2.8.1
• deps: update quay.io/medik8s/node-maintenance-operator Docker tag to v0.17.0
• deps: update Terraform constellation to v2
• deps: update Terraform openstack to v2
• deps: update cachix/install-nix-action action to v27
• deps: update docker/build-push-action action to v6
• deps: update fedora Docker tag to v41
• deps: update ubuntu Docker tag to v24
• deps: update Python dependencies (Pillow, cryptography, matplotlib, pycparser, sev-snp-measure)
• deps: update dependency @mdx-js/react to v3
• deps: update dependency clsx to v2
• deps: update dependency numpy to v2
• deps: update docusaurus monorepo to v3 (major) (@docusaurus/core, @docusaurus/module-type-aliases, @docusaurus/plugin-google-gtag, @docusaurus/preset-classic, @docusaurus/theme-mermaid)
• 🔐 Create all rate-limited PRs at once 🔐

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• deps: update rules_oci digest to 360df7a

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• deps: update Constellation containers to v2.17.0-pre.0.20240619151941-9cd11842442d (ghcr.io/edgelesssys/constellation/filebeat-debugd, ghcr.io/edgelesssys/constellation/logstash-debugd, ghcr.io/edgelesssys/constellation/metricbeat-debugd)
• deps: update gazelle digest to 88fb1cf
• deps: update ubuntu:22.04 Docker digest to 19478ce
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• deps: update github.com/daniel-weisse/go-cryptsetup digest to fd0874f
• deps: update react monorepo to v18 (major) (react, react-dom)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Issue description

I am following first steps (local) on TDX enabled kernel, but constellation create fails.

constellation version:

Version:        v2.13.0 (Enterprise build; see documentation for license agreement)
GitCommit:      ea1fe82682889056d1b5ede058927ed5960ccb01
GitTreeState:   clean
BuildDate:      2023-11-14T08:51:53
GoVersion:      go1.21.4
Compiler:       bazel/gc
Platform:       linux/amd64

os:
Ubuntu 22.04.3 LTS
kernel:
5.19.17

Steps to reproduce the behavior

Run constellation mini up --debug

Creating cluster in QEMU
Error: creating cluster: creating terraform variables: fetching image reference: sending request for versionsapi.ImageInfo: Get "https://cdn.confidential.cloud/constellation/v2/ref/-/stream/stable/v2.13.0/image/info.json": context canceled

I'm using my company machine, it connects to internet through http proxy, and I configured the proxy correctly before running the command, will constellation client tool uses the proxy？

export https_proxy=http://proxy-host:proxy-port
export http_proxy=http://proxy-host:proxy-port

but I still got the error, here is the details with debug:

2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:35    Checked arch and os
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:40    Checked that /dev/kvm exists
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:48    Checked CPU cores - there are 192
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:66    Scanned for available memory
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:74    Checked available memory, you have 1006GB available
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:84    Checked for free space available, you have 117GB available
A config file already exists in the configured workspace.
2023-11-23T06:40:40Z    DEBUG   cmd/miniup.go:187       Creating mini cluster
Error: creating cluster: creating terraform variables: fetching image reference: sending request for versionsapi.ImageInfo: Get "https://cdn.confidential.cloud/constellation/v2/ref/-/stream/stable/v2.13.0/image/info.json": dial tcp 13.225.103.76:443: connect: connection timed out

However i can download the json file with wget

Proxy request sent, awaiting response... 200 OK
Length: 1937 (1.9K) [application/octet-stream]
Saving to: ‘info.json’

info.json                          100%[================================================================>]   1.89K  --.-KB/s    in 0s

2023-11-23 07:21:11 (854 MB/s) - ‘info.json’ saved [1937/1937]

iptables:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -j LIBVIRT_INP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT

Version

v2.13.0

Constellation Config

auto generated

Use case

According to the doc, Constellation OS images are not available in region eu-west-1. Could you please make it available?

Describe your solution

Make Constellation images available in region eu-west-1

Additional context

Need to deploy in this region

Use case

making state of confidentiality really transparent / have a good chance to optimize it

inspired from

• #2786 and
• latest news about vulnerabilities like cachewarp see https://www.google.de/search?q=amd+sev-snp+cachewarp

Describe your solution

Since confidentiality is the main selling point of constellation,
are there already any plans to make this visible via something like
a Dashboard or analyzing script showing the state of confidentiality?

depending on

• Constellations version
• Config
• Hardware type
• supported hardware features (CPU/Platform)
• Cloud providers features (subset of Hardware features)
• kernel versions used
• Firmware version (Patches)
• ...

This is not only about the current state but also to give concrete advice, how its possible to optimize confidentiality, e.g.

• move to Azure
• use instances with newer hardware gens
• update Constellation
• change config to
  ...

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Document how Constellation can be used with resource provisioning via Terraform instead of constellation create.

Hi,

My case is, i'm looking of a secure setup for k8s like constellation provides only i need it to wrap around my virtual cluster inside my actual cluster, i don't any cloud provider for this and i'm not sure the machine supports the needed encryption features you rely on from the processor. also from what i understand you only support a local cluster version for testing and only the cloud versions are production ready with only 3 providers supported, correct?

So my question is, would it be possible to have a local less secure version of what you offer using some configuration or does my use case requires a different product\solution? if it's the latter then maybe you can suggest something? if not then i thank you for you time anyway

Issue Description

We recently observed virtual machines on AWS with AMD SEV-SNP enabled to not reliably contain a (functioning) SEV-SNP device. Machines where this is the case will not be able to join or bootstrap a Constellation cluster, as they are not able to hand out a valid attestation report. Therefore, the issue is not impacting Constellation's security guarantees.

The issue may show different symptoms, depending on which part of a Constellation cluster the broken VM is.

• When a machine trying to bootstrap the Constellation cluster is broken, the CLI will show an error stating that an invalid attestation report has been supplied when trying to apply the initial Constellation cluster configuration on it.
• When a machine trying to join a Constellation cluster, be that within a cluster in its bootstrapping process or a cluster being upgraded, the machine will be rejected by Constellation's join-service, as it is not able to supply a valid attestation report. When bootstrapping a cluster, this will lead to the node simply not being able to join the cluster. On an upgrade, where Kubernetes operators manage the VM lifecycle, this rejection will lead to nodes being re-provisioned until a VM with a working device is received.

The issue has already been reported to the AWS team and they are working on fixing it.

Possible Workarounds

The issue is not present on all machines, so it is still possible to create a functioning Constellation cluster in most cases. If you should run into the issue on a machine, the following workarounds can help.

• Try to provision another VM. It is recommended to provision VMs on the same region until you get a working one, and then terminate all non-working VMs to not receive the same machine again when re-provisioning. The same can be achieved by provisioning a VM in another region, but as AWS does not provide SEV-SNP machines on all regions, you might run into availability issues, depending on which region is used.
  To do so, you can navigate to the constellation-terraform directory in your Constellation workspace (or the directory containing the infrastructure configuration, if not using the Constellation CLI) to destroy and re-apply the instance group, which contains the VMs, and apply the Constellation configuration again.

  cd constellation-terraform
  terraform destroy
  terraform apply
  constellation apply

• If the deployment is non-production, you can also use AWS NitroTPM attestation instead of SEV-SNP. To do so, remove the attestation.awsSEVSNP block from constellation-conf.yaml and insert the following the following block instead:

  awsNitroTPM:
    measurements: {}

  After that, destroy the cluster, fetch the measurements for machines with NitroTPM attestation, and recreate the cluster.

  constellation terminate
  constellation config fetch-measurements
  constellation create
  constellation apply

Issue description

The latest constellation cli release (2.1.0) has a bug when displaying the version of the constellation binary.

When running constellation version the following version is returned: 0.0.0

Refer to screenshot below:

To reproduce

Steps to reproduce the behavior:

1. Download and install latest constellation binary for macOS (intel) (version 2.1.0 at the time of writing this)
2. Execute the following command: constellation version
3. Notice how the displayed version on STD out is 0.0.0 instead of 2.1.0

Hi,

I love the look of this project and am considering trying it out on a cluster for myself.
However I was wondering if Constellation currently has support for Bare Metal clusters? I'd like to bootstrap it to replace some microk8s clusters. Is something like this in the pipeline?

Thanks,
Nick

Use case

more and more people adapt Crossplane to setup and manage "everything" resp. make the specialized knowledge of how to set up the needed resources in a simplified way available to every one who could benefit from this.

=> Would be pretty handy to have a Crossplane composition to setup - not only a standard kubernetes cluster in any cloud- but also a confidential one based on constellation (in the clouds with supported hardware)

her is just a quick video on what is possible this way:
Create And Manage GitOps-Ready Kubernetes Clusters With Crossplane
https://www.youtube.com/watch?v=AVHyltqgmSU
and the doc to compositions:
see https://docs.crossplane.io/latest/concepts/compositions/

Describe your solution

No response

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Use case

Describe your solution

Additional context

The CLI should have a command to automatically create the required IAM resources.

• Add constellation iam create command
• Add documentation for automatic IAM creation

Use case

Constellation, working as the typical confidential cluster that could run on either cloud environment or local machine across platforms, need to fetch measurements/evidence against different type of TEEs/TPM to prove its trustworthiness. Once a new confidential computing environment get supported in CSP's environment or a new technology revealed to the market, Constellation must make addition to the current code space to enable the evidence fetching or replaying function for the platform.

In the meanwhile, different platform or confidential computing technologies varies in use, which requires the Constellation developers to have knowledge and understandings on different architectures. Maintaining these code seems another burden for the project, as efforts are required once there's change in API or Specifications of the underlying technologies.

Describe your solution

Instead of maintaining the code within Constellation, it seems more efficient to leverage an utility which provides the capability for application to do evidence fetching or replaying using a set of simple APIs across all kinds of platforms.

CC Trusted API is a nice approach to streamline the effort that Constellation requires on this side. As a project that aims to collect confidential primitives (i.e., measurement, event log, quote) for zero-trust design, it provides the capability to fulfill this need using some vendor agnostic and TCG compliant APIs in multiple deployment environments (e.g. firmware/VM/cloud native clusters).

By leveraging these APIs, Constellation can perform with evidence fetching on different platforms through a unified API and requires little effort on maintenance of code related to platform features.

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Issue description

The Darwin ARM build for v2.8.0, fails to run with a message

“constellation” can’t be opened because Apple cannot check it for malicious software.
This software needs to be updated. Contact the developer for more information.

To reproduce

Steps to reproduce the behavior:

1. Following installation instructions
2. Run curl -LO https://github.com/edgelesssys/constellation/releases/latest/download/constellation-darwin-arm64
3. The signature verification step fails (see below in additional info)
4. Run the installation sudo install ./constellation-darwin-arm64 /usr/local/bin/constellation
5. Run constellation

Environment

• md5sum 8020bd3b379454336718c6f68030cc81 constellation-darwin-arm64
• constellation version: not available
• constellation-conf.yaml: not applicable
• VM type used to run Constellation: not applicable
• uname -a gives Darwin myhostname 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64

Expected behavior

Expected the release build to be properly codesigned

Additional info / screenshot

Cosign step failure:

cosign verify-blob --key https://edgeless.systems/es.pub --signature constellation-darwin-arm64.sig constellation-darwin-arm64

Error: searching log query: [POST /api/v1/log/entries/retrieve][400] searchLogQueryBadRequest  &{Code:400 Message:verifying signature: invalid signature when validating ASN.1 encoded signature}
main.go:74: error during command execution: searching log query: [POST /api/v1/log/entries/retrieve][400] searchLogQueryBadRequest  &{Code:400 Message:verifying signature: invalid signature when validating ASN.1 encoded signature}

Use case

In case of an error during create/init/terminate as much information as possible is helpful. Currently developers have to add debug prints or use a debugger to get more information of what went wrong.

Describe your solution

Add logging statements / prints that are only called if a dedicated debug flag is set. These logging statements should include information on what actions are performed on which objects. The more information, the better.

You can use this logger implementation as architecture blueprint.

Additional context

• Global flags are set here: cli/cmd/root.go
• Example for output behind a CLI flag: cli/internal/cmd/create.go

Use case

When interacting with cloud provider APIs there are actions that take multiple minutes. In those cases the user might become uncertain if the program is still working. Some kind of animation, like a spinner, would improve UX.

Describe your solution

Implement a small animation that shows that the program is still running, while waiting for the init response to come back.

Feel free to also implement this for other commands like create or terminate.

Additional context

• Lots of ASCII Art Spinner inspiration and example code: https://stackoverflow.com/questions/2685435/cooler-ascii-spinners
• The init request is sent here: cli/internal/cmd/init.go

Issue description

When using the workspace flag (--workspace or -C) to run constellation upgrade apply in a directory other than the current working directory, the command fails when applying Helm chart upgrades with the following error:

constellation upgrade apply -C <workspace>
# ...
Error: upgrading services: reading master secret: open <workspace-path>/constellation-mastersecret.json: no such file or directory

Workaround

Change into the workspace manually and run the command without setting the workspace flag:

pushd <workspace>
constellation upgrade apply
popd

Fix

Fixed on main after #2249 is merged.
Fixed in v2.11, or a patch release (if any) of v2.10

Issue description

The documentation page https://docs.edgeless.systems/constellation/overview/clouds calls out that in AWS EC2, instance firmware is not reviewable. For SEV-SNP instances, this is incorrect. We publish the sources as well as reproducibly built binaries and programmatic (nix based) build instruction at https://github.com/aws/uefi. Please change the table accordingly :).

As a side comment, you can use this binary in combination with https://github.com/virtee/sev-snp-measure to generate launch digests for SEV-SNP instances.

Use case

According to the doc, cluster upgrade is not yet supported on AWS. This is a show-stopper for production usage.

Describe your solution

Please implement cluster upgrade feature for clusters deployed on aws

Additional context

Nothing more to mention!

Use case

I wish to configure the constellation terraform to use my own libvirt storage pool and network as constellation create doesn't run to completion with the defaults:

Error: error creating libvirt network: internal error: Network is already in use by interface virbr2

  with libvirt_network.constellation,
  on main.tf line 111, in resource "libvirt_network" "constellation":
 111: resource "libvirt_network" "constellation" {

if I change the terraform to use the default libvirt network, then there is a similar problem with the libvirt storage pool sharing the location of the default pool.

I would like to modify the terraform to just use the default network and storage pool, or provide it through configuration, but constellation create does not support that as far as I can tell.

Describe your solution

Allow editing the terraform with:
constellation create --only-terraform
which populates constellation-terraform and then
constellation create to apply the terraform.

OR

provide options to use an existing libvirt network and storage pool.

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Feature

Make the Kubernetes version of a Constellation cluster upgradable with zero downtime.
For more info see the RFC

• Automatically fetch hashes for Kubernetes artifacts
• Download and verify Kubernetes components with hashes
• Extend the JoinService for automatic updates
• Annotate nodes with their Kubernetes components hash
• Create a upgrade agent for automatic updates
• Add Support for Kubernetes components inside the Constellation node operator
• Expose Kubernetes cluster version in NodeVersion
• Reconcile the Kubernetes cluster version
• Add k8s upgrades to CLI

Constellation microservices should be upgradable in an existing cluster using the upgrade command.

Predecessor: #589

• Move/Refactor bootstrapper helm client to internal
• Implement Helm chart upgrade for CLI
• Ensure CLI upgrade does not overwrite newer releases
• Document helm upgrade cli cmd

E2E coverage is currently lacking some important features.

• Constellation verify
• Loadbalancer deployment
• Constellation recover ( #845 )
• MiniConstellation
• QEMU (#1490 )
• Node Image Upgrade (#1469 )

Use case

Originally it is about things running within kubernetes, but I think it's worth to share - maybe this idea can somehow be adapted for hardening constellation:

We can now assert two statements are true, our agent runs:

• On an AWS EC2 machine
• In a memory encrypted context

https://control-plane.io/posts/spiffe-confidential-computing-august-2023/

spiffe intros:
https://spiffe.io/
https://github.com/spiffe/spire
https://control-plane.io/posts/spiffe-keystone-of-cloud-native/

and the spiffe plugin:
RFC: SEV SNP Node Attestation Plugin
spiffe/spire#4469

Describe your solution

No response

Would you be willing to implement this feature?

• Yes, I could contribute this feature.

Issue description

after running constellation mini up, i still dont have any worker node available, this is the logs i get:

kubectl logs -n kube-system daemonsets/join-service -f

{"level":"INFO","ts":"2023-12-25T15:10:49Z","caller":"cmd/main.go:57","msg":"Constellation Node Join Service","version":"v2.14.0","cloudProvider":"QEMU","attestationVariant":"qemu-vtpm"}
{"level":"INFO","ts":"2023-12-25T15:10:49Z","logger":"validator","caller":"watcher/validator.go:72","msg":"Updating expected measurements"}
{"level":"FATAL","ts":"2023-12-25T15:11:19Z","caller":"cmd/main.go:90","msg":"Failed to get IP in VPC","error":"Get \"http://10.42.0.1:8080/self\": context deadline exceeded"}

then i tried to check the pods and i found out that the join service crash looped

kube-system   join-service-mkxdq  0/1     CrashLoopBackOff   22 (4m33s ago)   105m

so there are no worker nodes, only the control plane

so i deleted the join service pod and it restarted successfully however still no worker nodes joined

and here is the list of events kubectlt get events -A

NAMESPACE     LAST SEEN   TYPE      REASON             OBJECT                                 MESSAGE
kube-system   3m34s       Warning   FailedScheduling   pod/cilium-operator-7f8f557b9d-fqnl2   0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports..
kube-system   3m34s       Warning   FailedScheduling   pod/coredns-8956f444c-x26r2            0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules..
kube-system   39m         Normal    Pulled             pod/join-service-lxdzj                 Container image "ghcr.io/edgelesssys/constellation/join-service:v2.14.0@sha256:c5cb0644b6c0519d0db1fd1e0986083e84b16c7bc90812669a7dc89aeb89ba4c" already present on machine
kube-system   4m28s       Warning   BackOff            pod/join-service-lxdzj                 Back-off restarting failed container join-service in pod join-service-lxdzj_kube-system(bd1cd3a2-f203-4dca-9573-143cff075e51)

and here is the list of all pods:

NAMESPACE     NAME                                                           READY   STATUS             RESTARTS       AGE
kube-system   cert-manager-6dfc87675f-jp95c                                  1/1     Running            0              76m
kube-system   cert-manager-cainjector-79dd56cf68-874bb                       1/1     Running            0              76m
kube-system   cert-manager-webhook-7797df8bdb-cqfdd                          1/1     Running            0              76m
kube-system   cilium-operator-7f8f557b9d-fqnl2                               0/1     Pending            0              76m
kube-system   cilium-operator-7f8f557b9d-jtkdd                               1/1     Running            0              76m
kube-system   cilium-z8xzz                                                   1/1     Running            0              76m
kube-system   constellation-operator-controller-manager-85c66946c4-tbrbv     2/2     Running            0              70m
kube-system   coredns-8956f444c-5lwwf                                        1/1     Running            0              76m
kube-system   coredns-8956f444c-x26r2                                        0/1     Pending            0              76m
kube-system   etcd-control-plane-0                                           1/1     Running            0              76m
kube-system   join-service-lxdzj                                             0/1     CrashLoopBackOff   17 (69s ago)   76m
kube-system   key-service-5ntc8                                              1/1     Running            0              76m
kube-system   konnectivity-agent-qhbcb                                       1/1     Running            0              73m
kube-system   kube-apiserver-control-plane-0                                 1/1     Running            0              76m
kube-system   kube-controller-manager-control-plane-0                        1/1     Running            0              76m
kube-system   kube-scheduler-control-plane-0                                 1/1     Running            0              76m
kube-system   node-maintenance-operator-controller-manager-5b6dcf6d8-dn422   1/1     Running            0              70m
kube-system   verification-service-z4hp2                                     1/1     Running            0              76m

two of them is pending which are cilium-operator and coredns, dont know if that is relevant or not

intel-vx is enabled in bios and all pre-requisites are met

kubectl get nodes output:

NAME              STATUS   ROLES           AGE   VERSION
control-plane-0   Ready    control-plane   83m   v1.27.8

ps: same result with qemu

Steps to reproduce the behavior

constellation mini up in a new directory

Version

Version: v2.14.0 (Enterprise build; see documentation for license agreement)
GitCommit: facaa6a
GitTreeState: clean
BuildDate: 2023-12-19T07:37:24
GoVersion: go1.21.5
Compiler: bazel/gc
Platform: linux/amd64

Constellation Config

No response

• Rename kms to keyservice
• Refactor init and recovery regarding master secret usage
• Refactor keyservice to use go-kms-wrapping
• Add external KMS option to config file
• Implement init with external KMS
• Implement recovery with external KMS
• Auto recovery with cloud KMS
• Implement KMIP client as Constellation KMS backend

Hello everyone,
I've been trying to run the Mini Constellation on Ubuntu 18.0 and I'm getting the error:
Error: creating cluster: fetching image reference: fetching image reference: Get "https://cdn.confidential.cloud/constellation/v1/ref/-/stream/stable/image/v2.3.0/info.json": dial tcp: lookup cdn.confidential.cloud on 127.0.0.53:53: dial udp 127.0.0.53:53: connect: invalid argument

• I set my FORWARD policy to ACCEPT in iptables
• I tried to turn of the firewall
• There is 0 rules inside my ufw-reject-forward chain in iptables [even that the chain name is still in the table]
• restarted the firewall.
• I rebooted my machine as a final solution..

However, I'm still having the problem and Constellation can't fetch the image from the remote server.

Note: Here is the content of my iptables:
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N ufw-after-forward
-N ufw-after-input
-N ufw-after-logging-forward
-N ufw-after-logging-input
-N ufw-after-logging-output
-N ufw-after-output
-N ufw-before-forward
-N ufw-before-input
-N ufw-before-logging-forward
-N ufw-before-logging-input
-N ufw-before-logging-output
-N ufw-before-output
-N ufw-logging-allow
-N ufw-logging-deny
-N ufw-not-local
-N ufw-reject-forward
-N ufw-reject-input
-N ufw-reject-output
-N ufw-skip-to-policy-forward
-N ufw-skip-to-policy-input
-N ufw-skip-to-policy-output
-N ufw-track-forward
-N ufw-track-input
-N ufw-track-output
-N ufw-user-forward
-N ufw-user-input
-N ufw-user-limit
-N ufw-user-limit-accept
-N ufw-user-logging-forward
-N ufw-user-logging-input
-N ufw-user-logging-output
-N ufw-user-output
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-684007d48d62 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-684007d48d62 -j DOCKER
-A FORWARD -i br-684007d48d62 ! -o br-684007d48d62 -j ACCEPT
-A FORWARD -i br-684007d48d62 -o br-684007d48d62 -j ACCEPT
-A FORWARD -o br-1434c250795f -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-1434c250795f -j DOCKER
-A FORWARD -i br-1434c250795f ! -o br-1434c250795f -j ACCEPT
-A FORWARD -i br-1434c250795f -o br-1434c250795f -j ACCEPT
-A FORWARD -o br-0a34a52fe39c -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-0a34a52fe39c -j DOCKER
-A FORWARD -i br-0a34a52fe39c ! -o br-0a34a52fe39c -j ACCEPT
-A FORWARD -i br-0a34a52fe39c -o br-0a34a52fe39c -j ACCEPT
-A FORWARD -o br-f45042a12d9f -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-f45042a12d9f -j DOCKER
-A FORWARD -i br-f45042a12d9f ! -o br-f45042a12d9f -j ACCEPT
-A FORWARD -i br-f45042a12d9f -o br-f45042a12d9f -j ACCEPT
-A FORWARD -o br-788a928d0b87 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-788a928d0b87 -j DOCKER
-A FORWARD -i br-788a928d0b87 ! -o br-788a928d0b87 -j ACCEPT
-A FORWARD -i br-788a928d0b87 -o br-788a928d0b87 -j ACCEPT
-A FORWARD -o br-77ec4d63a6d7 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-77ec4d63a6d7 -j DOCKER
-A FORWARD -i br-77ec4d63a6d7 ! -o br-77ec4d63a6d7 -j ACCEPT
-A FORWARD -i br-77ec4d63a6d7 -o br-77ec4d63a6d7 -j ACCEPT
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A DOCKER -d 172.24.24.3/32 ! -i br-0a34a52fe39c -o br-0a34a52fe39c -p tcp -m tcp --dport 2000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-684007d48d62 ! -o br-684007d48d62 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-1434c250795f ! -o br-1434c250795f -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-0a34a52fe39c ! -o br-0a34a52fe39c -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-f45042a12d9f ! -o br-f45042a12d9f -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-788a928d0b87 ! -o br-788a928d0b87 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-77ec4d63a6d7 ! -o br-77ec4d63a6d7 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-684007d48d62 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-1434c250795f -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-0a34a52fe39c -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-f45042a12d9f -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-788a928d0b87 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-77ec4d63a6d7 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-skip-to-policy-forward -j ACCEPT
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-forward -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-forward -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 22 -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 9000 -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 8889 -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 22 -j ACCEPT
-A ufw-user-input -p udp -m udp --dport 22 -j ACCEPT
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT

I would appreciate your help with this!
Thanks a lot.

Create IAM resources for Constellation automatically with CLI using Terraform.

• Implement Azure IAM in Terraform
• Implement GCP IAM in terraform
• Add constellation iam create command
• Add documentation for automatic IAM creation
• Automatically add iam values to config
• Add documentation for automatic IAM config filling
• Add iam destroy to clean up resources
• Add documentation for automatic IAM resource deletion

Issue description

When terminating a Constellation on AWS the constellation terminate outputs an error and does not fully clean up the Constellation. Only when running constellation terminate a second time, all resources are cleaned up.

To reproduce

Steps to reproduce the behavior:

1. Create a Constellation on AWS
2. Run constellation terminate

 euler@work:~/projects/constellation/test$ ./constellation terminate
You are about to terminate a Constellation cluster.
All of its associated resources will be DESTROYED.
This action is irreversible and ALL DATA WILL BE LOST.
Do you want to continue? [y/n]: y
Terminating
Error: terminating Constellation cluster: exit status 1
Error: deleting EC2 EIP (eipalloc-005f6369d2d588052): disassociating: AuthFailure: You do not have permission to access the specified resource.
    status code: 400, request id: fdc85569-0a89-4900-9631-bec0685479bc 

Environment

• constellation version: 2.8.0

Expected behavior

The Constellation should be fully terminated without error.

Additional info / screenshot

To fully terminate your Constellation simply run constellation terminate again.
We are currently investigating this issue.

Issue description

When installing constellation using the new terraform provider <2.15.0 it was possible to leave the constellation_microservice_version unset using a default value matching the provider version. This was changed in #2791. When attempting to upgrade to v2.15.0 the terraform provider will fail with the error message Parsing microservice version: invalid semver: v because it is trying to parse the null value from TF state auto-prefixed with v by the internal/semver.New.

Steps to reproduce the behavior

1. Existing cluster with unset constellation_microservice_version in TF state.
2. Upgrade to 2.15.0
3. Will fail withParsing microservice version: invalid semver: v
4. Manually insert the current microservice version into the TF state
5. Upgrade will succeed

Version

v2.15.0

Constellation Config

No response

Feature

The constellation config should contain a declarative configuration for the desired Constellation OS image.
Currently, the image reference for every cloud provider is different (and even for one cloud provider, there can be multiple instances of the same image.)

• Build pipeline
• Measurement pipeline
• constellation create
• constellation config fetch-measurements
• constellation upgrade *

Add support for Kubernetes version 1.26

• #775

Use case

According to the doc, confidential storage is not yet supported on AWS. This is a show-stopper for production usage.

Describe your solution

Please implement confidential storage feature for clusters deployed on aws

Additional context

Nothing more to mention!

Issue description

The needed qemu image cannot be downloaded.

To reproduce

Steps to reproduce the behavior:

1. use the local guide
2. Start the installation

   constellation mini up 
   Downloading image to ./constellation.raw
   Error: preparing config: downloading image to ./constellation.raw: downloading image: 404 Not Found

Environment

• constellation version:

  constellation version
  Version:        2.2.1
  GitCommit:      15b612b4cbf6c92a889a55b995de56947ff321a9
  GitTreeState:   clean
  BuildDate:      2022-11-14T16:32:39Z
  GoVersion:      go1.19.3
  Compiler:       gc
  Platform:       linux/amd64

Expected behavior

the local installation should be usable

Additional info / screenshot

Issue description

I am following first steps (local) on AMD EPYC 7313 with SNP enabled kernel, but constellation create fails.

I followed the steps for qemu.

To reproduce

Steps to reproduce the behavior:

1. Follow the instructions

Environment

• constellation version: v2.11
• constellation-conf.yaml: created from constellation config generate qemu
• VM type used to run Constellation: qemu

Expected behavior

As shown in the tutorial page.

Additional info / screenshot

Terminal prints

Error: couldn't retrieve IP address of domain id: 6be4bc53-c51c-4313-8c5e-2b68e19207a2. Please check following:
1) is the domain running proplerly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 timeout while waiting for state to become 'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)

  with module.node_group["worker_default"].libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {

File in docker container constell-libvirt:/var/log/libvirt/libvirtd.log shows

2023-10-04 11:28:08.189+0000: 41: info : libvirt version: 8.10.0, package: 2.fc38 (Fedora Project, 2023-01-03-08:31:39, )
2023-10-04 11:28:08.189+0000: 41: info : hostname: jax
2023-10-04 11:28:08.189+0000: 41: error : virGDBusGetSystemBus:99 : internal error: Unable to get system bus connection: Could not connect: No such file or directory
2023-10-04 11:28:08.189+0000: 41: warning : networkStateInitialize:658 : DBus not available, disabling firewalld support in bridge_network_driver: internal error: Unable to get system bus connection: Could not connect: No such file or directory
2023-10-04 11:28:08.384+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb3'
2023-10-04 11:28:08.384+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb4'
2023-10-04 11:28:08.387+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb5'
2023-10-04 11:28:08.388+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-1'
2023-10-04 11:28:08.388+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-1.3'
2023-10-04 11:28:08.388+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-1.4'
2023-10-04 11:28:08.389+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-2'
2023-10-04 11:28:08.389+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-2.1'
2023-10-04 11:28:08.390+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on '5-2.2'
2023-10-04 11:28:08.391+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb6'
2023-10-04 11:28:08.397+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb1'
2023-10-04 11:28:08.397+0000: 49: error : udevGetUintProperty:214 : internal error: Missing udev property 'ID_VENDOR_ID' on 'usb2'
2023-10-04 11:28:28.224+0000: 26: error : virCgroupDetectControllers:451 : At least one cgroup controller is required: No such device or address
2023-10-04 11:28:28.390+0000: 27: error : virCgroupDetectControllers:451 : At least one cgroup controller is required: No such device or address
2023-10-04 11:28:28.526+0000: 29: error : virCgroupDetectControllers:451 : At least one cgroup controller is required: No such device or address
2023-10-04 11:28:28.529+0000: 30: error : virCgroupDetectControllers:451 : At least one cgroup controller is required: No such device or address
2023-10-04 11:33:31.032+0000: 25: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-10-04 11:33:31.519+0000: 25: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error

Issue description

When a K8s service of type LoadBalancer is deployed, the cluster cannot terminate cleanly.
When running constellation terminate, termination fails with an error like:

Error: deleting EC2 VPC (vpc-...): DependencyViolation: The vpc 'vpc-...' has dependencies and cannot be deleted.
	status code: 400, request id: ...

This is due to buggy cleanup code in the currently used AWS Operator. A fix is in progress and will be part of an upcoming release.

Workaround

Delete the load balancer of each service and their security groups manually before calling constellation terminate. You might also delete the VPC manually instead of the security groups. If you already ran terminate, you may delete the Elastic IP resources that are attached your Constellation cluster. You can correlate the IPs by either looking at the public IP contained in your constellation-id.yml or by looking at the Elastic IP ressources conntected to your offending VPC.

Expected behavior

constellation terminate finishes without errors.

Issue description

Some questiones about the description "When a Constellation node image boots inside a CVM, ..." on https://docs.edgeless.systems/constellation/next/architecture/attestation#node-attestation:

1、How to boot a Constellation node image in a CVM for Constellation ?

2、Is the launched Constellation node image treated as a Constellation node or is the CVM treated as a Constellation node?

3、Is Constellation running multiple k8s pods in the Constellation node？

Steps to reproduce the behavior

No response

Version

No response

Constellation Config

No response