Issue:

• We used the provider_meta section of the versions.tf file to maintain the versions of all the google-terraform modules we use
• However, this conflicts with the strict checks on the provider_meta section in the later terraform versions
• Thus, we should remove all the unsupported custom fields we have added to here.

I'm on AKS 1.18.14, and when running kubectl logs stackdriver-prometheus-k8s-0 -n kube-system stackdriver-prometheus-sidecar like recommended here I'm seeing the expected info:

level=info ts=2021-03-15T03:41:33.933Z caller=main.go:598 msg="Web server started"
level=info ts=2021-03-15T03:41:33.933Z caller=main.go:579 msg="Stackdriver client started"

But I'm also seeing a lot of warnings like this:

level=warn ts=2021-03-15T03:42:37.427Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0-3]"
level=warn ts=2021-03-15T03:43:10.559Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[85-87,91,92,98,100,103,105-156]; Unknown metric: kubernetes.io/anthos/coredns_cache_size: timeSeries[88,89]; Unknown metric: kubernetes.io/anthos/coredns_dns_request_count_total: timeSeries[90]; Unknown metric: kubernetes.io/anthos/coredns_dns_request_type_count_total: timeSeries[93-95]; Unknown metric: kubernetes.io/anthos/coredns_dns_response_rcode_count_total: timeSeries[96,97]; Unknown metric: kubernetes.io/anthos/coredns_forward_request_count_total: timeSeries[99]; Unknown metric: kubernetes.io/anthos/coredns_forward_response_rcode_count_total: timeSeries[101,102]; Unknown metric: kubernetes.io/anthos/coredns_panic_count_total: timeSeries[104]"

Is it expected? Am I missing something? Or is it an issue?

Thanks.

snippet-bot scan result

Life is too short to manually check unmatched region tags.
Here is the result:
Great job! No unmatching region tags found!

Report generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.

Are there ways to add throttle or extra filters to limit the network consumption?

I tried to add throttling to fluentd config map but the plugin is not available.

Description

• Where is the issue?
  • Install with Manual Loadbalancer
    • Interacting with the Baremetal cluster
      • Controlplane / API Server
• What's the issue?
  • In the step for SCP copy, the command has the cluster id hardcoded as cluster1 for the name of the VM

Change

gcloud compute scp \
    --project=${PROJECT} \
    --zone=${ZONE} \
    tfadmin@cluster1-abm-ws0-001:${KUBECONFIG_PATH_IN_ADMIN_VM} \
    ${KUBECONFIG_PATH_IN_LOCAL_WORKSTATION}

to

gcloud compute scp \
    --project=${PROJECT} \
    --zone=${ZONE} \
    tfadmin@${CLUSTER_ID}-abm-ws0-001:${KUBECONFIG_PATH_IN_ADMIN_VM} \
    ${KUBECONFIG_PATH_IN_LOCAL_WORKSTATION}

Description

The issue is related to the Anthos bare metal on GCP with Terraform sample.

• The main script for the sample takes in zone & region variables to represent the location in which the GCE VMs should be created
• However, the google_cloud_ instance_template module only uses the region variable when defining the template used to create the VM
• As a result the VMs can be created in any random-zone within the region and not necessarily the zone provided as the input variable.
• This has been working so far without an issue because probably everyone always used the default region us-central1 so the zone was defaulted to us-central1-a
• But if we try with something like us-east1 region and us-east1-c then the scripts will fail

ABM Terraform needs testing & test framework design document. The setup needs to be tested at few levels -

1. Write Test Cases and come up with few examples for the design
2. Acceptance Tests https://www.terraform.io/docs/extend/testing/acceptance-tests/index.html and Unit Tests https://www.terraform.io/docs/extend/testing/unit-testing.html
3. Release Versions, and Releases (Github Branches)

Issue

With the current setup if we run the ABM on GCE TF and reboot one of the gce vms, the vxlan interface used by the VM is wiped.

What should happen:

VXLAN interface created by terraform should persist after reboot.

Potential fix:

If you use ifconfig to configure network interfaces, the changes will be lost after reboot. To make your changes persistent, you need to modify configuration files like /etc/network/interfaces.

During the installation of latest version I found the following services weren't enabled by Terraform:

• "anthosaudit.googleapis.com",
• "opsconfigmonitoring.googleapis.com"

moreover the service account created doesn't have the following permission needed:
roles/opsconfigmonitoring.resourceMetadata.writer

Fixing those the installation of ABM 1.10 has been completed.

Once the Terraform ABM (GCE instances) are provisioned, the following setup takes a long time (~15mins) even for a 2 node cluster

sudo ./preflights.sh && \
  sudo bmctl create config -c anthos-gce-cluster && \
  sudo cp ~/anthos-gce-cluster.yaml bmctl-workspace/anthos-gce-cluster && \
  sudo bmctl create cluster -c anthos-gce-cluster

Investigate the reasons for slow cluster creation.

Issue:

• Currently any PR opened from a fork will fail CI tests that depend on environment variables.
• This is because the repository secrets are not accessible for forked workflows
• See here

Solution

• Unclear
• Maybe one of the comments from this issue

Issue:

• The sudo ./run_initialization_checks.sh step after the terraform script has completed has the following error
• As a result the shell has this script continuously running in the background
• It seems to be from this function that sets up signal handling for all the background tasks spawned by this script

./run_initialization_checks.sh: line 93: trap: $SIG: invalid signal specification

Description:

• The default storage class of the ABM clusters is defined as "standard" in the Terraform based guide.
• It is the same in the shell script based guide as well.
• However, in the Quickstart guide example config the storage class is set to "local-shared"
• As a result if someone creates an ABM cluster using one of the first two guides, then the default storage class in the cluster is names "standard"
• But for the virtctl plugin the default storage class is "local-shared"
• So if someone creates a VM in Anthos VMRuntime for ABM, the default storageclass will be "local-shared".
• Thus for someone running a cluster based off the first two methods above, VM creation will fail
• We need t o fix this by ensuring all of them have the same storage class value

Description

• Currently the integration tests only cover up to setting up the GCE infrastructure in GCP and configuring the GCE VMs
• It does not validate if the Anthos bare metal installation itself is successfull
• Thus we need to extend the integration tests to verify that the ABM install succeeds
• This will allow us to capture issue like: #158

anthos-bm-gcp-terraform script doesn't work when the project wide oslogin is enabled. It fails with ssh error.

Error: timeout - last error: SSH authentication failed ([email protected]:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

It have to be able to handle it properly or a custom metedata have to be added to the instance template.

  metadata = {
    enable-oslogin = "false"
  }

Description

• We don't have a specific area for hosting A4VMWare specific resources
• So we need to a new directory for this and enable pulling these material into the docs via region tags

Description

• The current anthos-bm-gcp-terraform contains two installation modes
• One of them is a two step process where the user runs the terraform script to create the GCE environment and a few additional steps to SSH into the created GCE to install ABM
• The other is an all-in-one install where running the terraform script not only creates the GCE environment but also triggers the ABM installation
• Both of these modes lead up to ABM being installed using the bundled loadbalancer mode.
• What this means is that whilst it emulates a truly baremetal environment, we cannot use this setup for demo purposes because we cannot reach the workloads running inside the cluster
• Thus, we need to add support for being able to hit the workloads inside the ABM cluster running inside the GCE VMs

What to be done?

• We update this sample to include an additional installation mode which enables setting up ABM using ManualLB mode
• We set a GCP loadbalancer as the manual LB used by ABM
• What this means is that once the installation is complete the control plane of the ABM cluster can be reached via a public IP
• In addition the workloads running inside the ABM cluster can also be reached via a public IP that routes traffic to the ingress controller setup in ABM by default

There should be an extra nest plugin to nest them under "logging.googleapis.com". Maybe we should consider move it to filter-kubernetes.conf so it can be effective for all included files.

[FILTER]
        # https://docs.fluentbit.io/manual/filter/nest
        Name    nest
        Match   *
        Operation nest
        Wildcard gke.googleapis.com*
        Nest_under logging.googleapis.com/labels

To ease use of collecting Anthos Clusters and uploading of snapshots created from create_snapshot.py provide functionality to upload snapshots to a Google Cloud Storage bucket.

Assuming that a user has the Google Cloud SDK the script should allow for specifying a bucket name and a service account key to upload the snapshot tar file.

Added arguments:
--upload-to
--service-account-key-file

@Shabirmean and I noticed flaky behaviour occurring.
We believe that it's due to github actions/checkout@v3, where the perms get overwritten
Every once in a while we have to ssh into the runners and change the permissions, or else the CI errors out with

Error: fatal: --local can only be used inside a git repository
Deleting the contents of '/home/$USER/actions-runner/_work/anthos-samples/anthos-samples'
Error: Command failed: rm -rf "/home/$USER/actions-runner/_work/anthos-samples/anthos-samples/.git"

During the install_abm.sh with mode=manuallb phase intermittently seeing:

Waiting for preflight check job to finish... OK
- Validation Category: machines and network
  - [FAILED] node-network (log: bmctl-workspace/cluster1/log/preflight-[...]/node-network)	
  - connecting to node 10.128.[...] through VIP [...external ip...] failed

Re-running sudo ./install_abm.sh seems to generally work, my initial suspicion is it might be related to the timing of the LB health-checks during the cluster deployment/testing window?

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

In my test on GCE I set firewall opening the ports used otherwise the ABM installation fails in the next step after terraform launch.

I would suggest to add firewall ports setting step.
If you agree I can prepare a PR for that.

Description

• We need add a tutorial that shows running a VM based workload in Anthos bare metal using Anthos VMRuntime (i.e.: Kubevirt)

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• fix(deps): update github.com/icrowley/fake digest to f797eb4
• chore(deps): update dependency go to v1.23.0
• chore(deps): update terraform hashicorp/terraform to < 1.10
• chore(deps): update terraform terraform-google-modules/gcloud/google to v3.4.1
• fix(deps): update module github.com/stretchr/testify to v1.9.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update terraform google to v6
• chore(deps): update terraform terraform-google-modules/project-factory/google to v16
• chore(deps): update terraform terraform-google-modules/vm/google to v11

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• fix(deps): update module github.com/gruntwork-io/terratest to v0.47.0

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/googlecloudplatform/cloud-foundation-toolkit/infra/blueprint-test to v0.16.1

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• fix(deps): update module gopkg.in/yaml.v2 to v3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

AppArmor is now supported in Anthos clusters on bare metal 1.8.2+.

It might be nice to have updates to init_args trigger a rebuild of the instances to keep the state consistent.

Add an option to provision an NFS server (Filestore) for use as Persistent Storage on the cluster.

The ABM Terraform installation needs GPU support in order to run ML and TensorFlow based GPU workloads. There are two options to do that -

• Terraform Resource
• Terraform Modules (Instance Template needs GPU support)

The example code below works for GCE with NVIDIA Tesla V100 using Terraform Resource:

resource "google_compute_instance" "gpu-vm" {
  count = 1
  name = "gpu-vm"
  machine_type = "n1-standard-4" // 1 CPU 16 Gig of RAM
  zone = "us-west1-a" // Call it from variable "region"
  tags = ["http"]
  boot_disk {
    initialize_params {
      image = "ubuntu-os-cloud/ubuntu-1604-lts"
      size = 50 // 50 GB Storage
    }
   }
  network_interface {
    network = "default"
      access_config {
        // Ephemeral IP - leaving this block empty will generate a new external IP and assign it to the machine
      }
  }
  guest_accelerator{
    type = "nvidia-tesla-v100" // Type of GPU attahced
    count = 1 // Num of GPU attached
  }
  scheduling{
    on_host_maintenance = "TERMINATE" // Need to terminate GPU on maintenance
  }
}

Once we have the GPU enabled we'll be installing the NVIDIA GPU and CUDA drivers.

We are using Anthos Attached in a cluster with Windows nodes. fluent-bit is not an option for Windows, however there is a fluentd image for Windows. Could you please provide a fluentd configuration to forward to your Stackdriver aggregator? Or could you provide another method to forward logs from Windows nodes?

Description

• Currently the terraform scripts for the anthos-bm-gcp-terraform sample has two steps.
• The terraform run will setup the GCE infrastructure and printout the next steps to actually log into the Admin VM and install Anthos bare metal
• We want to provide a new experience where the user can just run the Terraform script and have the entire setup completed which will include GCE setup + ABM installation

Tracking issue for migrating from CFT test framework to our new blueprints test framework. Both sets of tests can co exist as we migrate.

While testing GKE on AWS example I've hit an unexpected side effect:
Running terraform examples with aws-vault or other ways of passing my credentials - they were ignored and infrastructure was created in a "default" AWS project.
This happens due to the profile name being hardcoded in the provider configuration.
And provider configuration having the highest precedence.

I've prepared a PR for that.

Hardware virtualization provides better performance than software emulation when using Anthos VM Runtime on ABM. On GCE nested virtualization can be enabled in TF with the enable_nested_virtualization parameter.

Which sample?

• anthos-bm-gcp-terraform

Description

• The release process for the sample is based on the release-please bot. Currently we are using the terraform-module release type which is based off of the go/blueprints project
• However, we should upgrade this to add our own release type to achieve the following:
  • Upon release the versions of the other terraform modules under the provider_meta definition in the versions.tf file shall be updated to the correct versions used for each module during time of the release

Potential fix

• Add a new releaser and a updater that extends the existing terraform-module type to the release-please repo
• Use this new type in our release configs in the anthos-samples repo

Useful:

• Existing terraform-module releaser
• Existing terraform-module updater
• Release automation bot wrapper around release-please npm package
• All automation bots

Considerations about using the provider_meta field to maintain dependencies:

• Pros: All dependency information is maintained in the versions.tf file as a natural place where someone would look at
• Cons: If the definition and usage of provider_meta tag changes then the releaser would break. Thus, ideally it might be useful to store this information in a different file (e.g. dependency_matrix.txt)

Update CODEOWNERS to point to anthos-dpe team

Issue:

• There is a security vulnerability reported by dependant regarding the tidwall/gjson golang module.
• The issue is with version < 1.9.3
• Note that this dependency is not directly imported by any of the modules in this repo; it is an indirect dependency

Potential Fixes:

• Update the version for the tidwall/gjson module to >= 1.9.3 or to the latest
• Update the version of the go module in the repo that imports this module to a version where the tidwall/gjson version is >=1.9.3

Running the terraform init command gives this:

Initializing modules...
╷
│ Error: Unsupported Terraform Core version
│
│   on versions.tf line 28, in terraform:
│   28:   required_version = "~> 0.14.9"
│
│ This configuration does not support Terraform version 1.0.0. To proceed, either choose another supported
│ Terraform version or update this version constraint. Version constraints are normally set for good reason, so
│ updating the constraint may lead to other errors or unexpected behavior.
╵

I'm running terraform 1.0.0, but also got this error when running 0.14.8. It looks like the required_version just needs to be bumped. I'll poke at it and create a PR if I'm successful.

Issue:

• With the current setup if we run the ABM on GCE TF scripts twice with different cluster names the second one conflicts with the first

What should happen:

• Subsequent runs with different TF variables should produce 2 different clusters

Potential fix:

• Prefix the resources with the cluster_name from the variables
• Take in a separate prefix var to use with each resource name specific to a cluster

It would be ideal to delete any GKE Connect membership for the cluster during the Terraform destroy. Even better would be to also check during the create phase to ensure the bmctl create won't fail due to a stale GKE Connect membership.

I've observed an intermittent file provisioner error in anthos-bm-gcp-terraform. It appears that depending on timing, sometimes the module "init_hosts" {} will start before all the local_files are written to disk. For example:

│ Error: file provisioner error
│ 
│   with module.init_hosts["cluster1-abm-cp3-001"].null_resource.exec_init_script,
│   on modules/init/main.tf line 82, in resource "null_resource" "exec_init_script":
│   82:   provisioner "file" {
│ 
│ stat ./resources/.temp/.cluster1.yaml: no such file or directory

Update anthos-bm-gcp-terraform to ABM (bmctl) 1.10.2

Issue

• Linter is disabled on the license header for go files because of this issue.
• If that issue is resolved then the headers must be regenerated using the addlicense tool
• The TODO and the nolint flag has to be removed from the go files

Description:

• Currently we use terraform-docs to generate the docs for the terraform script/modules (see example)
• This is done manually and the developer will have to re-run the tool to regenerate the docs if they had made any changes.
• Ideally we would like to have it in the CI pipeline, so it automatically re-generates if the relevant files have changed.

What's the issue:

• However, we are unable to add this flow using the Github actions for terraform-docs because when running the tool it creates a license-header at the top of the docs.
• This license header is something we don't need/want
• Thus, if we use the Github actions in the CI then every PR is going to be considered having some docs related change since the tool is going to add this license header even if we remove it manually.
• Thus, we must monitor this issue (terraform-docs/terraform-docs#604) in the terraform-docs repo and if it gets addressed then update our CI to use Github actions for terraform-docs

Hey Y'all,

The anthos-bm-gcp-terraform scripts are failing. At first, I thought it was my forcing them to use the latest TF version, and my using Cloud Shell, but I just loaded up a VM with Terraform 0.15.5, as the scripts call for, and it's still failing. The creation of the machines seems to go fine, but the modules/init ssh connection seems to be failing. Essentially, the script times out:

module.init_hosts["abm-w1-001"].null_resource.exec_init_script: Still creating... [4m51s elapsed]
module.init_hosts["abm-w2-001"].null_resource.exec_init_script: Still creating... [4m51s elapsed]
module.init_hosts["abm-w1-001"].null_resource.exec_init_script: Still creating... [5m1s elapsed]
module.init_hosts["abm-cp1-001"].null_resource.exec_init_script: Still creating... [5m1s elapsed]

And there's an error for each machine, similar to:

│ Error: file provisioner error
│ 
│   with module.init_hosts["abm-w1-001"].null_resource.exec_init_script,
│   on modules/init/main.tf line 75, in resource "null_resource" "exec_init_script":
│   75:   provisioner "file" {
│ 
│ timeout - last error: SSH authentication failed ([email protected]:22): ssh: handshake failed: ssh: unable to authenticate,
│ attempted methods [none publickey], no supported methods remain

I did a little poking around, verified that they keys are being created etc. but I'm not seeing an obvious reason for the issue.

Cheers.

Issue

• The OpenStack TF sample is mostly hardcoded with additional steps only covered in an internal. Thus, we need to make it configurable and also complete the README so that anyone with a OpenStack deployment can try out the sample.

GCE admin VM needs service account permission (iam.serviceAccountKeyAdmin) to be able to complete the init.sh script. The script retrieves the service account json via gcloud command. We need to have a variable to set proper account permissions for the GCE instance to have that permission

To prevent conflicts with repeated usage, a randomized suffix could be automatically attached to the cluster name/resources.

Would also workaround #261