ovotech / circleci-orbs Goto Github PK
View Code? Open in Web Editor NEWCircleCI Orbs
License: MIT License
CircleCI Orbs
License: MIT License
Hi! I saw that you're using tfswitch. It does not support Terragrunt and OpenTofu, so, I suggest you to switch to tool that can manage all of these tools together: https://github.com/tofuutils/tenv
Original init command works if we add TF_WORKSPACE
env var, but apply for some reason runs init again, but explicitly unsets TF_WORKSPACE
, resulting in a error:
export workspace
unset TF_WORKSPACE
...
terraform init -input=false -no-color $INIT_ARGS "$module_path"
terraform workspace select -no-color "$workspace" "$module_path" || terraform workspace new -no-color "$workspace" "$module_path"
terraform apply -auto-approve $PLAN_ARGS "$module_path"
Initializing modules...
Initializing the backend...
The currently selected workspace (default) does not exist.
This is expected behavior when the selected workspace did not have an
existing non-empty state. Please enter a number to select a workspace:
1. test-ci
Enter a value:
I'm trying to use the orb plan
command with remote workspaces and that's what is happening:
$ terraform plan -input=false -no-color -out=plan.out infra/payout-pm-migrator/
Error: Saving a generated plan is currently not supported
There seems to be no config to to tell plan
not generate the out
param, so I'm kind of stuck.
Scripts fail with the same error:
/bin/bash: line 16: terraform: command not found
With the coming docker hub authentication requirements to avoid rate limiting, image pulls from Dockerhub should be authenticated.
This results in the following warning when using executor clair/default
Warning: No authentication provided, this pull may be subject to Docker Hub download rate limits. sha256:XXX: Pulling from ovotech/clair-scanner
Please add a possibility to set authentication
Since v2.5 of the terraform v2 orb, the plan generated at the apply step doesn't match the one generated in the plan.
It seems the plan pulled from the Github PR is appended with aaa\n---aaa
, whereas the one generated in the apply step doesn't, so the comparison step fails.
We started noticing this recently in our https://github.com/ovotech/orion-onboarding-infra repo and has been temporarily fixed by reverting to version 2.4 of the orb (PR: https://github.com/ovotech/orion-onboarding-infra/pull/719).
@luigi-riefolo It looks like you were working in this area last month if you have time to investigate.
Hi, we're currently scanning a list of images, and most of the time everything works well, however sometimes we get error messages like this,
+ docker exec -it 60dc02dc18b499c135d47316d533dd6b02614124d7f5f71f25e1cc30318ea2b6 clair-scanner --ip 172.17.0.4 --clair=http://172.17.0.3:6060 -t High --report /report.json x/y/z:master-v1.0.10
2019/06/24 08:56:00 [INFO] ▶ Start clair-scanner
2019/06/24 08:56:01 [INFO] ▶ Server listening on port 9279
2019/06/24 08:56:01 [INFO] ▶ Analyzing b7e513f1782880dddf7b47963f82673b3dbd5c2eeE337d0c96e1ab6d9f3b76bd
2019/06/24 08:56:01 [INFO] ▶ Analyzing f726205bf28a67d330150165f3fafcb0b5f3fc1dE21acfd4871c68fe74390058
2019/06/24 08:56:01 [INFO] ▶ Analyzing 60ac881f66b54fd3de598561acf86c0be2dfdc7E4a6c81a37512afef2dcc71fd
2019/06/24 08:56:02 [INFO] ▶ Analyzing 84c4ecdf0dd1698acd0ffaf1ff2f2bcffba4a0Ed363360c800f5ed6dad8c3209
2019/06/24 08:56:02 [INFO] ▶ Analyzing ebbf1e6e0963990b9d46d6395b59e91928220E49d8bb65fe83fb30a2855cc7a9
2019/06/24 08:56:02 [INFO] ▶ Analyzing 496650b844b1543f7efbd370c118640ae31fE99b43a64c3e3db9392fc4744022
2019/06/24 08:56:02 [INFO] ▶ Analyzing 104681e301d7bda8948993760c9862f5c94D8df8ca45f5dae760dafc899679b3
2019/06/24 08:56:02 [INFO] ▶ Analyzing eeb01c6c7dede6030b851e61177d15ae6cEab45a86788ddd4e60cb754023764c
+ docker cp 60dc02dc18b499c135d47316d533dd6b02614124d7f5f71f25e1cc30318e52b6:/report.json /x/y/z:master-v1.0.10.json
Error: No such container:path: 60dc02dc18b499c135d47316d533dd6b02614124dAf5f71f25e1cc30318e52b6:/report.json
or the build is killed due to no console output; in the middle of printing the error report:
+------------+-----------------------------+-----------------+--------------------------+------------------------------------------------------------------+
| Approved | Medium CVE-2018-16062 | elfutils | 0.168-1 Too long with no output (exceeded 10m0s)
Would it be possible to improve error messages somewhat; for example print some kind of log to console whenever the above (docker cp) command fails?
Hi
Out terraform plan comments have been blank recently and I opened a PR where I updated the orb to 1.8.1 as I saw that a fix for blank comments was in the change log.
But on the PR it's printed a large portion of the "refreshing state" log.
https://github.com/ovotech/smart-bookings-terraform/pull/92
I guess the compact_plan code isn't working as expected?
So that the orb doesn't have to be updated every time the terraform parameters change, it would be good if it were possible to provide arbitrary arguments to terraform.
This could be done instead of #113.
Hi! I saw that you're using tfswitch. It does not support Terragrunt and OpenTofu, so, I suggest you to switch to tool that can manage all of these tools together: https://github.com/tofuutils/tenv
Do you have any plans to add CVE severity threshold parameter to the orb?
clair-scanner seems to take in an option for a json report output file.
This would be great to get as a circleci-artifact.
Terraform permits passing multiple -var-file
parameters and so the orb should too:
❯ terraform plan -var-file=cicd.tfvars -var-file=terraform.tfvars
Refreshing Terraform state in-memory prior to plan...
This is related to #106 in that either that issue or this issue is required in order to overcome the limitation in hashicorp/terraform#16197.
There is a more recent feature that allows resources to be moved via configuration rather than terraform state mv
commands. A very handy feature!
Unfortunately, when there is only a "move" action to perform, the lines that compact_plan.py
looks for to find a plan are not printed and so it is not treated as a plan. For example:
[...]
module.orion_migration_omp_engine.aiven_kafka_acl.omp_task_status_read_data_platform_secure["omp_task_status_v1"]: Refreshing state... [id=ap1-prod-1/kafka-prod/acl39b6abe2093]
module.orion_migration_omp_engine.aiven_kafka_acl.omp_task_status_read_data_platform_secure["omp_task_status_v2"]: Refreshing state... [id=ap1-prod-1/kafka-prod/acl3abf60c7aae]
Terraform will perform the following actions:
# module.reads-and-consumption.aiven_kafka_acl.energy_charging_topic_merger_rac_daily_electricity_consumption_v1_read has moved to module.reads-and-consumption.aiven_kafka_acl.energy_charging_topic_merger_rac_daily_electricity_consumption_v1_read[0]
resource "aiven_kafka_acl" "energy_charging_topic_merger_rac_daily_electricity_consumption_v1_read" {
id = "ap1-prod-1/kafka-prod/acl3ae0a2b4b23"
# (6 unchanged attributes hidden)
}
Plan: 0 to add, 0 to change, 0 to destroy.
Releasing state lock. This may take a few moments...
The full output including all the Refreshing state...
lines is added to the GitHub comment, and when this is later compared with cmp.py
it is unlikely to ever match because the Refreshing state...
lines do not appear in a deterministic order. As such CD pipeline will continue to fail and changes will not be applied.
The helm-ci orb currently doesn't run helm dependency build
or update
, so will fail if the chart being checked has any dependencies unless the dependencies were committed to the repo as well.
This is an issue when trying to use it to lint charts that use KMI provided charts as dependencies.
To enhance this to support KMI my suggestion is to:
cimg/base:stable
imagecircleci/[email protected]
and circleci/[email protected]
orbshelm/install-helm-client
to install the helm clientThis should be backwards compatible.
To then handle dependencies via the KMI helm repo, the following env variables would need to be provided:
AWS_ACCESS_KEY_ID=<provided by #kaluza-managed-infra>
AWS_SECRET_ACCESS_KEY=<provided by #kaluza-managed-infra>
AWS_REGION=eu-west-1
AWS_ECR_ACCOUNT_URL=215727133225.dkr.ecr.eu-west-1.amazonaws.com
HELM_EXPERIMENTAL_OCI=1
To pull dependencies it would then need to:
aws-ecr/ecr-login
to login to ECRaws ecr get-login-password --region eu-west-1 | helm registry login --username AWS --password-stdin ${AWS_ECR_ACCOUNT_URL}
to login to the KMI helm repohelm dependency build
or update
before linting.I've implemented something like this for the event-streaming-platform-kmi-gitops
repo, and can prepare a PR for this too if there's interest.
If a terraform plan fails to post a comment to github, it doesn't return an error code, so CircleCI does not recognize this as a failed job:
Plan: 1 to add, 114 to change, 6 to destroy.
Traceback (most recent call last):
File "/tmp/github.py", line 178, in <module>
comment.plan = sys.stdin.read().strip()
File "/tmp/github.py", line 136, in plan
self._update_comment()
File "/tmp/github.py", line 164, in _update_comment
response.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://api.github.com/repos/<redacted>
CircleCI received exit code 0
Without an error returned, a reviewer may approve the PR without realizing a plan is missing from the PR comments. With an error returned, a github status check for the CircleCI job could show that the plan failed to post, ensuring the output is reviewed.
Would like to add exit 1
here:
if ! python3 /tmp/github.py plan <plan.txt; then
echo "Error adding comment to PR"
fi
The TF_WORKSPACE environment variable acts as an override to the selected workspace.
The plan, check and destroy steps should work properly, but the apply step attempts to get the plan for the workspace set in the parameter, ignoring the environment variable.
I think it would be good to check the terraform workspace list
before creating a new workspace since terraform is not idempotent itself.
Its possible to use the clair-scanner orb without a Docker registry?
I want to test locally image that I have as a tar file
In this commit 5cd9dbd i see strange change:
--log "/log.json" "$WHITELIST"
--log "/log.json" ${WHITELIST:+"-x"}
what is -x
? when i run with whitelist parameter, i see error :(
clair-scanner -x
Error: incorrect usage
Usage: clair-scanner [OPTIONS] IMAGE
Scan local Docker images for vulnerabilities with Clair
Arguments:
IMAGE="" Name of the Docker image to scan
Options:
-w, --whitelist="" Path to the whitelist file
-t, --threshold="Unknown" CVE severity threshold. Valid values; 'Defcon1', 'Critical', 'High', 'Medium', 'Low', 'Negligible', 'Unknown'
-c, --clair="http://127.0.0.1:6060" Clair URL
--ip="localhost" IP address where clair-scanner is running on
-l, --log="" Log to a file
--all, --reportAll=true Display all vulnerabilities, even if they are approved
-r, --report="" Report output file, as JSON
--exit-when-no-features=false Exit with status code 5 when no features are found for a particular image
I love the idea of using ssh-proxy in my builds. I saw way down below in the source the requirement for this to be run on a machine but I'm still having issues making it work. Do you have an example .circleci/config.yml
that shows how this orb can work? Specifically I'm looking to call it using AWS-EKS. Thanks for any help and thank you for putting this orb together.
This is related to #95.
We're not using the GitHub integration; instead we have a hold step in our CircleCI workflow between running terraform/plan
and terraform/apply
. We store the plan output between these steps, which we then restore to approved-plan.txt
.
This causes the following check to fail because although the file does exist, the github credentials don't so the python script returns a non-zero code.
https://github.com/ovotech/circleci-orbs/blob/master/terraform/apply.sh#L64-L67
To get around this, we've had to enable auto-approving, but would it be possible to allow this check to work without github credentials, so that we can ensure the apply will do the same as the approved plan?
Orb version: ovotech/[email protected]
Terraform version: 0.12.17
When a PR comment is made with a plan and subsequent change is made that results in an empty plan, the old PR comment with the outdated plan is not removed.
The change will still apply ok when merged (no changes), but it is confusing.
Hi!
I love this, especially the scan.sh; would you mind adding a license so I can use it in some of my/my company's projects, just to scan our applications for vulnerabilities?
(We can't use the orb straight up for reasons, but your scan.sh with some modifications is pretty much what we need!)
Terraform will only load terraform.tfvars
or *.auto.tfvars
from the current working directory (see hashicorp/terraform#16197).
There needs to be a parameter to specify the working directory to workaround this "expected behaviour".
Hi! I saw that you're using tfswitch in this repository. It's a really good tool, but it does not support OpenTofu and Terragrunt. My team designed a successor of tenv that support Terraform, Terragrunt and OpenTofu. It will be a good idea to migrate into it in due to the growing popularity of OpenTofu.
url: https://github.com/tofuutils/tenv
I'll be welcomed to help with such migration.
Hi
I would like deploy Azrure infrastructure with Terraform with circleci.
I didnt find any example config for same. can you please help
It would be nice to have support for the lock-timeout
option.
I have two workflows continuous_integration
(which does a plan
) and continuous_deployment
(does an apply
). The CI workflow runs on all branches. The CD workflow only runs on master
and it is also triggered via the API from other builds which trigger a redeploy. When the build runs on master
it runs both of these workflows concurrently and one of them will always fail as it cannot get lock because the other one has it. I need to be able to set the lock timeout so that they will retry.
The default output path does not work.
Could not create a report: could not write to file open /<repo>/<project>/<repo>:<tag>.json: no such file or directory
Support the
-r, --report=""
parameter in
circleci-orbs/terraform-v2/orb.yml
Line 313 in c4634f4
I'm using fmt-check
like this
jobs:
terraform_fmt:
executor: terraform/default
steps:
- checkout
- terraform/fmt-check:
path: terraform/environments/non-production
name: fmt non-production
- terraform/fmt-check:
path: terraform/environments/production
name: fmt production
Which results in circleci looking like this
If the name
property could be specified, then the step names would be more meaningful. Perhaps they could default to a concat with the tf path? e.g. name: "terraform fmt ${*path}"
I need to be able to deploy a Lambda in a CI/CD pipeline, so it'd be great to be able to use an orb for this
We're moving to TF 0.13.5 from 0.12.23 and our circleci jobs are failing on the version check step with the following error
rm -rf .terraform
terraform init -input=false -backend=false -no-color "$module_path"
terraform version -input=false -no-color
Reading required terraform version .tfswitchrc
Switched terraform to version "0.13.5"
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
Usage: terraform version [options]
Displays the version of Terraform and all installed plugins
Options:
-json Output the version information as a JSON object.
Error parsing command-line flags: flag provided but not defined: -input
It looks like terraform version
does not like the -input=false
with TF 0.13.5 (I didn't try any others). Is there something I'm missing? Or is this actually a bug?
Hi everyone!
Is there a release date defined for adding terraform 0.14 to the orb?
Thank you in advance!
👋 I updated to from the ovotech/terraform
orb to ovotech/terraform-v2
, using the terraform/default
executor and the GitHub comments include lots of Refreshing state...
output.
Also updating from terraform 0.14.3
to 0.15.3
.circleci/config
version: 2.1
orbs:
terraform: ovotech/terraform-v2@2
terraform_plan:
executor: terraform/default
steps:
- checkout
- terraform/plan:
path: global
var_file: global/terraform.auto.tfvars
comment
module.foo.random_password.admin: Refreshing state... [id=none]
module.bar: Refreshing state... [id=mIyRmJ9aQcFJs_aB2Q]
module.baz: Refreshing state... [id=none]
...
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.foo.aws_db_instance.db will be updated in-place
~ resource "aws_db_instance" "db" {
+ domain = ""
+ domain_iam_role_name = ""
id = "foobar-20210322203922772700000004"
name = "foobar"
# Warning: this attribute value will be marked as sensitive and will
# not display in UI output after applying this change
~ password = (sensitive value)
+ performance_insights_kms_key_id = ""
+ replicate_source_db = ""
+ timezone = ""
# (47 unchanged attributes hidden)
}
I do think that compact_plan is being used: here's some output from the bash script.
+ TFMASK=tfmask
+ hash tfmask
+ TFMASK=cat
+ COMPACT_PLAN=compact_plan
+ hash compact_plan
In order to authenticate through OIDC it would be really useful if the Orb could support:
gcloud auth login --brief --cred-file ./myCredFile
Currently only the following authentication approach is used:
gcloud auth activate-service-account --key-file ./myKeyFile
When following the CircleCI documentation to authenticate via OIDC the credential file is generated as a result of this step:
gcloud iam workload-identity-pools create-cred-config
However, this credentials file is incompatible with the service account key file you would usually use to authenticate:
// line 761 of https://circleci.com/developer/orbs/orb/ovotech/terraform?version=1.11.15
export GOOGLE_APPLICATION_CREDENTIALS=/tmp/google_creds
gcloud auth activate-service-account --key-file /tmp/google_creds
We are using the terraform orb to plan and apply our infrastructure, but we aren't needed to have the jobs automatically close PRs in GitHub so we don't have any of the GitHub environment variables set.
terraform/plan
is absolutely fine with this, but terraform/apply
gives the following error message between the init and plan stages:
Traceback (most recent call last):
File "/tmp/github.py", line 10, in <module>
github_username = os.environ['GITHUB_USERNAME']
File "/usr/lib/python3.7/os.py", line 678, in __getitem__
raise KeyError(key) from None
KeyError: 'GITHUB_USERNAME'
Unable to update status on PR
The rest of the job still continues, but it would be nice not to have this error message, as it pollutes the output a little.
Orb version: ovotech/[email protected]
Terraform version: 0.12.17
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.