Giter Club home page Giter Club logo

hashicorp / terraform-provider-external Goto Github PK

View Code? Open in Web Editor NEW
177.0 22.0 49.0 9.96 MB

Utility provider that exists to provide an interface between Terraform and external programs. Useful for integrating Terraform with a system for which a first-class provider does not exist.

Home Page: https://registry.terraform.io/providers/hashicorp/external/latest

License: Mozilla Public License 2.0

Makefile 1.52% Go 95.32% HCL 3.17%
terraform terraform-provider external

terraform-provider-external's Introduction

Terraform Provider: External

The external provider is a special provider that exists to provide an interface between Terraform and external programs. Using this provider, it is possible to write separate programs that can participate in the Terraform workflow by implementing a specific protocol.

Requirements

Documentation, questions and discussions

Official documentation on how to use this provider can be found on the Terraform Registry. In case of specific questions or discussions, please use the HashiCorp Terraform Providers Discuss forums, in accordance with HashiCorp Community Guidelines.

We also provide:

  • Support page for help when using the provider
  • Contributing guidelines in case you want to help this project

Compatibility

Compatibility table between this provider, the Terraform Plugin Protocol version it implements, and Terraform:

External Provider Terraform Plugin Protocol Terraform
>= 2.0.x 5 >= 0.12
>= 1.1.x 4 and 5 >= 0.12
<= 1.x.x 4 <= 0.12

Details can be found querying the Registry API that return all the details about which version are currently available for a particular provider. Here are the details for Time (JSON response).

Development

Building

  1. git clone this repository and cd into its directory
  2. make will trigger the Golang build

The provided GNUmakefile defines additional commands generally useful during development, like for running tests, generating documentation, code formatting and linting. Taking a look at it's content is recommended.

Testing

In order to test the provider, you can run

  • make test to run provider tests
  • make testacc to run provider acceptance tests

It's important to note that acceptance tests (testacc) will actually spawn terraform and the provider. Read more about they work on the official page.

Generating documentation

This provider uses terraform-plugin-docs to generate documentation and store it in the docs/ directory. Once a release is cut, the Terraform Registry will download the documentation from docs/ and associate it with the release version. Read more about how this works on the official page.

Use make generate to ensure the documentation is regenerated with any changes.

Using a development build

If running tests and acceptance tests isn't enough, it's possible to set up a local terraform configuration to use a development builds of the provider. This can be achieved by leveraging the Terraform CLI configuration file development overrides.

First, use make install to place a fresh development build of the provider in your ${GOBIN} (defaults to ${GOPATH}/bin or ${HOME}/go/bin if ${GOPATH} is not set). Repeat this every time you make changes to the provider locally.

Then, setup your environment following these instructions to make your local terraform use your local build.

Testing GitHub Actions

This project uses GitHub Actions to realize its CI.

Sometimes it might be helpful to locally reproduce the behaviour of those actions, and for this we use act. Once installed, you can simulate the actions executed when opening a PR with:

# List of workflows for the 'pull_request' action
$ act -l pull_request

# Execute the workflows associated with the `pull_request' action 
$ act pull_request

Releasing

The release process is automated via GitHub Actions, and it's defined in the Workflow release.yml.

Each release is cut by pushing a semantically versioned tag to the default branch.

License

Mozilla Public License v2.0

terraform-provider-external's People

Contributors

achinthagunasekara avatar apparentlymart avatar appilon avatar austinvalle avatar bendbennett avatar bflad avatar bookshelfdave avatar bsadvanon avatar claire-labry avatar danielmschmidt avatar dependabot[bot] avatar faultymonk avatar github-actions[bot] avatar grubernaut avatar grv87 avatar hashicorp-tsccr[bot] avatar hc-github-team-tf-provider-devex avatar jeanneryan avatar justinclayton avatar katbyte avatar kmoe avatar paultyng avatar pdecat avatar pedrosaxu avatar radeksimko avatar sbgoods avatar seriousben avatar stack72 avatar team-tf-cdk avatar zimbatm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terraform-provider-external's Issues

External does not use the module directory as the current working directory

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Terraform v0.11.1

Affected Resource(s)

  • provider.external v1.0.0

Expected Behavior

The external data resource should run inside the module directory

Actual Behavior

runs where the terraform plan is run

References

Can probably be fixed by https://github.com/grv87/terraform/commit/569eef0fd111e06b25e89104c89d187303e65f63 but seems to have been lost since march

External fails when -refresh=false

Shouldn't external always run even with -refresh=false? Since it's calling something that has side effects? Or at least run the first time?

Terraform Version

Terraform v0.11.7

  • provider.aws v1.27.0
  • provider.external v1.0.0

Affected Resource(s)

  • external

Terraform Configuration Files

data "external" "external" {
  program = ["${path.module}/generate.sh"]
}

resource "aws_key_pair" "key" {
  public_key = "${data.external.external.result.public_key}"
}

generate.sh:

#!/bin/bash
echo {\"public_key\": \"test\"}

Expected Behavior

It should work.

Actual Behavior

Error: Error running plan: 1 error(s) occurred:

* module.key.aws_key_pair.key: 1 error(s) occurred:

* module.key.aws_key_pair.key: Resource 'data.external.external' does not have attribute 'result.public_key' for variable 'data.external.external.result.public_key'

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform plan -refresh=false

Bump Expected Minimum Go Version to 1.18

Terraform CLI and Provider Versions

TF: v1.2.7
Provider: v2.2.2

Use Cases or Problem Statement

Following the Go support policy and given the ecosystem availability of the latest Go minor version, it's time to upgrade. This will ensure that this project can use recent improvements to the Go runtime, standard library functionality, and continue to receive security updates.

Proposal

  • Run the following commands to upgrade the Go module files and automatically fix outdated Go code:
go mod edit -go=1.18
go mod tidy
go fix
  • Ensure any GitHub Actions workflows (.github/workflows/*.yml) use 1.19 in place of any 1.18 and 1.18 in place of any 1.17 or earlier
  • Ensure the README or any Contributing documentation notes the Go 1.18 expected minimum
  • (Not applicable to all projects) Ensure the .go-version is at least 1.18 or later

How much impact is this issue causing?

Low

Additional Information

References

Code of Conduct

  • I agree to follow this project's Code of Conduct

"external" provider returns an invalid json

This issue was originally opened by @Phydeauxman as hashicorp/terraform#17632. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.11.3
+ provider.azurerm v1.2.0
+ provider.external v1.0.0

Terraform Configuration Files

variable ilbase_resourceId {
  default = "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"
}


data "external" "aseVip" {
  program = ["az", "resource", "show", "--ids", "${var.ilbase_resourceId}"]
}

Debug Output

Crash Output

Expected Behavior

The JSON object would be returned to terraform

Actual Behavior

Terraform config generates error:

data.external.aseVip: data.external.aseVip: command "az" produced invalid JSON: json: cannot unmarshal object into Go value of type string

Steps to Reproduce

Additional Context

If after I provision the App Service Environment I run the command below:

az resource show --ids "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"

It returns the JSON object below:

{
      "additionalProperties": {
        "internalIpAddress": "10.10.1.11",
        "outboundIpAddresses": [
          "52.224.70.119"
        ],
        "serviceIpAddress": "52.224.70.119",
        "vipMappings": []
      },
      "id": null,
      "identity": null,
      "kind": null,
      "location": null,
      "managedBy": null,
      "name": null,
      "plan": null,
      "properties": null,
      "sku": null,
      "tags": null,
      "type": null
}

References

data variables must be four parts: data.TYPE.NAME.ATTR in: ${data.external.script}

This issue was originally opened by @JasonOliverEmaze as hashicorp/terraform#20586. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.11.11 + provider.external v1.0.0

Terraform Configuration Files

  program = ["python", "${path.module}/secret.py", "-id", "SFTP-Key", "-keyname", "private-key"]
}

output "Private Key" {
  value = "${data.external.script}"
}

Expected Behavior

I am hoping for the script to output the variable as it would be if the python script was run (stdout).

Actual Behavior

I believe I need to include an attribute such as ${data.external.script.*}, ${data.external.script.all}, etc.?

`Error: Error loading C:\Users\oliverj\Documents\Code\github\test\test.tf: Error reading config for output Private Key: data.external.script: data variables must be four parts: data.TYPE.NAME.ATTR in:

${data.external.script}`

Steps to Reproduce

  1. terraform init
  2. terraform plan or terraform apply

References

does not have attribute result.foo

Terraform v0.9.8

${data.external.foo.result.bla}

will not actually work

* data.template_cloudinit_config.border: Resource 'data.external.border-gw-key' does not have attribute 'result.priv' for variable 'data.external.border-gw-key.result.priv'

the correct usage appears to be

${lookup(data.external.foo.result, "bla")}

it would be great if the documentation mentioned that. Took me quite some hairpulling to figure that out, as everything i found googling appears to be using result.attr

Error: Incompatible provider version (darwin_arm64)

Terraform CLI and Provider Versions

terraform version

Terraform v1.1.4
on darwin_arm64

Terraform Configuration

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding latest version of hashicorp/helm...
- Finding latest version of hashicorp/template...
- Finding latest version of hashicorp/kubernetes...
- Finding latest version of hashicorp/external...
- Finding datadog/datadog versions matching "~> 2.0"...
- Finding integrations/github versions matching "~> 4.0"...
- Finding latest version of hashicorp/random...
- Finding hashicorp/http versions matching "~> 2.0"...
- Finding latest version of hashicorp/null...
- Finding cloudflare/cloudflare versions matching "~> 3.0"...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/kubernetes v2.8.0...
- Installed hashicorp/kubernetes v2.8.0 (signed by HashiCorp)
- Installing hashicorp/http v2.1.0...
- Installed hashicorp/http v2.1.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.74.2...
- Installed hashicorp/aws v3.74.2 (signed by HashiCorp)
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (signed by HashiCorp)
- Installing hashicorp/null v3.1.0...
- Installed hashicorp/null v3.1.0 (signed by HashiCorp)
- Installing cloudflare/cloudflare v3.9.0...
- Installed cloudflare/cloudflare v3.9.0 (signed by a HashiCorp partner, key ID DE413CEC881C3283)
- Installing hashicorp/helm v2.4.1...
- Installed hashicorp/helm v2.4.1 (signed by HashiCorp)
- Installing hashicorp/external v2.2.0...
- Installing datadog/datadog v2.26.1...
- Installed datadog/datadog v2.26.1 (signed by a HashiCorp partner, key ID FB70BE941301C3EA)
- Installing integrations/github v4.20.0...
- Installed integrations/github v4.20.0 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)

Expected Behavior

Successful initialisation and installation of the related plugins

Actual Behavior

╷
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
╵

Steps to Reproduce

  1. terraform init

How much impact is this issue causing?

High

Logs

No response

Additional Information

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Bug in "external" data provider: Cannot extract value from python-produced json, also extracted value is neither blank nor non-blank

This issue was originally opened by @dkyrkou as hashicorp/terraform#12930. It was migrated here as part of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.9.1

Affected Resource(s)

Please list the resources as a list, for example:

  • data external provider

Terraform Configuration Files

provider.tf

provider "aws" {
  profile                 = "saml"
  region                  = "${data.external.aws_auth.result["region"]}"
}

data "external" "aws_auth" {
     program = ["python", "${path.module}myscript.py"]
}

myscript.py

#!/usr/bin/python
import json
import sys
test_dict = {}
test_dict['region'] =  "eu-west-1"
json.dump(test_dict, sys.stdout)

Debug Output

When I run "terraform plan", I get the following error

Error running plan: 1 error(s) occurred:

* provider.aws: Not a valid region: 

When the python script is run by itself, it produces the following valid json:

{"region": "eu-west-1"}

Expected Behavior

The string "eu-west-1" should be extracted from the python-produced json and get assigned to "region" in provider.aws

Actual Behavior

When I run "terraform plan", nothing gets assigned to "region". As you can see, nothing exists next to "Not a valid region:" output.

Steps to Reproduce

  1. terraform plan

Important Factoids

The truly weird thing is that if you modify provider.tf to the following

provider "aws" {
  profile = "saml"
  region = "${data.external.aws_auth.result["region"] == "" ? "eu-west-1" : "eu-west-2"}"
}

The assigned value is still nothing! How? As far as I know, the "?" operator is binary, which means that either "eu-west-1" or "eu-west-2" should get assigned to region, yet absolutely NOTHING is assigned!

Truly weird...

PS: Please no responses telling me that I should get rid of the script and assign the string directly to region, the actual production script reads from a configuration file, the provided script is just a small script that demonstrates the same problem.

external data source not updated

Terraform Version

Terraform v0.11.11

  • provider.azurerm v1.27.1
  • provider.external v1.1.2
  • provider.null v2.1.2

Affected Resource(s)

data.external

Terraform Configuration Files

data "external" "az_cosmos_db" {

  program = ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", "./external.data.cosmosdb.ps1"]

  query = {
    ResourceGroupName   = "${var.resource_group_name}"
    CosmosDbAccountName = "${var.cosmosdb_account_name}"
    DatabaseName        = "actions"
  }
}

resource "null_resource" "cosmos_actionsdb" {
  count = 1
  depends_on = ["azurerm_cosmosdb_account.default"]

  triggers = {
    db_exists = "${data.external.az_cosmos_db.result.DbExists}"
  }
}

terraform.tfstate

"data.external.az_cosmos_db": {
                    "type": "external",
                    "depends_on": [],
                    "primary": {
                        "id": "-",
                        "attributes": {
                            "id": "-",
                            "program.#": "6",
                            "program.0": "pwsh",
                            "program.1": "-NoLogo",
                            "program.2": "-NoProfile",
                            "program.3": "-NonInteractive",
                            "program.4": "-Command",
                            "program.5": "./external.data.cosmosdb.ps1",
                            "query.%": "3",
                            "query.CosmosDbAccountName": "test-sql-dev-cdb-we",
                            "query.DatabaseName": "actions",
                            "query.ResourceGroupName": "test-dev-rg-we",
                            "result.%": "5",
                            "result.AccountExists": "true",
                            "result.CosmosDbAccountName": "test-sql-dev-cdb-we",
                            "result.CreateDatabase": "false",
                            "result.DbExists": "true",
                            "result.ResourceGroupName": "test-dev-rg-we"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.external"
                },
                "null_resource.cosmos_actionsdb": {
                    "type": "null_resource",
                    "depends_on": [
                        "azurerm_cosmosdb_account.default",
                        "data.external.az_cosmos_db"
                    ],
                    "primary": {
                        "id": "4010272293405614768",
                        "attributes": {
                            "id": "4010272293405614768",
                            "triggers.%": "1",
                            "triggers.db_exists": "false"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.null"
                }
            },

Expected Behavior

when I run terraform refresh\apply I expected statefile to update db_exists with value from external data source

Actual Behavior

the first time I run terraform apply, it detects db_exists -> false, after I apply this state is saved to the statefile as "triggers.db_exists": "false"
the second time I run terraform apply, I expected "triggers.db_exists": "true"

Data source "external" fails on Linux platform

Terraform CLI and Provider Versions

Terraform CLI: 1.1.1

External provider: 2.2.2

Terraform Configuration

data "external" "tools_accounts" {
   program  = ["aws", "organizations", "list-accounts-for-parent", "--parent-id", local.deployment_ou, "--query", "Accounts[?Name==`${local.tools_account}`] | [0]"]
}

Expected Behavior

When that external AWS CLI command runs it returns result similar to below:

{
    "Id": "1234567890",
    "Arn": "arn:aws:organizations::1234567890:account/o-p12345e/1234567890",
    "Email": "[email protected]",
    "Name": "account-alias",
    "Status": "ACTIVE",
    "JoinedMethod": "CREATED",
    "JoinedTimestamp": "2022-06-15T14:00:22.158000-04:00"
}

External command returns map of string values which satisfies the data source requirement. Data source's result output should provide this map.

Actual Behavior

When running inside of AWS Codebuild/Linux environment it throws the error

Cannot unmarshal number into Go value of type string

Steps to Reproduce

  1. terraform apply

How much impact is this issue causing?

Medium

Logs

https://gist.github.com/YuriGal/fc74a9d87d3ab3148af58b569531a15d

Additional Information

The above configuration works on local laptop (Mac/Darwin, same terraform version, same provider version). I it just fails in AWS Codebuild/Linux environment.

Code of Conduct

  • I agree to follow this project's Code of Conduct

Switch Output to DynamicPseudoType

This also implies switching to the new Terraform Plugin framework. Switching to DynamicPseudoType will allow the output to be parsed from any value, whether JSON array or object, or if unparseable as the raw string.

Returned keys containing a "." create a nested map instead of keeping the "."

Terraform Version#

Terraform v0.11.7
+ provider.external v1.0.0
+ provider.null v1.0.0

Affected Resource(s)

External data provider

Terraform Configuration Files

data external ext {
  program = ["echo", "{\"foo.bar\": \"value\"}"]
}

resource "null_resource" "test" {
  triggers {
    foo = "${format("%v", data.external.ext.result)}"
  }
}

Debug Output

Does not contain anything useful related to the issue. I can provide it if needed.

Expected Behavior

data.external.ext.result should contain a flat map: {"foo.bar": "value"}.

Actual Behavior

The result of the external is converted into a nested map, where the keys are split at ".".

+ null_resource.test
    id:           <computed>
    triggers.%:   "1"
    triggers.foo: "map[foo:{Variable (TypeMap): map[bar:{Variable (TypeString): value}]}]"

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform plan

If this is intended behaviour, please regard this as a documentation bug, since I could not find any explanation of this behaviour in the docs.

Migrate Documentation to terraform-plugin-docs

The "standard library" Terraform Providers should implement nascent provider development tooling to encourage consistency, foster automation where possible, and discover bugs/enhancements to that tooling. To that end, this provider's documentation should be switched to terraform-plugin-docs, including:

  • Migrating directory structures and files as necessary
  • Enabling automation for documentation generation (e.g. make gen or go generate ./...)
  • Enabling automated checking that documentation has been re-generated during pull request testing (e.g. no differences)

Feature Request - Allow list/array in `query` and `result`

This issue was originally opened by @vikas027 as hashicorp/terraform#12249. It was migrated here as part of the provider split. The original body of the issue is below.


Terraform Version

0.8.7

Affected Resource(s)

external data source

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]

  query = {
    list_of_images = "${var.list_of_images}"
  }
}

Expected Behavior

List list_of_images should have been passed to the script.

Actual Behavior

Error refreshing state: 1 error(s) occurred:

* query: 1 error(s) decoding:

* '[list_of_images]' expected type 'string', got unconvertible type '[]interface {}'

Terraform Version

0.8.7

Affected Resource(s)

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]
}

create_policy.sh

#!/bin/bash
set -e
# This works
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
# This throws an error
jq -n '{"Version":"2008-10-17","Statement":[{"Sid":"repo_policy","Effect":"Allow","Principal":{"AWS":["arn:aws:iam::${account_id}:root","arn:aws:iam::${account_id}:role/ecr_restricted_admin"]},"Action":"*"}]}'

Expected Behavior

Terraform shouldn't thrown an error when the the bash script is producing a valid json

$ bash modules/ecr/create_policy.sh  | jq .
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "repo_policy",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::${account_id}:root",
          "arn:aws:iam::${account_id}:role/ecr_restricted_admin"
        ]
      },
      "Action": "*"
    }
  ]
}

Actual Behavior

* data.external.create_policy: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string

Steps to Reproduce

Create the resource and the script files and run terraform plan

Support option to output stderr regardless of failure

Terraform Version

Terraform v0.14.5
+ provider registry.terraform.io/hashicorp/aws v3.34.0
+ provider registry.terraform.io/hashicorp/external v2.1.0

Affected Resource(s)

  • external

Terraform Configuration Files

data "external" "test" {
  program = [
    "bash", "-ec",
    <<-EOF
    echo script making progress >&2
    echo '{}'
    # exit 1  # stderr gets forwarded to console when exit code != 0
    EOF
  ]
}

Debug Output

Plan
No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Panic Output

N/a

Expected Behavior

terraform {plan,apply} should (optionally) print the stderr output even when the exit code from the script is zero.

Actual Behavior

terraform {plan,apply} does not print the stderr output unless the exit code is nonzero.

Steps to Reproduce

  1. terraform apply or terraform plan

Important Factoids

N/a

References

scottwinkler/terraform-provider-shell#27, maybe

Terraform v0.11.15 pulls 2.2.0 by default instead of v1.2

Terraform v0.11.15 pulls 2.2.0 instead of v1.2

Terraform Version

0.11.15 (and other versions that use the 1.2 version)

Affected Resource(s)

Running terraform init with version 0.11.15 and the external provider

Terraform Configuration Files

provider "external" { }

While using terraform 0.11.15

Debug Output

/home # terraform -version
Terraform v0.11.15
...
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "external" (2.2.0)...

Expected Behavior

Terraform 0.11.15 downloads version 1.2 instead of 2.2.0

Actual Behavior

Terraform 0.11.15 downloads version 2.2.0

Steps to Reproduce

  1. terraform init

Important Factoids

Build logs show push failed for some part of the build/release:
https://github.com/hashicorp/terraform-provider-external/runs/4694624609?check_suite_focus=true

ansible-vault command not found

Terraform CLI and Provider Versions

$ terraform --version
Terraform v0.14.3
Your version of Terraform is out of date! The latest version
is 1.2.5. You can update by downloading from https://www.terraform.io/downloads.html
$ terraform init
Initializing the backend...
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...

  • Finding latest version of hashicorp/azurerm...
  • Finding latest version of hashicorp/external...
  • Installing hashicorp/azurerm v3.14.0...
  • Installed hashicorp/azurerm v3.14.0 (self-signed, key ID XXXXXXXXX)
  • Installing hashicorp/external v2.2.2...
  • Installed hashicorp/external v2.2.2 (self-signed, key ID XXXXXXXX)

Terraform Configuration

# variables.tf

data "external" "ansible" {
  program  = [
    "./vault.sh",
    "group_vars/dev/vars.yml"
  ]
}

# vault.sh
#!/usr/bin/env bash

set -e

vault_file_path=$1
if [ -z $vault_file_path ]; then
    echo You need to provide vault file path as the first argument. None received.
    exit 2
fi

ansible-vault decrypt --output - $vault_file_path --vault-password-file pass

Expected Behavior

data source should be able to run the script.

Actual Behavior

$ terraform plan -out=$PLAN -input=false
Acquiring state lock. This may take a few moments...
Error: External Program Execution Failed
on variables.tf line 26, in data "external" "ansible":
26: program = [
27: "./vault.sh",
28: "group_vars/dev/vars.yml"
29: ]
The data source received an unexpected error while attempting to execute the
program.
Program: ./vault.sh
Error Message: ./vault.sh: line 11: ansible-vault: command not found
State: exit status 127

Steps to Reproduce

  1. terraform plan -out=$PLAN -input=false
    Im running it from gitlab ci.

How much impact is this issue causing?

High

Logs

No response

Additional Information

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Enable go-changelog Automation

The "standard library" Terraform Providers should implement nascent provider development tooling to encourage consistency, foster automation where possible, and discover bugs/enhancements to that tooling. To that end, this provider's CHANGELOG handling should be switched to go-changelog, including:

  • Adding .changelog directory and template files
  • Enabling automation for regenerating the CHANGELOG (e.g. scripts or GitHub Actions)
  • (If enhancements are made available upstream in time) Enabling automation for checking CHANGELOG entries and formatting

command "Powershell.exe" produced invalid JSON: unexpected end of JSON input

Hi there,

i am trying to use terraform external provider to run a powershell script

i am not able to pass parameters to the script from terraform.

I use the below code to run the script

variable "file_path" {
type="string"
description="Enter the powershell file path"
default="../../Services/azurerm_iam_role/Powershell_Scripts/test.ps1"
}

data "external" "powershell_test" {
program = ["Powershell.exe", "${var.file_path}"]

query = {
foo = "asdf"
bar = "Hardcoded"
}
}

The below is my powershell script file code

Read stdin as string

$jsonpayload = [Console]::In.ReadLine()

Convert to JSON

$json = ConvertFrom-Json $jsonpayload

Access JSON values

$foo = $json.foo
$bar = $json.bar

while executing the terraform apply i got the below error

command "Powershell.exe" produced invalid JSON: unexpected end of JSON input

Can you please help me out with a fix or alternate solution.

Create provider design doc

Maintenance of the standard library providers prioritises stability and correctness relative to the provider's intended feature set. Create a design document describing this feature set, and any other design considerations which influence the architecture of the provider and what can and cannot be added to it.

External Not Run When Query Contains Computed Values

Terraform Version

Terraform v0.11.7
+ provider.aws v1.27.0
+ provider.external v1.0.0
+ provider.template v1.0.0

Affected Resource(s)

  • data.external

Terraform Configuration Files

In the real setup, I have a data.template_file injecting ARNs and then this python renderer transforming a DSL into a full Swagger spec. However, I was able to condense it down to just the lambda ARN going directly into the external provider.

Working

data "external" "swagger_renderer" {
  program = ["python3", "${path.module}/data/swagger_renderer.py"]
  query = {
    template_input = "${var.lambda_handler_invoke_arn}"
  }
}

Failing

data "external" "swagger_renderer" {
  program = ["python3", "${path.module}/data/swagger_renderer.py"]
  query = {
    # computed value on not yet created resource
    template_input = "${aws_lambda_function.api_handler.invoke_arn}"
  }
}

Debug Output

https://gist.github.com/danieladams456/76cb77fc649bc1dc40746889d8b14bf2

Expected Behavior

Plan should either:

  1. not call external provider
  2. use a placeholder

Based on this core PR (hashicorp/terraform#10670), I tried adding depends_on to defer evaluation until after the lambda function was created, but it didn't seem to change anything.

Actual Behavior

It sees there is no output variable, but the external script is not run. I verified this with file-based logging side effects.
aws_api_gateway_rest_api.api: Resource 'data.external.swagger_renderer' does not have attribute 'result.rendered' for variable 'data.external.swagger_renderer.result.rendered'

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform plan

References

hashicorp/terraform#10670

Workaround

Put anything the template depends on in a separate module and use outputs/variables to move the data between modules. The external provider works fine with simple variables.

External Data Source With Multiple Parameters

Hi there,

I'm running Terraform v0.11.11, and I’m calling an external tool that takes a total of 3 parameters in the following order: a regex expression and 2 strings. Since the regex being passed can be quite complex, I need to have it enclosed in double quotes. The trouble is that when terraform invokes this tool, things get messed up, as follows.

Using cmd.exe:
image

The tools is invoked, however the escape characters are send as-is, while an extra double quote is added at the very end for some unknown reason:
image

Using Powershell:
image

The tool is invoked, however no quotes are placed around the first parameter, as I’d want to. There’s just one quote, but that corresponds to the one with which the whole command line starts with; since the double quotes are enclosing the tool’s filename, it’s not a problem. But the regex itself is not surrounded by anything:
image

I believe there are 2 things I need to get right – the string as entered in config.tf needs to be syntactically correct from terraform’s standpoint, then it also needs to be parsed appropriately by the shell I’m invoking. I’m somehow missing one or both.

Adding depends_on to an external data source causes "apply" to prompt for changes each time

I've been using an external data source to do some work after resources are deployed, so I need the external data source to run at the end of the resource creation. When I add depends_on to the data source, it correctly runs after the dependent resource, but subsequent runs of apply cause Terraform to prompt for confirmation. If I comment out the depends_on, the data source still runs, but Terraform doesn't throw up the confirmation prompt.

Note that this example uses an aws_ecs_cluster as the dependent resource because that's how I discovered this issue, but it appears to affect other resources as well.

Terraform Version

Terraform v0.12.26
+ provider.aws v2.64.0
+ provider.external v1.2.0

Affected Resource(s)

  • External data sources

Terraform Configuration Files

terraform {
  backend "local" {}
}

terraform {
  required_version = ">= 0.12"
}

provider "aws" {
  region  = "us-west-2"
  version = "~> 2.64"
}

provider "external" {
  version = "~> 1.2"
}

resource "aws_ecs_cluster" "test" {
  name = "test-cluster1"
}

data "external" "noop" {
  program = ["python", "-c", "print('{}')"]
  depends_on = [aws_ecs_cluster.test]
}

Debug Output

Can be provided if absolutely necessary; its a lot to sanitize.

Expected Behavior

Terraform should run the external data source after the dependent resource without prompting. Here's an example showing the expected behavior with depends_on commented out:

data.external.noop: Refreshing state...
aws_ecs_cluster.test: Refreshing state... [id=arn:aws:ecs:us-west-2::cluster/test-cluster1]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Actual Behavior

Terraform prompts every time:

aws_ecs_cluster.test: Refreshing state... [id=arn:aws:ecs:us-west-2::cluster/test-cluster1]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.external.noop will be read during apply
  # (config refers to values not yet known)
 <= data "external" "noop"  {
      + id      = (known after apply)
      + program = [
          + "python",
          + "-c",
          + "print('{}')",
        ]
      + result  = (known after apply)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Steps to Reproduce

  1. Run terraform apply on the above sample
  2. Run terraform apply again and note the confirmation prompt
  3. Comment out the depends_on and run terraform apply again and note that the data source runs without prompting.

Broken 12/14/21: Error installing provider "external": openpgp: signature made by unknown entity

Hi there -- I have been a daily user of terraform and today we hit an odd issue without any change to our template files or infrastructure.

Terraform Version

Terraform v0.11.14
AWS Provider v2.70

and

Terraform v0.12.31
AWS Provider v2.70

Debug Output

https://gist.github.com/zbornstein/c8c98081b2ab7234ec5bfdfda0dc9234

Expected Behavior

Normal Execution

Actual Behavior

Execution Failure

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform init

Other github issues in terraform point to connectivity issues to download, but I have ruled that out -- and those errors highlight provider "aws" not provider "external". Would the recently published release have any impact to me today?

Provider Release Built With Go 1.16.12

Terraform Version

Terraform v1.1.0
on darwin_amd64
+ provider registry.terraform.io/hashicorp/external v2.1.0

Terraform Configuration Files

N/A

Description

Go has recently released versions 1.16.12 and 1.17.5, which includes (amongst other changes) mitigations for CVE-2021-44717.

syscall: don’t close fd 0 on ForkExec error

When a Go program running on a Unix system is out of file descriptors and calls syscall.ForkExec (including indirectly by using the os/exec package), syscall.ForkExec can close file descriptor 0 as it fails. If this happens (or can be provoked) repeatedly, it can result in misdirected I/O such as writing network traffic intended for one connection to a different connection, or content intended for one file to a different one.

For users who cannot immediately update to the new release, the bug can be mitigated by raising the per-process file descriptor limit.

The latest available external provider release was built with a Go version prior to these, so a new release should be done.

References

Bump Development/Build Minimum Go Version to 1.17

Terraform CLI and Provider Versions

N/A (main branch development)

Use Cases or Problem Statement

Following the Go support policy and given the ecosystem availability and stability of the latest Go minor version, it's time to upgrade. This will ensure that this project can use recent improvements to the Go runtime, standard library functionality, and continue to receive security updates

Proposal

  • Run the following commands to upgrade the Go module files and remove deprecated syntax such as //+build:
go mod edit -go=1.17
go mod tidy
go fix
  • Ensure any GitHub Actions workflows (.github/workflows/*.yml) use 1.18 in place of any 1.17 and 1.17 in place of any 1.16 or earlier
  • Ensure the README or any Contributing documentation notes the Go 1.17 expected minimum
  • (Not applicable to all projects) Ensure the .go-version is at least 1.17 or later
  • Enable the tenv linter in .golangci.yml and remediate any issues.

How much impact is this issue causing?

Medium

Additional Information

Code of Conduct

  • I agree to follow this project's Code of Conduct

The terraform-provider-external_v2.2.0_x5 plugin crashed!

Terraform CLI and Provider Versions

terraform v1.1
hashicorp/external v2.2.0

Terraform Configuration

terraform {
  required_providers {
    acme = {
      source = "vancluever/acme"
    }
    aws = {
      source = "hashicorp/aws"
    }
    cloudflare = {
      source = "cloudflare/cloudflare"
    }
    datadog = {
      source = "datadog/datadog"
    }
    external = {
      source = "hashicorp/external"
    }
    gitlab = {
      source = "gitlabhq/gitlab"
    }
    google = {
      source = "hashicorp/google"
    }
    null = {
      source = "hashicorp/null"
    }
    random = {
      source = "hashicorp/random"
    }
    tls = {
      source = "hashicorp/tls"
    }
    vault = {
      source = "hashicorp/vault"
    }
  }
  required_version = ">= 0.14"
  experiments = [module_variable_optional_attrs]
}
terraform {
  backend "gcs" {
    bucket = "tf_state_gcp_projects_prod"
  }
  required_version = ">= 0.13"
}

Expected Behavior

No errors

Actual Behavior

│ Error: Plugin did not respond

│ with module.kiwibase_develop.data.external.codeowners,
│ on gcp_project/gitlab_protect.tf line 43, in data "external" "codeowners":
│ 43: data "external" "codeowners" {

│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadDataSource call. The plugin logs may contain
│ more details.


│ Error: Request cancelled

│ with module.booking_production.data.external.codeowners,
│ on gcp_project/gitlab_protect.tf line 43, in data "external" "codeowners":
│ 43: data "external" "codeowners" {

│ The plugin.(*GRPCProvider).ValidateDataResourceConfig request was
│ cancelled.

Stack trace from the terraform-provider-external_v2.2.0_x5 plugin:
panic: interface conversion: interface {} is nil, not string
goroutine 26 [running]:
github.com/terraform-providers/terraform-provider-external/internal/provider.dataSourceRead(0xd35b68, 0xc0003871a0, 0xc0001ffa00, 0x0, 0x0, 0xc00028cce0, 0xc00029b8b8, 0x40e0f8)
github.com/terraform-providers/terraform-provider-external/internal/provider/data_source.go:88 +0x1b66
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000158380, 0xd35af8, 0xc000153140, 0xc0001ffa00, 0x0, 0x0, 0x0, 0x0, 0x0)
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:358 +0x17f
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc000158380, 0xd35af8, 0xc000153140, 0xc0001ff900, 0x0, 0x0, 0x0, 0xc0001ff900, 0x0, 0x0)
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:569 +0xfd
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0xc000138768, 0xd35af8, 0xc000153140, 0xc00075c4a0, 0xc53093, 0x12, 0x0)
github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:1133 +0x518
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0xc0001fe500, 0xd35ba0, 0xc000153140, 0xc00019ca00, 0x0, 0x0, 0x0)
github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:478 +0x2f5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler(0xc1b500, 0xc0001fe500, 0xd35ba0, 0xc000282060, 0xc000386d20, 0x0, 0xd35ba0, 0xc000282060, 0xc000286000, 0xa1)
github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:416 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00016e540, 0xd3c778, 0xc000582c00, 0xc000370100, 0xc000447140, 0x11429b0, 0x0, 0x0, 0x0)
google.golang.org/[email protected]/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00016e540, 0xd3c778, 0xc000582c00, 0xc000370100, 0x0)
google.golang.org/[email protected]/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000042220, 0xc00016e540, 0xd3c778, 0xc000582c00, 0xc000370100)
google.golang.org/[email protected]/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/[email protected]/server.go:857 +0x1fd
Error: The terraform-provider-external_v2.2.0_x5 plugin crashed!

Steps to Reproduce

  1. terraform plan

How much impact is this issue causing?

Medium

Logs

No response

Additional Information

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Go 1.19 os/exec Error on Executable in Current Directory

Terraform CLI and Provider Versions

N/A and N/A

Use Cases or Problem Statement

Go 1.19 was recently released and one of its minor version changes is to the os/exec package when the executable is found in the current directory. This will now return an error to prevent potentially unexpected or malicious code execution.

Additional information from the updated os/exec documentation:

The functions Command and LookPath look for a program in the directories listed in the current path, following the conventions of the host operating system. Operating systems have for decades included the current directory in this search, sometimes implicitly and sometimes configured explicitly that way by default. Modern practice is that including the current directory is usually unexpected and often leads to security problems.

To avoid those security problems, as of Go 1.19, this package will not resolve a program using an implicit or explicit path entry relative to the current directory. That is, if you run exec.LookPath("go"), it will not successfully return ./go on Unix nor .\go.exe on Windows, no matter how the path is configured. Instead, if the usual path algorithms would result in that answer, these functions return an error err satisfying errors.Is(err, ErrDot).

For example, consider these two program snippets:

...

cmd := exec.Command("prog")
if err := cmd.Run(); err != nil {
	log.Fatal(err)
}

These will not find and run ./prog or .\prog.exe, no matter how the current path is configured.

Code that always wants to run a program from the current directory can be rewritten to say "./prog" instead of "prog".

Just upgrading to Go 1.19 without code changes could introduce regressions for existing Terraform configurations that are dependent on the current directory lookup behavior.

Proposal

Multiple proposals will mention the os/exec package error workaround, which I'll describe up front here:

Code that insists on including results from relative path entries can instead override the error using an errors.Is check:

...

cmd := exec.Command("prog")
if errors.Is(cmd.Err, exec.ErrDot) {
	cmd.Err = nil
}
if err := cmd.Run(); err != nil {
	log.Fatal(err)
}

Proposals should not include the GODEBUG workaround as:

Setting the environment variable GODEBUG=execerrdot=0 disables generation of ErrDot entirely, temporarily restoring the pre-Go 1.19 behavior for programs that are unable to apply more targeted fixes. A future version of Go may remove support for this variable.

Proposal 1

Do nothing in the provider code after Go 1.19 upgrade, note change in CHANGELOG/release notes, and release new minor version. Typically, Go version upgrades are considered as "technical debt" for provider codebases and this change could be treated more as a bug fix in the name of security. This will potentially break existing Terraform configurations that do not update to .//.\ syntax after upgrading the provider, which is not a great practitioner experience.

Proposal 2

Do nothing in the provider code after Go 1.19 upgrade, note change in CHANGELOG/release notes, and release new major version. This will treat the change as a breaking change, but provide practitioners no advanced notice about potential issues in existing configurations. This will signal to practitioners the change may require configuration updates, but could stifle rollout in environments where it is hard to determine if the change will cause issues.

Proposal 3

Update the provider code with the os/exec workaround when performing the Go 1.19 upgrade, release new minor version, and leave workaround in place. This will prevent practitioners from potential configuration updates and will never notify practitioners of the potential security issue. In general, providers should promote a better security posture when possible for practitioners, so this is likely a non-starter.

Proposal 4

Update the provider code with the os/exec workaround when performing the Go 1.19 upgrade, release new minor version, and remove the os/exec workaround when releasing a new major version. This will prevent practitioners from potential configuration updates until a major version release and enables the provider to upgrade its Go version as expected. Practitioners will not be notified of the potential security issue or have advanced notice of the change.

Proposal 5

Similar to proposal 4, but when implementing the os/exec workaround raise a warning diagnostic instead of skipping over the error completely.

How much impact is this issue causing?

Medium

Additional Information

Code of Conduct

  • I agree to follow this project's Code of Conduct

data external does not allow me to run any linux command

This issue was originally opened by @xfustero as hashicorp/terraform#18524. It was migrated here as a result of the provider split. The original body of the issue is below.


Hi there,

I am using Terraform v0.11.7 and AWS as cloud provider. I am using an ec2 instance type m5 and I have a problem when using multiple EBS volumes. It seems the instance maps the devices NVMe randomly so I can not hardcode on TF user_data the nvm device name to a certain mountpoint.

To solve it I defined a data "external" in the templates.tf that runs script mapNVMdevices.sh. This scripts found each NVM we have (2) to which device /dev/xvdN is associated and set variables in template_file defining the nvme1n1 and the nvme2m1 device name associated (/dev/xvdX, /dev/xvdY) and their mount points.

These variables are used in user_data.yml to mount it correctly.

templates.tf

data "external" "nvm_devices" {
  program = ["bash", "${path.module}/mapNVMdevices.sh"]
}

data "template_file" "user_data" {
  template = "${file("${path.module}/user_data.yml")}"
    device_nvme1n1 = "${data.external.nvm_devices.result.device_nvme1n1}"
    device_nvme2n1 = "${data.external.nvm_devices.result.device_nvme2n1}"
    mnt1           = "${data.external.nvm_devices.result.mnt1}"
    mnt2           = "${data.external.nvm_devices.result.mnt2}"
  }
}

user_data.yml

mounts:
  - [ ${device_nvme1n1}, ${mnt1}, "ext4", "defaults", "0", "0" ]
  - [ ${device_nvme2n1}, ${mnt2}, "ext4", "defaults", "0", "0" ]

However a terraform plan is complaining about command not found for commands that are on the instance I am provisioning. I can run the bash script in that instance with no problem:

Error: Error refreshing state: 1 error(s) occurred:

* module.central-ftp.data.external.nvm_devices: 1 error(s) occurred:

* module.central-ftp.data.external.nvm_devices: data.external.nvm_devices: failed to execute "bash": ../mapNVMdevices.sh: line 27: ebsnvme-id: command not found
.../mapNVMdevices.sh: line 28: ebsnvme-id: command not found
../mapNVMdevices.sh: line 42: jq: command not found

ebsnvme-id and jq are in the path and even writing absolute path in the bash script I get this error.

Feature Request: Allow Pre-formatted Strings in Query Argument

Terraform Version

0.11.3

Affected Resource(s)

Please list the resources as a list, for example:

  • data.external

Terraform Configuration Files

data "external" "transform" {
  program = [ "jq",
              "--from-file",
              "${path.module}/assets/query.jq" ]

  query   = "{ \"foo\":\"bar\" }"
}

Expected Behavior

The string should be passed directly to the program via stdin.

Actual Behavior

An error is received:

Error: module.iam_roles.data.external.ec2_roles: query: should be a map

Steps to Reproduce

  1. terraform init
  2. terraform plan

Important Factoids

n/a

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

Notes

This could be a way to work around the request in #2 by using jsonencode() and passing the string directly to the external program.

Provide method to hide sensitive query data in Diffs

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.

Terraform v0.13.1

Affected Resource(s)

Please list the resources as a list, for example:

  • data.external

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

This will show the API key in the diff, which is undesirable.

data "external" "submit_complex_process" {
  query = {
    API_KEY = var.API_KEY
  }
}

A Potential fix:

data "external" "submit_complex_process" {
  sensitive_query = {
    API_KEY = var.API_KEY
  }
}

or:

data "external" "submit_complex_process" {
  query = {
    API_KEY = var.API_KEY
  }
  sensitive_keys = ["API_KEY"]
}

external provider does not log what its trying to do

Terraform-provider-external v1.0.0
Terraform all versions

I'm running a complicated external command. It's not working. What would be really useful in debugging why is that if I set TF_LOG to trace, the provider logs what its attempting to do, so I can reconcile the output, but it doesnt log anything.

This makes debugging so much more difficult than it needs to be. Could we have some debug please?!

External resource provider

Hi,

Are you are working on/considering adding support for resources (next to data) to this provider?

While you can use a null_resource to create a sort of 'scripted' provider using provisioners. The support for running a script on destruction is very limited there (if you set the count to 0 it is executed, if you just remove it the script is never executed). It also lacks the option to set output variables based on the output of your script.

Feature Request: Allow arrays from external datasource result

Terraform Version

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/aws v2.68.0
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/github v3.1.0
+ provider registry.terraform.io/hashicorp/random v2.2.1

Affected Resource(s)

  • data external

Terraform Configuration Files

Given the following Terraform:

data "external" "bastions" {
    program = ["bash", "${path.module}/external/bastion-ips.sh"]
}


# Create our security group referencing the IP ranges
resource "aws_security_group" "ssh" {
  name        = "ssh"
  description = "Allow SSH ingress from bastions"
  vpc_id      = var.vpc_id

  ingress {
    description      = "SSH Access"
    from_port        = 22
    to_port          = 22
    protocol         = "-1"
    cidr_blocks      = data.external.bastions.result.ipv4
    ipv6_cidr_blocks = data.external.bastions.result.ipv6
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

}

and the following redacted script snippet:

# Query for the addresses
IPV4_ADDRESSES=$(curl -s "${URL_IPV4}" | jq '[.SomeKey[] | .Address]')
IPV6_ADDRESSES=$(curl -s "${URL_IPV6}" | jq '[.SomeKey[] | .Address]')

# Output the addresses for terraform
jq -n \
    --argjson ipv4 "${IPV4_ADDRESSES}" \
    --argjson ipv6 "${IPV6_ADDRESSES}" \
    '{"ipv4":$ipv4,"ipv6":$ipv6}'

And the script's output (with actual ranges redacted):

{
  "ipv4": [
    "192.168.1.1/32",
    "192.168.1.1/32",
    "192.168.1.1/32"
  ],
  "ipv6": [
    "dead:beef::/56"
  ]
}

Expected Behavior

The expected behavior for a terraform plan is to see the IP ranges that are pulled from the bash script.

Actual Behavior

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.external.bless-bastions: Refreshing state...

Error: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string

Steps to Reproduce

  1. terraform apply

References

I've seen a couple of closed issues similar to this, but none specially cover the use case of ingesting a JSON array as the data source's result. Instead I've seen people trying to provide an array to the external script.

Switch to GitHub Actions and goreleaser Release Process

The "standard library" Terraform Providers should implement nascent provider development tooling to encourage consistency, foster automation where possible, and discover bugs/enhancements to that tooling. To that end, this provider's release process should be switched to goreleaser to match the documented Terraform Registry publishing recommendations. This includes:

  • Creating necessary .goreleaser.yml and .github/workflows/release.yml configurations for tag-based releases (see also: TF-279 RFC)
  • Ensuring necessary GitHub or Vault tokens are in place to fetch release secrets
  • Ensuring provider and internal release process documentation is updated

Long running external program always fails

Terraform CLI and Provider Versions

Terraform v1.3.1
on darwin_amd64
+ provider registry.terraform.io/hashicorp/external v2.2.2

Terraform Configuration

variable "sleep_mins" {
  type = string
  default = "0"
}

data "external" "test" {
  program = ["go", "run", "main.go"]

  query = {
    sleepMins = var.sleep_mins
  }
}

output "nap_duration" {
  value = format("Slept for %s minutes", data.external.test.result["sleepMins"])
}

Expected Behavior

Terraform (and external program) run successfully.

Returns:-

Outputs:

nap_duration = "Slept for 30 minutes"

Actual Behavior

  • When the invoked external program (main.go) runs for 30 mins or 60 mins, it returns the below error 100% of the time.
│ Error: External Program Execution Failed
│ 
│   with data.external.test,
│   on data.tf line 2, in data "external" "test":
│    2:   program = ["go", "run", "main.go"]
│ 
│ The data source received an unexpected error while attempting to execute the program.
│ 
│ The program was executed, however it returned no additional error messaging.
│ 
│ Program: /usr/local/go/bin/go
│ State: signal: killed
  • For execution durations at 0mins, 10mins, 15mins, 20mins, it runs successfully 100% of the time.

Steps to Reproduce

  1. terraform init
  2. terraform apply -var sleeps_mins=30

How much impact is this issue causing?

The only way around the problem is to drop back to v2.1.0 of the external provider. That's fine, and low impact for now.

Logs

https://gist.github.com/seventieskid/5db0865d84d0796e6893f32560b9997a

Additional Information

TF trace showing error (30 minute sleep in external program): https://gist.github.com/seventieskid/5db0865d84d0796e6893f32560b9997a#file-tf_30mins_fail-log

TF trace showing success (no sleep in external program):-

https://gist.github.com/seventieskid/406baf458ff50dcb9adb6ac3ff870435#file-tf_nosleep-pass-log

External program is here:-

https://gist.github.com/seventieskid/440cd4f949030f0c17dddb0c2061c5ea#file-main-go

Use Case

We have an external program (main.go) that invokes a tool that takes 50 mins to complete. So we must wait for it to complete. Terraform needs to interpret the output of that tool, hence we've opted for the TF external provider.

Code of Conduct

  • I agree to follow this project's Code of Conduct

invalid character 'a' looking for beginning of value

Terraform CLI and Provider Versions

Terraform v1.1.3
on linux_amd64

  • provider registry.terraform.io/hashicorp/external v2.2.0

Terraform Configuration

data "external" "go_lambda_packer" {
  program = ["bash", "${path.module}/scripts/go_lambda_packer.sh"]

  query = {
    source_path          = var.source_path
    output_path          = var.output_path
    install_dependencies = var.install_dependencies
  }
}

Expected Behavior

Seems like there's nothing wrong with the output, it's valid json (by the looks of the TRACE logs). I would expect to not run into any errors.

Actual Behavior

2022-01-20T23:12:41.735+0100 [TRACE] Executing graph transform *terraform.OrphanResourceInstanceCountTransformer
2022-01-20T23:12:41.735+0100 [TRACE] Completed graph transform *terraform.OrphanResourceInstanceCountTransformer (no changes)
2022-01-20T23:12:41.735+0100 [TRACE] Executing graph transform *terraform.AttachStateTransformer
2022-01-20T23:12:41.735+0100 [DEBUG] Resource instance state not found for node "module.go_lambda_packer.data.external.go_lambda_packer", instance module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] Completed graph transform *terraform.AttachStateTransformer (no changes)
2022-01-20T23:12:41.735+0100 [TRACE] Executing graph transform *terraform.TargetsTransformer
2022-01-20T23:12:41.735+0100 [TRACE] Completed graph transform *terraform.TargetsTransformer (no changes)
2022-01-20T23:12:41.735+0100 [TRACE] Executing graph transform *terraform.ReferenceTransformer
2022-01-20T23:12:41.735+0100 [INFO] ReferenceTransformer: reference not found: "path.module"
2022-01-20T23:12:41.735+0100 [INFO] ReferenceTransformer: reference not found: "var.source_path"
2022-01-20T23:12:41.735+0100 [INFO] ReferenceTransformer: reference not found: "var.output_path"
2022-01-20T23:12:41.735+0100 [INFO] ReferenceTransformer: reference not found: "var.install_dependencies"
2022-01-20T23:12:41.735+0100 [DEBUG] ReferenceTransformer: "module.go_lambda_packer.data.external.go_lambda_packer" references: []
2022-01-20T23:12:41.735+0100 [TRACE] Completed graph transform *terraform.ReferenceTransformer (no changes)
2022-01-20T23:12:41.735+0100 [TRACE] Executing graph transform *terraform.RootTransformer
2022-01-20T23:12:41.735+0100 [TRACE] Completed graph transform *terraform.RootTransformer (no changes)
2022-01-20T23:12:41.735+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer": entering dynamic subgraph
2022-01-20T23:12:41.735+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer": starting visit (*terraform.NodePlannableResourceInstance)
2022-01-20T23:12:41.735+0100 [TRACE] readResourceInstanceState: reading state for module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] readResourceInstanceState: no state present for module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to prevRunState for module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] states.SyncState: pruning module.go_lambda_packer because it is empty
2022-01-20T23:12:41.735+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] readDataSource: Re-validating config for module.go_lambda_packer.data.external.go_lambda_packer
2022-01-20T23:12:41.735+0100 [TRACE] GRPCProvider: ValidateDataResourceConfig
2022-01-20T23:12:41.735+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Received request: tf_data_source_type=external tf_proto_version=5 tf_req_id=68c6d46d-997d-ef51-7cbd-c03ed1750d28 tf_rpc=ValidateDataSourceConfig @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:445 @module=sdk.proto tf_provider_addr=provider timestamp=2022-01-20T23:12:41.735+0100
2022-01-20T23:12:41.735+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Calling downstream: tf_provider_addr=provider tf_req_id=68c6d46d-997d-ef51-7cbd-c03ed1750d28 tf_rpc=ValidateDataSourceConfig tf_proto_version=5 @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:452 @module=sdk.proto tf_data_source_type=external timestamp=2022-01-20T23:12:41.735+0100
2022-01-20T23:12:41.736+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Called downstream: @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:458 tf_proto_version=5 tf_req_id=68c6d46d-997d-ef51-7cbd-c03ed1750d28 tf_rpc=ValidateDataSourceConfig @module=sdk.proto tf_data_source_type=external tf_provider_addr=provider timestamp=2022-01-20T23:12:41.735+0100
2022-01-20T23:12:41.736+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Served request: @module=sdk.proto tf_data_source_type=external tf_req_id=68c6d46d-997d-ef51-7cbd-c03ed1750d28 @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:464 tf_proto_version=5 tf_provider_addr=provider tf_rpc=ValidateDataSourceConfig timestamp=2022-01-20T23:12:41.736+0100
2022-01-20T23:12:41.736+0100 [TRACE] readDataSource: module.go_lambda_packer.data.external.go_lambda_packer configuration is complete, so reading from provider
2022-01-20T23:12:41.736+0100 [TRACE] GRPCProvider: ReadDataSource
2022-01-20T23:12:41.736+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Received request: @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:470 @module=sdk.proto tf_proto_version=5 tf_provider_addr=provider tf_data_source_type=external tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 tf_rpc=ReadDataSource timestamp=2022-01-20T23:12:41.736+0100
2022-01-20T23:12:41.736+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Calling downstream: @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:477 tf_data_source_type=external tf_provider_addr=provider @module=sdk.proto tf_proto_version=5 tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 tf_rpc=ReadDataSource timestamp=2022-01-20T23:12:41.736+0100
2022-01-20T23:12:41.736+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Executing external program: tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 tf_rpc=ReadDataSource @module=provider @caller=github.com/terraform-providers/terraform-provider-external/internal/provider/data_source.go:136 program="/usr/bin/bash ../scripts/go_lambda_packer.sh" tf_data_source_type=external tf_provider_addr=provider timestamp=2022-01-20T23:12:41.736+0100
2022-01-20T23:12:43.146+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Executed external program: tf_rpc=ReadDataSource @caller=github.com/terraform-providers/terraform-provider-external/internal/provider/data_source.go:140 output="all modules verified
/home/bruno/GitHub/terraform-aws-go-lambda-packer/example
adding: my-lambda (deflated 47%)
adding: lambda.go (deflated 24%)
adding: go.mod (deflated 10%)
adding: go.sum (deflated 49%)
{
"source_code_hash": "NDFhOTllYzMwOWZjMjY0MTJjMTczZWEwNzQ2MjJjNzUyYmQ4MTAzZjdiMDcwZjQ1ZTE0OThlNmJi\nZGM4NTIxNgo=",
"output_path": "./my-lambda/my-lambda.zip"
}
" program="/usr/bin/bash ../scripts/go_lambda_packer.sh" tf_data_source_type=external tf_provider_addr=provider tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 @module=provider timestamp=2022-01-20T23:12:43.146+0100
2022-01-20T23:12:43.146+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Called downstream: tf_rpc=ReadDataSource @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:483 @module=sdk.proto tf_data_source_type=external tf_proto_version=5 tf_provider_addr=provider tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 timestamp=2022-01-20T23:12:43.146+0100
2022-01-20T23:12:43.146+0100 [TRACE] provider.terraform-provider-external_v2.2.0_x5: Served request: tf_proto_version=5 tf_rpc=ReadDataSource @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:489 @module=sdk.proto tf_data_source_type=external tf_provider_addr=provider tf_req_id=d4212c9f-a232-f289-00f5-17ec4ddc3195 timestamp=2022-01-20T23:12:43.146+0100
2022-01-20T23:12:43.147+0100 [ERROR] vertex "module.go_lambda_packer.data.external.go_lambda_packer" error: Unexpected External Program Results
2022-01-20T23:12:43.147+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer": visit complete, with errors
2022-01-20T23:12:43.147+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer": dynamic subgraph encountered errors: Unexpected External Program Results
2022-01-20T23:12:43.147+0100 [ERROR] vertex "module.go_lambda_packer.data.external.go_lambda_packer" error: Unexpected External Program Results
2022-01-20T23:12:43.147+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer": visit complete, with errors
2022-01-20T23:12:43.147+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer (expand)": dynamic subgraph encountered errors: Unexpected External Program Results
2022-01-20T23:12:43.147+0100 [ERROR] vertex "module.go_lambda_packer.data.external.go_lambda_packer (expand)" error: Unexpected External Program Results
2022-01-20T23:12:43.147+0100 [TRACE] vertex "module.go_lambda_packer.data.external.go_lambda_packer (expand)": visit complete, with errors
2022-01-20T23:12:43.147+0100 [TRACE] dag/walk: upstream of "provider["registry.terraform.io/hashicorp/external"] (close)" errored, so skipping
2022-01-20T23:12:43.147+0100 [TRACE] dag/walk: upstream of "module.go_lambda_packer.output.output_path (expand)" errored, so skipping
2022-01-20T23:12:43.147+0100 [TRACE] dag/walk: upstream of "module.go_lambda_packer.output.source_code_hash (expand)" errored, so skipping
2022-01-20T23:12:43.147+0100 [TRACE] dag/walk: upstream of "module.go_lambda_packer (close)" errored, so skipping
2022-01-20T23:12:43.147+0100 [TRACE] dag/walk: upstream of "root" errored, so skipping
2022-01-20T23:12:43.147+0100 [INFO] backend/local: plan operation completed

│ Error: Unexpected External Program Results

│ with module.go_lambda_packer.data.external.go_lambda_packer,
│ on ../main.tf line 2, in data "external" "go_lambda_packer":
│ 2: program = ["bash", "${path.module}/scripts/go_lambda_packer.sh"]

│ The data source received unexpected results after executing the program.

│ Program output must be a JSON encoded map of string keys and string values.

│ If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level. Terraform documentation on logging: https://www.terraform.io/internals/debugging

│ Program: /usr/bin/bash
│ Result Error: invalid character 'a' looking for beginning of value

2022-01-20T23:12:43.147+0100 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2022-01-20T23:12:43.147+0100 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2022-01-20T23:12:43.148+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-01-20T23:12:43.149+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/external/2.2.0/linux_amd64/terraform-provider-external_v2.2.0_x5 pid=739152
2022-01-20T23:12:43.149+0100 [DEBUG] provider: plugin exited

Steps to Reproduce

  1. terraform plan

How much impact is this issue causing?

Medium

Logs

https://gist.github.com/bschaatsbergen/b1e8aea48c72ad51aada389cfdae1b25

Additional Information

I've created the following script that I ran:

#!/bin/bash

# echo '{"source_path": "example/my-lambda", "output_path": "example/my-lambda/my-lambda.zip", "install_dependencies": true}' | scripts//go_lambda_packer.sh

function error_exit() {
  echo "$1" 1>&2
  exit 1
}

function check_deps() {
  test -f $(which go) || error_exit "go command not detected in path, please install it"
  test -f $(which jq) || error_exit "jq command not detected in path, please install it"
  test -f $(which zip) || error_exit "zip command not detected in path, please install it"
}

function parse_input() {
  eval "$(jq -r '@sh "export source_path=\(.source_path) output_path=\(.output_path) install_dependencies=\(.install_dependencies)"')"
  if [[ -z "${source_path}" ]]; then export source_path=none; fi
  if [[ -z "${output_path}" ]]; then export output_path=none; fi
  if [[ -z "${install_dependencies}" ]]; then export install_dependencies=none; fi
}

function build_executable() {
  cd ${source_path} || exit

  if $install_dependencies; then 
    go mod verify
    go mod tidy;
  fi

  GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build .
}

function pack_executable() {
  cd - || exit # go back to previous directory
  zip -r ${output_path} ${source_path} --junk-paths
}

function build_stable_base64_hash() {
  executable=$(find . -executable -type f) # find executable files in current directory

  sha256=$(sha256sum "${executable}" | awk '{print $1}')
  # echo "DEBUG: sha256 ${sha256}" 1>&2

  base64sha256=$(echo "${sha256}" | base64)
  # echo "DEBUG: base64sha256 ${base64sha256}" 1>&2
}

function produce_output() {
  # echo "DEBUG: source_code_hash ${base64sha256}" 1>&2
  # echo "DEBUG: output_path ${output_path}" 1>&2
  jq -n \
    --arg source_code_hash "$base64sha256" \
    --arg output_path "$output_path" \
    '{"source_code_hash":$source_code_hash,"output_path":$output_path}'
}

check_deps
parse_input
build_executable
build_stable_base64_hash
pack_executable
produce_output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Allow HCL output output for resource "external" or introduce resource "external_hcl"

It would be cool to do one of these two:

  1. Allow resource "external" to return HCL syntax - not just JSON, or
  2. Introduce resource "external_hcl" that allows HCL output

Currently, HCL output for resource "external" won't be accepted:

Error refreshing state: 1 error(s) occurred:

* data.external.users: 1 error(s) occurred:

* data.external.users: data.external.users: command "ruby" produced invalid JSON: invalid character '/' looking for beginning of value

Enhance Diagnostics When Unable to Execute Program

Hi,

I've got a "can't find external program" issue when I execute the terraform scripts in CI/CD (through Jenkins), but locally it's working.

Terraform Version

Terraform v0.11.7
+ provider.aws v1.13.0
+ provider.external v1.0.0

Terraform Configuration Files

data "external" "converted_template" {
  program = ["node", "${path.module}/terraform-data.js"]
  query = {
    template = "${file("${path.module}/data_pipeline.json")}",
    name = "${var.data_pipeline_name}",
    description = "",
    activate = "true",
    values = "${jsonencode("${var.values}")}"
  }
}

External data source is used through a module.

Expected Behavior

Terraform apply successfully worked.

Actual Behavior

* module.data_pipeline.data.external.converted_template: data.external.converted_template: can't find external program "node"

Read when there are no changes

Terraform Version

Terraform v0.13.3

  • provider registry.terraform.io/hashicorp/external v1.2.0

Affected Resource(s)

  • data "external"

Terraform Configuration Files

Debug Output

Panic Output

Expected Behavior

No changes == no confirmation and no data source info

Actual Behavior

  # module.some_module.data.external.some will be read during apply
  # (config refers to values not yet known)
 <= data "external" "some"  {
      + id          = "-"
      + program     = [
          + "python3",
          + "some.py",
        ]
      + query       = {
          + "output_secret_arn" = "arn:aws..."
        }
      + result      = {
          + "output" = "null"
        }
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "some"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

Steps to Reproduce

  1. Create external data source with some script output inside of a module
  2. (NOT SURE IF REQUIRED) Add depends_on block that will wait for some resource (for example null_resource)
  3. (NOT SURE IF REQUIRED) Add module output from external data source
  4. terraform apply to save everything to the state
  5. Rename the module folder and rename the actual import via module "some_new" { src = new path }
  6. terraform init
  7. terraform apply to save everything to the state
  8. terraform apply again and hopefully you will encounter same problem.

Important Factoids

References

Similar issue to hashicorp/terraform#25511

value passed through query contains dodgy code/characters

I have an external data program to create a resource for which there is not a Terraform resource for yet, however I need to pad 0's in front of a single digit for formatting of a name automatically.

Therefore, I have tried using the Terraform "format" interpolation function to do this, then passing that value into the query parameter of the external data program however this causes the python to fail due to some sort of code evaluation not happening within the external data source.

I'm not sure if it's an issue with this provider or an issue with the interpolation function or the way a string interpolation is performed before being passed into this provider, so I'm happy to raise the issue in the terraform core if required.

Terraform Version

$ terraform -v
Terraform v0.11.1

Terraform Configuration Files

variable "network_number_8" {
  default = "8"
}

data "external" "test" {
  program = ["python", "test.py"]

  query {
    test_good = "${var.network_number_8}"
    test_bad  = "${format("%03d",var.network_number_8)}"
  }
}

Python code for external data source

import json
import sys

args = json.load(sys.stdin)
json.dump(args, sys.stderr)
exit(1)

Actual Behavior

$ terraform apply
data.external.test: Refreshing state...

Error: Error refreshing state: 1 error(s) occurred:

* data.external.test: 1 error(s) occurred:

* data.external.test: data.external.test: failed to execute "python": {"test_bad": "%!d(string=008)", "test_good": "8"}

Steps to Reproduce

  1. terraform init
  2. terraform apply

unexpected end of JSON input

Terraform Version

Terraform v0.11.8

Affected Resource(s)

external

Terraform Configuration Files

data "external" "webhook" {
  program = ["az", "webapp", "deployment", "list-publishing-profiles", "--name", "fa-we-shared-pvs-prod", "--resource-group", "rg-we-shared-prod", "--query", "\"[].{PWD:userPWD}\"", "--output", "json"]  
}

Expected Behavior

being able to reference the password, for an azure webapp webhook and use that with GitLab provider to define integrations.

Actual Behavior

* module.function_nesk.data.external.hooks: data.external.hooks: command "az" produced invalid JSON: unexpected end of JSON input

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Additional Context

Running the command in Azure Cli, gives the following output, with Json encoding.

{
  "PWD": "mnvQcfGqw3F3fpj0wtw12345678901234567891023456987xxxxxxxxxxxxxxxxx"
}

or by changing the outputs in cli --output to tsv gives single string, but this also gives unexpected end of JSON input with terarform

mnvQcfGqw3F3fpj0wtw12345678901234567891023456987xxxxxxxxxxxxxxxxx

References

#13

Terraform 0.12 support

I didn't see anything in the works to make this provider compatible with terraform 0.12 so I wanted to get this in for tracking purposes. I understand terraform 0.12 may make some of the functionality of this provider obsolete but it will take a lot of effort to rewrite our configs to eliminate our need for it.

'result' attribute on external data source not set during a destroy operation

This issue was originally opened by @dotariel as hashicorp/terraform#16008. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.10.4-dev

Terraform Configuration Files

variable "users" {
  default = {
    "john.doe"  = "Users"
    "jane.doe"  = "Users,Administrators"
    "jack.frost = "Users,PowerUsers"
  }
}

data "external" "transpose" {
  program = [ "python", "${path.module}/transpose.py" ]
  query   = "${var.users}"
}

locals {
  default_groups = {
    "Admins"        = ""
    "ReadOnlyUsers" = ""
    "PowerUsers"    = ""
  }
  groups = "${merge(local.default_groups, data.external.transpose.result)}"
}
  • transpose.py takes the input and inverts the map to return values as keys and keys as values.

Expected Behavior

Destroy operation should succeed without error.

Actual Behavior

Destroy operation fails with:

Error applying plan:

1 error(s) occurred:

* local.groups: local.groups: Resource 'data.external.transpose' does not have attribute 'result' for variable 'data.external.transpose.result'

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

  1. terraform init
  2. terraform destroy --force

References

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.