I have noticed that by default I have to connect by DNS name (using digitalocean_droplet); it would be nice to be able to pass a switch (maybe --ipv4) and connect directly by IP instead of using DNS.

This is a problem because if you directly provision with TF and then try to run ansible using terraform.py as the "dynamic inventory" you're not able to connect because the DNS records do not exist.

I receive the following error:

ERROR: Inventory script (../terraform.py/terraform.py) had an execution error: Traceback (most recent call last):
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 658, in
main()
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 643, in main
output = query_list(hosts)
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 585, in query_list
for name, attrs, hostgroups in hosts:
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 67, in iterhosts
yield parser(resource, module_name)
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 83, in inner
name, attrs, groups = func(_args, *_kwargs)
File "/home/d060983/poc_automation/terraform.py/terraform.py", line 323, in aws_host
'tenancy': raw_attrs['tenancy'],
KeyError: u'tenancy'

In my .tfstate file the attribute tenancy is not included.

As written in the official doc on https://www.terraform.io/docs/providers/aws/r/instance.html the attribute is optional.

Receiving this error:

Traceback (most recent call last):
File "./terraform.py", line 562, in
main()
File "./terraform.py", line 547, in main
output = query_list(hosts)
File "./terraform.py", line 489, in query_list
for name, attrs, hostgroups in hosts:
File "./terraform.py", line 68, in iterhosts
yield parser(resource, module_name)
File "./terraform.py", line 84, in inner
name, attrs, groups = func(_args, *_kwargs)
File "./terraform.py", line 330, in aws_host
'ansible_ssh_host': raw_attrs['public_ip'],
KeyError: u'public_ip'

Which is I believe due to the fact I have an instance without a public_ip (living in a private network):

"attributes": {
    "ami": "ami-96a818fe",
    "availability_zone": "us-east-1a",
    "ebs_block_device.#": "0",
    "ebs_optimized": "false",
    "ephemeral_block_device.#": "0",
    "id": "i-57efa6f7",
    "instance_type": "t2.micro",
    "key_name": "REDACTED",
    "monitoring": "false",
    "private_dns": "ip-10-0-10-213.ec2.internal",
    "private_ip": "10.0.10.213",
    "public_dns": "",
    "root_block_device.#": "1",
    "root_block_device.0.delete_on_termination": "false",
    "root_block_device.0.iops": "0",
    "root_block_device.0.volume_size": "8",
    "root_block_device.0.volume_type": "standard",
    "security_groups.#": "1",
    "security_groups.112399896": "sg-49b1572f",
    "source_dest_check": "true",
    "subnet_id": "subnet-53b88d0a",
    "tags.#": "3",
    "tags.Environment": "internal",
    "tags.Name": "jenkins_master",
    "tags.Owner": "rnd",
    "tenancy": "default",
    "vpc_security_group_ids.#": "1",
    "vpc_security_group_ids.112399896": "sg-49b1572f"
}

Not quite sure how you want to solve that problem, but perhaps just ignore instances in .tfstate files if there is no public_ip?

My current working directory is ~/coding/terraform1 which is where I am trying to run my ansible-playbook command using terraform.py.

When the command runs it is three other terraform.tfstate files that are not in the working directory, or below.

For example:

• ~/coding/project1
• ~/coding/test
• etc.

The documentation says that it only reads the state file in the working directory or below. So, am I misreading the documentation or is this undesired behaviour?

Thanks!

As per the ansible best practices, I keep different inventories for different environments (or, at my current place of work, stages)

In keeping with that premise and with things I've been reading about terraform on the web (primarily this post), I have also separated by terraform out into different tfstate files on a per-stage basis.

So, my directory structure looks like:
site/
├── group_vars
├── inventory
│   ├── base
│   ├── dev
│   └── prod
├── playbooks
│   └── includes
├── terraform
│   ├── base
│   ├── dev
│   ├── modules
│   │   ├── app_instance
│   │   └── db_instance
│   ├── prod
└── vault

So, I need my dynamic inventory to match my terraform state file. If I use -i inventory/dev, then my inventory should parse only terraform/dev.

I've hacked this into the current version:

def get_stage_root(tf_dirname=None, root=None):
    """ look for a matching terraform root directory """
    root = root or os.getcwd()
    tf_dirname = tf_dirname or 'terraform'
    inv_dir = os.path.dirname(__file__).split(os.path.sep)[-1]
    terraform_base = os.path.join(root, tf_dirname)
    if inv_dir in os.listdir(terraform_base):
        return os.path.join(terraform_base, inv_dir)
    else:
        return root

and in main()

    staged_root = get_stage_root(root=args.root)
    if staged_root != args.root:
        args.root = staged_root

As I have this written, it's probably only useful to me. However, I thought it might be useful to other folks if it were made more generic.

I use this primarily for amazon. It seems like there's enough config variation that you might want to either consider adding a config stanza to the ansible config, or to add a custom terraform_inventory.cfg at either the root directory level or the file level.

Would be great with native support for Azure Stack. All terraform resources are the same, with the same name. Main exception is that Azure Public resources are named azurerm and Azure Stack resources are named azurestack.

Instance user-data can choke ansible. Filter out user-data from metadata.

I was trying to use this with Windows hosts in AWS and ran into a problem because the ansible_ssh_port value was set and this overrode the value I had set in a group_vars yml to configure WinRM.

I was able to fix the problem by removing the lines which defined the property and things started working at that point.

It also looks like this configuration key's name has been changed in Ansible 2.0. From http://docs.ansible.com/ansible/intro_inventory.html:

Ansible 2.0 has deprecated the “ssh” from ansible_ssh_user, ansible_ssh_host, and ansible_ssh_port to become ansible_user, ansible_host, and ansible_port. If you are using a version of Ansible prior to 2.0, you should continue using the older style variables (ansible_ssh_*). These shorter variables are ignored, without warning, in older versions of Ansible.

I have a terraform script that creates multiple "types" of ec2 instances... so think "web server" "database" type groups.

If i hack the ansible inventory script it would look like this:

[webserver]
node1.example.com
node2.example.com

[database]
node3.example.com

and then my ansible play books would classify tasks based on groups, i.e., install httpd on the "web server" assets only.

Is there a way to generate the hosts using terraform.py such that these groups get created based on some metadata in the terraform resource scripts? ie.. i think maybe role tags do this? https://github.com/CiscoCloud/terraform.py/blob/master/terraform.py#L330

Are there plans to make available in pip?

After provisioning with mantl's default aws.tf file, I got the following traceback with terraform.py. I've saved my tfstate as well, but didn't post it due to it's length.

ERROR! The file plugins/inventory/terraform.py is marked as executable, but failed to execute correctly. If this is not supposed to be an executable script, correct this with `chmod -x plugins/inventory/terraform.py`.
ERROR! Inventory script (plugins/inventory/terraform.py) had an execution error: Traceback (most recent call last):
  File "/home/siddharthist/Dropbox/code/asteris/mantl/plugins/inventory/terraform.py", line 545, in <module>
    main()
  File "/home/siddharthist/Dropbox/code/asteris/mantl/plugins/inventory/terraform.py", line 533, in main
    print(json.dumps(output, indent=4 if args.pretty else None))
  File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.5/json/encoder.py", line 180, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: dict_values([]) is not JSON serializable

ERROR! plugins/inventory/terraform.py:16: Error parsing host definition '"""\': No escaped character

Something like terraform.py --hostfile. It would output the hosts between comment blocks for easy addition and removal: terraform.py --hostfile >> /etc/hosts should be a reasonable thing to do.

Would it be beneficial when outputting the hosts file to be able to group through a property in Terraform? For instance, adding a tag or property to the resource would group servers in the Ansible inventory when output...

Example:

resource "vsphere_virtual_machine" "node" {
  count     = 3
  # other properties....
  tags {
    Name = "my-ansible-group"
  }
}

Output of inventory file using terraform.py --hostfile

[my-ansible-group]
192.168.99.100     node1
192.168.99.100     node2
192.168.99.100     node3

If this seems like a good idea, I could work on a PR.

In #13 it is shown how AWS resouces can be grouped into ansible groups. How can this be accoplished for OpenStack instances? I tried to add a role attribute within the metadata block, but this does not seem to be working. Any ideas?

Thanks!

when the dynamic inventory script runs, i get the following error:

piwi@jumphost:~/Documents/kubebify$ ansible all -i inventory/dev --list-hosts
ERROR! Attempted to execute "/home/piwi/Documents/kubebify/inventory/dev/terraform_inventory.sh" as inventory script: Inventory script (/home/piwi/Documents/kubebify/inventory/dev/terraform_inventory.sh) had an execution error: Traceback (most recent call last):
File "/home/piwi/.local/bin/ati", line 11, in
load_entry_point('ati==0.4.4.dev0', 'console_scripts', 'ati')()
File "/home/piwi/.local/venvs/ati/local/lib/python2.7/site-packages/ati/cli.py", line 81, in cli
output = query_list(hosts)
File "/home/piwi/.local/venvs/ati/local/lib/python2.7/site-packages/ati/terraform.py", line 940, in query_list
for name, attrs, hostgroups in hosts:
File "/home/piwi/.local/venvs/ati/local/lib/python2.7/site-packages/ati/terraform.py", line 106, in iterhosts
for module_name, key, resource in resources:
File "/home/piwi/.local/venvs/ati/local/lib/python2.7/site-packages/ati/terraform.py", line 54, in iterresources
for source in sources:
File "/home/piwi/.local/venvs/ati/local/lib/python2.7/site-packages/ati/terraform.py", line 48, in iter_states
yield json.loads(output[start_index:])
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

PEP 0394 states that "python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3."

As such, the first line of terraform.py should read

#!/usr/bin/env python2

A question about support for terraform remote state came up in the mantl gitter channel. As of right now, terraform doesn't support loading in remote tfstate files. Should it? And if the answer to that question is no, should we add some documentation to the README about how to use remote state files in the workflow?

For now dynamic inventories are able to have them, but in 2.4 they removed it from normal inventories. It's nice to be able to use the same role names on bare metal and terraform provisioned nodes, so I would recommend getting rid of the = from group names.

See ansible/ansible#30624 (it's about dashes, but equal signs are affected also).

If an EIP is used, that IP should be output by the script, otherwise Ansible can't connect to the instance.

As of terraform 0.6.11 (possibly even 0.6.10 but I haven't confirmed), the gce provider should use assigned_nat_ip instead of nat_ip to populate ansible_ssh_host and public_ipv4.

See hashicorp/terraform#4896 for the background on the change.

Currently, when used along with terraform 0.6.11, terraform.py will return blank values for ansible_ssh_host and public_ipv4 .

Re: mantl/mantl#529

Add support for Ansible 2.0.

terraform.py/src/ati/terraform.py has some tabs around lines 55 to 59 which causes problems with Python 3.

How should I annotate my google cloud terraform files?
Tags, different from the examples with azure and aws are lists not maps.
Can you provide a working example for GCP as well?
Thanks!

If Terraform configuration consists of one (or more?) DigitalOcean droplets, terraform.py exists with error in case user_data key is missing in *.tfstate file. I haven't checked if this is a regression introduced in a recent version of Terraform.

Terraform and Dynamic Inventory script versions

$ terraform --version
Terraform v0.6.14
$ ./dynamic-inventory/terraform.py --version
./dynamic-inventory/terraform.py 0.3.0pre

Traceback

$ ./dynamic-inventory/terraform.py --list
Traceback (most recent call last):
  File "./dynamic-inventory/terraform.py", line 658, in <module>
    main()
  File "./dynamic-inventory/terraform.py", line 643, in main
    output = query_list(hosts)
  File "./dynamic-inventory/terraform.py", line 585, in query_list
    for name, attrs, hostgroups in hosts:
  File "./dynamic-inventory/terraform.py", line 67, in iterhosts
    yield parser(resource, module_name)
  File "./dynamic-inventory/terraform.py", line 83, in inner
    name, attrs, groups = func(*args, **kwargs)
  File "./dynamic-inventory/terraform.py", line 153, in digitalocean_host
    'metadata': json.loads(raw_attrs['user_data']),
KeyError: u'user_data'

Terraform *.tfstate file contents

{
    "version": 1,
    "serial": 54,
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
            },
            "resources": {
                "digitalocean_droplet.galois": {
                    "type": "digitalocean_droplet",
                    "primary": {
                        "id": "<DROPLET ID>",
                        "attributes": {
                            "id": "<DROPLET ID>",
                            "image": "ubuntu-14-04-x64",
                            "ipv4_address": "<DROPLET IPv4>",
                            "ipv6": "true",
                            "ipv6_address": "<DROPLET IPv6>",
                            "ipv6_address_private": "",
                            "locked": "false",
                            "name": "galois.example.com",
                            "region": "fra1",
                            "size": "512mb",
                            "ssh_keys.#": "2",
                            "ssh_keys.0": "<SSH KEY #0>",
                            "ssh_keys.1": "<SSH KEY #1>",
                            "status": "active"
                        }
                    }
                }
            }
        }
    ]
}

Proposed fix

I found a fix that works for me, but could possibly break things for other people. Honestly I don't have time nor appropriate knowledges to deeply test this, so I'm just posting my patch, in case it is helpful to somebody.

diff --git a/terraform.py b/terraform.py
index cab214d..bd53589 100755
--- a/terraform.py
+++ b/terraform.py
@@ -150,7 +150,7 @@ def digitalocean_host(resource, tfvars=None):
         'image': raw_attrs['image'],
         'ipv4_address': raw_attrs['ipv4_address'],
         'locked': parse_bool(raw_attrs['locked']),
-        'metadata': json.loads(raw_attrs['user_data']),
+        'metadata': json.loads(raw_attrs.get('user_data', '{}')),
         'region': raw_attrs['region'],
         'size': raw_attrs['size'],
         'ssh_keys': parse_list(raw_attrs, 'ssh_keys'),

Setting incredibly common variables such as id, public, private, key_name, etc are very likely to collide with other packages. It might be more practical to namespace them under an ati variable or something similar.

I'm confused how to make this work with the vsphere provider as it doesn't support tags? Can someone point me in the right direction, please?

Hey,

Kudos for this project, hopefully it'll replace my quick'n'dirty bash script for generating a static inventory from tf state!

I tried to use it on an existing state file for an aws-only infrastructure, but got the following error:

terraform.py --list
Traceback (most recent call last):
  File "terraform.py", line 545, in <module>
    main()
  File "terraform.py", line 530, in main
    output = query_list(hosts)
  File "terraform.py", line 472, in query_list
    for name, attrs, hostgroups in hosts:
  File "terraform.py", line 67, in iterhosts
    yield parser(resource, module_name)
  File "terraform.py", line 83, in inner
    name, attrs, groups = func(*args, **kwargs)
  File "terraform.py", line 322, in aws_host
    'ansible_ssh_user': raw_attrs['tags.sshUser'],
KeyError: u'tags.sshUser'

Maybe it's worth mentioning i don't specify in terraform and ansible the ssh user, I rely on my .ssh/config for that.

Let me know if you need more information.

It would be helpful if the online help provides better information. For instance the --noterraform flag stops the script from looking for state files in the .terraform directory. At least that's what I deduce from the source code. This is not well explained in the help output.

--noterraform         do not use terraform from path (default: False)

The only way i can get terraform.py to read tfstate is by setting the --noterraform option or if a .terraform directory exists.

Seems like terraform state pull works great for local or remote states. Should terraform state pull be the default? And possibly change --noterraform to --localtfstate or --noremotestate if its required for some reason.

Hi there,

How can integrate the terraform.py script with ansible tower . My plan is to run playbook from tower and i need to sync all my hostname information to ansible inventory. can you please help on this?