hashicorp / packer Goto Github PK

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.

License: Other

Makefile 0.44% Go 71.48% Shell 1.06% JavaScript 0.25% PowerShell 0.16% Batchfile 0.01% Python 0.11% HCL 3.51% Dockerfile 0.19% MDX 22.80%

packer's Introduction

Packer

Packer is a tool for building identical machine images for multiple platforms from a single source configuration.

Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer supports various platforms through external plugin integrations, the full list of which can be found at https://developer.hashicorp.com/packer/integrations.

The images that Packer creates can easily be turned into Vagrant boxes.

Quick Start

Packer

There is a great introduction and getting started guide for building a Docker image on your local machine without using any paid cloud resources.

Alternatively, you can refer to getting started with AWS to learn how to build a machine image for an external cloud provider.

HCP Packer

HCP Packer registry stores Packer image metadata, enabling you to track your image lifecycle.

To get started with building an AWS machine image to HCP Packer for referencing in Terraform refer to the collection of HCP Packer Tutorials.

Documentation

Comprehensive documentation is viewable on the Packer website at https://developer.hashicorp.com/packer/docs.

Contributing to Packer

See CONTRIBUTING.md for best practices and instructions on setting up your development environment to work on Packer.

Unmaintained Plugins

As contributors' circumstances change, development on a community maintained plugin can slow. When this happens, HashiCorp may use GitHub's option to archive the plugin’s repository, to clearly signal the plugin's status to users.

What does unmaintained mean?

The code repository and all commit history will still be available.
Documentation will remain on the Packer website.
Issues and pull requests are monitored as a best effort.
No active development will be performed by HashiCorp.

If you are interested in maintaining an unmaintained or archived plugin, please reach out to us at [email protected].

packer's People

Contributors

Stargazers

Watchers

Forkers

garethr kyleconroy ccaum pierreozoux botimer hghazal nyo16 rubiojr justincampbell calebfornari benwoody arturhoo jcrosby msabramo aivaturi timothydavidellis netconstructor openhosts hosts-xx scraping-xx englishm buzzra netavanza gzapico antonlindstrom gitlisted markpeek markjaquith cwebberops ddoc svenne smerrill qur rgarcia appliedcode priteau rsampaio dqminh johncosta mikezter mkb218 lzy7199 greenalto jasonberanek jameswhite jvandyke xushiwei sit sgirones robinbowes philipcristiano benlovell rvoicilas jbronn whiteley hannson eedgar dougm psi savaki jstamerj iancmcc tobli bionix stephenrjohnson sarahhodne lusis phobos182 bluemarble rach mrmichalis mies gnomix ahawkins louisstamour masteinhauser libin mwhooker tekacs andyleclair tehranian justinsb oker1 transcendcomputing st63jun marcw spheromak whostolebenfrog mackrorysd cournape andrewschleifer mindjiver jerryclinesmith chasballew justinclayton khangtoh dlovell sharpie alouche timmow

packer's Issues

aws + vagrant post-processor intermediary artifact being deleted

I spent a while trying to figure out what was going on before I realized that:

If you use the aws-ebs builder
If you have the vagrant post-processor

The ami created gets deleted.

If you use the keep_input_artifact option to the vagrant post-processor then the artifact is properly kept.

The reason I'd expect the intermediate artifact to be preserved is this part from the docs:

With creating AWS images, however, the AMI is kept around, since Vagrant needs it to function.

VirtualBox: Can try exporting when state is still "stopping"

http://d.pr/i/klQD

Build failing with http port min and max

When specifying the http_port_min and http_port_max to be the same, the build fails with this. And when it does work after I specify different ports, the preseed URL of http://10.0.2.2:8262/preseed.cfg fails...

Plugin Architecture

We need a plugin architecture in place. Because Golang doesn't support dynamic linking of any kind, we'll have to use IPC for this. To keep things as simple as possible, let's go with the net/rpc package to get this done.

Ideas:

Given a binary, such as packer-build, the plugin lib must execute the underlying executable using os/exec or whatever. It must then initiate and abstract the RPC stuff.
To most of Packer, plugins must appear just as normal implementers of interfaces such as Command or Builder. Keep it simple!
Multiple packer instances must be able to run at one time, so static ports for listening on RPC is not possible.

Other notes:

I had an idea to use stdin/stdout for the communication mechanism for net/rpc. If this gets done, can the plugin call back into Packer? There are cases where this is necessary, so it is important that is possible.

provisioners: Access Builder Configuration

Combined with #27 this could be pretty cool. Something like:

{
  "type": "shell",
  "inline": [
      "echo {{ .Builder-region-id }}"
    ]
}

I looked briefly at text/template and couldn't see how to do nested configuration, so just did Builder-configuration-name.

Don't know if this drifts too far into dealing with handholding for individual platforms, but I saw execute_command, so, maybe.

Git ref for 0.1.0?

I am building a packer homebrew formula, but they prefer to compile from source. Can you tell me which ref corresponds to the 0.1.0 release that is available in the downloads?

Would be a great idea to tag each release. ;) thx

Post-processor configuration to remove intermediaries

Add a special packer-understood configuration value to post-processors that tells Packer to delete intermediary artifacts. By default this should be false.

The use case for this: Imagine you run a builder and you want to compress all the output, but you don't want to keep the original output. You can run all of them through a "compress" post-processor, and flag to delete intermediaries, which will delete the artifacts created by the builders in the process.

Handle signals (specifically ctrl-C)

We need a clean way to handle signals, specifically SIGINT.

This will suck.

VirtualBox error with default .ovf file name

tl;dr: It looks like the Vagrant post-processor for VirtualBox should name the .ovf file it creates "box.ovf" instead of "packer.ovf".

I am using the following software versions:

Vagrant 1.2.2
VirtualBox 4.2.12
OS X 10.8.4
Packer 0.1.1

I used the same template.json from #63 to output a Vagrant box for VirtualBox.

 ┌┤smerrill@lilliputian-resolution [Jun 29 11:04:38] ~/Documents/Development/packer-boxes/centos-6.4-x86_64 on master +
 └╼ packer build -only=virtualbox template.json
virtualbox output will be in this color.

==> virtualbox: Downloading VirtualBox guest additions. Progress will be shown periodically.
==> virtualbox: Copying or downloading ISO. Progress will be reported periodically.

[ many lined snipped ]

==> virtualbox (vagrant): Creating Vagrant box for 'virtualbox' provider
    virtualbox (vagrant): Copying: virtualbox/packer-disk1.vmdk
    virtualbox (vagrant): Copying: virtualbox/packer.ovf
    virtualbox (vagrant): Compressing box...
Build 'virtualbox' finished.

==> Builds finished. The artifacts of successful builds are:
--> virtualbox: 'virtualbox' provider box: packer_virtualbox.box
 ┌┤smerrill@lilliputian-resolution [Jun 29 11:13:13] ~/Documents/Development/packer-boxes/centos-6.4-x86_64 on master +?
 └╼ vagrant box add packerbox packer_virtualbox.box
Downloading or copying the box...
Extracting box...te: 371M/s, Estimated time remaining: 0:00:01)
Successfully added box 'packerbox' with provider 'virtualbox'!

That worked fine, but then upon doing a vagrant up with the box, it could not find the proper .ovf file:

 ┌┤smerrill@lilliputian-resolution [Jun 29 11:13:39] ~/Documents/Development/packer-test
 └╼ vagrant init packerbox
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
 ┌┤smerrill@lilliputian-resolution [Jun 29 11:13:47] ~/Documents/Development/packer-test
 └╼ vim Vagrantfile
 ┌┤smerrill@lilliputian-resolution [Jun 29 11:14:09] ~/Documents/Development/packer-test
 └╼ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'packerbox'...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["import", "/Users/smerrill/.vagrant.d/boxes/packerbox/virtualbox/box.ovf"]

Stderr: 0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Appliance read failed
VBoxManage: error: Could not read OVF file 'box.ovf' (VERR_FILE_NOT_FOUND)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component Appliance, interface IAppliance
VBoxManage: error: Context: "int handleImportAppliance(HandlerArg*)" at line 306 of file VBoxManageAppliance.cpp

When I renamed ~/.vagrant.d/boxes/packerbox/virtualbox/packer.ovf to box.ovf, I could vagrant up without issue.

packer build panics on example.json

Came back to try re-running #41 (to see if it's an intermittent AWS thing, maybe) and get this surprise:

$ packer build example.json
panic: gob: unknown type id or corrupted data [recovered]
        panic: gob: unknown type id or corrupted data

goroutine 1 [running]:
log.Panic(0x6709b0, 0x1, 0x1)
        /Users/mitchellh/go/src/pkg/log/log.go:307 +0xaa
github.com/mitchellh/packer/packer/plugin.(*cmdCommand).checkExit(0xc200103700, 0x255c80, 0xc2000bb5a0, 0x6709f8)
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/command.go:49 +0x9c
github.com/mitchellh/packer/packer/plugin.func·011()
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/command.go:26 +0x8c
github.com/mitchellh/packer/packer/rpc.(*command).Run(0xc200000598, 0xc2000d7600, 0xc2000d75a0, 0xc200097020, 0x1, ...)
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/rpc/command.go:48 +0x18a
github.com/mitchellh/packer/packer/plugin.(*cmdCommand).Run(0xc200103700, 0xc2000d7600, 0xc2000d75a0, 0xc200097020, 0x1, ...)
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/command.go:29 +0xc2
github.com/mitchellh/packer/packer.(*coreEnvironment).Cli(0xc2000d75a0, 0xc200097010, 0x2, 0x2, 0x0, ...)
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/environment.go:262 +0x85d
main.main()
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer.go:76 +0xac4

goroutine 2 [syscall]:

goroutine 4 [syscall]:
os/signal.loop()
        /Users/mitchellh/go/src/pkg/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
        /Users/mitchellh/go/src/pkg/os/signal/signal_unix.go:27 +0x2f

goroutine 6 [chan receive]:
main.func·001()
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/signal.go:18 +0x46
created by main.setupSignalHandlers
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/signal.go:28 +0x115

goroutine 13 [IO wait]:
net.runtime_pollWait(0x4fde60, 0x72, 0x0)
        /Users/mitchellh/go/src/pkg/runtime/znetpoll_darwin_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000cb7d0, 0x23, 0xc2000ff2a0)
        /Users/mitchellh/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2000cb750, 0x32ddc8, 0x0, 0xc2000ff2a0, 0x23, ...)
        /Users/mitchellh/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc200000708, 0x18, 0xc200152010, 0xb3284)
        /Users/mitchellh/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc200000708, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/mitchellh/go/src/pkg/net/tcpsock_posix.go:239 +0x25
github.com/mitchellh/packer/packer/rpc.func·005()
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/rpc/server.go:79 +0x65
created by github.com/mitchellh/packer/packer/rpc.serveSingleConn
        /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/rpc/server.go:85 +0xb8
$

Vagrant post-processor fails for VMWare boxes with "vmware: is a directory" error

I am using Packer 0.1.0 on OS X 10.8.4 with VMWare Fusion Professional 5.0.3.

I have the following template.json with two builders that was converted from a Veewee template using the veewee-to-packer gem.

{
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/base.sh",
        "scripts/ruby.sh",
        "scripts/chef.sh",
        "scripts/puppet.sh",
        "scripts/vagrant.sh",
        "scripts/virtualbox.sh",
        "scripts/cleanup.sh",
        "scripts/zerodisk.sh"
      ],
      "override": {
        "virtualbox": {
          "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'"
        },
        "vmware": {
          "execute_command": "echo 'vagrant'|sudo -S sh '{{.Path}}'"
        }
      }
    }
  ],
  "builders": [
    {
      "type": "virtualbox",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
      ],
      "boot_wait": "10s",
      "disk_size": 10140,
      "guest_os_type": "RedHat_64",
      "http_directory": "http",
      "iso_md5": "4a5fa01c81cc300f4729136e28ebe600",
      "iso_url": "http://mirror.rit.edu/centos/6/isos/x86_64/CentOS-6.4-x86_64-minimal.iso",
      "ssh_username": "vagrant",
      "ssh_password": "vagrant",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > shutdown.sh; echo 'vagrant'|sudo -S sh 'shutdown.sh'",
      "guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
      "virtualbox_version_file": ".vbox_version",
      "vboxmanage": [
        [
          "modifyvm",
          "{{.Name}}",
          "--memory",
          "480"
        ],
        [
          "modifyvm",
          "{{.Name}}",
          "--cpus",
          "1"
        ]
      ]
    },
    {
      "type": "vmware",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
      ],
      "boot_wait": "10s",
      "disk_size": 10140,
      "guest_os_type": "rhel6-64",
      "http_directory": "http",
      "iso_md5": "4a5fa01c81cc300f4729136e28ebe600",
      "iso_url": "http://mirror.rit.edu/centos/6/isos/x86_64/CentOS-6.4-x86_64-minimal.iso",
      "ssh_username": "vagrant",
      "ssh_password": "vagrant",
      "ssh_port": 22,
      "ssh_wait_timeout": "10000s",
      "shutdown_command": "echo '/sbin/halt -h -p' > shutdown.sh; echo 'vagrant'|sudo -S sh 'shutdown.sh'",
      "vmx_data": {
        "memsize": "480",
        "numvcpus": "1",
        "cpuid.coresPerSocket": "1"
      }
    }
  ],
  "post-processors": [
    "vagrant"
  ]
}

When I try to build a VMWare Vagrant box with it, everything works well until it gets to the post-processor step, whereupon it errors out:

 ┌┤smerrill@lilliputian-resolution [Jun 29 10:21:51] ~/Documents/Development/packer-boxes/centos-6.4-x86_64 on master +
 └╼ packer build -only=vmware template.json
vmware output will be in this color.

==> vmware: Copying or downloading ISO. Progress will be reported periodically.
==> vmware: Creating virtual machine disk

[ many lines snipped ]

==> vmware: Gracefully halting virtual machine...
==> vmware: Running post-processor: vagrant
==> vmware (vagrant): Creating Vagrant box for 'vmware' provider
    vmware (vagrant): Copying: vmware
Build 'vmware' errored: 1 error(s) occurred:

* Post-processor failed: read vmware: is a directory

==> Some builds didn't complete successfully and had errors:
--> vmware: 1 error(s) occurred:

* Post-processor failed: read vmware: is a directory

==> Builds finished but no artifacts were created.

Think through post-processors

Outputs need to be thought through. The first question of course is what should these be named, since "output" is a bit generic.

The purpose of an output is to take an artifact from a build and transform it into some other format. An example output is a Vagrant one. This takes an artifact of a prior build and turns it into a Vagrant box.

Failed provisioners should fail the build

If a provisioner fails for any reason, the entire build should fail.

`packer build` fails on Windows

packer build
https://gist.github.com/thasmo/e9a75819f2513a130bac

PACKER_LOG=1 packer build
https://gist.github.com/thasmo/f564f101ed05891db824

vagrant post-processor failed: read vmware: is a directory

Using the vagrant post-processor fails on VMWare Fusion:

==> vmware: Running post-processor: vagrant
==> vmware (vagrant): Creating Vagrant box for 'vmware' provider
    vmware (vagrant): Copying: vmware
Build 'vmware' errored: 1 error(s) occurred:

* Post-processor failed: read vmware: is a directory

No issue with virtualbox — it creates the .box file.

Panic on downloading large isos with a pre-existing cached copy

@mitchellh has this report already but just to post the details on the record and some easy steps to reproduce. In my case I'm using the vmware builder, on OS X 10.8.4 with Go 1.1.1.

There seems to be some threshold at which the filesize of an iso_url is big enough so as to cause a runtime panic when it tries to first verify a cached copy before downloading.

I can specify a iso_url that's 2 GB in size, with a correct checksum, and it will download and begin the build. I can cancel this and repeat the build and it will use the cached copy as expected.

However, if I use an iso that's 3 GB (or larger), the build will proceed only when the cached copy didn't already exist first. If I repeat the build and it tries to use the cached copy, there's a runtime panic. I don't see any requests reach the webserver in this case.

I don't know the exact breaking point, but it's somewhere between 2 and 3GB, on this particular machine, an iMac 2012 i7 with a Fusion Drive. In fact, I've even seen it panic once with the 2 GB file.

I also verified that the cached files both match the served files in size.

To set up the test, I wrote two empty files using dd:

dd if=/dev/zero of=2gb bs=1m count=2048
dd if=/dev/zero of=3gb bs=1m count=3072

Served them using python -m SimpleHTTPServer on the default port 8000, although I've also tested this using a non-local webserver.

Relevant lines in the templates:

"iso_url": "http://localhost:8000/3gb",
"iso_md5": "c698c87fb53058d493492b61f4c74189",

"iso_url": "http://localhost:8000/2gb",
"iso_md5": "a981130cf2b7e09f4686dc273cf7187e",

Full output with PACKER_LOG=1 here:

https://gist.github.com/timsutton/5889179

0.1.0 doesn't display the list of commands on Mac OS X (64 bits)

I'm on OS X 10.8.4. Do you need more information to reproduce this?

It just hangs there, after a couple minutes I've exited it using control+C

$ packer
usage: packer [--version] [--help] <command> [<args>]

Available commands are:
^C^CInterrupt signal received twice. Forcefully exiting now.

Idea: New Builder interface method: Rescue

Perhaps there should be a new builder interface method: Rescue. This method would be responsible in the case of a panic to make a best effort to clean itself up.

Let's be honest, problems are going to happen. In that case, it'd be nice for Packer to be like "I'm trying to clean shit up."

Build hooks

The system for "hooks" in builds must be implemented. Hooks are a mechanism for builders to mark various points in the build so that plugins can come in and instrument them in some way.

Hook should be a method on Build that is passed to the Builder as it runs. The interesting thing here is re: #1 because the builder might be remote but it still needs to call back onto the Hook. This is more of a detail for #1, rather than for this issue, but it is important to keep that in mind.

I'm unsure what parameters a hook should take. Return values I gather inspiration from ZNC, which allows hooks to return things such as HALT, CONTINUE, etc. It'd be nice for hooks to be able to stop the build, perhaps.

Provisioners

We need a special kind of hook for templates that specifies provisioners. These are special interfaces that are responsible for installing and configuring software within the machine.

My idea is that these are exposed as template configuration parameters, but are ultimately turned into basically a special case of hooks that runs at a certain time of the build.

Support Artifact Cleanup

Add a new method to the Artifact interface that handles cleanup for the artifact. This will be used for certain configurations to remove intermediary artifacts. I imagine the function would look like this:

Destroy() error

Make shell provisioner args more consistent

A minor nit:

The shell provisioner supports the path option for naming a single script and the scripts option for naming multiple scripts.

It would be more clear if these options were named path and paths or script and scripts.

Listing commands on Windows fails

packer --help gives me:

usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build        Error loading command: exec: "C:\\Users\\...\\packer-command-build": file does not exist
    validate     Error loading command: exec: "C:\\Users\\...\\packer-command-validate": file does not exist

Actually, the files are called "command-build.exe" and "command-validate.exe" and are not prefixed with "packer-".

But if I rename them it says:

usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build        Error loading command: dial tcp <nil>:10000: ConnectEx tcp: Das Format des angegebenen Netzwerknamens ist unzulässig.
    validate     Error loading command: plugin exited before we could connect

After renaming the files (adding the prefix "packer-") and using PACKER_DEBUG=1 it gives me this: https://gist.github.com/thasmo/4220bd9024a56332a87e

Hope that helps!

Docs: not clear where HTTPIP values come from

From IRC:

[09:14:20] <scottyfred>  hi all
[09:15:58] <scottyfred>  so, I have a question about config templates and boot_command. I'm trying to use the example boot_command at the bottom of this page verbatim: http://hc.packer.io/docs/builders/vmware.html
[09:16:38] <scottyfred>  it is not clear to me how {{ .HTTPIP }} and {{ .HTTPPort }} get resolved, but they aren't working for me.

Generic utility for map => struct using Go reflection

Every builder in existence will need to turn arbitrary map[string]interface{} into structs for configuration. Currently the AWS builder does this through a json Marshal/Unmarshal, which works but is kind of silly and produces poor error messages if things aren't perfect.

Let's write a generic utility for doing this, with nice error messages like "configuration field Foo wanted an int but got a string"

Remote machine communication

We need a way to communicate (SSH) with remote machines. This should be an interface so that depending on the underlying builder, the best thing is done. Things:

The builder is responsible for initializing this connection.
Hooks should use this connection to communicate with the machine, rather than trying to handroll.

Timeouts on VirtualBox builder are different between documentation and code.

The documentation mentions the boot_wait timeout to be 10s by default and the ssh_wait_timeout to be 20m. Both are none according to the code.

As well, the boot_command is missing an <enter> at the end.

Post-processor: compress

Build a post-processor that takes Artifacts with files and compresses them into "tar.gz" format (to start).

amazon-ebs build fails with RequestLimitExceeded

After successfully building an AMI, packer received a RequestLimitExceeded error while polling for the AMI to become ready. In this scenario, the AMI has been created, but no artifacts were created.

Terminal output:

==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating the AMI: Packer POC 1372452869
==> amazon-ebs: AMI: ami-deadbeef
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Error querying images: Request limit exceeded. (RequestLimitExceeded)
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Error querying images: Request limit exceeded. (RequestLimitExceeded)

Error Handling

All over the packer codebase right now are TODOs with error handling. As I learned with Vagrant, error handling must be TOP NOTCH. All error cases must be handled and raised to the user in a human-friendly way.

Key things:

Must be able to output human-friendly error message.
Must never crash the system (panics are out of the question, and aren't correct anyways).

Thoughts:

Put it on the Ui, such as Ui.Error or something.

provisioner/shell: Environment Variables

I think it'd be handy to be able to inject environment variables for the shell provisioner.

{
  "type": "shell",
  "inline": [
      "echo $FOO",
      "echo $BAZ"
    ],
  "env_vars": [
      "FOO=bar",
      "BAZ=foo"
    ]
}

Think through "Artifacts"

In Packer, a Template compiles to Builds, and a Build is responsible for producing an Artifact of some sort. An "artifact" is just the build result of a builder. For AWS this is an AMI. For VirtualBox this is an exported VirtualBox VM, etc.

What is the commonality between artifacts such that we can build that into the build struct?

Note that artifacts will have to be parameters, of such to "outputters" (name tbd) which are responsible for things such as creating a Vagrant box out of the prior thing.

redis example fails to build

with error: "Unable to locate package redis-server"

example.json has these contents:

{
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "REDACTED",
    "secret_key": "REDACTED",
    "region": "us-east-1",
    "source_ami": "ami-de0d9eb7",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example {{.CreateTime}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sudo apt-get update",
      "sudo apt-get install -y redis-server"
    ]
  }]
}

Running the commands from the getting started guide:

$ packer validate example.json
Template validated successfully.
$ packer build example.json
amazon-ebs output will be in this color.

==> amazon-ebs: Creating temporary keypair for this instance...
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing SSH access on the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Waiting for instance to become ready...
==> amazon-ebs: Connecting to the instance via SSH...
==> amazon-ebs: Provisioning with shell script: /var/folders/gp/gn1043nx4fn5k3db55x6w9l40000gn/T/packer-shell151021730
Ign http://security.ubuntu.com precise-security InRelease
Ign http://archive.ubuntu.com precise InRelease
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://archive.ubuntu.com precise-updates InRelease
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Hit http://archive.ubuntu.com precise Release.gpgse 0 B
Get:3 http://archive.ubuntu.com precise-updates Release.gpg [198 B]
Hit http://archive.ubuntu.com precise Release0%] [2 Rel
Get:4 http://archive.ubuntu.com precise-updates Release [49.6 kB]
Get:5 http://security.ubuntu.com precise-security/main amd64 Packages [295 kB]
Hit http://archive.ubuntu.com precise/main amd64 Packages
Hit http://archive.ubuntu.com precise/restricted amd64 Packages
Hit http://archive.ubuntu.com precise/universe amd64 Packages
Hit http://archive.ubuntu.com precise/multiverse amd64 Packages
Hit http://archive.ubuntu.com precise/main i386 Packages
Hit http://archive.ubuntu.com precise/restricted i386 Packages
Hit http://archive.ubuntu.com precise/universe i386 Packages
    amazon-ebs: Get:6 http://security.ubuntu.com precise-security/restricted amd64 Packages [4627 B]
Hit http://archive.ubuntu.com precise/multiverse i386 Packages
Get:7 http://security.ubuntu.com precise-security/universe amd64 Packages [77.1 kB]
Hit http://archive.ubuntu.com precise/main TranslationIndex
Hit http://archive.ubuntu.com precise/multiverse TranslationIndex
Get:8 http://security.ubuntu.com precise-security/multiverse amd64 Packages [2182 B]
Get:9 http://security.ubuntu.com precise-security/main i386 Packages [309 kB]
Hit http://archive.ubuntu.com precise/restricted TranslationIndex
Hit http://archive.ubuntu.com precise/universe TranslationIndex
Get:10 http://archive.ubuntu.com precise-updates/main amd64 Packages [644 kB]
Get:11 http://security.ubuntu.com precise-security/restricted i386 Packages [4620 B]
Get:12 http://security.ubuntu.com precise-security/universe i386 Packages [79.8 kB]
Get:13 http://security.ubuntu.com precise-security/multiverse i386 Packages [2367 B]
Get:14 http://security.ubuntu.com precise-security/main TranslationIndex [74 B]
Get:15 http://security.ubuntu.com precise-security/multiverse TranslationIndex [71 B]
Get:16 http://security.ubuntu.com precise-security/restricted TranslationIndex [72 B]
Get:17 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:18 http://security.ubuntu.com precise-security/main Translation-en [143 kB]
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Get:19 http://security.ubuntu.com precise-security/restricted Translation-en [1253 B]
Get:20 http://security.ubuntu.com precise-security/universe Translation-en [49.1 kB]
Get:21 http://archive.ubuntu.com precise-updates/restricted amd64 Packages [10.1 kB]
Get:22 http://archive.ubuntu.com precise-updates/universe amd64 Packages [207 kB]
Get:23 http://archive.ubuntu.com precise-updates/multiverse amd64 Packages [13.6 kB]
Get:24 http://archive.ubuntu.com precise-updates/main i386 Packages [662 kB]
Get:25 http://archive.ubuntu.com precise-updates/restricted i386 Packages [10.0 kB]
Get:26 http://archive.ubuntu.com precise-updates/universe i386 Packages [211 kB]
Get:27 http://archive.ubuntu.com precise-updates/multiverse i386 Packages [13.8 kB]
Get:28 http://archive.ubuntu.com precise-updates/main TranslationIndex [3564 B]
Get:29 http://archive.ubuntu.com precise-updates/multiverse TranslationIndex [2605 B]
Get:30 http://archive.ubuntu.com precise-updates/restricted TranslationIndex [2461 B]
Get:31 http://archive.ubuntu.com precise-updates/universe TranslationIndex [2850 B]
Hit http://archive.ubuntu.com precise/main Translation-en
Hit http://archive.ubuntu.com precise/multiverse Translation-en
Hit http://archive.ubuntu.com precise/restricted Translation-en
Hit http://archive.ubuntu.com precise/universe Translation-en
Get:32 http://archive.ubuntu.com precise-updates/main Translation-en [296 kB]
Get:33 http://archive.ubuntu.com precise-updates/multiverse Translation-en [7834 B]
Get:34 http://archive.ubuntu.com precise-updates/restricted Translation-en [2432 B]
Get:35 http://archive.ubuntu.com precise-updates/universe Translation-en [122 kB]
Fetched 3280 kB in 8s (387 kB/s)       bzip2 0 B] [35 T
Reading package lists... Donege lists... 0%
Reading package lists... Donege lists... 0%
Building dependency tree... 66%ency tree... 0%
Reading state information... Doneormation... 0%
    amazon-ebs: E: Unable to locate package redis-server
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Script exited with non-zero exit status: 100

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 100

==> Builds finished but no artifacts were created.
$

Make paths in templates relative to the template

I used this to convert a Veewee template to Packer: https://github.com/mitchellh/veewee-to-packer

The result went into an output subdirectory.

$ packer validate output/template.json 
Template validation failed. Errors are shown below.

Errors validating build 'virtualbox'. 1 error(s) occurred:

* Bad script 'scripts/postinstall.sh': stat scripts/postinstall.sh: no such file or directory

Errors validating build 'vmware'. 1 error(s) occurred:

* Bad script 'scripts/postinstall.sh': stat scripts/postinstall.sh: no such file or directory

But then:

$ cd output && packer validate template.json
Template validated successfully.

Shouldn't those paths be resolved as relative to the template path?

Plugin logging missing output

Currently, the plugin output is missing some log data. For example, in the output below, it should output "Serving connection" or something like that at some point but it never does. WAT?!

This will be critical to investigate and fix. I have no ideas.

packer master → bin/packer
usage: packer [--version] [--help] <command> [<args>]

Available commands are:
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer.go:28: Loading command: build
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Preparing to serve a command plugin...
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Plugin minimum port: 10000
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Plugin maximum port: 10000
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Plugin address: :10000
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Waiting for connection...
    build     build image(s) from tempate
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:41: waiting for all plugin processes to complete...
2013/05/08 12:44:00 /Users/mitchellh/go/src/pkg/net/rpc/client.go:145: rpc: client protocol error: unexpected EOF
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:95: plugin process exited
2013/05/08 12:44:00 /Users/mitchellh/code/go/src/github.com/mitchellh/packer/packer/plugin/client.go:168: 2013/05/08 12:44:00 Preparing to serve a command pl

Support SSH key-auth for first-contact, VMware/VBox

To help in using the same ks.cfg between AMIs and VMware/VBox assets, it would be nice to support SSH key-auth instead of just password auth, so we can default PasswordAuthentication to 'no' in the ks.cfg.

Post-processors should have access to a Ui

They should be able to use the Ui, because that just makes sense.

Build Errors and User Feedback

When a build encounters an error and a builder returns ActionHalt, can packer know this and notify the user that:

An error has occurred in the build process
Some steps may be reversed (specific language for that could be something like 'Error encountered in build process. Cleanup up...')
Modify the "completion" text to something like "There was an error building ..." (instead of the existing: ==> The build completed! The artifacts created were:)
Possibly color cleanup steps in a color other then green in this case?

Color code and prefix parallel builds

Because builds are meant to run in Parallel, let's color code and prefix the builds so that they're easier to read. An example (without color) of what this might look like:

amazon-ebs: Beginning build process
vmware: Beginning build process
amazon-ebs: Launching the source EC2 instance
vmware: Creating hardware metadata
vmware: Mounting virtual hard drive
amazon-ebs: Waiting for instance to become ready
vmware: Booting VM

This should be easy since we should be able to actually provider a wrapper around a packer.Ui that implements packer.Ui that just color codes things. Easyyyyyy.

Listing command on Windows still fails

packer --help

usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build        build image(s) from template
    validate     Error loading command: plugin exited before we could connect

PACKER_LOG=1 packer --help
https://gist.github.com/thasmo/69790543f4ed145f8f53

Using Cygwin, by the way. ;)

VirtualBox builder: needs an Artifact

VirtualBox 4.2.10 build fails with "The checksum for the file 'VBoxGuestAdditions_.iso' could not be found."

I'm trying to build a VirtualBox image.

My configuration is:

{
    "builders": [{
        "type": "virtualbox",
        "guest_os_type": "RedHat_64",
        "iso_url": "http://centos.serverspace.co.uk/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-minimal.iso",
        "iso_md5": "4a5fa01c81cc300f4729136e28ebe600",
        "ssh_username": "vagrant",
        "ssh_wait_timeout": "30s"
    }]
}

The output of PACKER_LOG=1 packer build:

2013/06/29 20:50:11 Detected home directory from env var: /home/zedr
2013/06/29 20:50:11 Attempting to open config file: /home/zedr/.packerconfig
2013/06/29 20:50:11 File doesn't exist, but doesn't need to. Ignoring.
2013/06/29 20:50:11 Packer config: &{PluginMinPort:0 PluginMaxPort:0 Builders:map[amazon-ebs:packer-builder-amazon-ebs digitalocean:packer-builder-digitalocean virtualbox:packer-builder-virtualbox vmware:packer-builder-vmware] Commands:map[build:packer-command-build validate:packer-command-validate] PostProcessors:map[vagrant:packer-post-processor-vagrant] Provisioners:map[shell:packer-provisioner-shell]}
2013/06/29 20:50:11 Setting cache directory: /home/zedr/Packer/packer_cache
2013/06/29 20:50:11 Environment.Cli: []string{"build", "example.json"}
2013/06/29 20:50:11 command + args: []string{"build", "example.json"}
2013/06/29 20:50:11 Loading command: build
2013/06/29 20:50:11 Creating plugin client for path: /home/zedr/Apps/packer/packer-command-build
2013/06/29 20:50:11 Starting plugin: /home/zedr/Apps/packer/packer-command-build []string{"/home/zedr/Apps/packer/packer-command-build"}
2013/06/29 20:50:11 Waiting for RPC address for: %s /home/zedr/Apps/packer/packer-command-build
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Preparing to serve a command plugin...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Plugin minimum port: 10000
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Plugin maximum port: 25000
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Plugin address: 127.0.0.1:10000
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Waiting for connection...
2013/06/29 20:50:11 Executing command: build
2013/06/29 20:50:11 Loading builder: virtualbox
2013/06/29 20:50:11 Creating plugin client for path: /home/zedr/Apps/packer/packer-builder-virtualbox
2013/06/29 20:50:11 Starting plugin: /home/zedr/Apps/packer/packer-builder-virtualbox []string{"/home/zedr/Apps/packer/packer-builder-virtualbox"}
2013/06/29 20:50:11 Waiting for RPC address for: %s /home/zedr/Apps/packer/packer-builder-virtualbox
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Serving a plugin connection...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Reading template: example.json
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Parsing template...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Creating build: virtualbox
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Preparing to serve a builder plugin...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Plugin minimum port: 10000
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Plugin maximum port: 25000
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Plugin address: 127.0.0.1:10002
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Waiting for connection...
2013/06/29 20:50:11 ui: �[1;32;40mvirtualbox output will be in this color.�[0m
�[1;32;40mvirtualbox output will be in this color.�[0m
2013/06/29 20:50:11 ui: 

2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Serving a plugin connection...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Build debug mode: false
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Preparing build: virtualbox
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 VBoxManage path: /usr/bin/VBoxManage
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Waiting on builds to complete...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Starting build run: virtualbox
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Running builder: virtualbox
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Downloading guest addition checksums: http://download.virtualbox.org/virtualbox//SHA256SUMS
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Parsed URL: &url.URL{Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"download.virtualbox.org", Path:"/virtualbox//SHA256SUMS", RawQuery:"", Fragment:""}
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Downloading: http://download.virtualbox.org/virtualbox//SHA256SUMS
2013/06/29 20:50:11 ui error: �[1;31;40mBuild 'virtualbox' errored: The checksum for the file 'VBoxGuestAdditions_.iso' could not be found.�[0m
�[1;31;40mBuild 'virtualbox' errored: The checksum for the file 'VBoxGuestAdditions_.iso' could not be found.�[0m
2013/06/29 20:50:11 ui error: 
==> Some builds didn't complete successfully and had errors:

==> Some builds didn't complete successfully and had errors:
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: 2013/06/29 20:50:11 Builds completed. Waiting on interrupt barrier...
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Checksum file parts: []string{"<HEAD><META", "HTTP-EQUIV=\"Content-Type\"", "CONTENT=\"text/html;charset=ISO-8859-1\"><TITLE>Not", "Found</TITLE></HEAD>"}
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: 2013/06/29 20:50:11 Checksum file parts: []string{"<H1>Not", "Found</H1>", "The", "requested", "object", "does", "not", "exist", "on", "this", "server.", "The", "link", "you", "followed", "is", "either", "outdated,", "inaccurate,", "or", "the", "server", "has", "been", "instructed", "not", "to", "let", "you", "have", "it."}
2013/06/29 20:50:11 ui error: --> virtualbox: The checksum for the file 'VBoxGuestAdditions_.iso' could not be found.
--> virtualbox: The checksum for the file 'VBoxGuestAdditions_.iso' could not be found.
2013/06/29 20:50:11 ui: 
==> Builds finished but no artifacts were created.

==> Builds finished but no artifacts were created.
2013/06/29 20:50:11 waiting for all plugin processes to complete...
2013/06/29 20:50:11 rpc: client protocol error: unexpected EOF
2013/06/29 20:50:11 rpc: client protocol error: unexpected EOF
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-builder-virtualbox: plugin process exited
2013/06/29 20:50:11 rpc: client protocol error: unexpected EOF
2013/06/29 20:50:11 /home/zedr/Apps/packer/packer-command-build: plugin process exited

Logging

Logging is critical in my ability to debug things on Vagrant, and it wasn't added until like... 0.7. I don't want to make that same mistake with Packer. Let's put logging in RIGHT AWAY.

Key ideas:

PACKER_LOG environmental variable for controlling verbosity.
Multiple levels: debug, info, warn, etc. I don't actually care what they're named, just that they exist.
Must be easily available to all parts of the packer program and libraries.

Thoughts on implementation:

I think a global "Logger" like variable would be nice so ANY library component can hook into loggers.
However, it might be nice to have one logger per packer Environment. But is an Environment necessary for using other components that might want to log? Probably not. Ugh.
If we are running multiple packer builds in parallel, we'll want to be able to differentiate log output from the different builds.

Build should manage Ui prefixing

The way things are architected right now with the build command handling Ui prefixing, everything within a build is prefixed with the "==> BUILDNAME: " prefix. For things like post-processors it would be nice if we were able to prefix it ourselves so we can do things like:

==> vmware: blah
==> vmware (compress): bah
==> vmware (upload): blah

etc.

So I think we should keep colors out to the commands and such, but lock prefixing into the Build object itself.

Dependency Updates

I recently tried to make build packer and hit an issue with one of the external libraries not having a method we were calling from within packer.

I quickly realized that it was and out-of-date dependency, and in this case ran:

go get -u github.com/mitchellh/goamz/ec2

Forcing go get to hit the network and install a new version of the lib.

Is there a way to pull this into a make command, or automate it in some fashion?

Global configuration file

For the actual packer binary, we want to have a global configuration file. This file should be some simple format that describes the available components in Packer. Example of what I imagine this looking like:

[commands]
build = packer-build

[builders]
aws-ebs = packer-builder-aws-ebs
virtualbox = packer-builder-virtualbox
vmware = packer-builder-vmware --maybe-a-flag

The mapping should be self explanatory I hope but its name => binary where binary should be an absolute path, relative path, or nothing and it is looked up on the PATH.

This file should parse into an EnvironmentConfig I'm guessing. or something like that.

Validator should provide line number of invalid syntax

Current:

packer validate template.json
Failed to parse template: invalid character ']' looking for beginning of value

Would be nice:

packer validate template.json
Line 19: Failed to parse template: invalid character ']' looking for beginning of value

Still hunting my file for this misplaced ']'...but I'll find it.

Data races

go test -race ./... finds 37 data races. Some of these may be isolated to test code, but this results in flaky tests, and makes it hard to find production races.

Full output: https://gist.github.com/titanous/5887208

vmware: Timeout while waiting for machine to shut down.

VMWare Fusion 5.0.3 on OS X 10.8.4

==> vmware: Timeout while waiting for machine to shut down.
==> vmware: Stopping virtual machine...
==> vmware: Deleting output directory...
Build 'vmware' errored: Timeout while waiting for machine to shut down.

Post-processor: Vagrant

Before release we need to have a post-processor to convert these things to Vagrant boxes. People will be asking for it right away, and I've always talked about Packer as a potential tool for building Vagrant boxes, so it may be confusing for those expecting it to be missing that feature.

hashicorp / packer Goto Github PK

packer's Introduction

Packer

Quick Start

Packer

HCP Packer

Documentation

Contributing to Packer

Unmaintained Plugins

packer's People

Contributors

Stargazers

Watchers

Forkers

packer's Issues

Recommend Projects

Recommend Topics

Recommend Org