redis example fails to build about packer HOT 23 CLOSED

This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list says:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.

Indeed apt-get update should be using a mirror like http://us-east-1.ec2.archive.ubuntu.com. The normal sequence of events is:

1. Instance boots up and the ssh service starts.
2. The cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
3. sudo apt-get update runs against the new mirror.
4. sudo apt-get install -y redis-server runs against the new mirror and everything works.

However, apt-get update is still using http://archive.ubuntu.com when the error happens, which makes me think the problematic sequence of events is:

1. Instance boots up and the ssh service starts.
2. sudo apt-get update starts running against the old mirror.
3. Meanwhile, the cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
4. sudo apt-get install -y redis-server runs against the new mirror, but errors out because apt-get update has never been run against this particular mirror.

The workaround I'm using is:

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }]

I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a sleep statement for now...

from packer.

I'm running through the docs myself, and I got the same error. I added the sleep, and got the same.

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sudo apt-get update",
            "sleep 1",
            "sudo apt-get install -y redis-server"
        ]
    }]

I also manually launched the demo AMI, and sudo apt-get update; sudo apt-get install -y redis-server worked fine. How do I see the script that packer is executing on the server?

Edit: I modified provisioner.go to log the script, and it's pretty much what's in the JSON file:

2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Opening /tmp/packer-shell394007326 for reading
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Read script: sudo apt-get update
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: sudo apt-get install -y redis-server
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Uploading /tmp/packer-shell394007326 => /tmp/script.sh

from packer.

I haven't yet checked what packages are available from the source AMI, but this seems pretty odd.
From another Ubuntu VM I have running right now:

vagrant@redacted:~$ sudo apt-get install -y redis-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  redis-server
0 upgraded, 1 newly installed, 0 to remove and 34 not upgraded.
Need to get 204 kB of archives.
After this operation, 523 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe redis-server amd64 2:2.2.12-1build1 [204 kB]
Fetched 204 kB in 0s (297 kB/s)
Selecting previously unselected package redis-server.
(Reading database ... 51084 files and directories currently installed.)
Unpacking redis-server (from .../redis-server_2%3a2.2.12-1build1_amd64.deb) ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db ...
Setting up redis-server (2:2.2.12-1build1) ...
Starting redis-server: redis-server.
vagrant@vwhitfoot:~$ uname -a
Linux vwhitfoot 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
vagrant@redacted:~$

from packer.

Ugh this is the weirdest thing. I've seen this happen too and simply don't know what causes it. Why... why does this happen?

from packer.

I think this is actually a timing issue with apt. I added another line (cat /etc/apt/sources.list) to the inline shell provisioning steps and now redis-server sometimes gets installed. Running into other intermittent errors now. Possibly related to #42?

from packer.

In hindsight, I should have had had PACKER_LOG=1 while testing this, but here are some examples of errors I still get intermittently with 0.1.1:

https://gist.github.com/englishm/e12fff3aa975d956e5c7

from packer.

Yeah it is intermittent for me too. I wonder if you put a sleep 1 in there if this completely solves the problem? But why is there a timing issue? I execute the commands in serial and wait for an exit status from the prior one before executing the next one. :(

from packer.

Yeah, I was wondering how related it might be to #42 as well. It does seem to be some kind of race condition. Out of curiosity is it any more stable on an m1.small? If there was ever a box prone to AWS weirdness it's the t1.micro

from packer.

I'm headed out for vacation today and can't check this right now.
I'll have limited Internet connectivity while I'm gone though and I'll
update this if I have time/bandwidth to retest at all.

On Sat, Jun 29, 2013 at 10:23 AM, Matthew Surabian <[email protected]

wrote:

Yeah, I was wondering how related it might be to #42https://github.com/mitchellh/packer/issues/42as well. It does seem to be some kind of race condition. Out of curiosity
is it any more stable on an m1.small? If there was ever a box prone to AWS
weirdness it's the t1.micro

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-20230622
.

Mike English
Atomic Object | http://www.atomicobject.com
[email protected]
[Ph] +1.616.776.6020

from packer.

@mitchellh I am confused, I thought Packer was creating a single shell script for the whole provisioner? How could this be caused by a race within Packer when the whole script is executed in one command?

Instead, could it be an issue with the VM image itself? I saw it happen often with the AMI from the documentation, but not with Raring (ami-e995e380).

from packer.

@priteau It does create a shell script, it can't be a race within Packer. That comment was just speculation.

@MattSurabian Unfortunately, it is unrelated to #42. Packer makes a single shell script from your inline calls, so it can't be a race within Packer itself.

from packer.

@ericlathrop You can see here what Packer does: https://github.com/mitchellh/packer/blob/06108b4f96cb899c9bf9c40808170e0ea8143c8e/provisioner/shell/provisioner.go#L127-L150

Basically it just takes each inline command and puts them in a script one line at a time. It then saves it. Line 140 is particularly where things happen.

from packer.

@grosskur your detective work is excellent! We can check if cloudinit has finished by checking for the existence of /var/lib/cloud/instance/boot-finished.

from packer.

I've updated the getting started guide to get it working. Thanks!

from packer.

Thanks, @grosskur, @mitchellh!

Would be nice to also see a more robust "instance readiness" check like @priteau suggests.
Maybe as a config option?

from packer.

@englishm Maybe, but I think it fits into the provisioner's role.

from packer.

@mitchellh fair enough.

from packer.

I am unable to find documentation of the above in the docs anymore, and this seems to still be a problem.

from packer.

@davclark It is still present in http://www.packer.io/intro/getting-started/provision.html, and also described in the shell provisioner documentation at the very end of http://www.packer.io/docs/provisioners/shell.html.

from packer.

OK - I see the sleep lines. I was looking for something more cloud-init specific that would look for the existence of a file (which is what I'm doing). I suppose sleep 30 is a good general solution, though.

Thanks @priteau!

from packer.

Some thoughts on waiting for cloud-init more effectively than a hardcoded sleep:

• Sufficiently new cloud-init versions (0.7.5 and newer, I believe) have json status files in /var/lib/cloud/data/ and /run/cloud-init that are documented at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/status.txt
• All cloud-init versions (or at least 0.6.3 and newer which shipped with Ubuntu 12.04) update /var/lib/cloud/instance/boot-finished. That file is persistent across boots, though, so it only works once. That might be ok for packer, but could lead to weirdness if it's left lying around in some upstream AMI. Documented in https://bugs.launchpad.net/cloud-init/+bug/1258113
• uvtool (which appears to be an Ubuntu-focused packer-like image creation tool) uses the following code to wait for cloud-init to finish: http://bazaar.launchpad.net/~uvtool-dev/uvtool/trunk/view/head:/remote-wait.sh

from packer.

This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list says:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.

Indeed apt-get update should be using a mirror like http://us-east-1.ec2.archive.ubuntu.com. The normal sequence of events is:

1. Instance boots up and the ssh service starts.
2. The cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
3. sudo apt-get update runs against the new mirror.
4. sudo apt-get install -y redis-server runs against the new mirror and everything works.

However, apt-get update is still using http://archive.ubuntu.com when the error happens, which makes me think the problematic sequence of events is:

1. Instance boots up and the ssh service starts.
2. sudo apt-get update starts running against the old mirror.
3. Meanwhile, the cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
4. sudo apt-get install -y redis-server runs against the new mirror, but errors out because apt-get update has never been run against this particular mirror.

The workaround I'm using is:

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }]

I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a sleep statement for now...

This was very helpful, thank you very much. We had the same issue while installing expect on Ubuntu using packer.

from packer.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

from packer.