Comments (23)
This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list
says:
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
Indeed apt-get update
should be using a mirror like http://us-east-1.ec2.archive.ubuntu.com
. The normal sequence of events is:
- Instance boots up and the ssh service starts.
- The cloud-init service starts and overwrites
/etc/apt/sources.list
with an EC2-specific mirror. sudo apt-get update
runs against the new mirror.sudo apt-get install -y redis-server
runs against the new mirror and everything works.
However, apt-get update
is still using http://archive.ubuntu.com
when the error happens, which makes me think the problematic sequence of events is:
- Instance boots up and the ssh service starts.
sudo apt-get update
starts running against the old mirror.- Meanwhile, the cloud-init service starts and overwrites
/etc/apt/sources.list
with an EC2-specific mirror. sudo apt-get install -y redis-server
runs against the new mirror, but errors out becauseapt-get update
has never been run against this particular mirror.
The workaround I'm using is:
"provisioners": [{
"type": "shell",
"inline": [
"sleep 30",
"sudo apt-get update",
"sudo apt-get install -y redis-server"
]
}]
I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a sleep
statement for now...
from packer.
I'm running through the docs myself, and I got the same error. I added the sleep, and got the same.
"provisioners": [{
"type": "shell",
"inline": [
"sudo apt-get update",
"sleep 1",
"sudo apt-get install -y redis-server"
]
}]
I also manually launched the demo AMI, and sudo apt-get update; sudo apt-get install -y redis-server
worked fine. How do I see the script that packer is executing on the server?
Edit: I modified provisioner.go to log the script, and it's pretty much what's in the JSON file:
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Opening /tmp/packer-shell394007326 for reading
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Read script: sudo apt-get update
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: sudo apt-get install -y redis-server
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Uploading /tmp/packer-shell394007326 => /tmp/script.sh
from packer.
I haven't yet checked what packages are available from the source AMI, but this seems pretty odd.
From another Ubuntu VM I have running right now:
vagrant@redacted:~$ sudo apt-get install -y redis-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
redis-server
0 upgraded, 1 newly installed, 0 to remove and 34 not upgraded.
Need to get 204 kB of archives.
After this operation, 523 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe redis-server amd64 2:2.2.12-1build1 [204 kB]
Fetched 204 kB in 0s (297 kB/s)
Selecting previously unselected package redis-server.
(Reading database ... 51084 files and directories currently installed.)
Unpacking redis-server (from .../redis-server_2%3a2.2.12-1build1_amd64.deb) ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db ...
Setting up redis-server (2:2.2.12-1build1) ...
Starting redis-server: redis-server.
vagrant@vwhitfoot:~$ uname -a
Linux vwhitfoot 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
vagrant@redacted:~$
from packer.
Ugh this is the weirdest thing. I've seen this happen too and simply don't know what causes it. Why... why does this happen?
from packer.
I think this is actually a timing issue with apt. I added another line (cat /etc/apt/sources.list
) to the inline shell provisioning steps and now redis-server sometimes gets installed. Running into other intermittent errors now. Possibly related to #42?
from packer.
In hindsight, I should have had had PACKER_LOG=1
while testing this, but here are some examples of errors I still get intermittently with 0.1.1:
https://gist.github.com/englishm/e12fff3aa975d956e5c7
from packer.
Yeah it is intermittent for me too. I wonder if you put a sleep 1
in there if this completely solves the problem? But why is there a timing issue? I execute the commands in serial and wait for an exit status from the prior one before executing the next one. :(
from packer.
Yeah, I was wondering how related it might be to #42 as well. It does seem to be some kind of race condition. Out of curiosity is it any more stable on an m1.small? If there was ever a box prone to AWS weirdness it's the t1.micro
from packer.
I'm headed out for vacation today and can't check this right now.
I'll have limited Internet connectivity while I'm gone though and I'll
update this if I have time/bandwidth to retest at all.
On Sat, Jun 29, 2013 at 10:23 AM, Matthew Surabian <[email protected]
wrote:
Yeah, I was wondering how related it might be to #42https://github.com/mitchellh/packer/issues/42as well. It does seem to be some kind of race condition. Out of curiosity
is it any more stable on an m1.small? If there was ever a box prone to AWS
weirdness it's the t1.micro—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-20230622
.
Mike English
Atomic Object | http://www.atomicobject.com
[email protected]
[Ph] +1.616.776.6020
from packer.
@mitchellh I am confused, I thought Packer was creating a single shell script for the whole provisioner? How could this be caused by a race within Packer when the whole script is executed in one command?
Instead, could it be an issue with the VM image itself? I saw it happen often with the AMI from the documentation, but not with Raring (ami-e995e380).
from packer.
@priteau It does create a shell script, it can't be a race within Packer. That comment was just speculation.
@MattSurabian Unfortunately, it is unrelated to #42. Packer makes a single shell script from your inline calls, so it can't be a race within Packer itself.
from packer.
@ericlathrop You can see here what Packer does: https://github.com/mitchellh/packer/blob/06108b4f96cb899c9bf9c40808170e0ea8143c8e/provisioner/shell/provisioner.go#L127-L150
Basically it just takes each inline command and puts them in a script one line at a time. It then saves it. Line 140 is particularly where things happen.
from packer.
@grosskur your detective work is excellent! We can check if cloudinit has finished by checking for the existence of /var/lib/cloud/instance/boot-finished.
from packer.
I've updated the getting started guide to get it working. Thanks!
from packer.
Thanks, @grosskur, @mitchellh!
Would be nice to also see a more robust "instance readiness" check like @priteau suggests.
Maybe as a config option?
from packer.
@englishm Maybe, but I think it fits into the provisioner's role.
from packer.
@mitchellh fair enough.
from packer.
I am unable to find documentation of the above in the docs anymore, and this seems to still be a problem.
from packer.
@davclark It is still present in http://www.packer.io/intro/getting-started/provision.html, and also described in the shell provisioner documentation at the very end of http://www.packer.io/docs/provisioners/shell.html.
from packer.
OK - I see the sleep lines. I was looking for something more cloud-init specific that would look for the existence of a file (which is what I'm doing). I suppose sleep 30
is a good general solution, though.
Thanks @priteau!
from packer.
Some thoughts on waiting for cloud-init more effectively than a hardcoded sleep:
- Sufficiently new cloud-init versions (0.7.5 and newer, I believe) have json status files in /var/lib/cloud/data/ and /run/cloud-init that are documented at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/status.txt
- All cloud-init versions (or at least 0.6.3 and newer which shipped with Ubuntu 12.04) update /var/lib/cloud/instance/boot-finished. That file is persistent across boots, though, so it only works once. That might be ok for packer, but could lead to weirdness if it's left lying around in some upstream AMI. Documented in https://bugs.launchpad.net/cloud-init/+bug/1258113
- uvtool (which appears to be an Ubuntu-focused packer-like image creation tool) uses the following code to wait for cloud-init to finish: http://bazaar.launchpad.net/~uvtool-dev/uvtool/trunk/view/head:/remote-wait.sh
from packer.
This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of
/etc/apt/sources.list
says:## Note, this file is written by cloud-init on first boot of an instance ## modifications made here will not survive a re-bundle.
Indeed
apt-get update
should be using a mirror likehttp://us-east-1.ec2.archive.ubuntu.com
. The normal sequence of events is:
- Instance boots up and the ssh service starts.
- The cloud-init service starts and overwrites
/etc/apt/sources.list
with an EC2-specific mirror.sudo apt-get update
runs against the new mirror.sudo apt-get install -y redis-server
runs against the new mirror and everything works.However,
apt-get update
is still usinghttp://archive.ubuntu.com
when the error happens, which makes me think the problematic sequence of events is:
- Instance boots up and the ssh service starts.
sudo apt-get update
starts running against the old mirror.- Meanwhile, the cloud-init service starts and overwrites
/etc/apt/sources.list
with an EC2-specific mirror.sudo apt-get install -y redis-server
runs against the new mirror, but errors out becauseapt-get update
has never been run against this particular mirror.The workaround I'm using is:
"provisioners": [{ "type": "shell", "inline": [ "sleep 30", "sudo apt-get update", "sudo apt-get install -y redis-server" ] }]
I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a
sleep
statement for now...
This was very helpful, thank you very much. We had the same issue while installing expect on Ubuntu using packer.
from packer.
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
from packer.
Related Issues (20)
- Failed getting the "github.com/hashicorp/azure" plugin HOT 15
- When using Azure Compute Gallery image as base, using image_id fails but using image_name/image_version works HOT 3
- can't install latest packer HOT 3
- Ansible playbook become: true not working when using with packer HOT 4
- Remove local plugin loading recommendation from v1.11.x version of developer documentation
- Salt Provisioner HOT 6
- Packer failed to run target: googlecloud HOT 9
- Please add support to alltrue function
- Please fix documentation pages to remember code sample choice selected by user and use HCL2 as the default format HOT 2
- [ CVE-2024-6104 ] Update github.com/hashicorp/go-retryablehttp package HOT 3
- Hasem HOT 1
- Go 1.22.0 toolchain introduces panic on go generate HOT 3
- Packer qemu builder crashes sometimes (yamux keepalive, qemu EOF) HOT 6
- Terraform-Like Optional Attribute Support
- Can not provide arn for organization HOT 2
- The copied image does not have name if error occurs HOT 2
- Improve legenda for packer's output HOT 5
- Win2022server OS corrupted/broken HOT 1
- Unable to install plugin - Invalid arguments, expected either 1 or 2 positional arguments, got 3 HOT 3
- Packer crashed while building EKS Anywhere node image for cloudstack HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from packer.