Comments (5)
The problem is with stop ecs
which fails the first time since the service is not running.
Having the following in your user data script will also work without needing a upstart job.
#!/bin/bash
# Create filesystem on the new EBS device:
mkfs -t ext4 /dev/sdb
mkdir /srv/data/
mount /dev/sdb /srv/data
echo "/dev/sdb /srv/data ext4 defaults,nofail 0 2" >> /etc/fstab
# Register as an ECS instance:
yum install -y aws-cli
aws s3 cp s3://<my-bucket>/ecs.config /etc/ecs/ecs.config
echo ECS_CLUSTER=default >> /etc/ecs/ecs.config
# Restart docker service to allow it to see new EBS volume
service docker restart
start ecs
from amazon-ecs-agent.
Hello @ms-ati. I believe this issue is the same as the one in this forum thread.
Per my response there, can you try either restarting the docker daemon after making a mount change (sudo stop ecs && sudo service docker restart && sudo start ecs
) or launching docker without the unshare -m
command?
Best,
Euan
Edit: To add a little more information, I believe the unshare
is there in the first place to work around an issue in docker (see issue and workaround). Please note that if you remove the unshare
command rather than restarting docker to propagate mount changes, there might be issues removing stopped containers.
from amazon-ecs-agent.
If you're curious, here's how I ended up solving the problem. Add this to your user data for the ECS instance:
#upstart-job
description "Pre-ECS agent initialization"
start on (starting ecs or starting docker)
task
script
# Create filesystem on the new EBS device:
mkfs -t ext4 /dev/sdb
mkdir /srv/data/
mount /dev/sdb /srv/data
echo "/dev/sdb /srv/data ext4 defaults,nofail 0 2" >> /etc/fstab
# Register as an ECS instance with access to our private docker repositories:
yum install -y aws-cli
aws s3 cp s3://<my-bucket>/ecs.config /etc/ecs/ecs.config
echo ECS_CLUSTER=default >> /etc/ecs/ecs.config
end script
The trick here is to tie the mounting & formatting step to ecs & docker start via an upstart task, ensuring this task runs before ecs or docker are started on the ECS instance.
from amazon-ecs-agent.
@euank Thank you kind sir. Restarting the docker daemon after mount change, as you describe above, does indeed fix the issue 👏
from amazon-ecs-agent.
@euank Is there any way to do this at ECS instance boot time? When I add the following to my autoscaling configuration user data:
#!/bin/bash
# Create filesystem on the new EBS device:
mkfs -t ext4 /dev/sdb
mkdir /srv/data/
mount /dev/sdb /srv/data
echo "/dev/sdb /srv/data ext4 defaults,nofail 0 2" >> /etc/fstab
# Register as an ECS instance:
yum install -y aws-cli
aws s3 cp s3://<my-bucket>/ecs.config /etc/ecs/ecs.config
echo ECS_CLUSTER=default >> /etc/ecs/ecs.config
# Restart docker service to allow it to see new EBS volume
stop ecs && service docker restart && start ecs
I don't see the volume working. SSH'ing into the ECS instance and running stop ecs && service docker restart && start ecs
works, but this is much more difficult to automate. I've tried simple variations such as adding stop ecs && service docker stop
to the beginning of the user data script and then starting the services at the end, but to no avail.
from amazon-ecs-agent.
Related Issues (20)
- Upgrade minimum docker client api to 1.24 to maintain compatibility with upcoming docker engine v26 release HOT 3
- Task Health Status wrongly reported as HEALTHY HOT 1
- Update not supported on ARM architecture HOT 1
- Add retries for publishing metrics & health checks
- ECS Deployment Fails Due to Premature Resource Availability Reporting HOT 8
- Add support for custom ECS-Agent and ECS-Telemetry Endpoints HOT 1
- Upgraded ecs agent causes Error loading previously saved state from BoltDB HOT 4
- ECS control plane not compatible with ECS-A and Docker v26 requirements for API version HOT 6
- AWS ECS agent does not start in EC2 instance HOT 3
- Agent is Failing to Add com.amazonaws.ecs.capability.logging-driver.journald Attribute to the Container Instance HOT 1
- Docker client doesn't support zstd compression HOT 1
- ECS agent on windows does not work for more than 10 CPU despite setting 'ECS_ENABLE_TASK_CPU_MEM_LIMIT' to true HOT 4
- Secret in US region, and ECS cluster in Asia pacific region HOT 2
- Run Security Updates without failing long-running tasks HOT 2
- Unable to delete Docker image due to multiple repository references HOT 1
- Specifying docker image for caching during ecs-init
- nvidia-gpu-info.json not being generated since v1.82.4
- Docker tags are not shown for pulled images where tag is specified in task definition HOT 1
- More descriptive log message for "Resources not consumed, enough resources not available" HOT 1
- ECS Instances stuck with "Agent Disconnected" HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-ecs-agent.