Comments (14)
cf-gitbot
commented on July 21, 2024
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/156569866
The labels on this github issue will be updated when the story is started.
from garden-runc-release.
karampok
commented on July 21, 2024
Hi team!
We think the problem is around grootfs because it started when we changed to grootfs.
Also because the file does actually exist in the image
$ mkdir /tmp/grafana && docker export $(docker create grafana/grafana) | tar -C /tmp/grafana -xvf -
$ ls etc/grafana/grafana.ini
etc/grafana/grafana.ini
so it might be related.
Any advice on how to debug it more?
Thx!
from garden-runc-release.
williammartin
commented on July 21, 2024
Hey @karampok @renelehmann Is it possible to give us the image to play with? Oh, is it grafana/grafana on Dockerhub?
from garden-runc-release.
renelehmann
commented on July 21, 2024
Hey @williammartin , yes, right and sorry not mentioned that. Thanks for looking into it. https://hub.docker.com/r/grafana/grafana/
from garden-runc-release.
williammartin
commented on July 21, 2024
Looks like I can reproduce this locally:
± ek+wm |develop {1} U:1 ✗| → cf push grafana_fail --docker-image grafana/grafana
± ek+wm |develop {1} U:1 ✗| → cf logs grafana_fail --recent
Retrieving logs for app grafana_fail in org teddyking / space dev as admin...
2018-04-06T16:32:32.45+0100 [API/0] OUT Created app with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a
2018-04-06T16:32:32.84+0100 [API/0] OUT Creating build for app with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a
2018-04-06T16:32:33.01+0100 [API/0] OUT Updated app with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a ({"state"=>"STARTED"})
2018-04-06T16:32:34.25+0100 [STG/0] OUT Creating container
2018-04-06T16:32:34.71+0100 [STG/0] OUT Successfully created container
2018-04-06T16:32:35.05+0100 [STG/0] OUT Staging...
2018-04-06T16:32:35.14+0100 [STG/0] OUT Staging process started ...
2018-04-06T16:32:36.45+0100 [STG/0] OUT Staging process finished
2018-04-06T16:32:36.45+0100 [STG/0] OUT Exit status 0
2018-04-06T16:32:36.45+0100 [STG/0] OUT Staging Complete
2018-04-06T16:32:36.49+0100 [STG/0] OUT Stopping instance 5b801228-48e2-4565-9fbb-5b8a7ab31aa8
2018-04-06T16:32:36.49+0100 [STG/0] OUT Destroying container
2018-04-06T16:32:36.61+0100 [API/0] OUT Creating droplet for app with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a
2018-04-06T16:32:36.89+0100 [STG/0] OUT Successfully destroyed container
2018-04-06T16:32:37.02+0100 [CELL/0] OUT Creating container
2018-04-06T16:33:15.42+0100 [CELL/0] OUT Successfully created container
2018-04-06T16:33:15.68+0100 [CELL/0] OUT Starting health monitoring of container
2018-04-06T16:33:15.88+0100 [APP/PROC/WEB/0] OUT t=2018-04-06T15:33:15+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"
2018-04-06T16:33:15.90+0100 [APP/PROC/WEB/0] OUT Exit status 1
2018-04-06T16:33:15.91+0100 [CELL/SSHD/0] OUT Exit status 0
2018-04-06T16:33:15.98+0100 [CELL/0] OUT Stopping instance fa4239a2-7cbe-41be-70ad-8b50
2018-04-06T16:33:15.98+0100 [CELL/0] OUT Destroying container
2018-04-06T16:33:16.01+0100 [CELL/0] OUT Creating container
2018-04-06T16:33:16.20+0100 [API/0] OUT Process has crashed with type: "web"
2018-04-06T16:33:16.24+0100 [API/0] OUT App instance exited with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a payload: {"instance"=>"fa4239a2-7cbe-41be-70ad-8b50", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>1, "crash_timestamp"=>1523028795931561938, "version"=>"0faf0105-679c-41ef-8367-c5fc9e0554d0"}
2018-04-06T16:33:16.35+0100 [CELL/0] OUT Successfully destroyed container
2018-04-06T16:33:43.25+0100 [CELL/0] OUT Successfully created container
2018-04-06T16:33:43.89+0100 [CELL/0] OUT Starting health monitoring of container
2018-04-06T16:33:44.04+0100 [APP/PROC/WEB/0] OUT t=2018-04-06T15:33:44+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"
2018-04-06T16:33:44.06+0100 [APP/PROC/WEB/0] OUT Exit status 1
2018-04-06T16:33:44.07+0100 [CELL/SSHD/0] OUT Exit status 0
2018-04-06T16:33:44.13+0100 [API/0] OUT Process has crashed with type: "web"
2018-04-06T16:33:44.13+0100 [CELL/0] OUT Stopping instance 80920a0a-5ce3-4729-6c6f-603e
2018-04-06T16:33:44.13+0100 [CELL/0] OUT Destroying container
2018-04-06T16:33:44.15+0100 [API/0] OUT App instance exited with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a payload: {"instance"=>"80920a0a-5ce3-4729-6c6f-603e", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>2, "crash_timestamp"=>1523028824077578920, "version"=>"0faf0105-679c-41ef-8367-c5fc9e0554d0"}
2018-04-06T16:33:44.19+0100 [CELL/0] OUT Creating container
2018-04-06T16:33:44.57+0100 [CELL/0] OUT Successfully destroyed container
2018-04-06T16:34:11.36+0100 [CELL/0] OUT Successfully created container
2018-04-06T16:34:12.01+0100 [CELL/0] OUT Starting health monitoring of container
2018-04-06T16:34:12.15+0100 [APP/PROC/WEB/0] OUT t=2018-04-06T15:34:12+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"
2018-04-06T16:34:12.16+0100 [APP/PROC/WEB/0] OUT Exit status 1
2018-04-06T16:34:12.17+0100 [CELL/SSHD/0] OUT Exit status 0
2018-04-06T16:34:12.26+0100 [CELL/0] OUT Stopping instance d94e7e24-65f7-4944-4131-7f4f
2018-04-06T16:34:12.26+0100 [CELL/0] OUT Destroying container
2018-04-06T16:34:12.29+0100 [API/0] OUT Process has crashed with type: "web"
2018-04-06T16:34:12.32+0100 [API/0] OUT App instance exited with guid 44325887-1c90-456a-8ce3-9ebe29c2b56a payload: {"instance"=>"d94e7e24-65f7-4944-4131-7f4f", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>3, "crash_timestamp"=>1523028852181288811, "version"=>"0faf0105-679c-41ef-8367-c5fc9e0554d0"}
2018-04-06T16:34:12.63+0100 [CELL/0] OUT Successfully destroyed container
from garden-runc-release.
williammartin
commented on July 21, 2024
Reproducible without CF as well:
Shed
gaol create -n cake -r docker:///grafana/grafana
gaol shell cake
root@cake:~# stat /etc/grafana/grafana.ini
File: '/etc/grafana/grafana.ini'
Size: 13660 Blocks: 32 IO Block: 4096 regular file
Device: 30h/48d Inode: 192 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 107/ grafana)
Access: 2018-04-06 15:44:09.064130489 +0000
Modify: 2018-03-28 12:44:40.000000000 +0000
Change: 2018-04-06 15:44:18.312130386 +0000
Birth: -
GrootFS
gaol create -n cake -r docker:///grafana/grafana
gaol shell cake
root@cake:~# stat /etc/grafana/grafana.ini
stat: cannot stat '/etc/grafana/grafana.ini': No such file or directory
This definitely looks like a regression. Sorry y'all!
from garden-runc-release.
williammartin
commented on July 21, 2024
Looks like in this containers bundle json there is a bind mount over /etc/grafana:
"mounts": [
{
"destination": "/etc/grafana",
"options": [
"bind"
],
"source": "/var/vcap/data/grootfs/store/unprivileged/images/cake/vol-e467f3f074213307736e036be19677355874a789c53064b1e60453703ee52889",
"type": "bind"
},
...
When I look at this directory on the host it is empty:
garden/dc44951d-6a7d-4139-8841-3d63164d9266:/var/vcap/data/garden/depot/cake# ll /var/vcap/data/grootfs/store/unprivileged/images/cake/vol-e467f3f074213307736e036be19677355874a789c53064b1e60453703ee52889
total 0
drwxr-xr-x 2 4294967294 4294967294 6 Apr 6 15:46 ./
drwx------ 8 4294967294 4294967294 293 Apr 6 15:46 ../
So for some reason we are bind mounting an empty directory here, which is why the files are disappearing in the container.
from garden-runc-release.
renelehmann
commented on July 21, 2024
Hey @williammartin Thanks a lot for your great research and the finding.
Do we have any chance we could get around this issue in short-term as long a final solution is not around?
from garden-runc-release.
williammartin
commented on July 21, 2024
@renelehmann I don't have a full understanding right now of the issue to be able to suggest anything, the only thing I can think of would involve forking the image. I'd be pretty interested to see what happened if we rebuilt the image without this line: https://github.com/grafana/grafana-docker/blob/master/Dockerfile#L16
I'll keep investigating though and I'm sure @julz will prioritise on Monday.
from garden-runc-release.
williammartin
commented on July 21, 2024
If I read this correctly, this code seems very wrong: https://github.com/cloudfoundry/grootfs/blob/master/store/image_cloner/image_cloner.go#L212-L222
It looks to me like this will use an empty folder as the source of a bind mount but when I read https://docs.docker.com/v17.09/engine/reference/builder/#volume, I would expect the source to be the container filesystem, making the files available on the host (e.g. for other containers to share).
Edit: ^^ No bind mounts necessary for the VOLUME directive, host filesystem directories should only be decided at run time by the creator.
from garden-runc-release.
williammartin
commented on July 21, 2024
Hey @renelehmann I did some more digging on how we treat Docker VOLUME directives. Bottom line is I don't understand why we are bind mounting here at all (will check in on Monday with the team) because it doesn't seem correct. In garden-shed all we did was create a directory in the container filesystem:
https://github.com/cloudfoundry/garden-shed/blob/master/rootfs_provider/layer_creator.go#L64-L68
https://github.com/cloudfoundry/garden-shed/blob/master/rootfs_provider/volume_creator.go
I think the TL;DR here is that:
The grafana.ini file is disappearing because there is a VOLUME directive in the grafana/grafana Dockerfile, in response to which, GrootFS is creating a folder on the host filesystem and using it as a source in a bind mount. This results in an empty directory being bind mounted over the directory containing the grafana.ini file.
I think it's likely we will fix this on Monday and ship asap but that's up to @julz
Sorry for the difficulties. I'll update this when I know more.
from garden-runc-release.
williammartin
commented on July 21, 2024
@renelehmann We have pulled a story into the backlog to fix this regression https://www.pivotaltracker.com/story/show/156569866 I'm starting to think these changes might have been a result of moving to support unprivileged (rootless) GrootFS.
The only workaround I have for you right now is to fork grafana/grafana and remove the VOLUME directive. That should give you the same behaviour as previously on CF.
I'll update this issue when we get to the story in the backlog.
from garden-runc-release.
renelehmann
commented on July 21, 2024
@williammartin Great effort and support. Thanks again. Meanwhile I already did what you suggested also and removed the volume directive. That works!
Regarding the grafana/grafana image they already have plans to remove the volume directive in a future release (5.1.x) to eliminate another impact:
grafana/grafana-docker#50
from garden-runc-release.
BooleanCat
commented on July 21, 2024
@renelehmann Thanks for reporting! garden-runc-release 1.13.1 contains the fix for this issue.
from garden-runc-release.
Related Issues (20)
- Use containerd-style stdin closer instead of exponential backoff stdin close HOT 2
- Get the protobuf duplicate fix registration warning/panic fixed in log-cache-release HOT 3
- GrootFS additional metrics HOT 7
- Support exporting garden-runc-release on windows HOT 4
- Uninitialized constant when rendering job template HOT 15
- Add support in CFAR for per-docker-app seccomp profiles HOT 6
- Upgrade busybox to 1.34.1 HOT 1
- Question: now the app container started by garden, the PID 1 process is app process? HOT 3
- gdn binary is gone in 1.20.9 release assets HOT 3
- Gdn failed to run on ubuntu bionic HOT 33
- Release gdn binary for ARM HOT 2
- release 1.22.9 doesn't include gdn binary HOT 2
- containerd and runc are included in two places that cause versions falling out of sync HOT 1
- Pinned dependecies should have a reason or unpinned
- Test issue. Please Ignore.
- Change default for garden spec to be containerd mode
- Stop Testing for and remove rootless mode
- Stop testing for containerd-for-processes in CI HOT 6
- Missing gdn binaries in release assets for 1.46 HOT 2
- Noisy neighbours causing CPU entitlement usage of consistent load apps to increase
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from garden-runc-release.