Problem with grootfs store for unprivileged causes garden failure about garden-runc-release HOT 4 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/156314215

The labels on this github issue will be updated when the story is started.

from garden-runc-release.

Thanks for opening the issue with us. The most suspicious output here is the I/O error and the very strange directory listing for the unprivileged directory.

~# ls -al /var/vcap/data/grootfs/store/
ls: cannot access /var/vcap/data/grootfs/store/unprivileged: Input/output error
total 113824
drwxr-xr-x 4 root root         4096 Mar 26 17:52 .
drwxr-xr-x 3 root root         4096 Mar 26 17:51 ..
drwx------ 9 root root          118 Mar 26 17:53 privileged
-rw------- 1 root root 118579802112 Mar 26 17:55 privileged.backing-store
d????????? ? ?    ?               ?            ? unprivileged
-rw------- 1 root root 118579802112 Mar 26 17:53 unprivileged.backing-store

Preliminary investigations would suggest a problem in your cluster with the backing data store:

https://unix.stackexchange.com/questions/39905/input-output-error-when-accessing-a-directory/39908
https://unix.stackexchange.com/questions/158443/files-in-ls-l-output

I can't really imagine anything that GrootFS could do to result in this, the kernel is seems unable to get the right information from disk.

As we discussed on Slack, let's see if deleting the cell in the IAAS and letting BOSH bring it back results in a healthy VM. It might be worth doing some disk diagnostics on your cluster regardless.

from garden-runc-release.

@sparameswaran Sounds like from further investigation, the IAAS delete resolved this. Do you think you need anything more in Garden or can we close this out?

from garden-runc-release.

Deleting the vm from iaas and redeploying appears to have fixed the problem. We can close this issue. Thanks.

from garden-runc-release.