Potential crash consistency bug - fsync fails to persist directory entry about fscq HOT 9 CLOSED

First, this shouldn't happen; the final fsync should persist at least all the directory entries (the pwrite might not make it to disk, but it's all zeros so that wouldn't have an observable effect).

I couldn't reproduce this; I get both A/ and foo, which has size 5.5K as expected.

Are you using the master branch of FSCQ (c4930e0 is fine; I just pushed a commit that mistakenly wasn't on GitHub, but it's a Makefile-only change)?

Can you reproduce this manually rather than from automated tests?

What operating system are you running? I'm testing on Arch Linux, kernel 4.20.13.

from fscq.

Yes, this shouldn't happen, and that's why I thought it could be a potential consistency bug.

I manually reproduced it on Ubuntu 16.04, kernel 4.15.0. However, I am on the SOSP17 branch, because I could not get the master branch compiled on any of the machines I have...

Using coqc 8.8.1 and ghc 8.0.1, I get the following error during compilation of the master branch.

COQC -quick Word.v
File "./Word.v", line 1139, characters 8-13:
Error:
Syntax error: 'Equivalent' 'Keys' expected after 'Declare' (in [vernac:command]).

Makefile.coq:664: recipe for target 'Word.vio' failed
make[1]: *** [Word.vio] Error 1

Do you have a workaround for this issue?
And, could you also try reproducing it on the SOSP17 branch? I just want to know whether this is an once-existed bug that is patched, or merely a false positive.

from fscq.

Can you try again with master? I pushed a fix (there was some code for Coq master which is incompatible with Coq v8.8 and v8.9).

from fscq.

I reproduced the bug on the sosp17 branch, though I haven't debugged why it's happening.

from fscq.

Got it. Confirmed that the master branch doesn't have this bug.

Have you had a chance to figure what was the issue on the sosp17 branch?
I am really interested in the details behind how a verified file system can have such bug.

from fscq.

On the master branch, I was able to reproduce the bug by checking out commit f0a0122, which means the commits 145cd1c, 31cf5f7, 97b50ec, and c4930e0 have something to do with crash consistency guarantees. Could you shed some light on what these patches do?

from fscq.

Closing since I can no longer reproduce this bug on Coq master or Coq 8.9.1, on either FSCQ sosp17 or master. Please re-open if you can reproduce, particularly on master.

As far as a verified file system having such a bug - this should not be surprising. FSCQ has an unverified component connecting the verified parts to FUSE, and FUSE itself is unverified; any part of this TCB could be buggy, like any other software.

Furthermore, many odd behaviors in FSCQ are not bugs according to our specification. Other filesystems have only an informal notion of correctness - FSCQ has a documented specification for its core functionality (the parts implemented in Coq), and this behavior does not always strive to match either POSIX or what other filesystems provide. This behavior in this example is indeed surprising and might indicate a bug in one of the unverified components (or something wrong with the test setup), but for example #14 is clearly allowed behavior according to our specification.

from fscq.

@squizz617 can you still trigger this bug on your machine? I still don't understand how this behavior is possible or why changing how direct writes work would affect persistence of the directory.

from fscq.

We finally figured out what was wrong. The directory creation, file write, and crash safety aspects of this test case are all red herrings; it turns out that fruncate was broken and used an unverified helper function that wasn't the right thing; see #15 for a much simpler way to produce a bug. Switching to direct writes instead of log writes does happen to fix this test case, but it merely masks the ftruncate bug.

Thanks for the report! In the end it did help us find and this bug in the Haskell code.

from fscq.