jvilk / bfsbench Goto Github PK

Write a wrapper program that captures interactions with Node's fs module, and logs them to a file.

When finished, we should be able to automagically log programs' file system behavior by replacing their require('fs') line with require('logfs').

Challenges:

• Programs that access many files will generate log files that are too large to be processed in JavaScript with JSON.parse (e.g. Doppio's javap benchmark suite generates 50+ MB of log file using a naive strategy).
  • Might need to 'chunk' log files, and/or use efficient logging strategies.
• Need to infer data dependencies. Which async calls can we execute in parallel?
  • Can determine when in the event stream callbacks are triggered.
  • Events triggered in the same JS 'turn' are guaranteed to be independent.
    • Node makes it easy to determine when turns end through process.nextTick.
• What about the actual data written/read?
  • Only need two things:
    • Initial set of files operated on.
    • Offset and size of reads/writes in the file.
• What about efficient Buffer-reuse strategies?
  • Could interpose on Buffer constructor, attach an id to each, and use that to identify buffers passed to fs.

One current limitation of BrowserFS: We do not support Node's writestreams.

Thankfully, all I need to do is emulate the createWriteStream method to construct a writeStream, the write method, the on method, and the end method.

Once I do this, I can use the event recorder in the browser with BFS.

Read-heavy

• Many small files. (Easy to generate using generate_random_files)
• Large files. (Easy to generate using generate_random_files)
• Bimodal mix (Mostly smaller files, a few large files)

Should figure out different access patterns, e.g.:

• Random
• Random groups (programs with 'phases')

Write-heavy

• Many small files. (Easy to generate using generate_random_files)
• Large files. (Easy to generate using generate_random_files)

Move-heavy

• Perform many move operations, and check results.

Real workloads

The most important part of all of this. Real programs, real workloads.

• Capture workloads from Doppio
• Capture workloads from Emscripten
• Capture workloads from Node applications