Giter Club home page Giter Club logo

browserfs's Introduction

DEPRECATED

Active fork: ZenFS

22 March 2024

Hey there, this is the current maintainer of BrowserFS.

I've been working on BrowserFS for over a year now, and have made some big improvements.

A few months ago, I began seperating some backends from BrowserFS to be placed into different packages. Along with this, I created the browser-fs organization on Github and the browserfs organization on NPM. I made new organizations to keep the repositories and packages organized. Another reason I did so was because John Vilk, the original author of the project, no longer had the time to administer the project. This means at the time of writing this, I still do not have access to the NPM package.

I feel that now, the project has changed so significantly it is no longer the same BrowserFS. Even the name, BrowserFS, implies it is meant for browsers, which it has outgrown. For that reason, I think that the project should be something new, but still carry the legacy of BrowserFS. I've decided to call it ZenFS, since a core goal of mine is ease of use and peace of mind.

In a letter to Robert Hooke in 1675, Isaac Newton famously wrote "If I have seen further it is by standing on the shoulders of giants". This is most certainly true in the case of ZenFS. Without the creation of BrowserFS, I would not have found it and been amazed at a complete file system in Typescript/Javascript.

I would like to extend my deepest thanks to Dr. Emery Berger. Shortly after submitting my first pull request to BrowserFS, I reached out about becoming a maintainer of the project. Dr. Berger welcomed my maintainership of the project, and greatly helped in some other matters.

Thank you very much to the community for helping me with this project by submitting issues and pull requests.

The NPM organization browserfs has had all of its packages deprecated, with a message pointing to the new package as well as this notice. All of the versions published under @browserfs have also been published under @zenfs.

I hope that ZenFS can continue the legacy of BrowserFS, and can reach the same popularity and reliability.

Until next time,
James P.
a.k.a. Dr. Vortex
BrowserFS maintainer
ZenFS creator

BrowserFS

BrowserFS is an in-browser file system that emulates the Node JS file system API and supports storing and retrieving files from various backends. BrowserFS also integrates nicely into the Emscripten file system.

Backends

BrowserFS is highly extensible, and ships with many filesystem backends:

  • HTTPRequest: Downloads files on-demand from a webserver using fetch.
  • LocalStorage: Stores files in the browser's localStorage.
  • IndexedDB: Stores files into the browser's IndexedDB object database.
  • Dropbox: Stores files into the user's Dropbox account.
  • InMemory: Stores files in-memory. Thus, it is a temporary file store that clears when the user navigates away.
  • ZipFS: Read-only zip file-backed FS. Lazily decompresses files as you access them.
  • IsoFS: Mount an .iso file into the file system.
    • Supports Microsoft Joliet and Rock Ridge extensions to the ISO9660 standard.
  • WorkerFS: Lets you mount the BrowserFS file system configured in the main thread in a WebWorker, or the other way around!
  • MountableFileSystem: Lets you mount multiple file systems into a single directory hierarchy, as in -nix-based OSes.
  • OverlayFS: Mount a read-only file system as read-write by overlaying a writable file system on top of it. Like Docker's overlayfs, it will only write changed files to the writable file system.
  • AsyncMirror: Use an asynchronous backend synchronously. Invaluable for Emscripten; let your Emscripten applications write to larger file stores with no additional effort!
    • Note: Loads the entire contents of the file system into a synchronous backend during construction. Performs synchronous operations in-memory, and enqueues them to be mirrored onto the asynchronous backend.
  • FolderAdapter: Wraps a file system, and scopes all interactions to a subfolder of that file system.
  • Emscripten: Lets you mount Emscripten file systems inside BrowserFS.

More backends can be defined by separate libraries, so long as they extend they implement BrowserFS.FileSystem. Multiple backends can be active at once at different locations in the directory hierarchy.

For more information, see the API documentation for BrowserFS.

Building

Prerequisites:

  • Node and NPM
  • Run npm install (or the equivilent command using your package manager) to install local dependencies.

After running npm run build, you can find built versions in the dist directory.

Custom builds (not recommended):

If you want to build BrowserFS with a subset of the available backends, change src/core/backends.ts to include only the backends you require, and re-build.

Usage

๐Ÿ›ˆ The examples are written in ESM. If you aren't using ESM, you can add <script src="browserfs.min.js"></script> to your HTML and use BrowserFS via the global BrowserFS object.

BrowserFS provides a convient configure function which you can use to easily configure BrowserFS to use a variety of file system types.

Here's a simple usage example using the LocalStorage-backed file system:

import { configure, BFSRequire } from 'browserfs';

// you can also add a callback as the last parameter instead of using promises
await configure({ fs: 'LocalStorage' });

const fs = BFSRequire('fs');

// Now, you can write code like this:

fs.writeFile('/test.txt', 'Cool, I can do this in the browser!', function (err) {
	fs.readFile('/test.txt', function (err, contents) {
		console.log(contents.toString());
	});
});

The following code mounts a zip file to /zip, in-memory storage to /tmp, and IndexedDB browser-local storage to /home:

import { configure, BFSRequire } from 'browserfs';
import Buffer from 'buffer';

const zipData = await (await fetch('mydata.zip')).arrayBuffer();

await configure({
	fs: 'MountableFileSystem',
	options: {
		'/mnt/zip': {
			fs: 'ZipFS',
			options: {
				zipData: Buffer.from(zipData)
			}
		},
		'/tmp': { fs: 'InMemory' },
		'/home': { fs: 'IndexedDB' }
	}
};

Using with Browserify and Webpack

BrowserFS is published as a UMD module, so you can either include it on your webpage in a script tag or bundle it with your favorite JavaScript module bundler.

You can also use BrowserFS to supply your application with fs, path, and buffer modules, as well as the Buffer and process globals. BrowserFS contains shim modules for fs, buffer, path, and process that you can use with Webpack and Browserify.

Webpack:

module.exports = {
	resolve: {
		// Use our versions of Node modules.
		alias: {
			fs: 'browserfs/dist/shims/fs.js',
			buffer: 'browserfs/dist/shims/buffer.js',
			path: 'browserfs/dist/shims/path.js',
			processGlobal: 'browserfs/dist/shims/process.js',
			bufferGlobal: 'browserfs/dist/shims/bufferGlobal.js',
			bfsGlobal: require.resolve('browserfs'),
		},
	},
	// REQUIRED to avoid issue "Uncaught TypeError: BrowserFS.BFSRequire is not a function"
	// See: https://github.com/jvilk/BrowserFS/issues/201
	module: {
		noParse: /browserfs\.js/,
	},
	plugins: [
		// Expose BrowserFS, process, and Buffer globals.
		// NOTE: If you intend to use BrowserFS in a script tag, you do not need
		// to expose a BrowserFS global.
		new webpack.ProvidePlugin({ BrowserFS: 'bfsGlobal', process: 'processGlobal', Buffer: 'bufferGlobal' }),
	],
	// DISABLE Webpack's built-in process and Buffer polyfills!
	node: {
		process: false,
		Buffer: false,
	},
};

Browserify:

var browserfsPath = require.resolve('browserfs');
var browserifyConfig = {
	// Override Browserify's builtins for buffer/fs/path.
	builtins: Object.assign({}, require('browserify/lib/builtins'), {
		buffer: require.resolve('browserfs/dist/shims/buffer.js'),
		fs: require.resolve('browserfs/dist/shims/fs.js'),
		path: require.resolve('browserfs/dist/shims/path.js'),
	}),
	insertGlobalVars: {
		// process, Buffer, and BrowserFS globals.
		// BrowserFS global is not required if you include browserfs.js
		// in a script tag.
		process: function () {
			return "require('browserfs/dist/shims/process.js')";
		},
		Buffer: function () {
			return "require('buffer').Buffer";
		},
		BrowserFS: function () {
			return "require('" + browserfsPath + "')";
		},
	},
};

Using with Node

You can use BrowserFS with Node. Simply add browserfs as an NPM dependency, and require('browserfs'). The object returned from this action is the same BrowserFS global described above.

If you need BrowserFS to return Node Buffer objects (instead of objects that implement the same interface), simply require('browserfs/dist/node/index') instead.

Using with Emscripten

You can use any synchronous BrowserFS file systems with Emscripten! Persist particular folders in the Emscripten file system to localStorage, or enable Emscripten to synchronously download files from another folder as they are requested.

Include browserfs.min.js into the page, and configure BrowserFS prior to running your Emscripten code. Then, add code similar to the following to your Module's preRun array:

/**
 * Mounts a localStorage-backed file system into the /data folder of Emscripten's file system.
 */
function setupBFS() {
	// Grab the BrowserFS Emscripten FS plugin.
	var BFS = new BrowserFS.EmscriptenFS();
	// Create the folder that we'll turn into a mount point.
	FS.createFolder(FS.root, 'data', true, true);
	// Mount BFS's root folder into the '/data' folder.
	FS.mount(BFS, { root: '/' }, '/data');
}

Note: Do NOT use BrowserFS.install(window) on a page with an Emscripten application! Emscripten will be tricked into thinking that it is running in Node JS.

If you wish to use an asynchronous BrowserFS backend with Emscripten (e.g. Dropbox), you'll need to wrap it into an AsyncMirror file system first:

/**
 * Run this prior to starting your Emscripten module.
 * @param dropboxClient An authenticated DropboxJS client.
 */
function asyncSetup(dropboxClient, cb) {
	// This wraps Dropbox in the AsyncMirror file system.
	// BrowserFS will download all of Dropbox into an
	// InMemory file system, and mirror operations to
	// the two to keep them in sync.
	BrowserFS.configure(
		{
			fs: 'AsyncMirror',
			options: {
				sync: {
					fs: 'InMemory',
				},
				async: {
					fs: 'Dropbox',
					options: {
						client: dropboxClient,
					},
				},
			},
		},
		cb
	);
}
function setupBFS() {
	// Grab the BrowserFS Emscripten FS plugin.
	var BFS = new BrowserFS.EmscriptenFS();
	// Create the folder that we'll turn into a mount point.
	FS.createFolder(FS.root, 'data', true, true);
	// Mount BFS's root folder into the '/data' folder.
	FS.mount(BFS, { root: '/' }, '/data');
}

Testing

To run unit tests, simply run npm test.

Citing

BrowserFS is a component of the Doppio and Browsix research projects from the PLASMA lab at the University of Massachusetts Amherst. If you decide to use BrowserFS in a project that leads to a publication, please cite the academic papers on Doppio and Browsix:

John Vilk and Emery D. Berger. Doppio: Breaking the Browser Language Barrier. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (2014), pp. 508โ€“518.

@inproceedings{VilkDoppio,
	author		= {John Vilk and
							 Emery D. Berger},
	title		 = {{Doppio: Breaking the Browser Language Barrier}},
	booktitle = {Proceedings of the 35th {ACM} {SIGPLAN} Conference on Programming Language Design and Implementation},
	pages		 = {508--518},
	year			= {2014},
	url			 = {http://doi.acm.org/10.1145/2594291.2594293},
	doi			 = {10.1145/2594291.2594293}
}

Bobby Powers, John Vilk, and Emery D. Berger. Browsix: Bridging the Gap Between Unix and the Browser. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (2017), pp. 253โ€“266.

@inproceedings{PowersBrowsix,
	author		= {Bobby Powers and
							 John Vilk and
							 Emery D. Berger},
	title		 = {{Browsix: Bridging the Gap Between Unix and the Browser}},
	booktitle = {Proceedings of the Twenty-Second International Conference on Architectural
							 Support for Programming Languages and Operating Systems},
	pages		 = {253--266},
	year			= {2017},
	url			 = {http://doi.acm.org/10.1145/3037697.3037727},
	doi			 = {10.1145/3037697.3037727}
}

License

BrowserFS is licensed under the MIT License. See LICENSE for details.

browserfs's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

browserfs's Issues

Correct backend file writing implementation

Certain backends will generate parent directories that don't exist when a file is written.

They should be updated so that writes only succeed when the parent directory of the file in the given path already exists. This will probably be achieved by editing the sync method of their custom File subclass.

Backends currently known to have this issue:

  • Dropbox
  • WebStorage
  • In-memory

Serious XmlHttpRequest listings.json performance bug

You'll laugh, but due to a bug, this is how we currently download listings.json:

  • Download listings.json as text.
  • Put it into a buffer, which will convert the file into the binary representation of UTF8 character-by-character.
  • Run JSON.parse on the Buffer. This implicitly runs Buffer.toString, which converts the buffer contents from the binary representation of UTF8 strings back into a JavaScript string.

These conversions are all unnecessary.

And here I was, wondering why Doppio with BFS hangs IE8/IE9 on load! Doppio's listings file is something like 900KB.

Emulate `process.on('exit',...)` for unit tests

I'm not sure if this is possible, but it would make things nicer.

Currently, here's how our tests run:

  • Kick off test.
  • Wait until all fs callbacks are handled.
  • End test.

It would be nice to insert a second part of the unit test where the function registered with process.exit is run. I'll need to look at the Jasmine docs.

Build a browserfs.js standalone library

This is a requirement for the newly-minted doppio-demo repository.

I can see this happening in two ways:

  1. Select which FS types you want, then bundle them into a compressed JS file.
  2. Throw the whole kitchen sink of FS types in, and let the library user select the configuration at startup.

mkdir calls in the fixture loader are synchronous

Currently they are executed as

fs.mkdir("./test", mcb);
fs.mkdir("./test/fixtures", mcb);
fs.mkdir("test/fixtures/node", mcb);

fs.mkdir is an asynchronous method, meaning that, for example, /test/fixtures/ could be created before /test/, which would trigger an error since the parent of fixtures would not exist at that point. These calls should be made asynchronous by moving each one into the callback of its predecessor, to avoid these race conditions.

Take a modular approach to backends

I opted to make my Dropbox backend a separate repo, because it seems like it should be an optional plug-in rather than core functionality. The four existing filesystems are universal enough to be part of the core, but I propose that any future ones be modular plugins.

We could create a method for registering a new backend on the BrowserFS namespace, like _.mixin, so validation can be performed.

Others could then add their own backends without needing to send PRs to this repo.

We could have a wiki page with a list of all backends.

I can then add other cloud storage providers (Box, Google Drive, etc.) this way.

Backends can list this repo as a dependency in their bower.json.

This would also suggest separating the test suite into its own repo. I'm trying to hook the Dropbox backend up to the Karma test suite but because the tests are in this repo, I'm just doing a messy ../dropbox-fs/dropbox.js hack in karma.conf.js.

Since the external API is always the same, only the implementation differs between backends, the one test suite should be enough for all plugins (plugins can obviously have their own test suites as well for testing internal behaviour).

Specify and document ProxyFile API

How do ProxyFile stubs work when the filesystem uses no specialization?

How can they work with specialization?

Does the filesystem know ahead of time if the request is going to be proxied, or does it only support translating existing FDs into ProxyFiles (which I think is ideal)?

Maybe it would be good to have a FileSystem.GetProxyFd(fd) function. ๐Ÿ˜„

Compress localStorage file names

We currently store files in localStorage like so:

localStorage[/path/to/file] = binary string representation of data

We should be able to convert the path into a binary string:

localStorage[binary string representation of /path/to/file] = binary string representation of data

The keys are counted against the localStorage storage quota, so this could be a decent savings.

We'd still repeat parent directories, though, but it's better than what we currently do.

Write a filesystem that supports mounting

Write a simple in-memory filesystem that allows you to mount other filesystems at arbitrary locations.

This provides a simple mechanism for interacting with files across persistent storage mechanisms.

Make cloud storage accounts under an organisation name

Currently the Dropbox and Google Drive apps are registered under my accounts. We should create accounts with these services with some BrowserFS email address, so that we can all log in and access the app consoles, generate new API keys etc.

Any feedback users have would then go to the shared email account too, which is better than it going to my personal account.

Convert to TypeScript

After experimenting with TypeScript with Doppio, I have determined that it is very suitable for projects such as this one.

As a result, sometime later in the summer I will likely move to TypeScript. I do not anticipate that it will take longer than a week of work, and the payments for switching over will be worth the amount of churn required to perform the conversion.

List of benefits:

  • Static guarantees that FileSystem implementations are appropriately defined to the interface.
  • Easy support for customized library compilations that only contain the filesystems you will be using -- TypeScript knows your dependencies among files.
  • Refactoring becomes tractable once we start increasing the amount of FS implementations.
  • New contributors get compilation errors when they accidentally misuse an interface rather than silent failures.

Tool to load in test fixtures

I'm already working on writing this.

Add a tool that emits a JavaScript script that loads a directory of test files into the current file system.

This will allow us to use Node's fixtures.

Note that this will need to be synchronous for now due to how our tests are run, so I'll likely hack it in specifically for the localStorage test. Later on, if we can make the tests asynchronous, we can alter the tool to emit asynchronous loading code that triggers the test start when it completes.

HTML5 FileSystem API backend

Currently this is only supported in Chrome, but apparently it's coming in 'near future' versions of all other desktop browsers.

I found a library which wraps it in a UNIX-like API, which should make tying it to Node's own UNIX-like API super easy. Of all the non-cloud backends, this looks like the best, feature-wise, so I think we should plan for the future when this has better support.

Chrome is the most popular browser already, so IMO it would be worth starting on it now, rather than waiting for the other browsers.

Fix IE9

IE9 passes all of our unit tests, but fails to load classes properly in Doppio (magic number failure). It's also absurdly slow due to the TypedArray implementation; so slow, that I was unable to get IE9's debug mode past preloading.

Solution (for slowness):

We only need a polyfill for DataView, which is what Buffers use. It should be easy to modify the DataView polyfill to directly use methods to convert its internal array of bytes into other forms. Right now, it constructs a new TypedArray and ArrayBuffer for each time you call a get function -- just for the small segment of the array that you are processing.

They did this to reuse code, but it's very very very slow.

It would be manually intensive, but probably simple, to change the DataView polyfill to use the individual get and set functions that the typed arrays use directly to bypass this expensive object construction (which a profile reveals is the most expensive operation that's going on).

I am OK with absolutely ditching the TypedArray polyfills in favor of a shaved-down DataView-only polyfill. In fact, that's my favored solution right now.

This is high priority for the Doppio merger, but low priority for me for now. As far as I can tell, this is the only issue blocking the Doppio merger.

Note to @perimosocordiae: I would be OK with performing the merger as-is and leaving this bug for later fixing.

XmlHttpRequest Filesystem

Add an XmlHttpRequest filesystem that grabs files from a server. To start, it will use a JSON-based index file that contains any relevant file properties (mainly: file size, whether or not it's a directory, and potentially permissions information).

I'm working on this now, as it's required for Doppio integration.

Fix `binary` StringUtil behavior

It appears that using the ASCII string util for the binary string format is incorrect, and the Node test cases don't stress the issue.

I figured this out when I realized that Doppio used to use BINARY to get an array of numbers that represents the file. Our ASCII behavior actually removes the highest bit from the data like Node does, as Node defines ASCII as 7-bit ASCII.

I think this is a matter of copying+pasting the ASCII behavior without the truncation, but I'm not completely certain yet. This is low priority for now.

Add optional synchronous API support

We should make it possible for file systems to implement a synchronous API.

Like system properties, this will be optional.

The default FileSystem implementation can be augmented to automatically provide the asynchronous API if the file system implements the synchronous API -- it'll just call the synchronous equivalent and then run the callback.

This should happen after asynchronous support is solid.

Stat object

We need a stat object to mirror Node's stat.

I need to investigate if this is mutable in node. We can do clever things with getters/setters if needed.

Calling stat with an empty string as the path doesn't throw an error

The BrowserFS core is meant to validate and normalise paths, so that backends can depend on the paths they receive being valid.

The empty string is not a valid path, however when the stat method is called with this as the path, the string reaches the backend's method, and that method must perform a check for this and throw an error, as I have done in the Dropbox and HTML5 FS backends.

This should be fixed in the core, and the workarounds removed from these two backends.

Here's the relevant test case that exposes this bug.

Performance tests

I think it would be good to get some performance tests added to the test suite before spending any time implementing new methods to improve performance.

This would help us prevent regressions, and even in cases like lavelle/BrowserFS#11 where it's fairly obvious that it will improve performance, it's nice to be able to quantify and measure the improvement.

This would also allow us to make comparisons of the relative speeds of the different backends, to have some quantitative evidence for users when they're weighing up the advantages and drawbacks of each.

I couldn't find any performance tests in the Node repo, but it wouldn't be too difficult to write our own.

Handle write/read streams

The Node API encourages using these when queueing up writes. Should be simple to implement generically as a queue of requests, right?

Make test launching asynchronous

Right now, Karma loads each script into the browser, and they execute immediately.

It would be better if we can dynamically trigger test start so we can better use Jasmine to kick-off tests (and maybe to emulate process.exit when all callbacks finish...). This would also make it simple to load in test files asynchronously by dogfooding our asynchronous API.

There are two ways to do this -- the good way, and the easy-but-bad way:

  • Good: Wrap each test in a closure, and deposit its runner in a known object (e.g. window.bfs_tests['test_name'] = function() {...)
  • Bad: Make two files that are concatenated directly before and directly after all of the test files that wraps everything in a single closure.

The good way would allow us to set up individual Jasmine tests for each Node test, which would be awesome!

Build system

I need a simple build system to compile and compress BrowserFS into a single module.

Make test registration asynchronous

OK, tests are launched properly now, but they are registered on page load.

If we can figure out how to register tests asynchronously, then we can trigger test registration after loading in any test files for #19.

Right now, the issue is that Karma's Jasmine adapter appears to trigger Jasmine tests once page load finishes. I wonder if there is a way to trigger it manually?

Async solution

Many methods in this library, and in APIs upon which it depends, are asynchronous, which could lead to some messy callback situations down the line.

I know the plan is to move to TypeScript some time. Async support in TypeScript is a while off (the 1.x series at least). Is there any way we can use some nice await/defer constructs in the meantime?

ES6's native async support is obviously ages away, but moving to IcedCoffeeScript would be pretty easy (it's a superset).

IndexedDB Filesystem

Add an IndexedDB file system that is the following:

  • Asynchronous-only (the synchronous API isn't widely supported, and will only be usable from webworkers -- an uncommon use case right now).
  • Supports permissions.

I may eventually add symlink support (just to have a symlink-capable file system), but it's much less important right now than the above.

I imagine this will be the most useful browser-local backend for BrowserFS, as IndexedDB is decently supported across browsers and can store large files.

Unit tests

I need to use a unit testing framework that I can run in the browser, and potentially outside of the browser.

When designing this, maybe I should specify how to write an encrypted file with authentication information / test configuration for various filesystems. It would be password protected, so a password would be required to start the test.

Alternatively: Prompt for username/password before test begins, then maybe a browser-side password saver could handle saving things for convenience.

Track 'dirty' content in PreloadFile

Currently, PreloadFile stores no metadata relating to what has changed.

At the least, PreloadFile should track:

  • If any new data has been written.
  • If any properties have been altered.

Fix file paths in mountable file system error messages

Right now, error messages thrown by the MFS are created by the mounted file systems themselves, which have no idea where they are mounted.

If we standardize our error messages, then the MFS can interpose on error messages to append the mount point to them.

This is low priority for now.

Fix Arbitrary wildcard makefile build order (causes dev doppio builds to fail)

Hi,
I'm ran into some build issues with the latest version of doppio and browserfs.

Everything is fine if I used the pre-built git versions of vendor/browserfs/dist

browserfs also passes its tests.

However when I build my own browserfs and copy the new versions from browser/lib into browserfs/dist,

then I can no longer run doppio debug in the browser.
but make doppio test passes! Grrrr. Here's the original symptom.

Line 699
BrowserFS.node.fs.Stats = function() {
Uncaught TypeError: Cannot read property 'node' of undefined
Stats.FILE = 1;
Stats.DIRECTORY = 2;
Stats.SYMLINK = 3;
Stats.SOCKET = 4;

The underlying cause: a build order / dependency issue because Make 'wildcard' does not gauarnantee a build order.
Using Make's sort function is a reaonabale workaround, if you're happy requiring a modern version of Make .

FYI steps to finding the problem.

In the official 'dist' builds Browser.node is defined early ...

grep 'BrowserFS.node.*=' dist-orig/browserfs.js

        BrowserFS.node = {};
    BrowserFS.node.Buffer = function() {
    BrowserFS.node.fs = function() {
    BrowserFS.node.fs.Stats = function() {
    BrowserFS.node.path = function() {
                    resolved = BrowserFS.node.path.normalize(cwd + (cwd !== "/" ? path.sep : "") + resolved);
    BrowserFS.node.process = function() {
                data = BrowserFS.node.Buffer((_ref2 = req.response) != null ? _ref2 : 0);

In my locally built version, BrowserFS.node = {} appears too late...

grep 'BrowserFS.node.*=' dist/browserfs.js

    BrowserFS.node.fs.Stats = function() { <-- fs_stats TOO EARLY!
    BrowserFS.node.process = function() {
    BrowserFS.node.path = function() {
                    resolved = BrowserFS.node.path.normalize(cwd + (cwd !== "/" ? path.sep : "") + resolved);
    BrowserFS.node.Buffer = function() {
    BrowserFS.node.fs = function() { <-- SHOULD BE BEFORE fs.Stats above
        BrowserFS.node = {};  <-- 000-browserfs.js SHOULD BE FIRST
                data = BrowserFS.node.Buffer((_ref2 = req.response) != null ? _ref2 : 0);

My suspicion fell on the following ...
SRCS_CORE := $(wildcard src/core/*.coffee)

FYI On my OSX machine with Make v3.82
SRCS_CORE := $(wildcard src/core/*.coffee)
is not sorted
In fact, a google search shows this is true for 3-.82 onwards, despite what the docs say -

https://bugzilla.redhat.com/show_bug.cgi?id=635607
"The NEWS file for make-3.82 says:

  • WARNING: Future backward-incompatibility!
    Wildcards are not documented as returning sorted values, but up to and
    including this release the results have been sorted and some makefiles are
    apparently depending on that. In the next release of GNU make, for
    performance reasons, we may remove that sorting. If your makefiles
    require sorted results from wildcard expansions, use the $(sort ...)
    function to request it explicitly.

If I'm reading "up to and including this release" right, the order was NOT supposed to change in this version. But it did.

"

Fortunately, Make supports a sort command, so I'll send a pull-request with the following -
#From Make 3.82 onwards, wildcard returns filenames in arbitrary order.
#Alphabetically sorting the values is sufficient for browserfs build dependencies.
SRCS_CORE_UNSORTED := $(wildcard src/core/*.coffee)
SRCS_CORE := $(sort $(SRCS_CORE_UNSORTED))

Alternatively one could split the core, or just explicitly name the files in the Makefile.
Best,
Lawrence.

FileBuffer API

Implement a buffer, which is how node users specify what to write to a file.

Define how to instantiate the API

You should be able to instantiate the API in some way.

For example, for an easy one-filesystem case:
fs.instantiate(BrowserFS.BrowserStorage())

For multiple in a hierarchy, you'd have to use something more complex.

Make a BrowserFS GitHub organisation and migrate the repos to it

I think we've agreed on keeping things modular and having separate repos for each backend (or each one not currently this repo, at least). To make it easier for users to find all the backends, it would be good to put them all under one account.

This account would have:

  • This core repo
  • The cloud backends
    • Dropbox
    • Google Drive
  • The local backends, if we extract them from this repo
  • The test suite, if we extract it from this repo (IMO a good idea, because otherwise the only way to test is to import the backends into the core repo, which is the wrong way round)
  • The website/homepage if we make one

Currently

[Core + Test suite] <- [Backends]

Proposed

[Test suite] <- [Backends] <- [Core]

Support for backend-specific unit tests

We need a mechanism to say 'run these tests for this particular backend'.

Some example tests:

  • localStorage: Fill up localStorage and ensure an appropriate error is thrown. I've heard some versions of IE silently fail; this would be a useful test.
  • FileSystem: Try to use a FS that isn't appropriately provisioned.
  • Dropbox: Try to use Dropbox when the user hasn't authenticated.

The easiest way to do this: Define a generateUnitTest function on file systems with specific tests. We include this as part of the testing code; it's not distributed with BFS at all. We pass it the structures it needs to generate its specific tests.

We could even potentially use this as a factory method for constructing backends for testing. Meaning, it's responsible for constructing the backend, generating generic unit tests, generating specific unit tests, etc. We can define generateUnitTest on FileSystem which handles generating generic unit tests.

Synchronous API support for LocalStorage FS

LocalStorage's API is synchronous, so I should add synchronous support to its file system.

Doing this will require adding synchronous API stubs in node_fs.coffee, filesystem.coffee, and file.coffee, which means this is blocked by #23.

Add a directory of filesystem templates

This will make it even easier to make a custom BrowserFS file system (and I'd use them, too).

I should start by making templates for the following cases:

  • Read-only filesystems
  • Read/write filesystems

Define how to set up BrowserFS for webworkers

Both sides need to set up the message passing interface correctly to handle browser FS requests without disturbing any existing messages.

Ideal solution: User can encapsulate the default supplied BrowserFS send/receive functions how they want, or use them directly. Allows users to use BrowserFS easily, or in place of webworker messages.

Store directories in localStorage

We're already storing a header of stat information for files, so it's natural to store a few bytes for directories as well. This allows us to persist empty directories, and it also paves the way for eventual symlink support.

Add generic properties support to `PreloadFile`

PreloadFile should have a generic implementation for property-related functions, like chmod.

Since PreloadFile has a reference to the file system that created it, these implementations should be guarded with:

unless @_fs.supportsProps()
  cb new BrowserFS.ApiError BrowserFS.ApiError.NOT_SUPPORTED

Tightly bind DataView polyfill and Buffer

Doppio is unusably slow in IE9 due to all of the copying we are doing. To download a binary file using XmlHttpRequest, we copy the entire file eagerly this many times:

  • Download file
  • Convert file from VB safe array into JavaScript array
  • Convert JavaScript array into our Node Buffer polyfill, which:
    • Instantiates a DataView, which writes an array the size of the file and instantiates each individual byte to 0.
    • Copies JavaScript array into the DataView byte-by-byte.

We can skip the third and fourth copies if we instantiate our Buffer directly with the JavaScript array, which is in the same format that our polyfill internally stores the data as. This requires the Buffer implementation to be aware of the polyfill.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.