fs.rename would be great and I imagine it might not be super trick to implement.

Is a PR welcome? I understand you would like to keep the API slim.

Does this project support OPFS or does it need to be added?

blocking a fellow who wants to use isogit to download https://github.com/mooculus/calculus

need to implement:

• fs.symlink(target, path[, type], callback)
• fs.readlink(path[, options], callback)

Hi - thanks for the great projects.

I'm having trouble with readFile. I'm getting the following error:

Error: ENOENT: /TypescriptGeneratorTest
    at CacheFS._lookup (CacheFS.js:105)
    at CacheFS.writeFile (CacheFS.js:154)
    at superblockPromise.then (index.js:146)

with the code posted below. Note: I tried the utf8 option directly from a string and, per the code below, I'm now encoding the data to Uint8Array. The path fed to the writeFile is actually /TypescriptGeneratorTest/index.js thought the error only shows the directory - which definitely exists. What am I missing?

export function writeFileToFS (contentMetaData, content) {
    var filePath = contentMetaData.filePath
    if (!("TextEncoder" in window)) {
        alert("Sorry, this browser does not support TextEncoder...")
    } else {
        var enc = new TextEncoder(); // always utf-8
        var encodedContent = enc.encode(content)
        console.log('encoded content: ')
        console.log(encodedContent)
        window.pfs.writeFile(filePath, encodedContent, ((err) => {
            if(err) {
                console.error('could not write file to file system due to the following error: ')
                console.log(err)
            } else { // if successful, replace the rawContentItem in the store with the new one from the filesystem
                console.log('the following file was written succesfully to the file system: ' + filePath)
                readFileToStore(filePath)
            }
        }))
    }
}

writeFile will happily write to a path that is currently a directory. The children of the directory will be kept, but its type will become 'file' and thus any subsequent operation will throw ENOTDIR.

Here is a minimal example.

import FS from "@isomorphic-git/lightning-fs";

const fs = new FS("demo");
const pfs = fs.promises;

const main = async () => {
  await pfs.mkdir("/foo");
  await pfs.writeFile("/foo", "bar");
  await pfs.readdir("/foo"); // throws ENOTDIR
};

main();

In Node, the following error is thrown by writeFile: Uncaught [Error: EISDIR: illegal operation on a directory, open 'test']

This seems related to #23. If oldFilepath and newFilepath are equal, then the file is deleted; I'm not sure if this is the default node fs behavior, but we can see this is because the old file is deleted after the new file is inserted here:

  rename(oldFilepath, newFilepath) {
    let basename = path.basename(newFilepath);
    // Note: do both lookups before making any changes
    // so if lookup throws, we don't lose data (issue #23)
    // grab references
    let entry = this._lookup(oldFilepath);
    let destDir = this._lookup(path.dirname(newFilepath));
    // insert into new parent directory
    destDir.set(basename, entry);
    // remove from old parent directory
    this.unlink(oldFilepath)
  }

A simple fix would be to check for that condition, and return without doing anything:

  rename(oldFilepath, newFilepath) {
    if (oldFilepath == newFilepath)
      return;
    ...
  }

I'm trying to use my old service worker for server like file access but it don't work with lightingFS because:

fs.stat('/foo/', (e) => { console.log(e)});

show error:

ENOENT: /foo/

the directory exists because fs.stat('/foo', ...) is working fine.

I need this to be working because I'm distinguishing directories from files using slash at the end of the directory. Probably my worker can be refactored but it would be nice if the library work the same as Node fs.

In theory, a native implementation of exists that doesn't throw an Error and immediately catch it might be ever-so-slightly faster. Not sure what the overhead of creating an Error object is... it can be "expensive" because it creates a stack trace - although I think most browsers compute the stack trace only if needed nowadays. Hmm.

Originally posted by @wmhilton in #21 (comment)

Hello @wmhilton and team 😉

One liner

I think the docs should make it very important that your app needs to have one and only one instance of new FS('fs').

Details

We only tested on Firefox, but in one of our codebase we had several instances of fs (in different modules) with const fs = new FS('fs'). We hoped they would be in sync since it's the same IndexedDB behind. It seems like there is a cache layer or something with the transactions. Maybe it comes from how idb-keyval works.

Anyway, if you don't use the same fs instance when you try to readdir or any other fs operations than the instance you used with git.plugins.set('fs', fs); you will have sync problems.

I know the getting started for browser mentions a global var but it does not say much more.

Questions

• Is my diagnose wrong?
• What do you think about this?
• Should we improve the docs?
• Should we find a way for several new FS('fs') to reuse the same wrapper behind the scenes?

Thanks 😄

Can you please let me about using this in TypeScript. currently there are no typings available.

Error:

  Try `npm install @types/isomorphic-git__lightning-fs` if it exists or add a new declaration (.d.ts) file containing `declare module '@isomorphic-git/lightning-fs';`ts(7016)

Hi.

There are mkdir, readdir, but the docs don't mention any way to check if directory exists. Is there such a function?

Thanks.

Hi,
A recently installed isomorphic-git into my project via "npm install --save isomorphic-git". But after installing there is no directory "lightning-fs" and so i cant access lightning-fs.
Is there ani solution for this ?
Thanks in advance for your answer.

Hello, first of all thank you for the great libraries lightning-fs and isomorphic-git!

I am interested in implementing something like Git's push --force-with-lease which allows you to require that a ref has a particular hash before changing it (to prevent multi-tab conflicts), similar to how a compare-and-swap works.

How tricky do you think would it be to do something like this, due to the caching Lightning-FS does? Would an approach of grabbing a Web Lock, syncing the FS, and doing a read and write (while holding the lock) work?

The FS should be able to tell what is the current name with which it was initialized the last time (And maybe the options). Right now the only way to know the name is by querying fsInstance.promises._backend._store and check if it is undefined or if it has a _storeName this would be useful to not initialize the same database twice when you have multiple databases in the same thread that depend on user options

I've been using isomorphic-git with lightning-fs and it's working great so far, but the next stage of my project is calling an emscripten-built assembler. Unfortunately, lightning-fs doesn't seem to implement the mount functionality emscripten expects for integration like BrowserFS.

Is it already possible to utilize lightning-fs within emscripten? If not, I think it would be really useful for lightning-fs to work with emscripten out of the box similar to BrowserFS. The full integration from Git to executing build tools all within the same file system is really appealing.

I ran into unexpected behavior in an app because LightningFS' readdir can return non-sorted strings. The following example shows the issue in Node.js (make sure you npm install --save-dev fake-indexeddb first), but the issue was first identified in the browser (Firefox and Safari):

require("fake-indexeddb/auto"); // HAS to come first for Node.js to have indexedDB
const FS = require('@isomorphic-git/lightning-fs');

const fs = new FS("testfs", {wipe: true});
const pfs = fs.promises;

(async function() {
  await pfs.mkdir('/hi');
  const max = 15;
  const randN = Array.from(Array(50), _ => Math.floor(Math.random() * max));
  // write & append fifteen files in random order with random data
  for (const r of randN) {
    const fname = `/hi/${r}`;
    pfs.writeFile(fname, Math.random());
  }
  const contents = await pfs.readdir('/hi');
  console.log(contents);
})();

The script above writes to a set of fifteen files (whose names are between '0' and '14') in random order, and then prints the output of readdir. Running this script several times shows the issue: an unsorted list of files:

$ node demo.js
[
  '13', '2', '14', '9',
  '4',  '3', '8',  '11',
  '0',  '1', '7',  '12',
  '10', '6', '5'
]
$ node demo.js
[
  '7',  '14', '4',  '1',
  '2',  '8',  '10', '3',
  '9',  '13', '11', '5',
  '12', '6',  '0'
]
$ node demo.js
[
  '8',  '11', '14', '0',
  '13', '3',  '4',  '5',
  '6',  '9',  '7',  '12',
  '10', '2'
]

This behavior persists whether I use numeric or alphanumeric filenames.

In contrast, if I use Node's fs module, readdir returns sorted strings.

Is this an IndexedDB limitation? If it is, a sort on each readdir may be too expensive. But if not, maybe there's a workaround in the library itself?

Loving the LightningFS + isomorphic-git lifestyle, thank you for your continued hard work on these projects!

target: browser as defined in https://isomorphic-git.org/docs/en/browser

we are getting the following on using isomorphic-git and lightning fs in our app - is this an expected condition?

If you think this is possible, I'll be happy to work on this. or help you work on this.

https://developers.cloudflare.com/workers/reference/storage/overview/

It seems when a url option is passed, a .superblock.txt file must be provided at the url, listing the available files. Creating this file is not always possible, and when it is missing, the _cache.stat() check in readFile at https://github.com/isomorphic-git/lightning-fs/blob/master/src/PromisifiedFS.js#L150 throws an error, preventing the reading of any remote files.

It might be nice if there were a way to specify that files are present when the superblock file is not, or when it is missing content. A method like fs.backFile() maybe that would add an entry to the cache.

Orthogonally, that readFile line could be skipped when there is no superblock file, maybe a HEAD request performed instead.

An internal error was encountered in the Indexed Database server

Screenshot:

To reproduce, run the previous step once and this step twice from the quickstart page. The first time works, but the second time fails, and afterwards all other fs operations throw the same error.

You can also reproduce from scratch by starting with a fresh incognito window.

I found this bug in a search, but that seems to be iOS Safari. Anyone know of a workaround for this?

https://github.com/isomorphic-git/lightning-fs/blob/master/src/CacheFS.js#L202
The oldFile is unlinked before the newFile is set correctly, when lookup fail, e.g. the newFilepath doesn't exist. the old file will be lost.

This is a minor thing, but since I just spent an embarrassingly long amount of time puzzling over it, I figured I’d mention it.

The documentation for fs.writeFile in the README mentions that "data should be a string of a Uint8Array."—which I interpreted as any data has to be converted into a Uint8Array and then encoded into a string. Looking through the source I noticed, however, that the function actually does the encoding for us if a string is passed.

Maybe this could be clarified? It might be just me, but the wording along with the amounts of time I have seen "or" misspelled as "of" really confused me here. 😅

Either way, thank you so much for this library (and the rest of Isomorphic Git!).

Happy holidays, if you celebrate.

First, thanks for your work on this project 🙂

The current implementation is fairly slow with large repos, for instance vscode, which has around 5000 files or typescript, which has around 50k. It takes about a minute to clone vscode with --singleBranch and --depth 1, and doesn't manage to clone typescript in the ~15 minutes I waited.

By adding batching to the indexdb writes (Put all writes into a single transaction rather than one transaction per file) and changing the autoinc in the cachefs to increment a counter rather than search for the highest inode (the search means writing N files is N^2 time), I am able to see vscode clone in ~20 seconds and typescript clone in about 2 minutes. this is approx 3x slower than native for vscode and 6x slower than native for typescript.

Batching:

diff --git a/idb-keyval.ts b/idb-keyval.ts
index 45a0d97..94920ef 100644
--- a/idb-keyval.ts
+++ b/idb-keyval.ts
@@ -2,10 +2,12 @@ export class Store {
   private _dbp: Promise<IDBDatabase> | undefined;
   readonly _dbName: string;
   readonly _storeName: string;
+  readonly id: string
 
   constructor(dbName = 'keyval-store', readonly storeName = 'keyval') {
     this._dbName = dbName;
     this._storeName = storeName;
+    this.id = `dbName:${dbName};;storeName:${storeName}`
     this._init();
   }
 
@@ -44,6 +46,31 @@ export class Store {
   }
 }
 
+class Batcher<T> {
+  private ongoing: Promise<void> | undefined
+  private items: { item: T, onProcessed: () => void }[] = []
+
+  constructor(private executor: (items: T[]) => Promise<void>) { }
+
+  private async process() {
+    const toProcess = this.items;
+    this.items = [];
+    await this.executor(toProcess.map(({ item }) => item))
+    toProcess.map(({ onProcessed }) => onProcessed())
+    if (this.items.length) {
+      this.ongoing = this.process()
+    } else {
+      this.ongoing = undefined
+    }
+  }
+
+  async queue(item: T): Promise<void> {
+    const result = new Promise<void>((resolve) => this.items.push({ item, onProcessed: resolve }))
+    if (!this.ongoing) this.ongoing = this.process()
+    return result
+  }
+}
+
 let store: Store;
 
 function getDefaultStore() {
@@ -58,10 +85,17 @@ export function get<Type>(key: IDBValidKey, store = getDefaultStore()): Promise<
   }).then(() => req.result);
 }
 
+const setBatchers: Record<string, Batcher<{ key: IDBValidKey, value: any }>> = {}
 export function set(key: IDBValidKey, value: any, store = getDefaultStore()): Promise<void> {
-  return store._withIDBStore('readwrite', store => {
-    store.put(value, key);
-  });
+  if (!setBatchers[store.id]) {
+    setBatchers[store.id] = new Batcher((items) =>
+      store._withIDBStore('readwrite', store => {
+        for (const item of items) {
+          store.put(item.value, item.key)
+        }
+      }))
+  }
+  return setBatchers[store.id].queue({ key, value })
 }
 
 export function update(key: IDBValidKey, updater: (val: any) => any, store = getDefaultStore()): Promise<void> {

Counter:

diff --git a/src/CacheFS.js b/src/CacheFS.js
index ed26c57..0dc6950 100755
--- a/src/CacheFS.js
+++ b/src/CacheFS.js
@@ -5,6 +5,7 @@ const STAT = 0;
 
 module.exports = class CacheFS {
   constructor() {
+    this._maxInode = 0
   }
   _makeRoot(root = new Map()) {
     root.set(STAT, { mode: 0o777, type: "dir", size: 0, ino: 0, mtimeMs: Date.now() });
@@ -38,16 +39,7 @@ module.exports = class CacheFS {
     return count;
   }
   autoinc () {
-    let val = this._maxInode(this._root.get("/")) + 1;
-    return val;
-  }
-  _maxInode(map) {
-    let max = map.get(STAT).ino;
-    for (let [key, val] of map) {
-      if (key === STAT) continue;
-      max = Math.max(max, this._maxInode(val));
-    }
-    return max;
+    return ++this._maxInode;
   }
   print(root = this._root.get("/")) {
     let str = "";

Please let me know if you'd consider incorporating these changes... the batching should be safe, I'm not super sure about the autoinc, but I don't see a reason why it would cause issues (the main difference is deleting a file would free up its inode value in the original implementation but doesn't here, but that shouldn't be a problem AFAIK)

I'm using dist/lightning-fs.min.js from https://cdn.jsdelivr.net/ and it seems that path module is not exposed. It would be nice if you don't need to use browser-fs just to have path.join and path.split.

Or is there a way to access it?

I'm sorry, this library seems great but not having any experience with browser storage I'm finding the docs very counter-intuitive.
I managed to make it do most of what I need, but as the title suggests I'm trying to understand if there's a way to access a folders & its subfolders as a single file (to, for example, download it, or send it via webRTC, or anything else)

Please convert the codebase to TypeScript, as it may help contributors make changes, and it will help with type-checking and other errors. I am working on a PR for this, and it should be here very soon.

Hey. Do you have any experience using compressed data (like zip) with lightning-fs? My use case involves sending quite large file tree from the server and I want it to be synchronously available on the client. BrowserFS has ZipFS.

Are there any plans to support events? More specifically, has there been any interest in implementing the FSWatcher class from the fs API?

In Firefox, see https://observablehq.com/@a10k/hello-lightning-fs-isomorphic-git
Works perfectly in chrome but when creating the new filesystem, DOM refuses.

This is to make it easier to control the lifecycle of the filesystem

With browserFS you can call toString and get the value as string. With Lighting-FS it don't work like this. In node you can also use toString not sure what object readFile give in node.

check https://stackoverflow.com/questions/8936984/uint8array-to-string-in-javascript

The rmdir function in newer versions of Node allows for a recursive option that when set to true makes the command behave more like a rm -rf, which I think can be very useful in many situations (such as deleting a cloned but no longer needed git repo for example).

The documentation of lightning-fs suggests that its rmdir also takes an options object, but I browsing through the source I couldn’t find anything that seemed to be using that object (I might have missed something though 😅). I feel like adding a native recursive option could really benefit the library and perhaps would even allow for a more efficient way than a recursive "rimraf" implemented on top of the library, since if I understood the structure of lightning-fs correctly it should be possible to just gather a list of all inodes contained in the directory, delete them and then delete the directory itself in a single transaction, right?

Sorry if that suggested solution is useless, but I hope the rest of my point still stands.

Keep up the great work! 🎉

I am creating a wrapper of my fs using lighting fs backend option but I am not able to perform these stuff:

• create file
• write file
• mkdir etc

And all other operation work well like read, stat, etc...

What should I do to perform these create, write, etc stuff?

I have a code to read a real folder from File Access API into lightningFS.

The repo is about 800kb, has 300 files.

The code works very slowly, because of writeFile (~1.3 sec).

I've read that indexedDB throttles writeFile, is that so?
How to speedup?

Here's the code, it recursively reads all dirs/files and uses fs.promises.writeFile to write them to lightningFS.

This writeFile call is the main reason for the delay, even though the data is very small.

  let fs = new LightningFS('fs', {wipe: true});

  let relPath = [''];

  async function handle(dirHandle) {
    console.time(dirHandle.name);

    for await (const entry of dirHandle.values()) {
      if (entry.kind === "file") {
        const file = await entry.getFile();
        let data = new Uint8Array(await file.arrayBuffer());
        let filePath = [...relPath, file.name].join('/');
        await fs.promises.writeFile(filePath, data);
      }

      if (entry.kind === "directory") {
        const newHandle = await dirHandle.getDirectoryHandle( entry.name, { create: false } );
        relPath.push(entry.name);
        let dirPath = relPath.join('/');
        await fs.promises.mkdir(dirPath);
        await handle(newHandle);
        relPath.pop();
      }
    }
  }

P.S. Is there any other backend for lightningFS? I need a simple in-memory strorage.
It's quite ironic that lightningFS is so sluggish for a tiny test repo.
Maybe I'm doing something wrong?

The .prettierrc file specifies semi: true which means that a semi colon should finish each line, yet many files (for example HttpBackend.js) do not have trailing semicolons, or only have one or two.

Seems like prettier is not actually being used. Would you like a PR to fix this across all the files?

I noticed that current IDB interface had two errors:

1. The constructor type was typed as a regular function
2. The readFile was mis-typed as loadFile

/lightning-fs/index.d.ts

export interface IDB {
- constructor(dbname: string, storename: string): IDB
+ new (dbname: string, storename: string): IDB
  saveSuperblock(sb: Uint8Array): TypeOrPromise<void>
  loadSuperblock(): TypeOrPromise<FS.SuperBlock>
- loadFile(inode: number): TypeOrPromise<Uint8Array>
+ readFile(inode: number): TypeOrPromise<Uint8Array>
  writeFile(inode: number, data: Uint8Array): TypeOrPromise<void>
  wipe(): TypeOrPromise<void>
  close(): TypeOrPromise<void>
}

Hey. I see that some work was already done in https://github.com/isomorphic-git/lightning-fs/tree/feat/native. What is the timeline for this and what are the current blockers?

For reference: https://www.chromestatus.com/feature/6284708426022912.

Is there a way to reset or delete an FS (it's indexdb store)?

Hmmm, that doesn't look right:

Firefox,

{
  "@isomorphic-git/lightning-fs": "^3.0.3",
  "isomorphic-git": "^0.51.12"
}

I use webpack to bundle for the browser.

The default backend does not currently implement a destroy method to control the lifecycle

When two instances point to the same underlying IndexedDb they clobber each other. This is particularly frustrating when working with Workers, because only a single Worker can use the file system. But it also is the same underlying cause of #16.

fs.rm is a new API in NodeJS fs module, that allows running recursive delete of directories. I can be handy in isomorphic-git/isomorphic-git#1383

First of all, thank you very much for the awesome isomorphic-git and this library!

My questions is:
Is there a way to delete all the files in a git repo?
I tried to rimraf the repo directory like "/repo_name" but the files are still in IndexedDB.