Giter Club home page Giter Club logo

get-all-files's Introduction

get-all-files

A blazing fast recursive directory crawler with lazy sync and async iterator support.

Install

$ npm i get-all-files

Usage

import { getAllFiles, getAllFilesSync } from 'get-all-files'

// Lazily iterate over filenames asynchronously
for await (const filename of getAllFiles(`path/to/dir/or/file`)) {
  // Could break early on some condition and get-all-files
  // won't have unnecessarily accumulated the filenames in an array
  console.log(filename)
}

// Get array of filenames asynchronously
console.log(await getAllFiles(`path/to/dir/or/file`).toArray())

// Lazily iterate over filenames synchronously
for (const filename of getAllFilesSync(`path/to/dir/or/file`)) {
  // Could break early on some condition and get-all-files
  // won't have unnecessarily accumulated the filenames in an array
  console.log(filename)
}

// Get array of filenames synchronously
console.log(getAllFilesSync(`path/to/dir/or/file`).toArray())

API

Methods

getAllFiles(path[, options])

Returns a lazy async iterable/iterator that asynchronously iterates over the file paths recursively found at path in no particular order.

Calling toArray on the returned value returns a promise that resolves to an array of the file paths.

getAllFilesSync(path[, options])

Returns a lazy iterable/iterator that iterates over the file paths recursively found at path in no particular order.

Calling toArray on the returned value returns an array of the file paths.

Parameters

path

Type: string

A path to a file or directory to recursively find files in.

options

Type: object

Properties
resolve

Type: boolean
Default: false

Whether to resolve paths to absolute paths (relative to process.cwd()).

isExcludedDir

Type: (dirname: string) => boolean
Default: () => false

A predicate that determines whether the directory with the given dirname should be crawled. There is no isExcludedFile option because you can exclude files by checking conditions while lazily iterating usinggetAllFiles.sync or getAllFiles.async.

Contributing

Stars are always welcome!

For bugs and feature requests, please create an issue.

License

MIT © Tomer Aberbach

get-all-files's People

Contributors

prinzhorn avatar tomeraberbach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

get-all-files's Issues

ESM only support makes things difficult

I like the restructuring of the project in general, but moving to ESM makes it impossible / very hard to use the module for us at the moment. Could you add a parallel CommonJS build?

Node 16.13.0: getAllFiles is crashing when isExcludeDir has a match

I've found a case where invoking getAllFiles() crashes Node 16.13.0. The log output below should have lines for start and close on "cat", but it doesn't. Furthermore, the finally block at the end of the test file should be deleting the temporary directory - and it isn't.

Test file, saved as testGetAllFiles.mjs

import { getAllFiles } from 'get-all-files';
import path from "path";
import fs from "fs/promises";
import os from "os";

const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "test-get-all-files-"));
let tempDirResolve;
let tempDirPromise = new Promise(res => tempDirResolve = res);
tempDirPromise = tempDirPromise.then(() => fs.rm(tempDir, { recursive: true }));

/**
 * 
 * @param {RegExp?} excludeDirFilter
 */
async function runTest(excludeDirFilter) {
  const options = {};
  if (excludeDirFilter instanceof RegExp) {
    options.isExcludedDir = dirName => excludeDirFilter.test(dirName);
  }
  else if (typeof excludeDirFilter === "string") {
    options.isExcludedDir = dirName => dirName.includes(excludeDirFilter);
  }

  try {
    console.log("Start", excludeDirFilter || "no filter")
    let results = await getAllFiles(tempDir, options).toArray();
    console.log("Close", excludeDirFilter || "no filter", results);
  }
  catch (ex) {
    console.log(excludeDirFilter, "exception thrown");
    throw ex;
  }
}

try {
  await fs.mkdir(path.join(tempDir, "cat", "foo"), { recursive: true });
  await fs.mkdir(path.join(tempDir, "spec", "one"), { recursive: true });
  await fs.mkdir(path.join(tempDir, "spec", "two"), { recursive: true });
  await fs.mkdir(path.join(tempDir, "dog", "car"), { recursive: true });

  const fileList = [
    "cat/foo/module1.mjs",
    "cat/foo/module2.mjs",
    "dog/car/module3.mjs",
    "spec/one/module4.mjs",
    "spec/two/module5.mjs",
    "spec/two/module6.mjs",
  ];

  await Promise.all(fileList.map(relativePath => {
    return fs.writeFile(
      path.join(tempDir, relativePath),
      `console.log("Hello World");\n`,
      { encoding: "utf-8" }
    );
  }));

  console.log("tempDir: " + tempDir);
  await runTest();
  await runTest("elephant");
  await runTest("cat");
  await runTest("monkey");
  /*
  await runTest(/elephant/);
  await runTest(/cat/);
  await runTest(/monkey/);
  */
  console.log("Completed tests");
}

finally {
  tempDirResolve();
  await tempDirPromise;
  console.error("exiting");
}

Resulting output:

$ node testGetAllFiles.mjs 
tempDir: /tmp/test-get-all-files-1cWOqL
Start no filter
Close no filter [
  '/tmp/test-get-all-files-1cWOqL/cat/foo/module2.mjs',
  '/tmp/test-get-all-files-1cWOqL/cat/foo/module1.mjs',
  '/tmp/test-get-all-files-1cWOqL/dog/car/module3.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/one/module4.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/two/module6.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/two/module5.mjs'
]
Start elephant
Close elephant [
  '/tmp/test-get-all-files-1cWOqL/cat/foo/module2.mjs',
  '/tmp/test-get-all-files-1cWOqL/cat/foo/module1.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/one/module4.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/two/module6.mjs',
  '/tmp/test-get-all-files-1cWOqL/spec/two/module5.mjs',
  '/tmp/test-get-all-files-1cWOqL/dog/car/module3.mjs'
]
Start cat

This probably isn't your bug, but I'm filing it here so someone (hopefully not me) can reduce the testcase and file a bug against NodeJS, or (unlikely) V8.

ts-node compatibility

Hi,

Not sure this is the best place to ask, but I have this script.ts file:

import { getAllFilesSync } from "get-all-files";
getAllFilesSync(".")

and I run with ts-node script.ts but I get this error

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: C:\p\monorepo\node_modules\get-all-files\src\index.js
require() of ES modules is not supported.
require() of C:\p\monorepo\node_modules\get-all-files\src\index.js from C:\p\monorepo\tools\cli\prepush.ts is an ES module file as it is a .js file whose nearest
 parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from C:\p\monorepo\node_modules\get-all-files\package.j
son.

Do you have any ideas? I'm on ts-node v9

Many thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.