borisdiakur / memoize-fs Goto Github PK

I'm using memoize-fs in a crawler to store downloaded files. When the cache of 43k files (2GiB in total) got invalidated, I've found these two rows:

fnStr = String(fn),
hash = crypto.createHash('md5').update(fnStr + argsStr + salt).digest('hex');

I think that such behaviour should at least be mentioned in the manual, if not to say that there should be a way to turn it off.

Probably doesn't support es6?

Why not switch to Babel's parser?

async function fun(a) {
  return new Hello(a);
}

async function fun_(a) {
  return memoize.fn(fun, {}).then((memFn) => {
    return memFn(a);
  });
}

fun('meow').then(console.log);
fun_('meow').then(console.log);

First time this runs:

Hello { sound: 'meow' }
Hello { sound: 'meow' }

Second time this runs:

Hello { sound: 'meow' }
{ sound: 'meow' }

I'm having trouble importing memoize-fs using TypeScript, and I've been struggling for the past hour to figure out why it's failing. I made a minimal test case at https://github.com/joliss/import-problem so you can run it yourself if you like.

I have the following index.ts:

import memoizeFs from "memoize-fs";
memoizeFs({});

TypeScript can't find the import (both with tsc and in VS Code), even though memoize-fs is clearly installed:

~/src/import-problem $ tsc
index.ts:1:23 - error TS2307: Cannot find module 'memoize-fs' or its corresponding type declarations.

1 import memoizeFs from 'memoize-fs'
                        ~~~~~~~~~~~~

~/src/import-problem $ ls -l node_modules/memoize-fs/
total 20
-rw-r--r-- 1 ubuntu ubuntu  1079 Feb 29 15:44 LICENSE
-rwxr-xr-x 1 ubuntu ubuntu 12026 Feb 29 15:44 README.md
drwxr-xr-x 2 ubuntu ubuntu    41 Feb 29 15:44 dist
-rw-r--r-- 1 ubuntu ubuntu  2034 Feb 29 15:44 package.json

tsc nonetheless produces a compiled index.js file. If I run it, I get the following error from Node's module resolver:

~/src/import-problem $ node index.js
node:internal/modules/cjs/loader:598
      throw e;
      ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /home/ubuntu/src/import-problem/node_modules/memoize-fs/package.json
    at exportsNotFound (node:internal/modules/esm/resolve:303:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:593:13)
    at resolveExports (node:internal/modules/cjs/loader:591:36)
    at Module._findPath (node:internal/modules/cjs/loader:668:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1130:27)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/home/ubuntu/src/import-problem/index.js:6:38)
    at Module._compile (node:internal/modules/cjs/loader:1376:14) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Node.js v20.11.1

I can see the following exports declaration in the package.json:

~/src/import-problem $ cat node_modules/memoize-fs/package.json
{
   ...
  "module": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "exports": {
    ".": {
      "import": "./dist/index.mjs"
    }
  },
  ...
}

The dist/index.d.ts and dist/index.mjs files do exist in the distribution.

I'm not sure why exactly the exports field isn't being picked up properly by Node.

I'm running the latest stable Node and TypeScript versions:

~/src/import-problem $ node --version
v20.11.1
~/src/import-problem $ tsc --version
Version 5.3.3

For what it's worth, my actual workflow doesn't involve transpiling with tsc but rather running with ts-node, but it's showing the same error (ts-node index.ts and ts-node --transpile-only index.ts).

Hi @borisdiakur, thanks for your work. Have you thought of adding timed cached invalidation? It would be really useful to stop calling APIs whose results doesn't naturally change that often, particularly if a quota is set, too. Thanks!

I have a application, which only works with 0.1.7 (or lower) of memoize-fs.
More Information will follow.

My use case and confusion. I'm using memoize-fs to acquire jwt tokens for tests. Tokens living 1 hours, and their acquisition takes about 20 seconds, so memoize-fs effectively cutting those 20 seconds of from every test run.

maxAge is a timer based cache invalidation, however, because I'm using it to cache results across runs, I need invalidation happen based on the age of the cached data, not when memoization function was created.

This is at least to say confusion point, as documentation doesn't say anything about it. And I'd say, it's makes sense to store records age along with the data and verify this age on pull from disk. In this case maxAge would behave consistently with expectations (expectation is age of the data, not the memoization function instance in a process).

Hey @borisdiakur, thanks for the module! I was wondering if you'd be willing to accept a patch that would allow the comments/formatting of a function to change, without invalidating the cache. The idea is that instead of hashing the source code of the function, we would use a parser like babylon to build an AST (abstract syntax tree) of the function that represents its semantic behavior while disregarding comments/formatting/etc. The AST would be serialized into JSON and then included in the hash.

Awesome tool first of all!

I have an unusual case that I have a promisified function (returns a promise) but also takes a callback (although it's for a different purpose). This makes it incorrectly identify it as an async function.

As a workaround for now I'm just passing an extra non-function last argument, but it'd be better if this could be set manually like in options {promisified}, or if it were in the API like memoize.async/promisified.

I have fixed it by removing line 249 in my own branch, but I'm not sure what the intent of this was, so I haven't suggested a PR.

So, as a pseudo-example:

let callCount = 0;
const f = (_) => {
  callCount++;
}

const example = async () => {
  const memoizedF = await memoizer.fn(f, { force: true });
  
  // two calls.
  memoizedF(); 
  memoizedF();
}

example(); // call count is 1.
example(); // call count is 2.

This behavior gets weirder when you consider how it plays with argument level caching: only the first call after creating the memoizedF seems to have its cache renewed, that is, in the following code:

const memoizedF = await memoizer.fn(f);
memoizedF();
memoizedF(2);

const forceMemoizedF = await memoizer.fn(f, { force: true });
forceMemoizedF(); 
forceMemoizedF(2); // this will not have cache renewed.

As far as I can tell, removing line 249 fixes this with no side effects, but I haven't run your tests or otherwise figured out why you may wish for this run-once type behavior.

This will fix most of the "limitations", and also resolve my current problem.

I'm using it to cache configuration (which may include functions). Basically I memoize the function that loads a config file which returns some configuration that can have functions.

// tryLoadConfig returns an async function
const loadConfig = tryLoadConfig({
  ...ctxRest,
  testPath,
  config,
  start,
  runnerName,
});
const cfgFunc = await memoize(loadConfig, {
  cacheId: 'load-runner-config',
  astBody: true,
  salt: runnerName
});
const runnerConfig = (await cfgFunc()) || {};

In the fresh run, without cache, it writes a cache file with only the non-function things from this configuration object. The thing is that I later getting a function from this config (e.g. runnerConfig.postHook) but because on the second run it gets the config from cache it doesn't have the function there.

So. That's clear from the readme. But if we switch the cache files to be regular js file we'll be able to just put there whatever js the memoized function returns.

There's no problem for generating a hash in this way too.