Giter Club home page Giter Club logo

flat-cache's Introduction

flat-cache

A stupidly simple key/value storage using files to persist the data

NPM Version tests codecov npm

install

npm i --save flat-cache

Usage

const flatCache = require('flat-cache');
// loads the cache, if one does not exists for the given
// Id a new one will be prepared to be created
const cache = flatCache.load('cacheId');

// sets a key on the cache
cache.setKey('key', { foo: 'var' });

// get a key from the cache
cache.getKey('key'); // { foo: 'var' }

// fetch the entire persisted object
cache.all(); // { 'key': { foo: 'var' } }

// remove a key
cache.removeKey('key'); // removes a key from the cache

// save it to disk
cache.save(); // very important, if you don't save no changes will be persisted.
// cache.save( true /* noPrune */) // can be used to prevent the removal of non visited keys

// loads the cache from a given directory, if one does
// not exists for the given Id a new one will be prepared to be created
const cache = flatCache.load('cacheId', path.resolve('./path/to/folder'));

// The following methods are useful to clear the cache
// delete a given cache
flatCache.clearCacheById('cacheId'); // removes the cacheId document if one exists.

// delete all cache
flatCache.clearAll(); // remove the cache directory

Motivation for this module

I needed a super simple and dumb in-memory cache with optional disk persistance in order to make a script that will beutify files with esformatter only execute on the files that were changed since the last run. To make that possible we need to store the fileSize and modificationTime of the files. So a simple key/value storage was needed and Bam! this module was born.

Important notes

  • If no directory is especified when the load method is called, a folder named .cache will be created inside the module directory when cache.save is called. If you're committing your node_modules to any vcs, you might want to ignore the default .cache folder, or specify a custom directory.
  • The values set on the keys of the cache should be stringify-able ones, meaning no circular references
  • All the changes to the cache state are done to memory
  • I could have used a timer or Object.observe to deliver the changes to disk, but I wanted to keep this module intentionally dumb and simple
  • Non visited keys are removed when cache.save() is called. If this is not desired, you can pass true to the save call like: cache.save( true /* noPrune */ ).

License

MIT © Jared Wray

flat-cache's People

Contributors

amilajack avatar chenaaron3 avatar cktang88 avatar jaredwray avatar ljharb avatar nopnop avatar ozairp avatar patricksmms avatar royriojas avatar sgilroy85 avatar teamop avatar trysound avatar uncenter avatar webreflection avatar wojtekmaj avatar xuexb avatar ybiquitous avatar yumetodo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flat-cache's Issues

Feature Request: cache.all method

I think it would be nice to fetch the whole cache object.

const fcache = require('flat-cache')

const birthdays = fcache.load('birthdays')

birthdays.set('Alice', '01/01/98')
birthdays.set('Bob', '02/02/97')

birthdays.all() // returns { 'Alice': '01/01/98', 'Bob': '02/02/97' }

Where are the Error Handlers?

Hey just started using this library.

I am aware you want to keep it simple so may not implement my suggestion.

But where are the error handlers.

.setKey()
.save()

all return undefined
so there is no way to see if the data was saved or not

Would be nice if

  • they returned a promise or the data that was saved
  • we could pass a callback function for error handlers

I am having trouble debugging my application because of it.

I also searched this repository for "error" so for a module that is downloaded by the millions, I assume there are no errors on the module side and it has to be something in my code.

But still can't debug because of no error handlers.

However my console.logs indicate the functions are being called

Consider inline the `write` module.

This is something we need to consider carefully, I don't see a big deal having small modules as dependencies, but I have the feeling inlining the write module might make sense for this module instead of fetching that as a dependency.

Cache Reset

Is there something I am missing

const path = require('path')
const flatCache = require('flat-cache')
let appCache = flatCache.load('test', path.resolve('./cache'))

appCache.setKey('a', 'something')
appCache.save()

appCache.setKey('b', 'something else')
appCache.save()

only the b key exists in cache and I want both to exist

Don't rely on `this`

Love this module.

Using it in chains is less simple because it relies on this, I have to bind:

doSomethingAsync()
  .then(myCache.save.bind(myCache)

Instead of:

doSomethingAsync()
  .then(myCache.save)

Cache Expiration

Is there a way to set an expiration time on a cache item? For instance say I want to cache results from a weather API - it's not necessary to make the API request every 5 minutes if the page that needs the data is accessed that often. Something like this would be ideal:

cache.setKey('key', 'value');
cache.save(60); // cache lasts for 60 min

And to erase expired items:
cache.eraseExpired();

If nothing like this exists I will create a PR. Also this is my first time posting to GH, hopefully the formatting isn't too jacked up.

Why Keyv is necessary?

I was playing with the package and noticed that it doesn't really use keyv.get(), if it is indeed the case, why do we use keyv.set() if we won't access the items later on? Maybe I just need more clarification.
The package requires keyv and on load we initialize a new instance (new Keyv()).
In my local machine I commented out all keyv occurrences and flat-cache worked.

TLDR;
Why is keyv necessary?

Thanks

Keyv: `Invalid String Length`

Related, somewhat, to #94...

just chiming in as to the original discussion on keyv, that update has caused for me, as the version of json-buffer introduced causes

RangeError: Invalid string length
     at stringify (node_modules/json-buffer/index.js:32:35)
    at Object.stringify [as serialize] (node_modules/json-buffer/index.js:32:37)
    at node_modules/keyv/src/index.js:202:22

I'm testing on node lts/hydrogen (v18.19.1), npm v10.2.4.
This is specifically happening on >= 3.1.1.

I'm not able to reproduce the same issue when pinned to 3.0.4.

Remove dependency on `graceful-fs`

Since we only use fs.existsSync from that package and that checking for existence should be done now using fs.statSync we should just align and remove the dependency as well.

3.1.0 should be breaking

Release version 3.1.0 removes support of Node 10 - this should be a breaking change. Can we get a bugfix version where Node 10 is supported?

Uncaught exception when cache file contains invalid JSON

When you call readJSON() on this line, there is a chance that an exception will be thrown, since JSON.parse() is called under-the-hood.

JSON.parse() calls should be wrapped in a try-catch block in case the string can't be parsed as JSON. This can happen if another program creates/modifies the cache file, or if the node script using flat-cache crashes in the middle of a write.

This issue is currently affecting ESLint (eslint/eslint#7748) and standard (standard/standard#673). More details in the linked issues.

update del

The dependency is outdated and is using [email protected] that doesn't include licence file. Updating del to v3.0.0 solves this issue. And apparently doesn't break anything, the tests still pass.

Add a way to keep non visited keys

This is related to jaredwray/file-entry-cache#2, which in time is also related to eslint/eslint#6780

Basically the default behavior of this module was to prune non visited keys. This was done to prevent non visited keys to stay in the cache. Since the main use case of this module is just to load to memory the cache, do operations in memory and then persist the new changes back to disk at the end of the process. I assumed non visited keys were not relevant anymore so they were removed.

But as described in the issues linked above, there might be scenarios when we want to keep the non visited keys.

Data had been replaced with "NUL" characters

Hi,

I am using flat-cache, as a offline storage in an application developed using electron. Sometime, the content in cache file was completely or partially replaced with "NUL" characters. I could not identify the cause for this. Could you please suggest a solution for this?

thanks,
Partha

Update minimist to 1.2.3 or later

Hello, just found this out, so wanted to open an issue.

minimist released a security statement to use version 1.2.3 or later due to a prototype pollution bug

minimist is not a top level dependency (It's not in package.json), but it is present in package-lock.json (a word search reveals 21 occurrences)

This means projects with flat-cache as dependency and a build process that includes checking vulnerability (through $ npm audit) might have a fail build because of it. I was wondering if you can please take a look, thanks!

The cache is overwritten

var flatCache = require('flat-cache');
var cachePath = './__cache/';

// Written three key
var cache = flatCache.load('cacheId', cachePath);
cache.setKey('key', {
    foo: 'var'
});
cache.setKey('key2', {
    foo: 'var2'
});
cache.setKey('key3', {
    foo: 'var3'
});
cache.save();
// => {"key":{"foo":"var"},"key2":{"foo":"var2"},"key3":{"foo":"var3"}}


// I want to use only one key, but I 'll cover the other key when you save
var cache2 = flatCache.load('cacheId', cachePath);
cache2.setKey('key', {
    foo: 'var'
});
cache2.save();
// => {"key":{"foo":"var"}}

Under normal circumstances , I would like to use a key, but do not want to cover the rest, I do not know whether there is such a demand problem , thank you~

Switch from cjs to ESM?

Many modules are switching to ESM to support both Deno and browser imports without bundeling. Think you should make the switch as well...

Can make a PR if u like

would require at least node v12.17

Dependency updates

Hello, before I spend time preparing a PR I'd like to know if you willing to accept patches which make this and a few other modules require node 4? I'm working through some dependency trees for node modules and trying to eliminate situations where multiple versions of the same module are installed. The current versions of many dependencies now require node >=4.

`pinkie-promise` was deleted

Hi there,

Do you mind updating del to version 3.0.0, looks like the version we have pinned here depends on pinkie-promise and this one was deleted.

Cheers,

Oscar

Missing utils.js

v1.2.0 only includes cache.js as stated in package.json. Causing problems with ESLint.

> eslint --format=node_modules/eslint-formatter-pretty .

Cannot find module './utils'
Error: Cannot find module './utils'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/private/var/app/v1-customers/node_modules/eslint/node_modules/file-entry-cache/node_modules/flat-cache/cache.js:4:16)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

Clear Cache

When do i clear the cache?

I manually added a new row to a table and when I called the end point, it still showed me the old rows. So at what point do I clear the cache?

Circular References (related to persitify)

Hello,

I came across a problem with persistify that is related to flat-cache (persistify is really awesome by the way)

Sometimes, browserify cache may contain circular reference that break flat-cache serialization. I fixed this by replacing JSON.stringify by json-stringify-safe in cache.js#L109. But it may not be a good solution so I won't push a PR before having your perspective on this.

Maybe for a caching mechanism, highly related to development purpose, it's acceptable to fallback to something like json-stringify-safe. At least with a warning instead of a fatal Exception or with a lenient options to optionally use json-stringify-safe instead of JSON.stringify ?

Breaking change in a minor version

v3.0 of flat-cache has engines.node set to "^10.12.0 || >=12.0.0", but v3.1.0 sets it to ">=12.0.0", which is a breaking change. (it was broken in #74) This is causing the tests in one of my packages to break.

Please publish a v3 that restores ^10.12.0 compatibility, and then if you'd still like to drop node 10 support a v4 would be perfectly fine. Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.