lokijs-forge / lokidb Goto Github PK

I'm submitting a...

[X] Bug report

Current behavior

Today, when inserting a new object to a collection, LokiDb adds a meta property to that object on to which it writes a version property

            if (!this._disableMeta) {
                newDoc.meta.version = 0;
            }

If the object already has a meta property and it is set to null (as per my case) then loki will try and set null.version = 0
Which causes an error

Expected behavior

Loki should not reserve properties without some kind of prefix or suffix as for example 'meta' is a common property name.
but instead should maybe follow the $loki property name and be called $meta or just put the meta prop in $loki

Minimal reproduction of the problem with instructions

just create a collection and insert {meta: null}

Environment

Latest loki version

I'm submitting a

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Feature request
[x] Documentation issue or request
[ ] Support request

Current behavior

Cannot pass PartitioningAdapter as an adapter to initializePersistence

Error:

Type 'PartitioningAdapter' is not assignable to type 'StorageAdapter'.
  Types of property 'exportDatabase' are incompatible.
    Type '(dbname: string, dbref: import("/xxx/angular-pwa/node_modules/@lokidb/partitioning-adapter/types/loki/src/loki").Loki) => Promise<void>' 

    is not assignable to type 
    '(dbname: string, dbref: import("/xxx/angular-pwa/node_modules/@lokidb/loki/types/loki/src/loki").Loki) => Promise<void>'.

Expected behavior

Can pass PartitioningAdapter as an adapter to initializePersistence

Minimal reproduction of the problem with instructions

    IndexedStorage.register();
    PartitioningAdapter.register();
    let idbAdapter = new IndexedStorage('test.db');
    let paAdapter = new PartitioningAdapter(
      idbAdapter, 
      { 
        paging: true, 
        pageSize:5*1024*1024 
      }
    );
    let options: Loki.Options = {
      env: 'BROWSER',
      serializationMethod: 'normal'
    };
    
    let db = new Loki('test.db', options);

    let persistenceOptions: Loki.PersistenceOptions = {
      adapter: paAdapter, <-- Issue is here
      autosave: false,
      autosaveInterval: 1000,
      autoload: false,
      throttledSaves: false,
      persistenceMethod: 'indexed-storage'
    };
    this.db.initializePersistence(persistenceOptions).then(err => {
      console.log('initializePersistence complete');
    });

What is the motivation / use case for changing the behavior?

Be able to use PartitioningAdapter

Environment

LokiDB version: 2.0.0-beta.8
Browser/Node version: Chrome 71.x

The issue is that there are two Loki type definitions one in Loki & one in PartitioningAdapter. PartitioningAdapter extends the StorageAdapter which references its own copy of Loki and not the root copy here @lokidb/loki

@lokidb/partitioning-adapter/types/loki/src/loki
@lokidb/loki

If I try to use the reference to @lokidb/partitioning-adapter/types/loki/src/loki instead of @lokidb/loki I can pass the paAdapter to the PersistenceOptions but i get another error

Module not found: Error: Can't resolve '@lokidb/partitioning-adapter/types/loki/src/loki'

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiJS2/blob/master/CONTRIBUTING.md#question

Current behavior

The scoring result keys of a full-text search obtained by a ResultSet are the data row.

Expected behavior

Use the $loki instead inside the scoring result.

Something went wrong when publishing 2.0.0-beta.7 to npm...

https://travis-ci.org/LokiJS-Forge/LokiDB/builds/397214166

Maybe I republish this version or create a new version (2.0.0-beta.8) after I solved the problem with npm.

I'm submitting a...

[x] Feature request
[x] Documentation issue or request

Current behavior

Looking for the LokiDB equivalent of the LokiFsStructuredAdapter.

Expected behavior

I expected it to be a package like the @lokidb/fs-storage adaptor. I would like to re-implement the zip adapter I made last year ( techfort/LokiJS#758 ). I’m currently learning TypeScript, so it’d be great practice.

I wasn’t able to find docs for the @lokidb/partitioning-adapter package, though that may be what I’m looking for.

Any pointers in the right direction would be great 😄congrats on releasing the beta

Environment

LokiDB version: 2.0.0-beta.9 (2020-02-07)

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question
[x] Refactoring Discussion?

In LokJS, our ops 'evolved' when adding binary indices and to adapt to problems with javascript equality and relational ops when dealing with dirty data and mixed types. Recently we have even added ops for explicitly opting for js relational ops ($jgt, $jgte, $jlt, $jlte, $jbetween)... not for functional output differences but just because they are faster in some situations.

I also suggest me might add different 'modes' for dealing with relational operations... by default we should probably continue to expect that users will throw dirty data at it and we will need to rely on 'loki sorting' to establish a reliable 'range' over which our binary indices can work. But many people may be producing their own data internally and have confidence over the cleanliness of their data... especially if they also adopt typescript for code that creates documents they insert.... in those cases we might want to 'reroute' our $ltHelper and $gtHelper to utilize native javascript relational and equality operators... doing so should increase performance even more.

I will propose suggestions later but for now I will just attempt to quantify/itemize factors involved in this for discussion.

All other have it, we don't -> A logo for LokiDB.

My idea so far, because Loki is a shape shifter:

A database (maybe represented by 3 stacked boxes). morphing to animals (bird, snake, fish...) in the right half. ;)

I'm submitting a...

[x] Feature request

Current behavior

Currently, the only options to query at array property level is $contains and $size.

See techfort/LokiJS/issues/447

Expected behavior

Introduce a new operator (like $array) which enables all other operations ($neq, $lt...) at each element of an array.

Minimal reproduction of the problem with instructions

coll.insert({ar: [1, 2, 3, 4]});

coll.find({
  $array: ["ar", {
    $lt: 2
  }]
});

What is the motivation/use case for changing the behavior?

More useful operator.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

The readme published to npm contains HTML code which is not interpreted right.

Expected behavior

The readme should only contain markdown.

Minimal reproduction of the problem with instructions

https://www.npmjs.com/package/@lokidb/loki

What is the motivation / use case for changing the behavior?

Cleaner look.

I'm submitting a...

[x] Feature request

Current behavior

According to techfort/LokiJS#688, I tried to implement Full Outer Join in TypeScript, with success; and noticed performance improvement for no reasons. (I have 100K rows, and 4 tables' join is usually around 2-3 sec. It becomes like 1 sec.)

The code is

interface IJoinCollection<T> {
    data: T[];
    key: keyof T;
}

function fullJoin<T, U>(
    colL: IJoinCollection<T>,
    colR: IJoinCollection<U>,
    mapFn: (l: T, r: U) => any,
    isFull: boolean = true
): any[] {
    const joinMapL: any = {};
    const joinMapR: any = {};
    const result: any[] = [];

    for (const rowR of colR.data) {
        const v = rowR[colR.key];

        if (v) {
            joinMapR[v] = joinMapR[v] || [];
            joinMapR[v].push(rowR);
        } else {
            result.push({} as T, rowR);
        }
    }

    for (const rowL of colL.data) {
        const v = rowL[colL.key];

        if (v) {
            for (const vR of joinMapR[v] || [{}]) {
                result.push(mapFn(rowL, vR));
            }

            if (isFull) {
                joinMapL[v] = joinMapL[v] || [];
                joinMapL[v].push(rowL);
            }
        } else {
            result.push(mapFn(rowL, {} as U));
        }
    }

    if (isFull) {
        for (const rowR of colR.data) {
            const v = rowR[colR.key];

            if (v) {
                for (const vL of joinMapL[v] || [{}]) {
                    result.push(mapFn(vL, rowR));
                }
            }
        }
    }

    return result;
}

Environment

LokiDB version: 2.0.0-beta.8 with @lokidb/fs-storage 2.0.0-beta.8
Browser/Node version: Node v11.11.0 Yarn 1.15.2 MacOS

I'm currently using the old LokiJS, which is great, but when the new LokiDB releases, I'd love to switch. 🔄

I see there's a wonderful arsenal of adapters ready for LokiDB.
There is only one adapter that I'd like to request extra. 📌

• Meet loki-cordova-fs-adapter.

This adaptor is abandoned three years ago but still works on cordova v8 and cordova-ios v5 versions in my tests. It's amazing as it allows cordova apps to save Loki data to the File Storage. Which is persistent (unlike indexedDB which might get deleted by iOS during low memory).

If any one of you JS Gurus could look at the source code (only 100 lines!) and create an official version for LokiDB that'd be super cool! 😎

If any further information is required on Cordova File Storage, they have a great guide on this in their docs.

Happy easter! 🥚 🗿

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

I can load DB, load collection, insert, save, and load collection again with no issue but when I refresh the page or after auto-reload via ionic-serve is trigger that data does not persist

$ ionic serve
> ng run app:serve --host=0.0.0.0 --port=8100

[INFO] Development server running!
       
       Local: http://localhost:8100
       External: http://192.168.0.4:8100
       DevApp: ura@8100 on ****      
       Use Ctrl+C to quit this process

[INFO] Browser window opened to http://localhost:8100!

[ng] i ｢wdm｣: wait until bundle finished: /
[ng] Date: 2018-10-15T17:52:41.516Z
[ng] Hash: b4166270973e520f51ff
[ng] Time: 3211ms
[ng] chunk {main} main.js, main.js.map (main) 1.93 kB [initial] [rendered]
[ng] chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 681 bytes [initial] [rendered]
[ng] chunk {runtime} runtime.js, runtime.js.map (runtime) 5.22 kB [entry] [rendered]
[ng] chunk {styles} styles.js, styles.js.map (styles) 38.7 kB [initial] [rendered]
[ng] chunk {vendor} vendor.js, vendor.js.map (vendor) 326 kB [initial] [rendered]
[ng] ERROR in node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,36): error TS1005: ';' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,75): error TS1005: ';' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,119): error TS1005: ';' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,142): error TS1005: ':' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,153): error TS1005: ')' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,154): error TS1005: ';' expected.
[ng] node_modules/@lokidb/fs-storage/types/loki/src/result_set.d.ts(220,158): error TS1005: ';' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,36): error TS1005: ';' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,75): error TS1005: ';' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,119): error TS1005: ';' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,142): error TS1005: ':' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,153): error TS1005: ')' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,154): error TS1005: ';' expected.
[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,158): error TS1005: ';' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,36): error TS1005: ';' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,75): error TS1005: ';' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,119): error TS1005: ';' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,142): error TS1005: ':' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,153): error TS1005: ')' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,154): error TS1005: ';' expected.
[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,158): error TS1005: ';' expected.
[ng] i ｢wdm｣: Failed to compile.

Expected behavior

There should be no errors and data should persist after page refreshes.

Minimal reproduction of the problem with instructions

-------------database.service.ts----------------
import { Loki } from '@lokidb/loki';
import { IndexedStorage } from '@lokidb/indexed-storage';

private db = new Loki('test');
private options: Loki.PersistenceOptions = {
    adapter: new IndexedStorage(),
    autosave: false
};

load(collection_name: string, options?: object | null) {
     return this.db.initializePersistence(this.options)
    .then(() => {
      let collection = this.db.getCollection(collection_name);
      if (!collection) {
        collection = this.db.addCollection(collection_name, options);
      }
      return collection;
    });
  }

save(): Promise {
    return this.db.saveDatabase().then(() => {
      console.log('Saved!');
    });
  }

-------test.page.ts-------
export class TestPage implements OnInit {

  private authCollection:  Collection;

  constructor(private dbs: DatabaseService) {
    this.dbs.load('test').then((col) => {
      console.log(col); /// -_- ...is always empty after refresh
    });
  }

  ngOnInit() {
  }

  setItem(document: object): void {
    if (Object.keys(document).length !== 0 && document.constructor === Object) {
      this.dbs.load('test').then(() => {
          col.insert(document)// works
          this.dbs.save(); //works
      });
      this.dbs.save();
    }
  }

  getItem() {
     this.dbs.load('test').then((col) => {
      console.log(col); // works and shows inserted data ...BUT -_- ...is always empty after refresh
    });
  }

What is the motivation / use case for changing the behavior?

Environment

LokiDB version: 2.0.0-beta.8
"@lokidb/fs-storage": "^2.0.0-beta.8",
    "@lokidb/indexed-storage": "^2.0.0-beta.8",
    "@lokidb/loki": "^2.0.0-beta.8",


Browser/Node version: 
Google Chrome is up to date
Version 69.0.3497.100 (Official Build) (64-bit)

Others:
Ionic:

   ionic (Ionic CLI)          : 4.0.6 (/usr/local/lib/node_modules/ionic)
   Ionic Framework            : @ionic/angular 4.0.0-beta.0
   @angular-devkit/core       : 0.7.0-rc.3
   @angular-devkit/schematics : 0.7.0-rc.3
   @angular/cli               : 6.0.8
   @ionic/ng-toolkit          : 1.0.0
   @ionic/schematics-angular  : 1.0.1

Cordova:

   cordova (Cordova CLI) : 8.0.0
   Cordova Platforms     : none

System:

   ios-sim : 5.0.6
   NodeJS  : v8.11.3 (/usr/local/bin/node)
   npm     : 5.6.0
   OS      : macOS High Sierra
   Xcode   : Xcode 10.0 Build version 10L213o


--------tsconfig.json---------
{
  "compileOnSave": false,
  "compilerOptions": {
    "outDir": "./dist/out-tsc",
    "sourceMap": true,
    "declaration": false,
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "target": "es5",
    "lib": [
      "es2017",
      "dom"
    ]
  }
}

Any help would be appreciated thanks!!

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[x] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

In order to update the LokiDB stored in IndexedDB with the data returned from the server you must delete entire collection and re-insert all the data.

Desired behavior

Pass all server objects and insert/update/delete is handled by LokiDB

Minimal reproduction of the problem with instructions

Passing update server objects adds new entries

What is the motivation / use case for changing the behavior?

Make it easier to sync large amounts of data

Environment

LokiDB version: 2.0.0-beta.8


Browser/Node version: Chrome 71

I'm submitting a...

[x] Feature request

Current behavior

Currently, there is no function to do a simple update insert of a document.

Expected behavior

Provide a method to insert or update a document to the collection. The function should check by using unique indices if the document should be inserted or update an existing one.

See techfort/LokiJS/issues/158

Minimal reproduction of the problem with instructions

What is the motivation / use case for changing the behavior?

Long requested feature. Many databases support this.

I'm submitting a...

[x] Feature request

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ x?] Bug report
[ ] Feature request
[X ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Hi, I have some issues with fuzzy search, I'm not sure if I'm just using incorrectly as I couldn't find anything about it in the docs
Fuzzy search works great, some of the time I'l give some examples that work and some that don't

with the following data set ['Tanglefoot', 'Shield', 'Acid Splash', 'Prestidigitation]

The search term 'ash' should return Acid Splash but returns 'Shield'
The search term 'foot' should return 'Tanglefoot' but return 0 results
The search term 'digitation' should return 'Prestidigitation' but return 0 results

How I use the fuzzy search:

{ $fts: { query: { type: 'fuzzy', field: 'name', value: search.value, extended: true, prefix_length: 0 } } }

[X] Bug report
[X] Documentation issue or request

So this is either a bug or just a lack of documentation and me using this wrong.

Current behavior

When using autoload, if the database already exists everything works as expected,
the issue is then on a first time load of the website the DB is yet to be created and loki throws the error

DB ERROR TypeError: The persistence adapter did not load a serialized DB string or object.
    at eval (lokidb.loki.js:5577)

I've debugged it for the last hour, and it does seem like loki is trying to create a new DB if needed, the issue lays in getAppKey function, it receives the newly created catalog and tries to search for the appKey in the store index, but it is not there.

Expected behavior

To work with on first load with autoload on

Minimal reproduction of the problem with instructions

Here is how I Initialize my DB (works perfectly if autoload is off)

const db: Loki | undefined
let loading = false

export default () => {
  if (!db && process.client && !loading) {
    loading = true

    console.log('Initializing DB')

    FullTextSearch.register()
    IndexedStorage.register()
    const lokiDb = new Loki('notAMimic')

    lokiDb.initializePersistence({ adapter: new IndexedStorage(), autosave: true, autoload: true })
      .then(() => {
        db = lokiDb
      })
      .catch((e: Error) => {
        // eslint-disable-next-line no-console
        console.log('DB ERROR', e)
      })
  }

  return db
}

Environment

Latest

Browser/Node version: Chrome 84.0.4147.125

Others:
I doubt It's relevent, but, using typescript with vuejs

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question
[x] Discussion of build / toolchain process ?

First I would like to thank you for your efforts converting LokiJS to modern Typescript environment in addition to adding full text searching support... it may take a while for me to become aware of and appreciate all the work you have done.

I also do not want to suggest imposing any restrictions that would limit your productivity :)

I do have a few questions, and now might be a good time to ask them :

Elasticsearch : can we optimize a build path that does not involve this external dependency? I believe I can unit test, perform a build, and bundle by just running npm run test:web but the 'npm run test:node' script fails (usually because of elasticsearch). Is there benefit to the node tests outside of being able to use elasticsearch? If so, can we have a different test:node script which skips those unit tests when run locally, while still ensuring they will run in travis?

build.js (npm run build) : Is there any way to make this work better cross-platform? This script will not work in Windows environment, and I even am getting sed errors on mac osx. Not sure how important those steps are but it would be nice to have build path for local use that had minimal shell/os dependencies.

Not to rush any roadplans that might have been developed, but once we near a point where you feel comfortable with making an official release we will probably want to be ready for dealing with the wide variety of environments that people might be coding in. As a developer using Windows 10 as his primary development environment, I would probably need this to provide any significant contributions. I do have ubuntu/osx boxes but they are not daily drivers.

[X ] Bug report

Hya again ^^

When adding an fts field if the field is optional in the collection, fts will fail on insert, this will cause the insert to fail and stop at the last item inserted.

for example, let's say I have an Item collection and I want to be able to search in the field group but this field is optional, i.e sometimes it has a value, sometimes null

When fts tokenizes the field it will try to split it by whitespaces

and fails, as null.split is not a function

Edit:
After testing for 1 more minute 🤦
I found the offending line:

if (doc[fieldName] !== undefined && doc[fieldName] !== null) { ...

fixes the issue

Expected behavior

If a field value is not present, that is null or undefined, skip.
Maybe even check for more types and throw/convert to a string?
Number => toString ? throw invalid type error? expected string got X

Minimal reproduction of the problem with instructions

Something like this should work:

//Initialize DB
    FullTextSearch.register()
    IndexedStorage.register()
    const lokiDb = new Loki('notAMimic')

// initialize collection
   db.addCollection<T>(collectionName, {
   fullTextSearch: [
        { field: 'name' },
        { field: 'group' },
        { field: 'category' },
      ],
  })

// add data
collection.value.insert([
{ name: 'Sword of madness', group: 'sword', category: 'Martial' },
{ name: 'Club', group:'club', category: 'simple' },
{ name: 'Shield of awesome', group: null, category: 'to lazy to come up with something' } // Cry here
])

[X] Bug report

Current behavior

When using autoload to load the DB and then calling clear on a loaded collection the fulltext docstore of that collection is not cleared.

I checked this for a newly loaded collection (that is loaded from the server) calling clear clears the fulltext docstore
but If I refresh (allowing the autoload feature to work) and then call clear, when I try to insert new data into the collection I get the error Field already added. in the insert method of the fullTextSearch, when I examine the docstore in that error I can see that it is still full

Edit:
After debugging the code a bit more, I see that loki does call clear on the fulltext search but the issue is that with autoLoad for some reason the _docs for the inverted_index is empty even but the actual inverted_index is not, not sure why this is happening

Expected behavior

calling collection.clear should clear the fulltext search for that collection as well. no matter how the collection was created.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[x] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

I looked in the following places for documentation on how to use LokiDB:

• https://www.npmjs.com/package/@lokidb/loki
• https://github.com/LokiJS-Forge/LokiDB
• https://lokijs-forge.github.io/LokiDB/

While the API is documented, there are no examples for anyone wanting to start with LokiDB. Maybe LokiJS could be a fall back, but it is not clear. I am not seeing a wiki either.

Can anyone indicate some examples that are suitable starting points?

Dear @Viatorus @Cortys @VladimirTechMan.

It seems there has been no development in the past seven months. Care to share a statement on what the future holds for LokiDB?

I'm submitting a...

[x] Bug report

Current behavior

TL;DR: Unable to use persistence.
I am trying to manually load and save database, but I always get either error 'persistenceAdapter not configured' or no error but also no persistence. Lokijs on the other hand works with default options, but I need fulltext search. Is it possible that behavior/options changed since moving from lokisj?

Expected behavior

To be able to use persistence (loading and saving database from disk).

Minimal reproduction of the problem with instructions

STEPS TO REPRODUCE

const { Loki } = require('@lokidb/loki');
const loki  = new Loki('database.db');
loki.loadDatabase();

What did I try?

• default options
• with options.env: 'NODEJS'
• with options.adapter: new FSStorage() (const { FSStorage } = require('@lokidb/fs-storage');)
  • with and without FSStorage.register()
• with options.autoload: true and options.autoloadCallback (callback is not called)
• with options.autosave and options.autosaveInterval: 1000

Environment

LokiDB version: 2.0.0-beta.6
Browser/Node version: NODE 8.9.4

Guys, this project is great and I cannot wait for documentation :) Thanks for your hard work!

EDIT:

I managed to get something working 👍 ( I found documentation - and I noticed the sidebar :))
It seems lokidb needs to call:

database.initializePersistence({
  autoload: true
});

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

update a record
delete that record
save database
values left in database

Expected behavior

no values left in database

Minimal reproduction of the problem with instructions

save after (update record and delete record) leaves behind values in DB
save after update, save after delete record does not leave behind values

What is the motivation / use case for changing the behavior?

Explicitly saving database after an update and delete would leave behind records

Environment

LokiDB version: 2.0.0-beta.8

Browser/Node version: Chrome 71

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Expected behavior

Minimal reproduction of the problem with instructions

What is the motivation / use case for changing the behavior?

Environment

LokiDB version: X.Y.Z


Browser/Node version: Browser X.Y.Z.

Others:

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

No benchmarking exists.

Expected behavior

Add benchmarking to CI to compare newer and older implementation performance.

What is the motivation/use case for changing the behavior?

Verify that there are no big performance losses.

Environment

Benchmarks for NodeJS and some Browsers.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiJS2/blob/master/CONTRIBUTING.md#question

Current behavior

Unit test with Karma does not generate a code coverage report.

Expected behavior

Unit test with Karma should generate a code coverage report.

Minimal reproduction of the problem with instructions

npm run test:web

What is the motivation / use case for changing the behavior?

Unit test with jasmine generates a code coverage report.

Environment

LokiJS version: trunk


Browser/Node version: Browser any

Others:

[x] Documentation issue or request

Would it be possible to offer in-place upgrade from lokijs 1 serialization format to 2 in loadDatabase? I don't mind doing it myself as long as there would be a clear transformation I could do inside my own load handler.

What do you think about renaming LokiJS2 to a different name before releasing it?

Problem:
It is not easy to distinguish (using a search engine) between LokiJS and LokiJS2.
I know this problem from AngularJS and Angular 2...

I don't think this would dramatically decrease the popularity of this library.

@obeliskos @techfort
What is your opinion?

I am open for proposals for a Nordic name. ;)

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[x] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Unable to test full-text-search

Environment

LokiDB version: @lokidb/[email protected]



Others:

I am trying to import

import LokiFullTextSearch from '@lokidb/full-text-search';

and i'm getting

Module not found: Error: Can't resolve '@lokidb/full-text-search'

i can create new database with lokidb = new loki() but i'm unable to run this example for full-text-search

LokiFullTextSearch.register();

  const lokis = new loki();
  // Setup the collection with full-text search options.
  const coll = lokis.addCollection("test", { fullTextSearch: [{ field: "txt" }] });
  // Add documents.
  coll.insert([
    { txt: "Betty Botter some butter" },
    { txt: "But she said the butter’s bitter" },
    { txt: "If I put it in my batter, it will make my batter bitter" },
    { txt: "But a bit of better butter will make my batter better" },
    { txt: "So ‘twas better Betty Botter bought a bit of better butter" }
  ]);
  // Search.
  const query = {
    query: {
      type: "term",
      field: "txt",
      value: "butter"
    }
  };
  const result = coll.find({ $fts: query });
  console.log(result);

Thanks

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

ERROR in node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,36): error TS1005: ';' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,75): error TS1005: ';' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,119): error TS1005: ';' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,142): error TS1005: ':' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,153): error TS1005: ')' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,154): error TS1005: ';' expected.

[ng] node_modules/@lokidb/indexed-storage/types/loki/src/result_set.d.ts(220,158): error TS1005: ';' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,36): error TS1005: ';' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,75): error TS1005: ';' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,119): error TS1005: ';' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,142): error TS1005: ':' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,153): error TS1005: ')' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,154): error TS1005: ';' expected.

[ng] node_modules/@lokidb/loki/types/loki/src/result_set.d.ts(220,158): error TS1005: ';' expected.

Expected behavior

no errors and should be able to easily compile

Minimal reproduction of the problem with instructions

No matter what I do I keep getting the same errors. I've it seems to stem from result_set.d.ts whose types are present in all 4 adapaters and loki itself.

Environment

Ionic:

   ionic (Ionic CLI)          : 4.0.6 (/usr/local/lib/node_modules/ionic)
   Ionic Framework            : @ionic/angular 4.0.0-beta.0
   @angular-devkit/core       : 0.7.0-rc.3
   @angular-devkit/schematics : 0.7.0-rc.3
   @angular/cli               : 6.0.8
   @ionic/ng-toolkit          : 1.0.0
   @ionic/schematics-angular  : 1.0.1

Cordova:

   cordova (Cordova CLI) : 8.0.0
   Cordova Platforms     : none

System:

   ios-sim : 5.0.6
   NodeJS  : v8.11.3 (/usr/local/bin/node)
   npm     : 5.6.0
   OS      : macOS High Sierra
   Xcode   : Xcode 10.0 Build version 10L213o

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Expected behavior

Minimal reproduction of the problem with instructions

What is the motivation / use case for changing the behavior?

Environment

LokiDB version: X.Y.Z


Browser/Node version: Browser X.Y.Z.

Others:

• Remove inconsistency in ResultSet (see techfort/LokiJS/issues/558)
• Option to disable meta/revision in Collection (see techfort/LokiJS/issues/560) #80
• Add nested property sort functionality (see techfort/LokiJS/issues/574) #81
• Rename/Overthink StageAPI (see techfort/LokiJS/issues/687)
• support deep Array clone (see techfort/LokiJS/pull/692)

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

IndexedDB stores the database as a JSON string.

Expected behavior

IndexedDB stores the database as an object. Can be enabled via an option.

What is the motivation/use case for changing the behavior?

IndexedDB is an object storage so it can store hole objects (including Date, Map, Blobs etc.).
This would decrease storage space because e.g. Blobs don't have to be serialized using base64.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

When you remove a collection, the 'autosave' does not pick up the change.

Expected behavior

I would expect autosave to pick up the change, so that the deletion of the collection is properly persisted.

Minimal reproduction of the problem with instructions

const idbAdapter = new IndexedStorage(dbName);
const db = new Loki('dbfile');
let persistenceOptions = {
	adapter: idbAdapter,
	autosave: truep,
	autosaveInterval: 900, // 900ms.
	autoload: true,
	throttledSaves: false,
};
wdb.initializePersistence(persistenceOptions)
.then((r) => {
	console.log('Done loading.');
	const collection = db.addCollection(`Test_500`);
	collection.insert({ foo:'bar' });
	setTimeout(() => {
		// So autosave has run
		db.removeCollection('Test_500');
		setTimeout(() => {
			// so autosave has had time to do it's thing again
			console.log('Collections in db:', db._collections.length);
			console.log('Collections last persisted', db._persistenceAdapter._dbref._collections.length);
			console.log('Check DevTools  Application storage. The DB should be EMPTY.');
		}, 2000);
	},2000);
})

What is the motivation / use case for changing the behavior?

In my mind, deletion of data should "stick", and not show up again if the browser is refreshed. In my project I can work around it by 'clearing' a collection instead of deleting and re-creating it, but this does feel like a bug that should be tracked.

Environment

LokiDB version: 2.1.0
Browser/Node version: Browser X.Y.Z.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[x] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

I would like to:

a) Create my own custom adapter (IndexedEncryptedStorage) and use it
b) Submit a PR to add a new adapter to this project (I already have it written)

Something like this would be ideal

https://github.com/techfort/LokiJS/wiki/LokiJS-persistence-and-adapters#creating-your-own-reference-mode-persistence-adapter

I'm submitting a...

[x] Bug report

Current behavior

Using findOne on indexed collections (using default 'js' comparator) results in error:

TypeError: this.comparator is not a function at AvlTreeIndex.locate (<path>/node_modules/@lokidb/loki/webpack:/@lokidb/loki/packages/loki/src/avl_index.ts:849:20)

I thought that the 'js' comparator was default, so I would assume this to work out-of-the -box. Meaning I do not have to add anything to comparatorMap.

Collections without indexes don't have this problem.

So, either it's a bug, or I'm doing it wrong. But that is difficult to know due to the lack of documentation in this project.

Expected behavior

Creating a collection with an index should be searchable

Minimal reproduction of the problem with instructions

Create and search a collection like this:

FSStorage.register()
const db = new Loki('<path>/data`)
db.initializePersistence({ adapter: new FSStorage(), autosave: true, autoload: true })

interface Person {
    name: string
    email: string
}

const coll = db.addCollection<Person>('person', {
    rangedIndexes: {
           name: { indexTypeName: 'avl', comparatorName: 'js' }
     }
})

coll.insertOne({ name: 'Herbert', email: 'mail@com' })
coll.findOne({ name: 'Herbert' })

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request

Current behavior

This error shows up after saving 500,000 rows, then refreshing the browser and trying to save the next set of rows. Code is simple to replicate.

Error Code

Uncaught (in promise) TypeError: this._changes is undefined
    _createChange collection.ts:1148
    _createInsertChange collection.ts:1158
    _insertMetaWithChange collection.ts:1167
    insertOne collection.ts:726
    insert collection.ts:665

Expected behavior

No error, rows inserted into collection.

Minimal reproduction of the problem with instructions

https://jsbin.com/nejijiberu (Copy to local dev server if needed. it's hard to make LokiDB work on jsBin) Same code as below

		let testCollection; // we will assign it after persistence
		IndexedStorage.register();
		PartitioningAdapter.register();

		let idbAdapter = new IndexedStorage(dbName);
		let paAdapter = new PartitioningAdapter(
			idbAdapter,
		 	{
				paging: true, 
				pageSize: 5*1024*1024 
			}
		);

		wdb = new Loki(dbName);

		let persistenceOptions = {
			adapter: paAdapter,
			autosave: window.self === window.top,
			autosaveInterval: 1000,
			autoload: window.self === window.top,
			throttledSaves: true,
			persistenceMethod: 'indexed-storage',
		};

		wdb.initializePersistence(persistenceOptions)
		.then(() => {
			console.log('Loki Persistence initialized');
			// Initialize collection
			testCollection = wdb.getCollection('Test');
			if (!testCollection) testCollection = wdb.addCollection('Test');
			// Display number of rows IN collection
			const oldRecords = testCollection.find();
			console.log('Number of rows loaded from IndexedDB collection:',oldRecords.length);
			// Use verbose method to get maximum id of existing records.
			const oldIds = oldRecords.map((r) => parseInt(r.id));
			const oldIdsLength = oldIds.length;
			let maxId = oldIds[0] || 0;
			for (let i = 1; i < oldIdsLength; ++i) {
				if (oldIds[i] > maxId) {
					maxId = oldIds[i];
				}
			}
			// Fetch placeholder data
			fetch('https://jsonplaceholder.typicode.com/photos') 
				.then(response => response.json())
				.then(json => {
					console.log('Enlarging result set from jsonplaceholder...');
					let bSet = json;
					// Make set larger
					bSet = [
						...bSet,...bSet,...bSet,...bSet,...bSet,
						...bSet,...bSet,...bSet,...bSet,...bSet,
					];
					bSet = [
						...bSet,...bSet,...bSet,...bSet,...bSet,
						...bSet,...bSet,...bSet,...bSet,...bSet,
					];
					// bSet = [
					// 	...bSet,...bSet,...bSet,...bSet,...bSet,
					// 	...bSet,...bSet,...bSet,...bSet,...bSet,
					// ];
					console.log('Cloning',bSet.length,'records...');
					bSet = JSON.parse(JSON.stringify(bSet)); // clone so we can reset id field
					console.log('Resetting',bSet.length,'id fields, starting at',maxId+1);
					for (i = 0; i < bSet.length; i += 1) bSet[i].id = i+maxId+1; // reset id field
					console.log('Adding',bSet.length,'rows from jsonplaceholder into LokiDB...');
					testCollection.insert(bSet);
					console.log('Added rows from jsonplaceholder. Refresh to see if they stick around.');
				})

		})
		.catch((e) => console.error('initializePersistence error', e.message));

What is the motivation / use case for changing the behavior?

I need to save large numbers of records (about 2 mil) across 80 or so collections. The above bug was encountered while I was attempting to make an easy way to replicate a secondary bug.

The secondary bug: With 80 or so collections and ~2M records saved (and showing up in both IndexedDB and in memory), reloading the browser would cause the collections to vanish, even though I can still see the data sitting in IndexedDB. I figured it'd be best to focus on the primary bug first, then try to figure out why something between LokiDB, IndexedStorage, and PartitioningAdapter was failing to autoload the 80+ collections.

Environment

LokiDB version: 2.1.0

Exact same error in Firefox 81.0 and Chrome 85.0.4183.121.

I'm submitting a...

[x] Feature request

Current behavior

There are no integration tests (node require/import, browser <script>) after the library is bundled.

Expected behavior

Test exported function/classes in the bundled library.

What is the motivation/use case for changing the behavior?

Verify bundled library.

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Currently if I have a record like this:

{
  _id: 1,
  key: {
    subkey: 'some sub key 1',
  }
}

I can't seem to query for the subkey using any syntax options that I can come up with. I expected this to work with dot notation similar to the original Lokijs or Mongo. An example query would be collection.findOne( { 'key.subkey': 'some sub key 1' } )

Expected behavior

I would expect dot notation to provide a result when querying nested keys.

Minimal reproduction of the problem with instructions

https://next.plnkr.co/edit/YjcUXu94hbCIVdYv?open=lib%2Fscript.js

What is the motivation / use case for changing the behavior?

Environment

Node 8.10/Latest Chrome
LokiDB version: 2.0.0-beta.6

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiJS2/blob/master/CONTRIBUTING.md#question

Current behavior

All types (d.ts) for each package will be exported.

Expected behavior

Only necessary types should be exported.

Minimal reproduction of the problem with instructions

npm run build

What is the motivation / use case for changing the behavior?

Minimize code bloat.

Environment

LokiJS version: trunk


Browser/Node version: Browser any

Others:

I'm submitting a...

[ ] Regression
[ ] Bug report
[x] Feature request
[ ] Documentation issue or request
[ ] Support request

[Feature] MongoDB-style aggregation pipeline

So that we can have nice things like $group (advanced counting).

Mingo does it nicely, although it lacks the performance tweaks and indexes.

I'm submitting a...

[x] Feature request

Current behavior

I removed the deprecated function Object.observe from LokiDB.

Expected behavior

Investigate if Proxy could be an alternative to Object.observe so that Collection autoupdate works like in LokiJS.

I'm submitting a...

[X] Feature request
[X ] Documentation issue or request

As far as I can see from the code, LokiDb only works in sync methods, in my current work I can see some io blocking (2sec ~) while inserting just 250 entries, this is a bit troubling, as these are not a lot of records and in the future, I will need to insert around 2000.

Is there something I'm missing here? is there a way of making async operations?

I can't wait any longer. 😸

Querying nested properties with the dot-notation does not seem to work currently, since Resultset.find() treats property as a simple Object key. Has this changed compared to LokiJS?

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/LokiJS-Forge/LokiDB/blob/master/CONTRIBUTING.md#question

Current behavior

Internal token results are not cached. So an equal or similar full-text search query cannot be optimized from the previous query.

Expected behavior

• Cache token results of a full-text search query.
• Make option optional.
• Clear cache if any document is dirty.
• Maybe LokiDB query results can be cached too.

What is the motivation/use case for changing the behavior?

Better performance for search bars, if only some words have been changed.

I'm submitting a...

[x] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[x] Temporary fix / possible long term fix

Current behavior

Some filters used in find() does not work as it should with LokiJS or MongoDB. For example,

{$and: [
    {tag: "marked"},
    {$or: [
        {nextReview: {$exists: false}},
        {nextReview: {$in: [null, ""]}},
        {nextReview: {$lt: new Date()}}
    ]}
]}

What is the motivation / use case for changing the behavior?

I am not writing the filter object directly, but by parsing a string with parsimmon https://github.com/patarapolw/zhdiary-app/blob/master/src/backend/db/MongoQParser.ts.

Temporary fix? MongoToFilter function, and use JavaScript's filter in .where().

Environment

LokiDB version: 2.0.0-beta.8 with @lokidb/fs-storage 2.0.0-beta.8
Browser/Node version: Node v11.11.0 Yarn 1.15.2 MacOS

Trying to determine if it is ready for use in a production app.

Thanks!

Hey guys,

I just started defining the project structure of LokiJS2. Its the first time I am doing this, so maybe someone can help me out.

What I used so far:

• jsdoc (documentation)
• eslint (coding/formatting style, not configurated yet)
• webpack and babel
  • bundle src-files to one file (src/** -> lib/)
  • solve ES6 imports with babel
• karma + jasmin (test platform + unit test framework)
  • uses karma-webpack to enable import .. form ...
• uglify-js with specific branch to harmony -> supports ES6
• istanbul
• benchmark

What is missing:

• .travis.yml, bower, npm
• phantomjs (currently does not support ES6).

For me open questions:

• Do we want to code ES6 and transpile to ES5 (with babel)? For me: Just ES6.

Please feel free to give helpful hints/commits for the initial project structure. ;)

Started with: techfort/LokiJS/issues/525