Datastore refuses 'string' properties greater than 1500 characters. It will accept 'text' properties of any length. How do we set the property type to 'text'?

> var Test = require('./server/apis/test.model.js')
datastore.js
shioh.productArray.model.js
undefined
> var str = ''
undefined
> for (let i = 0; i<1502; i++){str = str.concat('a')}
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

> var entity = new Test({text : str})
undefined
> entity
ModelInstance {
  className: 'Entity',
  schema:
   Schema {
     instanceOfSchema: true,
     methods: {},
     statics: {},
     virtuals: {},
     shortcutQueries: {},
     paths: { text: {} },
     callQueue: { model: {}, entity: {} },
     options: { validateBeforeSave: true, queries: [Object] },
     __meta: {} },
  excludeFromIndexes: [],
  entityKey: Key { namespace: undefined, kind: 'test', path: [Getter] },
  entityData: { text: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaa' },
  pre: [Function: pre],
  post: [Function: post],
  hook: [Function: hook],
  __setupHooks: [Function: __setupHooks] }
> entity.save()
Promise { <pending> }
> (node:12000) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: The value of property "text" is lo
nger than 1500 bytes.

Isn't this inconsistent?

Model.findOne({foo: 'bar'}, (err, entity) => {
  // entity != Array
})

versus

Model.findOne({foo: 'bar'}).then((entities) => {
  // entites == Array
})

const studentSchema = new Schema({
name: { joi: Joi.string().required() },
scores:{ joi: Joi.array().required(), excludeFromIndexes: true }
})

I get this error

{ Error: Exclude from indexes cannot be set on a list value
at /Users/Mike/dev/current/datastore/node_modules/grpc/src/node/src/client.js:554:15 code: 400, metadata: Metadata { _internal_repr: {} } }

I'm reading the gstore-node docs & I'm not clear on how the __override property on discussed here https://sebelga.gitbooks.io/gstore-node/content/middleware-hooks/pre-hooks.html works (and what is it for)?

can you elaborate & give an example?

If I have a array

"test": [{"organizationName":"strge","familyName":"raju"},{"organizationName":"sand","familyName":"sand"}]

want to filter in organizationName=strge and familyName=sand and get only if both filters match in particular object not in different object of a array. #

I'm trying to use a custom validator to verify that a field is an array containing one or more objects. However, the custom validator function is getting passed a string as input instead of the original array being passed in. It seems as if the array is having .toString() called on it before it's passed into the validator. However, if I remove the custom validator function then the correct values are saved to the database.

Code, output, package, and node version information below.

Schema and validator:

const itemSchema = new Schema({
    name: { type: 'string', required: true },
    arrOfObjs: {
        type: 'array',
        validate: {
            rule: validateArrOfObjs,
        }
    },
    createdOn: { type: 'string', default: gstore.defaultValues.NOW, write: false, read: false }
});

function validateArrOfObjs(obj, validator) {
    console.log("Input type:");
    console.log(typeof obj);
    console.log("Input is array:");
    console.log(Array.isArray(obj));
    console.log("Array:");
    console.log(obj);
    console.log("First item:");
    console.log(obj[0]);
    console.log("First item type:");
    console.log(typeof obj[0]);

    if (!Array.isArray(obj)) {
        return false;
    }

    var allObjs = obj.every((item) => {
        return item !== null && typeof item === 'object';
    });

    return allObjs;
};

const Item = gstore.model('Item', itemSchema);

Testing code:

var arr = [
  { name: 'foo' },
  { name: 'bar' },
  { name: 'baz' }
];

var item = new Item( {name: 'test123', arrOfObjs: arr} );

console.log("Array:");
console.log(item.arrOfObjs);
console.log("Array first item:");
console.log(item.arrOfObjs[0]);
console.log("Array first item type:");
console.log(typeof item.arrOfObjs[0]);

console.log("Validating!");
const { error } = item.validate();
if (error !== null) throw error;

item.save();

Output:

Array:
[ { name: 'foo' }, { name: 'bar' }, { name: 'baz' } ]
Array first item:
{ name: 'foo' }
Array first item type:
object
Validating!
Input type:
string
Input is array:
false
Array:
[object Object],[object Object],[object Object]
First item:
[
First item type:
string

yarn package.json

{
  "name": "gstore-poc",
  "version": "1.0.0",
  "description": "POC for gstore library",
  "main": "index.js",
  "scripts": {
    "api": "babel-node server.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "babel-preset-env": "^1.6.1",
    "body-parser": "^1.18.2",
    "express": "^4.16.2",
    "gstore-node": "^2.1.2",
    "mongoose": "^5.0.0-rc1",
    "morgan": "^1.9.0"
  },
  "devDependencies": {
    "babel-cli": "^6.26.0",
    "babel-core": "^6.26.0"
  },
  "babel": {
    "presets": [
      "env"
    ]
  }
}

Node version:

node --version
v9.3.0

It would be awesome to be able to plug in a custom Promise library (i.e. bluebird), just like you would with mongoose.

I know you can set a custom lib for google-cloud/datastore, but it would be even cooler to have this feature for gstore-node

Sorry to be blowing up your issues list but when I launch my app with these two lines:

var gcloud = require('gcloud')(configGcloud);
var datastore = gcloud.datastore({"apiEndpoint": "127.0.0.1:8343"});

It loads in about 2 seconds, if I add:

var gstore = require('gstore-node');

it goes up to 10-15 seconds, any idea why it takes so long to import or what I'm doing wrong? Thanks!

It had taken me while to figure out how to get it working and was thinking a example with returning objects they can use would be very helpful to new comers. In the Gitbook it is on the models/methods/GET documentation, but I think that adding a example to map out the entities and use plain() would be a very common requirement for people. It would also be good to add it to the readme where the other function calls are. Me a year ago probably would have never figured it out :)

My function for fetching based on a array of _id's and returning objects
If there is a more performant way to do this please let me know!

export async function getPagesBy_id(_ids) {
  let response = null;

  try {
    response = await Page.get(_ids, null, null, null, { preserveOrder: true })
      .then((entities) => {
        return entities.map((entity)=> {
          return entity.plain();
        });
    });

  } catch (err) {
    console.log('getPagesBy_id err', err);
  }

  return response;
};

Hi,

Love what's been done here, this is just the kind of thing I've been looking for after moving from Java (using the objectify library for Google datastore) into Javascript Node development and finding the implementation a bit bare-bones in the base google-cloud library. The schema and model concepts really make it easy to work with.

The only thing I'd really love to see added is the option of having Promises returned instead of using callbacks (on the relevant such methods), such as the Model methods get, updated, delete etc. It could be added using a different name such as BlogPost.getAsync(1234). I like to make use of async functions and combined with promises it makes things amazingly easy to reason about.

For now I'll have to wrap all the base methods in Promises myself, but would be nice to have it out the box.

In model.js:824, the type of the property is validated. However, if the value of said property is equal to undefined, the property validation fails.

Hi,
I'm having some issues working with the library. Maybe I'm initializing in a bad way the gstore-node?

Saving is not working, or nothing happens at all when this method is executed:

    var data = ClientApp.sanitize(req.body);
    console.log(data);
    var clientApp = new ClientApp(data,data.appName);

    clientApp.save().then(function() {
        this.datastoreEntity().then((data) => {
            var entity = data[0];
            console.log(entity);
        });
    }).catch((err) => {
        console.log(err);
    });

Here are the files:
clientApp.controller.js

var ClientApp = require('../models/clientApp');

function create(req, res) {
    var data = ClientApp.sanitize(req.body);
    console.log(data);

    // {
    //     appName: 'Yikes',
    //     androidBundleId: 't.t.t',
    //     iosBundleId: 'a.a.a'
    // }

    var clientApp = new ClientApp(data, data.appName);

    var data = ClientApp.sanitize(req.body);
    console.log(data);
    var clientApp = new ClientApp(data,data.appName);

    clientApp.save().then(function() {
        this.datastoreEntity().then((data) => {
            var entity = data[0];
            console.log(entity);
        });
    }).catch((err) => {
        console.log(err);
    });
}


module.exports = {
    create: create
};

(Model)
clientApp.js

var uuid = require('uuid');

var datastore = require('@google-cloud/datastore')({
    projectId: 'tttttttt',
    keyFilename: './keyfile.json'
});

var gstore = require('gstore-node');
gstore.connect(datastore);

var Schema = gstore.Schema;

var schema = new Schema({
    appName: {
        type: 'string',
        required: true,
        read: false
    },
    androidBundleId: {
        type: 'string',
        required: true,
        read: false
    },
    iosBundleId: {
        type: 'string',
        required: true,
        read: false
    },
    modified: {
        type: 'boolean',
        default: false,
        read: false,
        write: false
    },
    createdOn: {
        type: 'datetime',
        default: gstore.defaultValues.NOW,
        write: false,
        read: false
    },
    appKey: {
        type: 'string',
        default: uuid.v4(),
        write: false
    }
});

schema.virtual('timestamp').get(function() {
    // the scope (this) is the entityData object
    return this.createdOn;
});

var ClientApp = gstore.model('ClientApp', schema);

module.exports = ClientApp;

STR

const gstore = require('gstore-node');
const datastore1 = require('@google-cloud/datastore')({
    projectId: 'my-google-project-id',
   namespace: `1`
});

// Then connect gstore to the datastore
gstore.connect(datastore1);

const datastore2 = require('@google-cloud/datastore')({
    projectId: 'my-google-project-id',
   namespace: `2`
});
gstore.connect(datastore2);

The second datastore datastore2 overwrites the first one and therefore calls to delete operation, for example, on datastore1 are done on namespace2 instead and therefore fails. The issue is that we export new Gstore at https://github.com/sebelga/gstore-node/blob/master/lib/index.js#L135 instead of ust Gstore so there is no way to create two separate instances of Gstore

Hi @sebelga,

First, thanks for this beautiful project. Eases the implementation by 400%. You're awesome, keep up the good work (and have a Happy New Year 🎉).

What's the issue:

I read that you use the validator.js - we can even see it in action in this line - but that implementation only supports simple validation rules that don't expect additional parameters (e.g. the isIn rule).

How could it be solved:

I suggest either to pass something extra on the object like:

{
  validate: 'isIn',
  validation_args: [ /* an array of multiple args */ ],
  // ...
}

or change the validate prop to support both string, object/array:

// simple case
{
  validate: 'isEmail'
}

// complex case
{
  validate: {
    rule: 'isIn',
    args: [ /* ... */ ]
}

// another approach
{
  validate: ['isIn', arg2, arg3, arg4, ..., argN]
}

Or just say, "No we only support simple cases" :)

As google-cloud datastore lib has changed their entity response format from
{ key: someKey, data: someData }
to
{ entityData }
in @google-cloud/datastore 0.5.0 (The change is actually in this commit ), this clearly will breaks model creation. For example, trying to fetch an entity with a key would result in the following error as entity.key in the response is no longer presence.

/MyProject/node_modules/gstore-node/lib/serializers/datastore.js:23

 data.id  = entity.key.path[entity.key.path.length - 1];

                      ^

TypeError: Cannot read property 'path' of undefined

May I ask whether there is any plan to support datastore 0.5.0 soon?

If you call the Model.get method with an array consisting of only one ID, nothing seems to be returned.

For example, calling the following returns an array of 2 entities:

MyModel.get([10, 15])

However calling the following doesn't return anything:

MyModel.get([10])

Because of this function:

function parseId(id) {
    return isFinite(id) ? parseInt(id, 10) : id;
}

The call User.get("1") tries to find an entity with a numeric ID instead of a String ID.
This is great for a codebase written from scratch, but causes problems when dealing with legacy.

Are there any plans to add support for named keys? It seems that createKey is always expecting ID to either be a numeric string or number, but I would like to be able to create an entity that builds keys using the string without parsing it. It would be nice to define the type of key when defining a schema.

On a related note, since the ID is being parsed with parseInt, IDs that are larger than 2⁵³ -1 will not parse properly 😢 . This could be a problem for anyone looking to interoperate with IDs generated from languages that properly support 64-bit integers. Instead of parsing as JS numbers, they should be wrapped with datastore.int().

order does not work in a datetime field

 const ProjectSchema = new Schema({
    name:     {type: 'string'},
    members:  {type: 'object'},
    createOn: {type: 'datetime', default: gstore.defaultValues.NOW},
    updateOn: {type: 'datetime', default: gstore.defaultValues.NOW},
    files: {type: 'object'}
  });

I tried the following query

const query =
        db
          .ProjectModel
          .query()
          .order('updateOn', {
            descending: true
          })
          .limit(20);

It is not delivered in the expected order.

Hello. I have the following schema for a Pose model:

{
  name: {type: 'string', required: true},
  skeleton: {values: ['HumanBody', 'HumanHand'], required: true},
  ...
}

When I try to create a new entity without skeleton attribute, it's automatically picking HumanBody as the default value:

console.log(new Pose({name: 'Whatever'}))
// will output: {name: 'Whatever', skeleton: 'HumanBody'}

Thus, the required: true option has no effect because it's setting a default value.

This behavior is confusing, and also undesired in my case, as I need the attribute to be explicitly defined, raising a ValidationError when not given. How can I achieve this?

Thanks.

I have been investigating a failing test in our test suite and am struggling with a test that basically validates some params and calls Model.update(). The call to Model.update() ends up throwing an unhandled rejection:

UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property '0' of undefined

After some debugging I found the error being thrown at model.js:258 (onEntityUpdated), where updateData is undefined. This sort of makes sense as model.save() doesn't return the created entity, but this seems like a common use case and I figured the tests would cover a case like this.

I made a bare test that has this issue. DevApp has just 2 fields: appName: string, type: number:

const newName = 'New Test App Name';
const fApp = new DevApp({appName: 'f-app', type: 0});

return fApp.save(function (err) {
  assert.ifError(err);
  assert.equal(fApp.appName, 'f-app');
  return DevApp.update(fApp.entityKey.id, {appName: newName})
    .then((retApp) => {
      assert.equal(retApp.appname, newName);
      done();
    });

This is a bit contrived as I couldn't get a sane, promise-only version working either:

return fApp.save()
  .then(() => {
    return DevApp.update(fApp.entityKey.id, {appName: newName});
  })
  .then((retApp) => {
    assert.equal(retApp.appname, newName);
    done();
  })
  .catch(console.error);

This errors with: { code: 400, metadata: Metadata { _internal_repr: {} } }

I would like to be able to save a model, update it, and see the changes updated (all in a promise chain).

Am I doing something unreasonable or wrong? Any help would be appreciated.

As far as I can see now, results from a query are the raw results from the query executed by the underlying gcloud-node library? What would be the preferred way to convert these entities back to gstore-node Model objects?

I.e my use case is the following roughly:
query(..., (entities) => { // do something with entities, update some properties entity.save(); });

For one of my kinds, I store custom string as the ID in the format of "user1:user2" (where user1 and user2 are numerical IDs) for easy and fast retrieving. To create this string, I'm using util.format and passing it to the second parameter of a new model instance. When I save this to the datastore and check to see how it saved, it shows only "user1" as a numerical ID. How can I force gstore to save the ID as a string even though one is passed to the new instance?

For now, I'm saving through gcloud which works fine.

Similar to #1, but with getting instead. Getting an object by number ID works fine, so does a regular string name (e.g: 'testobject'). However, in my case, a string name of "12345:12345" returns 404 every time through gstore-node. Tested in gcloud works successfully.

Thank you

Hello,

First of all thank you for your work.
I would have liked to know if you will soon migrate to typescript?

Thank you in advance

The line:

For info on how to configure gcloud read the docs here.

has a broken link, thanks!

From the code it looks like the global save https://github.com/sebelga/gstore-node/blob/master/lib/index.js#L100 is a wrapper around google-cloud/datastore's save method and doesn't use transactions.

Today I found super strange behavior

When I start the application with pm2, the save method returns undefined

Any idea that it may be happening?

For an entity like below?

id: string
results: {
  "success_rate": 90.0,
  "failure_rate": 50.5
}

I'd like to send documentation improvements, but cannot find where it is stored in this repo.

The code for // from inside a Transaction in https://sebelga.gitbooks.io/gstore-node/content/model/update-method.html is missing an {.

I get an error below when deleting a table with more than 500 entities. This is because the deleteAll function seems to be fetching the entries first and then trying to delete them in one go instead of breaking them down to a batch of 500. Workaround is do the same by the caller but that makes deleteAll function unnecessary.

cannot write more than 500 entities in a single call

Can't find any effect of this. Model shows excludeFromIndexes

The field shows that it is indexed. (No other properties falsely show it)

But then its values all state not indexed. (Note the array content type)

Not sure if its anything to pay attention to as it is not stored in the indexs (n0deEdge).
The note above showed that the content type of n0deEdge is an array (which is correct), and below it shows it is a string.

With the new gcloud-node update, they changed the whole way transactions are handled.

This means the runInTransaction alias is no longer valid, and the internal transaction handling should be updated to support gcloud v0.37.0

Hi there,

I see you've got the equivalent of the datastore.save( entity ) method, which we can access in this library the other way round with entity.save(); - returning a promise or passing in a callback if we like to save a single element.

I'm wondering if there is anything which would be the equivalent of datastore.save( entities[] )

For example, in my code:

async function resaveEntities(model, cursor = null) {
  const query = model.query().limit(50);

  if (cursor) {
    query.start(cursor);
  }

  const result = await query.run();
  const response = result[0];

  log.info(`fetched ${response.entities.length} entities of ${model.entityKind}`);

  // CODE HERE TO RE-SAVE ALL THE FETCHED ENTITIES

  if (response.nextPageCursor) {
    resaveEntities(model, response.nextPageCursor);
  } else {
    log.info(`END of re-saving of [${model.entityKind}]`);
  }
}

The reason I ask is because if I'm not mistaken, batch saving large groups of entities (as I'm trying to do here, basically running through all the entities of a kind and re-saving them- in order to re-index their properties if I changed the schema) can be optimised when its done in groups instead of individually every time.

I suppose I could use the google-cloud-node library to re-save them, but would I not lose the schema then? When I look at the data that is being returned in the query, it appears that it's just a bunch of plain data- no extra schema information or even functions so I'm unsure if entity.save() would actually even work either.

I'm going to play around with it for a bit. But let me know if there is a solution I may be overlooking.

I've been trying to query and re-save a bunch of entities lately, I noticed something that causes a little friction.

When doing data validation for an entity property which has a type of gcloud.datastore.geoPoint, it will fail, even if the structure of the date is completely correct. Such as:

{
  latitude:  37.305885314941406,
  longitude: -89.51815032958984
}

It would be great if it allowed for both of these use cases- or perhaps auto-converted the data to the correct gcloud.datastore.geoPoint object automatically if the structure is fine. As the data returned from a query is just plain and it doesn't convert back easily.

Im playing around with GAE, datastore and came across this package. Taking this along lines of mongoose is really good.

One of the major complaints of GAE datastore seems to be you will be locked into google-eco-system and cant migrate to other database engines.

So what do you think about having a 'new or this' package which will act as an abstraction layer on top of datastore, mongoose and mysql. One package - talk to mongoose or datastore or mysql

This means an application 'migration' is easy and more users.

I havent used mongoose or datastore intensively - so wanted to know what you feel about this proposal ?

Hey there! I think this is a bug, as the documentation does not mention having to create indexes for queries.

I have an example model that looks like this:

const gstore = require('gstore-node')();
const Schema = gstore.Schema;

const itemSchema = new Schema({
    name: { type: 'string', required: true },
    arrOfObjs: {
        type: 'array',
        validate: {
            rule: validateArrOfObjs,
        }
    },
    createdOn: { type: 'string', default: gstore.defaultValues.NOW, write: false, read: false }
});

function validateArrOfObjs(obj, validator) {
    if (!Array.isArray(obj)) {
        return false;
    }

    var allObjs = obj.every((item) => {
        return item !== null && typeof item === 'object';
    });

    return allObjs;
};

const itemQuerySettings = {
    order : { property: 'createdOn', descending: true }, // descending defaults to false and is optional
    select : ['name', 'createdOn']
};

// Add settings to schema
itemSchema.queries('list', itemQuerySettings)

const Item = gstore.model('Item', itemSchema);
export default Item;

If I create a few items and then try to list them with the following endpoint:

const listItems = (req, res) => {
  Item.list()
    .then((response) => {
      res.json({
        entities: response.entities,
        nextPageCursor: response.nextPageCursor
      });
    });
};

I get the following error:

(node:14987) UnhandledPromiseRejectionWarning: Error: no matching index found. recommended index is:
- kind: Item
  properties:
  - name: createdOn
    direction: desc
  - name: name

    at ./node_modules/grpc/src/client.js:554:15

On node v9.3.0, Yarn package.json:

{
  "name": "gstore-poc",
  "version": "1.0.0",
  "description": "POC for gstore library",
  "main": "index.js",
  "scripts": {
    "api": "babel-node server.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "babel-preset-env": "^1.6.1",
    "body-parser": "^1.18.2",
    "express": "^4.16.2",
    "gstore-node": "^2.1.4",
    "mongoose": "^5.0.0-rc1",
    "morgan": "^1.9.0"
  },
  "devDependencies": {
    "babel-cli": "^6.26.0",
    "babel-core": "^6.26.0"
  },
  "babel": {
    "presets": [
      "env"
    ]
  }
}

Hi!

I guess this is more of a question than an issue, but is there any easy way to convert these to JSON objects that can be returned by express routes? Thank you!

I am new to both datastore and gstore-node. Having a hard time figuring out how to write the schema for one-to-many and many-to-many relationships without using hierarchal relationships.

If we take a small example of a User having many Addresses, how should the schema look ?

const userSchema = new Schema({
    name: { type: 'string' }
});

const addressSchema = new Schema({
    city: { type: 'string' }
});

I apologize for posting a question here.

First off sweet project cant wait to get it working!

I am trying to get the minimal setup working and when I run my seed function to add a page to Datastore I get: cannot read property 'key' of undefined.

I have tried every different variation I could from going through the docs, and not sure if there a way I can define the key on a field in the model.

My create function
Note

• that I am passing a object directly to it so i changed req.body to req.
• The id I want to use is _id field on my model(I have this working without the use of gstore-node). I also chose to use _id because I would have naming conflict with my frontend relay/graphql
• Console.log does output the correct _id string of entityData._id

/* eslint no-unused-vars: 0 */
const gstore = require('gstore-node');
const Page = require('../../../models/datastore/pageSchema');

const createPage = (req, res, next) => {
  const entityData = Page.sanitize(req);
  const page = new Page(entityData, entityData._id);

  page.save()
    .then(entity => res.json(entity.plain()))
    .catch(err => next(err));
};

And the full error:

H:\Coding\Projects\react-starter-kit\node_modules\gstore-node\lib\entity.js:158
    return namespace ? self.gstore.ds.key({ namespace, path }) : self.gstore.ds.key(path);
                                                                               ^

TypeError: Cannot read property 'key' of undefined
    at createKey (H:\Coding\Projects\react-starter-kit\node_modules\gstore-node\lib\entity.js:158:80)
    at ModelInstance.Entity (H:\Coding\Projects\react-starter-kit\node_modules\gstore-node\lib\entity.js:25:30)
    at Model (H:\Coding\Projects\react-starter-kit\node_modules\gstore-node\lib\model.js:21:1)
    at ModelInstance (H:\Coding\Projects\react-starter-kit\node_modules\gstore-node\lib\model.js:23:31)
    at createPage (H:/Coding/Projects/react-starter-kit/src/data/queries/googleDatastore/page/controller.js:23:19)
    at Object.<anonymous> (H:/Coding/Projects/react-starter-kit/src/data/seed/index.js:20:1)
    at Module._compile (module.js:570:32)
    at loader (H:\Coding\Projects\react-starter-kit\node_modules\babel-register\lib\node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (H:\Coding\Projects\react-starter-kit\node_modules\babel-register\lib\node.js:154:7)
    at Module.load (module.js:487:32)
error Command failed with exit code 1.

My model

const gstore = require('gstore-node');

const Schema = gstore.Schema;

const pageSchema = new Schema({
  _id: {
    type: 'string',
    required: true
  },
  path: {
    type: 'string',
    optional: true
  },
  public: {
    type: 'boolean',
    required: true,
    default: false,
    excludeFromIndexes: true
  },
  ... rest of fields
});

const listSettings = {
  limit: 25,
  order: { property: '_id' },
};

pageSchema.queries('list', listSettings);

module.exports = gstore.model('Page', pageSchema);

By running the following code:

      db
          .ProjectModel
          .query()
          .select(['name', 'updateOn']);

You get the following error

Error: no matching index found. recommended index is:<br/>- kind: Project<br/>  properties:<br/>  - name: name<br/>  - name: updateOn<br/>
    at /Users/cristyansepulveda/visualive_website/node_modules/grpc/src/node/src/client.js:569:15 code: 412, metadata: Metadata { _internal_repr: {} } }

model:

const ProjectSchema = new Schema({
    name:     {type: 'string'},
    updateOn: {type: 'datetime', default: gstore.defaultValues.NOW},
  });

Hello,

while looking at the code to investigate why Gstore was returning only the first error as opposed to all of them (like ORMs normally do), I found this piece of code inside validateEntityData() in model.js:

                if (!validate.success) {
                    delete validate.success;
                    return validate.errors[Object.keys(validate.errors)[0]];
                }

As a result, Model.save() on validation failure returns only the first error.

Is there a reason for that? Couldn't we just return validate.errors in its entirety?

Seems like it should validate for "required: true" in the schema. validate.js doesn't accept nulls or zero-lengths so maybe those checks should be at the top of model.validate() and flip skip if value is null or zero-length after trimming.

I have upgraded my Node.js from 4.x.x to 6.11.0 and got a problem now. The entity method save() doesn't seem to work for me anymore. The code below throws this message in console: (node:7184) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'then' of undefined.

const faker = require('faker');
const User = require('./../models/user.model');
const entity = new User({
    displayName: faker.name.firstName() + ' ' + faker.name.lastName(),
    username: faker.internet.userName()
});

entity.save()
    .then(() => {
        res.send('OK');
    })
    .catch((err) => {
        res.send('Not OK');
    })
;

Looks like the method is not returning a promise, so the entity doesn't get saved in the datastore. On the other side if I use the general gstore method for saving, it works perfectly.

...

var gstore = require('gstore-node');

gstore.save(entity)
    .then(() => {
        res.send('OK');
    })
    .catch((err) => {
        res.send('Not OK');
    })
;

Is there a problem with my code or the library needs to be fixed? Thanks.

As a user of mongoose, wanting to start switching to google cloud datastore, this library is ideal for me, thank you so much for making it.

So I created a connection:

var gstore = require('gstore-node');
var gcloud = require('google-cloud')(config.googlecloud);
gstore.connect(gcloud.datastore());

Then I created a schema and made it into a model:

var gstore = require('gstore-node');
var Schema = gstore.Schema;

var urlSchema= new Schema({
    name: { type: 'String', excludeFromIndexes: true },
    subdomain: { type: 'String' }
});

module.exports = gstore.model('url', urlSchema);

I don't know what step I might be missing in the docs, but when I try to query my new model:

var Url = require('./url.model');
Url.query().run(function (error, results) {
    console.error(error);
    console.log(results);
});

I get the above error (it doesn't even make it to the console.error line. Specifically, it points to line 924 of model.js in gstore-node:

return self.ds.createQuery.apply(self.ds, createQueryArgs);

For some reason, the datastore is supposed to be passed in when creating the model, but is not. Given that there is a line 924 of model.js in the lib/ folder or gstore-node, at this point I'm pretty sure I'm in over my head and could use some guidance.

Did I do something wrong in the process of creating my model? Note that if I dump the Url object, it has a ds property with a datastore. object on it. In fact, if I dump the self object in model.js, It contains the Url complete with the attached datastore, but for some reason self.ds is undefined.

When giving a non-existent key to model.get(), I get

UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'Symbol(KEY)' of undefined

and I cannot catch the rejection of Promise, like

User.get('NON-EXISTENT-KEY')
.then((entities) => {
  // an entity found
})
.catch((err) => {
  // never happen
})

I am using gstore-node 0.8.0, google-cloud 0.44.2 and node v6.9.1. I guess something happening here?
https://github.com/sebelga/gstore-node/blob/master/lib/model.js#L110

Hello!

Does all the "save" operations works as upsert(which will overwrite an entity if it already exists in Cloud Datastore) or is there any way to use insert(which requires that the entity key not already exist in Cloud Datastore)?

Thanks in advance.

hello, i have an entity with the ff model.
model
first_name: { type: 'string' },
last_name: { type: 'string' },
address: { type: 'object' },

and as I save, I save the address as
{ number: 001, street: example street, City: example City }

AFTER I SAVE IT, i will update the City only...

how should i do that? because as i update it, it will overwrite all the values with
street: { City: overwriteValuehere }

i want it instead street: { number: 001, street: example street, City: overwriteValuehere }

tnx!

Hi, would like to know if anyone face this issue? All my properties default to Indexed even after I set excludeFromIndexes to true. Below is my schema and datastore gui.

how do I apply filter in key ?