Make PutMulti and DeleteMulti execute concurrent requests when over API limit. about nds HOT 15 CLOSED

Would this check both the number of records and the overall record size?

from nds.

I wrote this feature for goon, it's not as trivial as it sounds there's
lots of gotcha, particularly how you return on partial success.

On Fri, Apr 3, 2015, 2:25 AM Derek Perkins [email protected] wrote:

Would this check both the number of records and the overall record size?

—
Reply to this email directly or view it on GitHub
#6 (comment).

from nds.

@derekperkins I am not quite sure what you mean. appengine/datastore only allows you to GetMulti a maximum of 1000 entities in one go. PutMulti and DeleteMulti only allow 500 entities. Withnds.GetMultiit breaks the entities up into chunks of 1000 or less and then executes concurrently as many calls todatastore.GetMultias necessary to retrieve all the entities. This doesn't currently happen withnds.PutMultiandnds.DeleteMulti` but I would like to add code to make it happen.

In practical terms though if I were to try to nds.GetMulti 5000 entities, each 1Kb big then I would need to set my context timeout to 30 seconds from the default 5 seconds and even then would probably get timeout errors.

@mzimmerman is right about it being non trivial to implement. If he looks closely, he might see how similar my implementation is to his. Thanks for the help Matthew. 😜 I originally wrote my code with channels and realised Goon's looked much nicer.

from nds.

@mzimmerman I wasn't trying to make it sound trivial. We've looked at implementing this for a while now, but haven't precisely because it is complex.

@jongillham There are more reasons to chunk it up than just the record count. We have also run into issues where the total size of the PutMulti causes an error. I don't know if that is because it is the 10 MB transactional limit mentioned here https://cloud.google.com/appengine/docs/go/datastore/#Go_Quotas_and_limits or some undocumented limit.

We refer to that error internally as Too big for missiles, switching to guns, because we sometimes have to not use a PutMulti and instead do a lot of concurrent Puts. There's for sure some optimal middle ground there, but we haven't found it yet.

from nds.

@derekperkins I understand now. I've not run into that limit yet. I have a feeling that the maximum size an RPC response can be is 32Mb. So you would want to automatically chunk your Gets/Puts/Deletes if the size is > 10/32Mb or number of entities > 1000. That seems useful. It would almost certainly have to use some reflection to follow down struct members counting the size of each member.

from nds.

I just ran into this over the weekend and got verification that there is a 1 MB limit for a PutMulti.
"There's a per-API call limit of 1 MB for almost every API call, including datastore.*." - David Symonds
https://groups.google.com/forum/#!topic/google-appengine-go/kzSVWIwIbP0

from nds.

Ran into this just now. Possible to prioritize this? Or do you want a PR?

Thanks for nds btw.

from nds.

Hi @kidoman I'm happy to add the feature. I just need to do a bit of research as @mzimmerman is right. It isn't as trivial as it sounds due to the possibility of partial failure.

@mzimmerman how did you tackle the problem with Goon if one datastore.PutMulti succeeds and another one fails for a single call of nds.PutMulti? I believe that datastore.PutMulti can also have partial failure? If that's the case then I suppose I would just return an appengine.MultiError from my nds.PutMulti and mark all the errors associated with the failed datastore.PutMulti as non nil. Does that sound reasonable?

from nds.

Yes, that's exactly right, a MultiError.
https://github.com/mjibson/goon/blob/master/goon.go#L220

See maddyblue/goon#20 for the discussion around the
design decisions

On Sun, Oct 18, 2015, 5:39 AM Jonathan Gillham [email protected]
wrote:

Hi @kidoman https://github.com/kidoman I'm happy to add the feature. I
just need to do a bit of research as @mzimmerman
https://github.com/mzimmerman is right. It isn't as trivial as it
sounds due to the possibility of partial failure.

@mzimmerman https://github.com/mzimmerman how did you tackle the
problem with Goon https://github.com/mjibson/goon if one
datastore.PutMulti succeeds and another one fails for a single call of
nds.PutMulti? I believe that datastore.PutMulti can also have partial
failure? If that's the case then I suppose I would just return and
appengine.MultiError from my nds.PutMulti and mark all the errors
associated with the failed datastore.PutMulti as non nil. Does that sound
reasonable?

—
Reply to this email directly or view it on GitHub
#6 (comment).

from nds.

@mzimmerman thanks for the input.

@kidoman I've released the changes to the concurrent branch. Please let me know how you get on and I'll merge it to master in a few days if everything is OK.

from nds.

@jongillham I have looked the code and left a few comments. Thanks for putting this together.

from nds.

@kidoman here's the next instalment b240b98 with your suggestions. Thanks for spotting the potentially disastrous race condition bug!

from nds.

Looking forward to the merge!

from nds.

Commit 705d168 adds this functionality.

from nds.

Danke!

Sent from my iPhone

On 27-Oct-2015, at 12:55 AM, Jonathan Gillham [email protected] wrote:

Commit 705d168 adds this functionality.

—
Reply to this email directly or view it on GitHub.

from nds.