Trying to get this to work on a go1.10 project and I think the release needs to be tagged to work.

Before being executed Put and Delete methods lock their respective entities in the datastore for 32 seconds. What happens if a transaction takes longer than 32 seconds to execute? Can a transaction take longer than 32 seconds to execute?

Is time.Time not supported as an interface type? I'm getting this error on both Get and GetMulti requests.
WARNING: nds:loadDatastore marshal gob: type not registered for interface: time.Time

I'm having issues where nothing ever seems to be a cache hit, despite repeated Get calls for the same database key. Below is my appstats profile for 4 subsequent calls, which should involve the last 3 getting a cache hit.

Is there some way for me to check or turn on debugging flags to show me why the cache is being invalidated?

This is running on the devserver 1.9.12.

I recently had the problem that each time the Go SDK version was changed my Travis build script wouldn't work anymore until I manually upgraded it to the latest version. Since then I wrote a small script to automatically pull the latest, correct Go SDK. If you accept that it might break anytime (since the URL is not officially documented), it might be interesting to you: https://gist.github.com/stephanos/d48fd3500614bd83e63e.

Cheers! :)

appengine/memcache bug https://code.google.com/p/googleappengine/issues/detail?id=11382 was fixed in SDK 1.9.17. Code defending against this bug can be removed as it is now redundant.

We just had this show up in our logs from nds. memcache: server error Any idea what might be causing that?

A test needs to be written to ensure this is the case.

Probably can refactor into get.go, put.go, delete.go and transaction.go.

We have to now put on the big boy pants and start migrating some data which is already present in datastore. The struct has since then changed: a few columns added (which is not a problem) and some columns removed (which is a problem).

When trying to load such an entity from datastore (via nds; to keep the cache consistent) we will end up with an error.

Any recommendations on how to go about this?

I'm about to integrate my hrd library with nds and noticed that it is currently not possible to specify a custom memcache expiration time.

Is this something you think might be worthwhile?

I'm running https://github.com/mjibson/appstats to track my RPC calls. When I use datastore directly, it functions as expected and there is one call to the datastore.

As soon as I make the exact same nds call, it makes 4 calls to memcache instead of the two that I'd expect.

Do you know what could be causing that problem?

If memcache CAS fails for a specific key when updating in GetMulti, we should probably not update the local cache because that will likely have changed by another go routine as well. This needs some more analysis.

If the full datastore API were supported, nds would be a drop-in replacement. Is there any problem with the missing funcs being wrappers around the datastore equivalent?

The datastore timeout context does not work with nds caching.

When trying to run the tests using goapp test -v ./... I get the following error:

appengine: NewContext passed an unknown http.Request

I'm assuming this is due to the unspecified *http.Request given in https://github.com/qedus/nds/blob/master/nds_test.go#L31

Since App Engine will probably move more and more towards Managed VMs (currently beta) it might be a good idea to support both platforms, vanilla App Engine and Managed VMs.

Note: The problem is that the import path has changed, "appengine" becomes "google.golang.org/appengine", see related forum discussion.

I was remembering David Symond's talk on high performance apps and I went back and revisited it, and this slide popped out at me.
http://talks.golang.org/2013/highperf.slide#19

He shows memcache being 20x faster than datastore, and RAM 1000x faster than memcache. That got me thinking that it might be fantastic to add an in-memory cache option to nds. I'm thinking that you would initialize nds in the Init and provide a fixed memory size. Maybe I'm willing to allocate 5% of the total instance memory to this extra cache layer. Then any datastore lookup would first check RAM, then memcache and then finally datastore. If nds isn't ever initialized with a RAM cache, there would be no difference in how it functions.

I found this package earlier today that looks like it should handle all the heavy lifting of promoting and expiring cache items. Since it's from CloudFlare, I trust that it is highly performant and well coded, especially looking at their benchmarks.
http://godoc.org/github.com/cloudflare/golibs/lrucache

This is obviously isn't a must-have, nds has already made huge strides for us. I wanted to toss the idea out for future development. What are your thoughts?

I have a reasonably large App Engine Classic app using nds. In production, it works well without any issues. However, when running my tests via goapp test ./... I get a datastore failure with an error of datastore: invalid key.

If I however, run the particular test that failed directly via gapp test -run=FailingTestName ./..., the test that failed in the broad test passes.

If I modify the number of retries that my app will perform on a datastore fail to a higher number (say 6), the broad test goapp test ./... passes.

If I substitute datastore. for all my nds. function calls, the tests pass.

I am using the latest version of nds, with App Engine 1.9.40 on a 2014 Macbook Pro.

What further information can I give to help solve this problem?

When using GetMulti, if there is a difference between datastore and the struct used to load the data into, datastore returns the data, with an error type ErrFieldMismatch - cannot load field "X" into a "Y": no such struct field.
The result from nds for N structs that trigger this error, is only a single populated struct, and N-1 empty structs (no fields populated), with N errors.
It should be all the structs with their values, and N errors.

I believe that this can be fixed by changing https://github.com/qedus/nds/blob/master/get.go#L359 to cacheItems[index].err = err
I can provide a PR if needed.

NDB has at least one known consistency issue with its caching strategy. See https://code.google.com/p/appengine-ndb-experiment/issues/detail?id=84 for a write up and possible solution. As this package uses the same strategy as NDB therefore it also has the same issue.

The proposed fix of using a random number as a CAS ID looks fairly easy to implement.

For this to happen cloud.google.com/go needs to be compatible with App Engine Standard and the memcache package needs to be implemented.

Needs the same functionality as GetMulti.

This issue revolves around RunInTransaction which locks all memcache keys that use Put and Delete within the transaction prior to committing to the datastore.

The following scenario would produce stale results when calling Get.

• RunInTransaction locks memcache successfully.
• Memcache fails/locks are evicted.
• Memcache recovers.
• Another client calls Get on the entities that were supposed to be locked and repopulates memcache.
• RunInTransaction commits to the datastore and memcache is left with stale entity values.

I currently don't see any way of completely eliminating this consistency issue. Note that it does not affect Get, Put or Delete when used outside of RunInTransaction.

Note that this issue does not affect consistency guarantees when just using nds.Get* and nds.Put* functions.

We started randomly seeing memcache quota errors that ended up failing our Put operations. We're paying customers, so I'm not sure why those started showing, but it seems reasonable to me to have nds bypass memcache and go directly to datastore on this error. What do you think?

In production on a low load service, I'm getting the following error about once a day:

nds:lockMemcache AddMulti memcache: item not stored

My understanding is that is't simply a case of saving to memcache failing, in which case it only means that the next Get would have to get the contents from the datastore instead of memcache.

If this is indeed the case, please close this issue, but on the offchance that my understanding is incorrect, I thought it would be useful to have a record.

datastore.PropertyList used to serialize into local memory and memcache. This is probably less efficient than using GOB with memcache and just keeping the interface within local memory.
Note that the memcache namespace prefix will have to be changed if this is fixed.

We need to ensure that the API accepts all these types of arguments correctly.

Do you have any intention of supporting Queries via GetAll or Run? I can see how that would be much more challenging than Get/Put/Delete.

I've been getting warnings from the logger I placed in the onError function:

nds:loadCache setValue: datastore: cannot load field "rg" into a "models.UserAccount": multiple-valued property requires a slice field type

I think there's an opportunity to update the tests to ensure that any logs made are expected, otherwise to fail. This may help uncover situations in which we think we're pulling from the cache but are really failing at a later step (e.g. setValue) and falling back to the datastore.

I'm occasionally getting a warning of: nds:lockMemcache AddMulti memcache: item not stored

While I'm fine with the occasional Memcache error being thrown, this is happening when saving a single entity. I'm not sure why nds is calling AddMulti for a single entity instead of Add.

I get a panic with the stack trace shown below with "nds". It seems that "x*Record" is nil in "Load"

====================================
// Lädt einen Datensatz aus dem Datastore
func (x *Record) Load(ps []datastore.Property) error {
x.props = make([]datastore.Property, 0, len(ps)) <============== line storage.go:165
x.Created = time.Now()
x.Modified = x.Created
....

=================================
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x47b88a]

goroutine 31 [running]:
panic(0x8ff760, 0xc82000a070)
/home/tim/Dokumente/go_appengine/goroot/src/runtime/panic.go:481 +0x3e6
main66787.(*Record).Load(0x0, 0xc820285c70, 0x5, 0x5, 0x0, 0x0)
storage.go:165 +0x75a
github.com/qedus/nds.setValue(0x944b20, 0xc8201e23a0, 0x196, 0xc820285c70, 0x5, 0x5, 0x0, 0x0)
/home/tim/Dokumente/gopath/src/github.com/qedus/nds/nds.go:155 +0x1e4
github.com/qedus/nds.loadDatastore(0x7f81cd1d36e8, 0xc820270ae0, 0xc8201df720, 0x1, 0x1, 0x7f81cd1cf0a8, 0x800d60, 0x0, 0x0)
/home/tim/Dokumente/gopath/src/github.com/qedus/nds/get.go:339 +0x5c7
github.com/qedus/nds.getMulti(0x7f81cd1d36e8, 0xc820270ae0, 0xc8202424d0, 0x1, 0x15, 0x800d60, 0xc820273340, 0x97, 0x0, 0x0)
/home/tim/Dokumente/gopath/src/github.com/qedus/nds/get.go:164 +0x3cf
github.com/qedus/nds.GetMulti.func1(0x7f81cd1d36e8, 0xc820270ae0, 0xc8202768d0, 0x1, 0x1, 0xc8202768e0, 0x0, 0xc8202424d0, 0x1, 0x15, ...)
/home/tim/Dokumente/gopath/src/github.com/qedus/nds/get.go:78 +0x19c
created by github.com/qedus/nds.GetMulti
/home/tim/Dokumente/gopath/src/github.com/qedus/nds/get.go:81 +0x32b

nds.Delete currently locks memcache and then deletes its entity from the datastore. If the following situation occurs then stale cache will be left in the datastore:

• Delete is called and locks the memcache item to be deleted.
• The memcache lock is evicted from memcache.
• Get is called and loads the entity into memcache.
• Delete removes the entity from the datastore.

Therefore the entity will no longer be in the datastore but will be in memcache.

Note that this issue does not affect consistency guarantees when just using nds.Get* and nds.Put* functions.

Thought needs to go into which errors are appropriate before and after datastore calls. PutMulti for example need not generate errors if it fails to update memcache or local cache. Instead it should clean up appropriately.

I propose we use the github.com/qedus/nds/v2 directory to contain version 2 code instead of the v2 branch.

@someone1 I'm interested to hear your thoughts.

If ctx and key have a different namespace for any kind of request in nds, then nds will use the key's namespace for the datastore request, as intended, but will do the memcache requests without creating a new context with that namespace in it, causing the memcache requests to be against a different namespace

It's not uncommon that when I use nds.Put, I know that I'm going to be needing it soon after. This is especially true in my data pipeline. Since a Put doesn't add an item to memcache, I was thinking that it would be great to have CachePut / CachePutMulti functions to go ahead and preemptively stick them into memcache.

The alternative of course would be to have Put / PutMulti always insert items into Memcache.

Thoughts?

Access to the transaction strict member needs to be guarded by a mutex to allow safe concurrent access.

Take the following situation where I create a custom context:

type myContext struct {
    appengine.Context
} 

If I do this:

nds.RunInTransaction(c, func(tc appengine.Context) error {
    mc := &myContext{tc}

    // The following calls to the NDS API will not be executed
    // the transactional code path.
    nds.Get(mc, key, val)
    nds.Put(mc, key, val)
    nds.Delete(mc, key)
}, nil) 

There is no way of the nds.Get/Put/Delete functions knowing that the mc context is a transactional context in the current API. Note that if the context is not wrapped then nds.Get/Put/Delete works as expected.

The only nice way I can see around this subtle issue is to change the NDS API as shown in this context branch: https://github.com/qedus/nds/tree/context

I wrap appengine.Contexts a lot to add extra functionality that must be bound to a single context/request. The changes I propose to the API will allow me to continue doing this without fear that I am not in a transaction when I think I am.

There is some more discussion here: https://groups.google.com/forum/#!topic/golang-nuts/xrxjgCGzUfs

This is the code that doesn't quite make sense: https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/context.py#778.

This was driving me up a wall, but nds treats transactions differently than the datastore package.

Specifically, it keeps an internal lock which is never unlocked after a commit or rollback. This will keep executing code deadlocked and is different than how the datastore package handles things.

Comments around this specify:

// tx.Unlock() is not called as the tx context should never be called
// again so we rather block than allow people to misuse the context.

Should we reconsider this policy or better document the difference in behavior from the datastore package?

If I got it right currently entities are not stored to memcache on Put().

Suppose we run in transaction and need to update some entities. Then current execution order is:

1. User code calss nds.RunInTransaction()
2. Get entity by key from datastore (not stored to memcache)
3. Change entities
4. User code calls nds.PutMulti() with transactional context
5. nds.putMulti() add items to tx.lockMemcacheItems (memcache not changed)
6. nds.putMulti() call datastorePutMulti()
7. nds.putMulti() executes deferrer that removes from memcache tx.lockMemcacheItems.

First of all it's not clear why we create lockItem() at line 129 as it seems it is not used further in current implementation.

Second just removing items from memcache results in unnecessary paid Get() operation on following request while potentially results could be retrieved from the "free" memcache. For example I have a chat application and whenever user writes to the bot and bot sends response I do following:

1. On request Chat entity is retrieved outside of transaction
2. On response Chat entity updated within transaction (if state changed)
   Currently the first Get() is not leveraging memcache with always a miss. This increases response time & costs.

So the suggestion is:

1. Before putting nds should try to lock items in memcache.
2. If previous pre-put locks was successful nds tries to put items to memcaches using CAS.
3. If lock or putting to memcache failed try delete from memcache as currently.
4. Not related improvement: If delete from memcache failed create a deferred task for removing items from memcache. That could mitigate temporarily memcache unavailability.

I know caching is hard to get right and probably I'm missing something.

edit: fixed urls to permalinks