Based on discussion at http://mattjibson.com/blog/2013/06/26/go-read-open-source-google-reader-clone/#comment-944610909, memcache access is limited to 1000 objects at once, and the memcache library needs to be made aware of this limit.

Hi, I'd like to unit test my code that relies on datastore in a way that does not require the devserver to be started. Is it possible to use goon and test without it using aetest and starting a devserver behind the covers?

https://github.com/golang/go/wiki/Modules#modules

This repository has no tag.
I think we should add a semver tag for go modules.
We can use only commit hash designate now, It is inconvenient.

Is it safe to add a property to an entity? or should I migrate my data first?

For now I've edited goon.go:44:

func NewGoon(r *http.Request) *Goon {
    return FromContext(appstats.NewContext(r))
}

I am just wondering if there is a case when gob.Register is needed on the following lines:

https://github.com/mjibson/goon/blob/master/entity.go#L31
https://github.com/mjibson/goon/blob/master/entity.go#L41

PR #87 redesigned cache key generation to fix some issues, however in the process it removed the ability for people to determine the final cache key. This is something that people actually use, as evidenced by this comment. Thus the ability to know the final cache key should be reintroduced.

The conundrum is that we either have to surface it with a new function name, and thus breaking compatibility with existing code that expects MemcacheKey to return the final key or we surface the whole thing via MemcacheKey and not provide automatic bug fixes to people who override the function.

I'm happy with the recent changes to the memcache layer. However, the complexity of the serialization code prevented me from reading any of the changes. While the performance benefits are nice, I do not think they are enough to justify this complexity of code. I think it is more important that many people agree that it is good.

I propose that we remove this magic and revert back to the gob version. I definitely want to keep the new features that were added, but I want to understand how they work.

If there is strong disagreement to this proposal, I'd like to discuss it. My goal is to have the code in a state where I can understand it and have motivation to read through all changes. I thought I could ignore that, but goread broke so many times that this is clearly not possible.

Thoughts?

Hi,
Am trying to store a struct into datastore like below,

type Config struct {
Id int64 datastore:"id,noindex" goon:"id" json:"id"
Config map[string]float64 datastore:"config,noindex" json:"config,omitempty"
}

I have also impl PropertyLoaderSaver interface to convert the map into json string before saving to datastore, everything work fine with datastore, while fetching the json string is convert to back to map,

But when this entity is fetched from memcache am getting following error from goon.go file,

goon - goon.go:475 - goon: Failed to decode field config - extra data in buffer

Any help on how to fix this issue?

Note: Every First time fetch from datastore it works fine, only subsequent fetch (while it fetch from memcache) its throwing this error

I think this is so cool project. but, I can't use no license source code.

I found that when we fetch slice values from cache, goon appends the values into existing slice. Instead of appending values, goon should replace the slice.

Test to reproduce:
yuichi1004@ab412e6

Test Result:

$ go test -v -run Migration  ./
=== RUN   TestMigration
INFO     2018-04-28 09:43:06,511 devappserver2.py:105] Skipping SDK update check.
WARNING  2018-04-28 09:43:06,511 devappserver2.py:121] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING  2018-04-28 09:43:06,566 simple_search_stub.py:1196] Could not read search indexes from /var/folders/5j/2bmf8d2d3xgb9lmttlptxvm8p49fjs/T/appengine.testapp.01099286/search_indexes
INFO     2018-04-28 09:43:06,568 api_server.py:308] Starting API server at: http://localhost:51010
INFO     2018-04-28 09:43:06,638 dispatcher.py:255] Starting module "default" running at: http://localhost:51011
INFO     2018-04-28 09:43:06,640 admin_server.py:146] Starting admin server at: http://localhost:51013
--- FAIL: TestMigration (4.48s)
        goon_test.go:994: MC-false > Expected 1 car! Got: 2
FAIL
exit status 1
FAIL    github.com/yuichi1004/goon      4.501s

Note that we have extra cars on the above test.

See #55

Do you think it would make sense to add laterGet and laterPut using "appengine/delay" ?

laterGet can also be useful if you want to pull something from memcache you might need later in another request.

ref: http://talks.golang.org/2013/highperf.slide#13

I noticed that goon uses a *http.Request instead of a context.Context when creating a *Goon, which is only used to get a new context.Context. I'm not clear on the reasoning behind this decision as it breaks from the common Go and AppEngine style of passing in a context.Context as the first argument in a function call. Especially since the vast majority of functions will already have a context.Context, but it's usually only the handler function which has a *http.Request to pass in.

I would suggest requiring a context.Context as the argument instead of the *http.Request which is currently used. However I'd love to get some feedback on this.

Currently, goon.Get calls methods as below
.. => goon.Get => datastore.Get => goon.putMemcache => ..
or (if it's cached)
.. => goon.Get => memcache.Get => goon.putMemcache => ..

I think this flow has a problem.
PropertyLoadSaver.Load is called in only datastore.Get.
And goon.putMemcache ignores fields which tagged as datastore:"-".
Therefore the fields changed by PropertyLoadSaver.Load aren't cached.

The return of goon.Get should be same whether it's cached or not.

Just curious. Does it have any meaning such as GO cONtext or any?

If I try to create an entity in a unit test :

    u := &User{Name: "Joe", CName: "joe"}
    k, err := n.Put(u)
    if err != nil {
        t.Fatal(err)
    }

Parentless queries will fail because of a consistency issue (parentless puts may not be available right away.)

The problem can be avoided by sleeping for 1 second but the test becomes dependent on the machine load, etc.

    // Sleep a bit to wait for the HRD emulation to get out of our way
    time.Sleep(1000 * time.Millisecond)

According to this post the proper way to avoid this issue inside a test is to get the entity using its known key, right after.

My subsequent queries work perfectly if I perform a datastore.Get(c,k,u) first.

However they fail miserably if I use n.Get(u) even though the key is known and set:

I'm really enjoying goon, but something I think it needs is a .PutMemoryOnly (which puts to cache and memcache, excluding datastore).

I see a big need for this since I have applications doing a lot of .Puts to maintain a last_seen timestamp (which I'm more than happy to only write to dbstore every 30/60 seconds - However I require it to still be available "if" possible via cache or memcache).

Currently my only option is to maintain my own cache separately, so I'm caching the same thing twice :/

Add support for applications that use the google.golang.org/appengine packages. These packages work on App Engine Classic and managed VMs.

There are a couple of ways to do this. One is to fork Goon and replace all uses of appengine/* with google.golang.org/appengine/* and appengine.Context with the x/net Context. A second option is to write a shim layer above Goon.

TestEmbeddedStruct fails on my environment. It seems to be failed to fetch unexported fields following.

2220 type ContainerStruct struct {
2221     Id string `datastore:"-" goon:"id"`
2222    embeddedStructA
2223    embeddedStructB `datastore:"w"`
2224 }

I think private fields are expected to be ignored by datastore API. So this behavior looks as expected to me. Am I missing something?

The test passed if I make those fields public like this.

2220 type ContainerStruct struct {
2221     Id string `datastore:"-" goon:"id"`
2222    EmbeddedStructA
2223    EmbeddedStructB `datastore:"w"`
2224 }

Test Result

$ goapp test -v . -run TestEmbeddedStruct
2017/05/03 14:33:06 appengine: not running under devappserver2; using some default configuration
=== RUN   TestEmbeddedStruct
INFO     2017-05-03 05:33:07,983 devappserver2.py:756] Skipping SDK update check.
WARNING  2017-05-03 05:33:07,984 devappserver2.py:772] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING  2017-05-03 05:33:08,079 simple_search_stub.py:1146] Could not read search indexes from /var/folders/h0/c79kbs8x6jj8scsd6j22c5hh01hx69/T/appengine.testapp.yuichi.murata/search_indexes
INFO     2017-05-03 05:33:08,084 api_server.py:205] Starting API server at: http://localhost:50322
INFO     2017-05-03 05:33:08,087 dispatcher.py:197] Starting module "default" running at: http://localhost:50323
INFO     2017-05-03 05:33:08,090 admin_server.py:116] Starting admin server at: http://localhost:50324
--- FAIL: TestEmbeddedStruct (4.35s)
        goon_test.go:2266: #1 - Expected - 1, got 0
        goon_test.go:2269: #1 - Expected - 3, got 0
        goon_test.go:2272: #1 - Expected - 4, got 0
        goon_test.go:2266: #2 - Expected - 1, got 0
        goon_test.go:2269: #2 - Expected - 3, got 0
        goon_test.go:2272: #2 - Expected - 4, got 0
FAIL
exit status 1
FAIL    github.com/yuichi1004/goon      4.364s

Similar to the request at qedus/nds#54

Would it be possible to extend this to work outside of Google App Engine?

When you "put" an incomplete object and the Id is of type string, it's not populated and is instead left blank. This is inherently because the datastore will only generate an integer id.

I guess it would be proper to return an error back to the caller if they're trying to "Put" an incomplete key and their Id is of type String?

The following test fails on the last inequality check with:
demo_test.go:54: goonPutGet.Value should be 15 but is 0

package goon_test

import (
    "testing"

    "appengine/aetest"
    "appengine/datastore"

    "github.com/mjibson/goon"
)

type PutGet struct {
    ID    int64 `datastore:"-" goon:"id"`
    Value int32
}

func TestPutGet(t *testing.T) {
    c, err := aetest.NewContext(nil)
    if err != nil {
        t.Fatal(err.Error())
    }
    g := goon.FromContext(c)
    key, err := g.Put(&PutGet{ID: 12, Value: 15})
    if err != nil {
        t.Fatal(err.Error())
    }
    if key.IntID() != 12 {
        t.Fatal("ID should be 12 but is", key.IntID())
    }

    // Datastore Get
    dsPutGet := &PutGet{}
    err = datastore.Get(c,
        datastore.NewKey(c, "PutGet", "", 12, nil), dsPutGet)
    if err != nil {
        t.Fatal(err.Error())
    }
    if dsPutGet.Value != 15 {
        t.Fatal("dsPutGet.Value should be 15 but is",
            dsPutGet.Value)
    }

    // Goon Get
    goonPutGet := &PutGet{ID: 12}
    err = g.Get(goonPutGet)
    if err != nil {
        t.Fatal(err.Error())
    }
    if goonPutGet.ID != 12 {
        t.Fatal("goonPutGet.ID should be 12 but is", goonPutGet.ID)
    }
    if goonPutGet.Value != 15 {
        t.Fatal("goonPutGet.Value should be 15 but is",
            goonPutGet.Value)
    }
}

Looking at the code it appears that the in memory cached value of PutGet{ID:12} is not being transferred to goonPutGet. Note that if I create a new Goon struct for the g.Get(goonPutGet) everything works fine because the in memory cache is not used.

I was thinking, it would be quite useful to add a (writing this off the top of my head)

func (g *Goon) getContext() appengine.Context {
   return g.context
}

I keep having to pass the Context and Goon around at the same time (logging, creating queries, accessing memcache directly, etc), and Goon already has the Context already.

Let me know if there are any objections to this, and if not, I'll fire up a pull request.

There's a few people looking at this now, and it'd be awesome if someone wanted to implement control variance, as discussed in David Symonds' talk: http://talks.golang.org/2013/highperf.slide#26

I've seen memcache on goread take 100-200ms, both on read and write, which can triple the time of a request. If defaulted to 3ms or 5ms or something, and made configurable, I would be happy.

I don't think this is a goon issue, but instead that Google released a bugged SDK. However it has a serious effect on goon (tests don't work), so I think it's an issue worth keeping track of here.

Basically some tests seem to deadlock, and others throw up a wide variety of threading errors. I have filed a bug report to google.

I have only tested this on my own Windows setup, and I have only seen other reports with similar errors from Windows setups. So it would be interesting to know if it's a Windows specific problem.

@mzimmerman, you run arch, right? Perhaps you could try running the goon tests with Go SDK 1.9.3 a few times and see if they always succeed without errors, like they should and have been succeeding with previous SDK versions.

Hi there,

New to goon / appstats & go, came across a issue where the context Goon was using was not that of appstats'. So I was not getting any RPC calls logged, Goons documentation doesn't mention to use FromContext(c) rather than NewGoon(r) (I discovered it seeing the function in the function listing).

Might be easier for newer guys to be aware to use FromContext when using appstats.

Thanks,
Mike :)

Hello,
Thank you a lot for this package. it is always a great help for us.

By the way, I encountered its behavior as this Title, when I try to use the new function IgnoreFieldMismatch .
a3e1c61#diff-fd6485aca5f3b9fb50a56fbfc9f202b4R414

I think It's better to be improved, If it is not by design.

Thanks again