Comments (11)
I haven't done this because I can't think of a case where I would want anything other than the max cache expiry time.
What's your usecase?
from nds.
Now that I think about it you are right, if everything works as expected in nds it there should not be reason for it. At work we use Objectify which allows to customize the memcache expiration per entity. Apparently I've never questioned this strategy until now, but I can't come up with a good reason either (there is one entity with a 5 minute expiration I change maybe once a month through the datastore viewer since we didn't bother to create a UI for it yet, but that's not a good reason).
from nds.
Well, the only reason might be to remove an entity from memcache before the memory pressure needs to do it (the cache size is rather limited unless you have a dedicated memcache), for entities where you know that this will not be needed after x minutes. But that's probably premature optimization.
from nds.
On a related note: do you think there is a use case for ignoring the cache for specific datastore requests?
In Objectify:
ofy().cache(false).load().key(thingKey).now();
from nds.
Does Objectify have a strongly consistent cache strategy? If it doesn't then it would definitely need an option to specify cache expiry as a client could potentially fetch stale results forever. It would also need the option to fetch straight from the datastore.
If there is strongly consistent cache then there is no reason to not always use the cache for Get, Put and Delete methods assuming the memcache API is working correctly.
Note that NDS only adds entities to the cache on the first Get request. Therefore just using Put on thousands of entities does not affect memcache and memory pressure.
I am only aware of Python's ndb and this library that implements strong cache consistency. Do you know of any Java ones?
from nds.
From Objectify's documentation:
Objectify's global cache provides near-transactional consistency with the datastore, even under heavy contention. There is still, however, one circumstance in which the cache could go out of synchronization with the datastore: If your requests are cut off by DeadlineExceededException.
But I can certainly see your point about always using the cache. I don't know what the motivation in Objectify was.
One use case I could think of for skipping the cache for specific Get operations though is when a background task that runs maybe once a day iterates over all entities (e.g. to create robots.txt). Then it would fill the cache pretty quickly and possibly throw out entities that are accessed by the actual user.
from nds.
Thanks for the info. I am surprised that Objectify doesn't have a strongly consistent caching strategy! I wonder what Java library does. Libraries like Goon and, as you just showed, Objectify work for the majority of cases though.
I guess it depends on what type of app/functionality you are building and if you don't mind some junk being returned from cache requests every so often. App Engine instances frequently die/misbehave in all sorts of ways. I don't ever want to have to think about this in the code I write as cache consistency bugs always seem the hardest to find and potentially have devastating effects. This is why I created NDS.
As for your background task usecase. If you are just getting datastore entities and not mutating them then it would be easiest just to replace nds.Get
with datastore.Get
. They have the same signature. datastore.Query
also doesn't touch the cache. Keep trying though - maybe there is a reason to have the option to avoid memcache or set the expiry! :-)
from nds.
Agreed. Great discussion, thanks :)
from nds.
A max expiration would eventually self-heal #31 and #33. Otherwise, stale results can handarojnd forever if not changed/evicted.
I think it makes sense.
from nds.
You are correct, issues #31 and #33 would eventually heal. However, here's why I don't think adding a max expiration adds value for the majority of cases:
- Memcache doesn't stay around permanently. An entry can stay for days, maybe even weeks but it always gets evicted at some point in the future. Although I have to say there are no guarantees without specifying an eviction time.
- I can't think of a case where someone would expect strong cache consistency but be fine with that failing. Most people either want guaranteed cache consistency (which you get with using only nds.Get and nds.Put) or can tolerate not having it.
- Adding an extra setting increases complexity when there may only be a handful of uses.
from nds.
Reason #2 resonates with me the most. That's really why this a forced expiration approach would be a hack.
With that said, for those that would like to deploy this hack: a forced reload every X minutes (or whatever), the extra setting would help. We could set it as an optional parameter, so no additional complexity for anyone who doesn't want to use it.
from nds.
Related Issues (20)
- Remove zeroMemcache* code. HOT 1
- It is not possible to detect if nested contexts are transactional. HOT 5
- Make the transaction context safe for concurrent access. HOT 1
- Failing tests with "appengine: NewContext passed an unknown http.Request" HOT 2
- Different namespaces used for memcache and datastore HOT 4
- Migrating data when the underlying struct changes HOT 2
- Tests fail when using nds HOT 14
- nds item not stored warning HOT 4
- Calling AddMulti for a single entity instead of Add HOT 2
- Panic with datastore.PropertyLoadSaver HOT 3
- Make compatible with cloud.google.com/go HOT 8
- ErrFieldMismatch in GetMulti returns only one result HOT 4
- Gracefully handle memcache quota limits HOT 2
- Support saving of entities to memcache in putMulti() HOT 7
- Tag v2 branch as v2.0.0 HOT 3
- [v2] Possible marshal/unmarshaling bug HOT 2
- Change locking policy for Transaction HOT 1
- Change module versioning from git branches to subdirectories. HOT 2
- want new tag in v2 HOT 2
- failed to Put dial tcp 192.168.0.3:15127: connect: connection refused HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nds.