localdata / localdata-tiles Goto Github PK

see etag-cache lines around 109 for an example of how to do this

Either too large or too small

We're getting 0 results from the datasource for known good bounds. Not sure if something happened to projections or to queries yet.

What is going on I don't even computer.

If a survey has points that are unassociated with a base feature, then we need to pass along the response ID in the UTFGrid. Otherwise, we have no way of retrieving more information about the item.

Right now, this requires a small change both in tiles.js and in the MongoDB nodetiles datasource.

For checkbox questions, the first-level filter should color entries with two colors: one for checked, one for unchecked. The subfilter should only show entries where the box is checked.

The browser interprets it as a script anyway, but it's poor form. We use the same S3 caching middleware for tile PNGs and UTFGrid JSON, so I suspect that's the culprit. We send to S3 after we've sent the data to the client, so we should be able to just check the Content-Type there.

For consistency

cc @prashtx

If we have a cached tile, we currently send a 302 response and redirect the client to the cached tile's S3 URL. There's some overhead in waiting for that response and then initiate a new request. The time to retrieve the S3 object for the Heroku-hosted app should be small, though, so it might be faster to just send the data instead of the 302 response.

Thoughts:

• We already get the S3 header. If we will get the whole file later, it could make sense to get the whole file at the beginning, instead of the header. That would basically increase miss latency but decrease hit latency.
• What will this do to memory, CPU, and network usage?

I added some responses to https://app.localdata.com/index.html#surveys/my-test-survey.

When I clicked "check for new results", the tiles were refreshed but new responses did not appear. When I zoomed out, new tiles were rendered and the new responses showed. I think the code to check for new responses is failing in some way.

We will usually want to instrument a cache with both a hit rate metric and metrics for the hit latency and miss latency. We can simplify the instrumented code by combining these.

This requires coordination with the dashboard. We should be able to generate the list of valid questions, and the list of possible answers for each, from the current and past forms. Then we're dealing with a handful (or dozens at the high end) of objects rather than thousands or tens of thousands.

This should speed up the initial tiles that we render for each survey, and it should wreak less havoc on the mongodb cache, once we have a full-fledged instance where the cache is really being helpful.

If there is no data for a tile, we still check the S3 cache and potentially redirect the client to S3, even though the result will be identical to every other empty tile.

We should bypass S3 if there is no data. Ideally, we should skip the tile rendering and return a static, "blank" PNG. If we have a background color set for the map style (unlikely for this tile server), then we won't send a fully transparent PNG.

This ought to save the client from making several requests (since we won't redirect to S3), and it should reduce the latency in responding to the remaining requests.

We need to reproject them back to WGS84.

Similar to a CPU load average, which indicates the number of tasks waiting to run, the request load would measure the number of active requests.

The tile server tends to get requests in groups: multiple tile images and their UTFGrids. Understanding the "overlap" in our requests could be informative. For example, the default of 5 MongoDB connections might be inadequate.

Let's issue a "warning" rather than an "error" if we get a 503 from New Relic and will just try again later.

Probably occurred in a request for /filter/collectors/tile...

2014-02-06T19:39:47.691856+00:00 app[web.2]: 
2014-02-06T19:39:47.691960+00:00 app[web.2]: /app/lib/controllers/tiles.js:205
2014-02-06T19:39:47.699711+00:00 app[web.2]:       for (i = 0; i < answers.length; i += 1) {
2014-02-06T19:39:47.699711+00:00 app[web.2]:                              ^
2014-02-06T19:39:47.699711+00:00 app[web.2]: TypeError: Cannot read property 'length' of undefined
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at /app/lib/controllers/tiles.js:205:30
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.<anonymous> (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:177:8)
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.EventEmitter.emit (events.js:95:17)
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.emit (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:84:38)
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.fulfill (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:97:20)
2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.<anonymous> (/app/lib/models/Form.js:64:5)
2014-02-06T19:39:47.699895+00:00 app[web.2]:     at Promise.<anonymous> (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:177:8)

and shortly thereafter

2014-02-06T19:39:47.699711+00:00 app[web.2]:     at Promise.resolve (/app/node_modules/mongoose/lib/promise.js:108:15)
2014-02-06T19:39:47.697196+00:00 app[web.1]:                              ^
2014-02-06T19:39:47.697196+00:00 app[web.1]: TypeError: Cannot read property 'length' of undefined
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at /app/lib/controllers/tiles.js:205:30
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.<anonymous> (/app/lib/models/Form.js:64:5)
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.<anonymous> (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:177:8)
2014-02-06T19:39:47.691679+00:00 app[web.1]: 
2014-02-06T19:39:47.691918+00:00 app[web.1]: /app/lib/controllers/tiles.js:205
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.EventEmitter.emit (events.js:95:17)
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.emit (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:84:38)
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.fulfill (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:97:20)
2014-02-06T19:39:47.697196+00:00 app[web.1]:     at Promise.resolve (/app/node_modules/mongoose/lib/promise.js:108:15)
2014-02-06T19:39:47.697441+00:00 app[web.1]:     at Promise.<anonymous> (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:177:8)
2014-02-06T19:39:47.697441+00:00 app[web.1]:     at Promise.EventEmitter.emit (events.js:95:17)
2014-02-06T19:39:47.697196+00:00 app[web.1]:       for (i = 0; i < answers.length; i += 1) {
2014-02-06T19:39:47.697441+00:00 app[web.1]:     at Promise.emit (/app/node_modules/mongoose/node_modules/mpromise/lib/promise.js:84:38)

We allow filtering by collector, but we treat it just like any other question. The response isn't structured that way, though, so our generated styles are inconsistent with the data that nodetiles-core will actually see.

Right now we kick a response object over to nodetiles-core and let it do everything. Let's use nodetiles-core as the renderer, and then we can implement cache control responses, caching, etc. in the server code.

The MongoDB nodetiles datasource is just one file. We're not using it generically, and there may be advantages to adding some application-specific code to it. If changes and bugfixes might require modifications both to the tile server and the datasource, then I think it makes sense to pull it into the tile server repo.

We can control logging and instrumentation much more easily if we don't have to make changes to a referenced repo.

It seems reasonable to create application-specific nodetiles datasources, considering the lightweight interface.

http://localhost:3001/dbcb3590-0f59-11e2-81e6-bffd22dee0ec/filter/condition/tiles/14/4411/6055.png

I'm taking a look.

// Todo:
// - load from a template
// - use a sensible center (from the survey data)
// - better attribution etc. (use survey data)

npm install fails for reasons that are not apparent to me. I frequently get errors like this around canvas or bson (it's not entirely clear from the log who is the culprit)

npm ERR! cb() never called!
npm ERR! not ok code 0

Blowing away node_modules and re-installing generally fixes the problem -- but sometimes it takes multiple cycles to get everything working. Maybe things need to be installed or compiled in a specific order to play well together?

@prashtx, didn't you mention you had a random issue with dependencies on heroku, too, or was that on server-api?

It's bad enough that I have to attach this:

When we render a tile, we translate the tile name into a bounding box and then add a buffer. That lets us render objects that are technically outside our tile but will be in view of the tile due to rendered style (a thick border, for example).

When we confirm validity of the image cached in S3, we only look at the tile's bounding box, not the buffered bbox. If a new object pops up right next to our tile, we'll incorrectly serve the cached image.

Our styles are not so fat right now, so this isn't a major issue.

To fix, I think we should modify nodetiles-core to allow a configurable buffer. Then we can use the same buffer for our cache validation that we pass to nodetiles-core.

Seems to be a problem in the filters. Need to reproduce and fix.

Mongo related packages should be pegged to a specific version.

Request is very loose.

The map PNG and UTFGrid both use the same data for a given tile, but right now we make separate requests. We should be able to make one request and reuse the data.

We can cache the data with a short time-to-live, so that we don't explode our memory footprint. We likely need to use promises, since the database requests will go out at about the same time. The second one should see a promise in the cache, which in some cases may have been resolved already.

The effectiveness of an in-memory cache will decrease as we scale up the number of dynos. Ideally the approach should be general enough that we could drop in redis or similar and share the cache across processes.

How can we run localdata-tiles securely, locally? Fakeroku?

@hampelm : Using this issue for the discussion around filter architecture.

We need to "color" and "filter".

Color: Apply styles that color entries according to the answer for a specified question.
Filter: Only plot the entries where the answer matches the one specified. The color for those entries should be the same as if we hadn't filtered.

I don't think we need to alter nodetiles-core. We're just giving it data and styles in all of the cases. We should be able to generate a version of the datasource that selects the appropriate fields. We can still cache the resulting Map object, using the survey ID and a string description of the color/filter information.

I've started a filter branch that we can use for the server portion of this. Right now it just has some housekeeping changes.
https://github.com/LocalData/localdata-tiles/tree/filter

One of the dependencies is looking for request > 3 when I push to Heroku:

No compatible version found: request@'>=3.0.0-0'

Here's the full log output: https://gist.github.com/hampelm/6744849/edit

header with the total response count, datestamp

The heap frequently gets up to 500mb+ on Heroku.
It doesn't do that locally

https://hacks.mozilla.org/2012/11/tracking-down-memory-leaks-in-node-js-a-node-js-holiday-season/

We should get a handle on tile server performance. There are several pieces of work (DB, tile rendering, cache checking, etc.) and multiple paths (S3 cache vs generated tile, UTF grids), and I'd like to know how they each contribute to performance.

I think we can make the tile server faster, but we should know which pieces to work on, and we should track our progress.

StrongOps doesn't have the granularity needed. We could log the metrics, but then we have to do our own log processing, graphing, etc.

NodeTime supports custom metrics, and I think those will work for us:
http://docs.nodetime.com/#custom-metrics

Now what to record?

• Cache hits/misses for every cache we use (etag, S3, survey stats)
• Time to get data from database (will this require instrumenting nodetiles-core or nodetiles-mongodb?)
• Total time to serve tiles
• Total time to serve UTF grids
• Time to check validity for ETag cache and S3 cache

For the first few tiles for a given survey, each tile results in a request for the survey stats, which is a very expensive operation. Once we get the stats, we cache them for future tiles.

Instead, we should cache a promise for the stats. Then the initial tiles can all share the same request.

On master and filter, I get this error when running node tileserver.js

/Users/matth/projects/ld/tiles/localdata-tiles/node_modules/mongoose/lib/index.js:278
      throw new mongoose.Error.OverwriteModelError(name);
            ^
OverwriteModelError: Cannot overwrite `Form` model once compiled.
    at Mongoose.model (/Users/matth/projects/ld/tiles/localdata-tiles/node_modules/mongoose/lib/index.js:278:13)
    at Object.<anonymous> (/Users/matth/projects/ld/tiles/localdata-tiles/lib/models/Form.js:143:38)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/Users/matth/projects/ld/tiles/localdata-tiles/tileserver.js:37:13)
    at Module._compile (module.js:456:26)

We don't have any layers that are drawn on top of our main data layer. So if we find new entries for a tile, we could draw them on top of the existing image.

We check the count and time of last modification for cached image validation. We could instead check the count of documents within the bounding box and before the last modification time. If that count doesn't match, we had a deletion and should refetch and rerender. If that count matches, we fetch entries after the last modification time. If there are none, the cached image is good to go. If there are new entries, we can render those to a PNG and composite that with the cached image.

If the composite time is not large, this could save a lot of time when updating tiles at low zoom levels. Our system primarily has additions, so we would seldom have to fetch all of the data for a tile.

The UTFGrid data is cached as JSONP text. Cached data is served with a redirect to S3. S3 doesn't gzip responses, though.

One option is to store the raw and gzip'd responses on S3. Based on the Accept-Encoding header, we can redirect to the appropriate item.

Another option is to proxy the S3 data through the server and use the built-in conditional compression. The underlying zlib calls are async, so we should not end tying up the event thread. This option is appropriate if we find that the client-side overhead is high for redirects.