openimagerynetwork / oin-meta-generator Goto Github PK

osm-dynamic-tiler depends on this and currently uses a GitHub reference. It would be great to refer to a version and pull from npm instead.

I think https://github.com/openimagerynetwork/oin-meta-generator/blob/master/index.js#L78 is causing us to miss the last chunk of files, should it be?

  for (var i = 0; i <= limitParallel; i++) {
    var start = chunks * i;
    iterator(start, start + chunks, data.Contents);
  }

http(s).globalAgent.maxSockets defaults to 5 until 0.11 (at which point it becomes Infinity): https://nodejs.org/docs/v0.10.39/api/http.html#http_agent_maxsockets

node-s3-client includes a note about this: https://github.com/andrewrk/node-s3-client#tips

The generated metadata files in meta drop any folder prefix information they may have had on S3. This is a problem when multiple folders contain imagery with the same filenames (since the last one to be written wins).

I've been running this a bunch of times now and seem to be running into a magic wall at 548 files. As soon as it tries to process that many, I get the error below. Could the s3 module have some limit internally causing this?

/Users/flasher/workspace/oin-meta-generator/node_modules/s3/node_modules/aws-sdk/lib/request.js:32
          throw err;
                ^
TypeError: Cannot read property 'addListener' of undefined
    at Object.exports.execFile (child_process.js:781:15)
    at Object.exports.exec (child_process.js:642:18)
    at Object.module.exports.remote (/Users/flasher/workspace/oin-meta-generator/node_modules/gdalinfo-json/index.js:86:17)
    at generateMeta (/Users/flasher/workspace/oin-meta-generator/index.js:112:12)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:65:7)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:74:5)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:74:5)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:74:5)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:74:5)
    at iterator (/Users/flasher/workspace/oin-meta-generator/index.js:74:5)

AWS_ACCESS_KEY_ID appears to be the defacto environment variable used for this purpose (and may already be set in some peoples' environments). AWS_SECRET_KEY_ID is used here instead.

An error popped up after processing new imagery on production. Metadata is generated fine (example), but when the catalog indexes the new image, mongodb throws an error:

message: 'exception: Can\'t extract geo keys: { _id: ObjectId(\'58fffebeb0eae7f3b143c1a8\'), uuid: "http://oin-hotosm.s3.amazonaws.com/58fff92190a38300103c37c8/0/0cb3651f-d796-4e6a-ab58-ecaf12569d0b.tif", geojson: { bbox: [ 318567.4056300001, 7842361.890440002, 318843.0565999987, 7842607.837750001 ], coordinates: [ [ [ 318567.40563, 7842607.83775 ], [ 318843.056599999, 7842607.83775 ], [ 318843.056599999, 7842361.89044 ], [ 318567.40563, 7842361.89044 ], [ 318567.40563, 7842607.83775 ] ] ], type: "Polygon" }, meta_uri: "http://oin-hotosm.s3.amazonaws.com/58fff92190a38300103c37c8/0/0cb3651f-d796-4e6a-ab58-ecaf12569d0b_meta.json", footprint: "POLYGON ((318567.40563 7842607.83775,318843.056599999 7842607.83775,318843.056599999 7842361.89044,318567.40563 7842361.89044,318567.40563 7842607.83775))", bbox: [ 318567.4056300001, 7842361.890440002, 318843.0565999987, 7842607.837750001 ], gsd: 0.02512999999987835, projection: "PROJCS["WGS 84 / UTM zone 59S",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","63...", file_size: 24653540, uploaded_at: new Date(1493170469325), acquisition_end: new Date(1428188820000), acquisition_start: new Date(1428185220000), properties: { license: "CC-BY 4.0", sensor: "Unknown", wmts: "http://tiles.openaerialmap.org/58fff92190a38300103c37c8/0/0cb3651f-d796-4e6a-ab58-ecaf12569d0b/wmts", tms: "http://tiles.openaerialmap.org/58fff92190a38300103c37c8/0/0cb3651f-d796-4e6a-ab58-ecaf12569d0b/{z}/{x}/{y}.png", thumbnail: "http://oin-hotosm.s3.amazonaws.com/58fff92190a38300103c37c8/0/0cb3651f-d796-4e6a-ab58-ecaf12569d0b_thumb.png" }, contact: "Keiko Saito,[email protected]", provider: "Government of Vanuatu", platform: "uav", title: "Tanna Letobom Ortho 05042015_1007", __v: 0 } 
 longitude/latitude is out of bounds, lng: 318567 lat: 7.84261e+06',

It seems that for these images the bbox and footprint units are in meters, when mongo expects them to be in geographic coordinates, latitude and longitude.

cc @tombh

The AWS SDK supports multiple credential sources and transparently uses them. This means that if I have $HOME/.aws/credentials or $HOME/.aws-credentials-master configured, it will pull values from them if none are available in the environment.

This check should be less aggressive: https://github.com/openimagerynetwork/oin-meta-generator/blob/master/index.js#L20-L26

(I use require-env for this purpose, and assign values to constants with env.require("WHATEVS") early in to effect the same crash-first behavior.)

Wondering if this is meant as a standalone app or rather as part of an upload Web form?

From what I understand in the scenario described, image data is already existing in some bucket, but is missing metadata json files. This tools would allow extracting basic gdalinfo metadata plus letting the user add any OIN specific values.

Up till now we've been using a simplified expression of footprint that provides little more than that already offered by bbox. @mojodna has already implemented a more comprehensive format in oam-dynamic-tiler's tiler-prep stage. As he describes, it is;

non-rectangular and shows the image's footprint after it's been masked out, with nodata and otherwise [...] this samples at 1% (?) to keep the GeoJSON size down, since the end result is stair-stepped no matter what (because it's polygonizing pixels)

This is clearly a better solution. However, we need to formalise both the necessity and optimal format. Therefore:

1. What purpose does the footprint serve? Does it serve any purposes beyond oam-browser's use? Can the purposes be served by the bounding box or a centroid? Or perhaps having a detailed footprint means that a bounding box field is superfluous?
2. Currently the footprint payload is non-trivially large, for example, for a 325MB image, this footprint is 24kb. Of course, this can be simplified by down-sampling, simplifying polygons and using topojson. However, we need to define certain expectations. Can a single OIN meta data payload function reasonably as intended if it is, say, over 5kb? If not then can the footprint be useful with a limit of, say 3kb? If not then should the footprint be kept separately?

Follows on from discussion in #26.

In #10, I started to explain the layout of my terrain buckets. In addition to the 4326/ prefix, I also have a 3857/ prefix that contains derived data (for tiling purposes), both reprojected and resampled per zoom. Since that's not intended directly for publishing purposes, I'd like to limit the prefix that the tool uses to list data to be processed.

I have a bucket containing a bunch of zipped TIFFs (DEMs, in this case), all contained within a source/ prefix. For testing purposes, one of them is uncompressed: 4326/srtm_01_02.tiff

When configuring oin-meta-generator to point at the bucket in question (s3://cgiar-csi-srtm.openterrain.org), it logs the following messages but doesn't do anything further (I'm using the default config.json with the assumption that I'll fill in correct values later):

$ npm start

> [email protected] start /Users/seth/src/openimagerynetwork/oin-meta-generator
> node index.js

Successfully connected to S3 bucket, now retrieving data.

What should I expect to see / have produced for me?