localdata / localdata-tasks Goto Github PK

Newlines in text fields become newlines in the CSV.

If we ask for an export for a survey that doesn't exist, we should log an error or warning, since that's likely a sign of trouble/misconfiguration. Right now, we just happily process 0 responses.

We can do this on the side, while we stream in the responses (maybe a promises library is in order).

We only include the object id right now, which makes manipulating the data on a per-entry basis difficult.

Right now we assume there won't be a mix of Point with MultiPoint geometries, Polygon with MultiPolygon, LineString with MultiLineString. That might not always be true, though. We should be able to promote to multi-geometries with -nlt PROMOTE_TO_MULTI (http://www.gdal.org/ogr2ogr.html).

We get the following error exporting walkscope data. Walkscope has a mix of point + linestring data, which may be part of the problem?

Apr 17 13:45:23 localdata-tasks app/taskman.1:  { [OperationalError: SQLITE_RANGE: bind or column index out of range] 
Apr 17 13:45:23 localdata-tasks app/taskman.1:    name: 'OperationalError', 
Apr 17 13:45:23 localdata-tasks app/taskman.1:    cause: { [Error: SQLITE_RANGE: bind or column index out of range] errno: 25, code: 'SQLITE_RANGE' }, 
Apr 17 13:45:23 localdata-tasks app/taskman.1:    stack: 'Error: SQLITE_RANGE: bind or column index out of range' } 

With #19, we create 3 sqlite files and run ogr2ogr on them, even if there were no shapes of that geometry type. We should monitor the geometry-type-filtered streams and skip layers as appropriate.

From an emailed bug report:

Also, I recently came across an issue in the survey results I just downloaded: when I open up the survey results, all of the entries for “Matthew, S, Hampel” have been sliced into three separate columns, which moves all of the data in his results to the right one column. I’m guessing this is because he entered his name with commas, which is typically used in a CSV to separate columns, but I’m not certain why this just recently started happening

From a client:

I noticed that when we export the questions they don't always come out in the same order. For example, on 10/6 the question about the condition of the house appeared in column N. Now it is appearing in column W. This has made it a little tricky to merge files that we exported on different days.

Do you think this was a one-time change based on something adjusted in the survey or is this something that will be happening regularly?

When creating a Shapefile for a survey that has 60k+ entries, we can user over 1 GB of memory. This seems attributable to the ogr2ogr GeoJSON->Shapefile conversion command. The performance degradation causes the export to take several minutes, at which point we've long since reported an error to the user.

If the Shapefile format allows it, then let's see if we can get close to streaming data out. At the very least, we should avoid reading the entire dataset into memory.

Some customers would find a vector graphics export useful for visual design workflows. SVG is probably reasonably straightforward/well-supported option.

Accept a public flag that, when present, will strip collector information from exports.