getodk / aggregate Goto Github PK

Comparing export to csv with ODK Briefcase, the exported csv emits 'null' when a cell
is blank, whereas the ODK Briefcase export emits an empty cell.

The ODK Aggregate implementation should emit an empty cell.

It would be nice for American users to have the option to display the altitude units
in feet.
Also, maybe a choice in lat / lon units: DD MM.MMMMM or DD MM SS.SSSSS or DD.DDDDD.
Thanks a lot for the great work.

Using Aggregate 1.0, comment from user ( [email protected] ):

2) Fusion table names default to 'data'. Changing the name seems to
break the link. Furthermore having multiple fusion tables each linked
to a form breaks all but the first publishing link. Presumably there
is a conflict due the other tables also being named 'data'? When
publishing to Google Spreadsheet one can define the tablename, but not
with Fusion. Is there a way of defining it in the form?

---
We should verify that publishing to multiple fusion tables from the same form works
and that different fusion table names can be specified (assuming Fusion Tables allows
this).

The Aggregate install script should check for Java 7 before running. That'll prevent
this message (and the messages to the list)

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/appengine/tools/admin/AppCfg
: Unsupported major.minor version 51.0

Should also be documented on the website.

Once a FilterGroup is saved. If the filter group is not switched off of and then saved
again, the URI is may not be being set on all the dependent entities. It may only be
set on the FilterGroup element and not in all the entities that make up the FilterGroup.

What steps will reproduce the problem?
1. Go to https://cnp-rainfall.appspot.com/openid_login.html
2. Click the "sign in with google"
3. Error 400 appears, "OpenID auth request contains an unregistered domain: https://cnp-rainfall.appspot.com/"

What is the expected output? What do you see instead?
I should be able to login with Google but instead am diverted to a page telling me
OpenID is being / has been deprecated.

What version of the product are you using? On what operating system?
ODK 1.4.3 on windows 8


Please provide any additional information below.

I'm unsure how to migrate to a newer version of OpenID given that ODK runs all the
set up automatically.

What steps will reproduce the problem?
1. Log into Aggregate
2. Click "Site Administration"
3. Click in full name field and try to enter a space

What is the expected output? What do you see instead?
I should be able to enter a space, as in "Jane Smith," but I'm not.

What version of the product are you using? On what operating system?
ODKAggregate-1.0 RC1, accessed via Safari 5.1 (7534.48.3) and FireFox 4.0.1.

Please provide any additional information below.
None.

A community member discovered that urls in Tomcat are case sensitive and requested that
we update documentation

I updated the documentation on both
http://code.google.com/p/opendatakit/wiki/AggregateTroubleshooting
http://opendatakit.org/use/aggregate/tomcat-install

However it might make sense to warn people in the installer.


------------------------

Hi there,

just a useful note you may want to add to
http://code.google.com/p/opendatakit/wiki/AggregateTroubleshooting.

I set up a Tomcat server for ODK Aggregate which receives data from
ODK Collect via an SSL connection. I spent some time trying to
troubleshoot why my phone was not connecting to ODK Aggregate, when I
realised that the server URL is actually case sensitive in Tomcat.

So I set up my server at https://aomceldk.bounceme.net/ODKAggregate
(the site where I'm testing the system currently).
If you enter aomceldk.bounceme.net/odkaggregate into the browser it
can't find ODK Aggregate, only if you enter aomceldk.bounceme.net/
ODKAggregate. Since I had typed everything in lower case into my
phone, I kept receiving Not Found (404) error message.

I'm sure experienced web designers know that URLs are case sensitive,
but for a beginner like me this was quite a surprise - so maybe useful
to add under Step 1 of "Cannot Retrieve List of Forms".

Best,

Lu

What steps will reproduce the problem?
1. Publish to Google Spreadsheets
2. Open the spreadsheet and attempt to set a data validation on a date column.
3. Notice that Google Spreadsheets complains about the date values in the column.

What is the expected output? What do you see instead?

Should be able to have Google Spreadsheets recognized the date or datetime as a valid
date.

What steps will reproduce the problem?
1. Define a form with repeat groups (or groups)
2. Add a <bind> for the group with type="string"

What is the expected output? What do you see instead?

Aggregate should still detect that it is a real group and ignore the spurious type="string"


Validate should complain about this.

What steps will reproduce the problem?
1. upload (attached) form to latest Aggregate
2. load the form in ODK Collect
3. submit

What is the expected output? What do you see instead?
Submission to be successful, returning 201

What version of the product are you using? On what operating system?
Latest aggregate on appspot.com

Please provide any additional information below.
The same issue occurs for Infinity, -Infinity and NaN (though the latter is not a possible
output by ODK Collect)

ODK Aggregate uses GWT 2.5.x for its web page rendering. At the time we began development,
internationalization support was not yet built into GWT. It now is ( http://www.gwtproject.org/doc/latest/DevGuideI18n.html
).

However, to enable it, we would need a developer to go through the client code and
extract the display strings into a translatable-strings dictionary and make the simple
changes to the code to access that dictionary for the display string.

We lack the resources to do this at this time. If anyone has a strong desire for this,
and can contribute a developer's time to sift through the code and extract the translatable
strings, we'd be happy to begin supporting internationalized ODK Aggregate web pages.

Create a username without a password.

Try logging onto it.

This should present an Access-denied error or some sort of messaging that the user
is invalid.

Instead, it appears to be reporting a generic server error (out-of-quota).

Perhaps this is a regression with the migration to Java 7 ?

From Email:
---------------------------
There are two ways to go:

(1) Change Aggregate to show dates in a specified timezone (e.g., have a timezone choice
on the Site Admin / Preferences tab). Use that designated timezone for all exported
csv and kml data, and all published data to Google Spreadsheets or Fusion Tables.

This is roughly the path Aggregate is heading down now.  Since Google's App Engine
hosts are all configured with GMT time, we sort of standardized on preserving that
end-to-end. 

The change would be confined to the formatting layer, and would be straightforward
to do.

(2) Actually preserve the timezones of the incoming data (a systemic change).

This would change Build/XLSForm and Collect to emit metadata that communicates the
timezone of a filled-in submission as a preload metadata element (e.g., in the OpenRosa
block) so that date types (which lack timezone information) could be properly timezoned.
 If that were present, we could then preserve timezones for all datatypes into the
datastore layer.

The problematic datastore is then MySQL.  Google and PostgreSQL already provide storage
types that can preserve timezones.  For MySQL, it probably makes sense to store a timezoned
datetime as two columns -- the UTC datetime and the timezone. Complicated but a fairly
constrained change within the MySQL datastore layer.

This would be an Aggregate 1.1 feature, along with string-valued version and full OpenRosa
1.0 compliance.

What steps will reproduce the problem?
1. publish some submissions
2. choose 'Purge Submissions'
3.

What is the expected output? What do you see instead?

The purge submissions implementation does not pay attention to the URI of the last
record published.  It can therefore purge only up to before the last transmitted timestamp.

In heavily used systems, or ones that received bulk uploads, this can cause some published
submissions to remain in the datastore (those published in the same 1-second or shorter
interval, depending upon the datastore used).

What steps will reproduce the problem?
1. as site admin, sitting on Permissions tab with local changes
2. do other things for 15-20 minutes.
3. return to app.
4. navigate off -- notifies you of pending changes
5. save changes

What is the expected output? What do you see instead?

Should save changes or give reasonable error message.

Error message return is the body of the openid_login.html page
(indicating that the OpenId auth has expired).  Should be able 
to detect this redirect and present a more reasonable error message.

There are scenarios where you want to feed data from one Aggregate instance to another.
For example, you can have a hierarchical data collection campaign where data is reviewed
in each district before being forwarded on to the country level. In these scenarios,
it'd be great to have a way to forward data from one instance to another. I suppose
you could use Briefcase, but transfers would be a lot faster and easier if you go server
to server.

What steps will reproduce the problem?
1. create a UTF-8 XForm document with a BOM
2. upload to Aggregate or download to the phone.
3. watch things crash

It would be really really nice if the BOM character could be detected and skipped so
that the XML processing can succeed.  A lot of UTF-8 editors stick a BOM character
at the start of the file, causing many users to be puzzled.

I have a question about downloading the data from Agreegate (on appspot server) with
repeating elements - when I go to the 'download CSV' option, it doesn't download the
repeated element data, instead it just shows the URL for the extra data table.  I can't
seem to find a way to directly download that repeated data - is there a way for doing
so that I'm missing?

"I would like to know if ODK can pass the handset id of an android through
the server url. I would like to limit the xforms that can be accessed by
different groups of people based on the group to which the handset id has
been assigned."

It would be beneficial to be able to export submitted forms as PDF reports so that media
and captions are directly linked with recorded data.

After 4 weeks of error-free data exporting to CSV with ODK Aggregate, one of the forms
has stopped exporting to CSV.  Trying to export FormX fails - after many minutes of
the screen saying "Generating", the screen reads " "Failure - abandoned all retry attempts.”
I've gotten this error before, and usually it goes away if I try to export a few hours
later, but after 5 days, it will still not export. 

I'm using the most recent version of ODK Aggregate, there is not a billing problem,
I've adjusted AppEngine to allow for maximum capacity. There is no error that appears
in Systems on AppEngine, and there are errors in the Log report, but nothing that I
can interpret. Other forms on the site export with no errors. 

My current solution was to upload the same form that was failing to export as a new
version, and its working fine - so is it possible this form got corrupted? How can
I retrieve the data from this form? 

We have a use case where we want to use ODK Collect/Aggregate to collect health information.
That health information will be inserted into a third party health system. We anticipate
that there will be validation and other errors that could occur during form entry.
We would like to keep invalid forms in Collect until the field worker has corrected
them. After they have corrected them we assume they will be resubmitted.

I believe this will require changes on both Aggregate and Collect if it is accepted.
Ideally, Aggregate would communicate all invalid submissions, and a brief message as
to why the form submission failed. All valid forms would be deleted from the Collect
client.

I am also severely missing a possibility to export data to statistical programmes such
as Stata or R including variable labels and value labels, i.e. essentially the underlying
logic of the questionnaire. Before, we were using pencil paper questionnaires and the
free (but not open source) CSPro as data entry tool. It was possible to directly export
to Stata, which was extremely handy. Any change in the questionnaire was immediately
and correctly reflected in the Stata export. As any digital questionnaire includes
the information of (a) the question, (b) variable name, (c) answer categories in words
and (d) associated values in numbers. As far as I understood, (a)-(c) would be lost
in ODK .csv export and only (d) is kept. Is that correct? IMHO, this would mean much
(error-prone) duplication of work. 

How difficult is it to include (a)-(c) and to develop an export to statistical programmes?
Has already anyone started to write an export filter to one of these statistical programmes?

--Gerry

What steps will reproduce the problem?
1. Go to Site Admin -> Preferences
2. Configure Enketo Webform Integration
3. By entering API URL and API token

What is the expected output? What do you see instead?

It should show me the web form instead of it raise error "There was an error obtaining
the webform. (message:no account exists for this OpenRosa server)"

What version of the product are you using? On what operating system?

ODK Aggregate 1.4.1

Please provide any additional information below.

I had created account in Enketo with different gmail address than I had used to access
ODK Aggregate for first time.

Kindly guide me what wrong I am doing in configuration.

What steps will reproduce the problem?
1.use ODK Aggregate v1.0.1
2.Log into https://orrholly.appspot.com
Aggregate user: Student
Aggregate password: password
3.upload the attached file Lab-13v8_Biodiversity_v8.xml as a form
4.download form to Xoom Tablet on Kobo Collect App
5.send finished form back to Google App Engine site (https://orrholly.appspot.com)

6. Try to export submission to KML with the following attributes:
Geopoint: location
Title: name
Picture: repeat_observation/image

What is the expected output? What do you see instead?
Should export KML.  Error message: Failure - will retry later

What version of the product are you using? On what operating system?
ODK Aggregate v1.0.1
Desktop Window 7
KoBo collect 1.1.7-rc-1
Motorola Xoom / Android Version 3.2.1


Please provide any additional information below.

What tool are you using?
Android phone

What is frustrating, troubling, annoying or lacking?
It is frustrating that when you take a photo it gives us a link to somewhere on the
ODK server, which, when you are going through thousands of necessary photos it becomes
hard to distinguish where the photo came from to begin with.

How would you like it to work?
We would like to be able to either rename the link for the photo so we can recognize
it, or have a way to edit directly in ODK.


NOTE: the core team has limited ability to make software enhancements.
We rely heavily on community-contributed enhancements to the tools. If this
is something you very much need, consider hiring a developer to make the
changes with the stipulation that they work with the core team to specify
the details of the change and contribute the change back to the project
under an Apache 2 license.

in form submission results, it'd be nice to see the question text not the
data type.

What steps will reproduce the problem?
1. during the upload-to-appengine phase
2. enter an invalid username/password
3. upload fails but reports SUCCESS

What is the expected output? What do you see instead?

Google's script apparently returns zero on this failure condistion, rather than non-zero,
as is industry standard.

What steps will reproduce the problem?
 1. Setup IIS using the ARR module to generate url rewrite rules to  route requests
to Aggregate, but without the port number in the URL
 2. Sign into Aggregate
 3. the URL is not well formed and consists of e.g. http://myDomainwithoutPortNum/ODKAggregate/http://myDomainwithoutPortNum/ODKAggregate/Aggregate.html

What is the expected output? What do you see instead?
 Expected: http://myDomainwithoutPortNum/ODKAggregate/Aggregate.html
 Actual:http://myDomainwithoutPortNum/ODKAggregate/http://myDomainwithoutPortNum/ODKAggregate/Aggregate.html


What version of the product are you using? On what operating system?

Windows Server 2008, Chrome Browser; Aggregate 1.0.2

Please provide any additional information below.

Possible solution - Change the way that the post login URL is formed.  My guess is
that the url is tokenised, perhaps using the 8080 port as a way of splitting the URL.
 Instead, the / or \ could be used.

This is the only step where this is an issue, all the other links between pages work
fine.

What steps will reproduce the problem?
1.entering static IP address to network adapter OR; using 3g modem internet connection
with assigned static Public IP address.


What is the expected output? What do you see instead?
I was expecting the server to work on internet connection with universal static IP
address(specially that there is no fie wall except  f WINDOWS's) ,  ODK VM booting
without Network configuration, can't deploy on the internet network because of invalid
Network adapter setting. 

What version of the product are you using? On what operating system?
ODK VM aggregate v1.4.3.0, using win 7 professional

Please provide any additional information below.
no special Fire walls, ODK VM work perfectly on LAN setting.

What steps will reproduce the problem?
1. I uploaded all my forms well using a Samsung galaxy tab. I got the successful upload
message on the device. 
The problem is my administration team cannot see all of my forms that I uploaded to
that particular server.  Is there a way to diagnose or retrieve these forms? its puzzling

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
Android 4.x the latest

Please provide any additional information below.

When the export ends up in Failure - abandoned all retry attempts, Aggregate does not
delete the corresponding filter - "FilterForExport". Successful exports delete the
"FilterForExport".

We should chat to make sure it works out right.

I tried uploading data from our own instance of ODK Aggregate 1.0, on Google AppEngine,
to Fusion Tables, and getting an error message:

    Unauthenticated access is allowed only for SELECT andDESCRIBE statements

    Error 401

    CREATE TABLE 'zambia-ranq-round3'(...)

    Error: org.opendatakit.aggregate.exception.ODKExternalServiceException:
    Unauthenticated access is allowed only for SELECT andDESCRIBE statements

What steps will reproduce the problem?
1. ODK Aggregate, Visualize >> Geopoint to map
2. When you click in the map on one of the point locations (=red drops) the map moves
(shifts)
3. So this bug doesn't let you watch the metadata/attribute data in the map visualisation

What is the expected output? What do you see instead?
The map shifts all the time when you go near one of the point locations (red drops).
So you can't see the attribute data in the visualisation-map-mode.

What version of the product are you using? On what operating system?
I suppose the latest version...?

Please provide any additional information below.

From a user:

I have a ODK aggregate form that I am uploading to Google docs. There
is multiple select sections in it with numeric underlying values, i.e.
1, 2, 3, 4 ,5 etc. Now uploading gives me in the coloumn, just two
numbers, say "3 5". No problem here, sending to Google docs, google
change my data to a date. For the above I get 3/5/2011. Formatting the
column to text is no use either, since the original data is lost
already, at least I must manually change it back to "3 5".

How can I prevent this from happening, other than changing the
underlying value to text?

----------------
Will see what can be done to get Google Docs to not do this.

If you have a google-hosted domain, e.g., @camfed.org,
and you try to publish, publishing is apparently broken.

No error is captured in the log(!).

The check for form-deletion causes the misc tasks table to be scanned a lot.

Should perhaps handle this differently?  e.g., would be nice if FormInfo had a flag
indicating whether the form was marked for deletion. (v2.0?).

Performance could be impacted if you are deleting forms very frequently.

Would be helpful to have an option to display two columns of data, one 
for the english choice and one for the underlying value. An easy 
approach i've used is to build an english list with underlying names 
the same as the english names and then recreate the same list again 
with a second list of underlying values that equal the numerical 
values i'm after. But it makes the form a little cumbersome and 
redundant looking 

On GAE, it appears that deleting records can 'succeed' without actually completing.

Subsequent reads of BigTable can then encounter these records, causing the UI to enter
an infinite-refresh cycle until the FormInfo structures are manually cleaned up.

The fix would 
(1) add logic to automatically clean up these failures. 

_form_info_fileset -- when reading these, call delete on the oldest of any recorded
entries.

_form_info_xform_bin -- remove the oldest of these also, with the same logic.

(2) add a flag identifying a mis-configured form, and enable the user to delete that
form from the UI.

(3) make the delete-form process more robust and extensive, ensuring that any rows
related to the form are deleted.

Many of the unit tests of the persistence layer assume if you put() a value, then an
immediate get() will retrieve it. This is not true in the parallel datastore dev environment.


And production code shouldn't be doing immediate fetches anyways.

Need to look into how to correctly flush GAE datastore during unit tests so these random
errors don't occur.

From [email protected]

I have installed ODK aggregate on a server with Tomcat and Postgres. Everything works
fine, only publishing to GME gives error.

I have setup OAuth for a google account and uploaded the same in ODK aggregate. Now
I can publish to Fusion tables. Same google account is an admin for GME and created
Vector table in GME using following KML file.


<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2">
  <Document>
    <name>GME Geo Tagger</name>
    <Folder>
      <!-- Placemark is needed to build the data store -->
      <Placemark>
        <ExtendedData>
          <Data name="/data/meta/instanceId"/>
          <Data name="/data/meta/instanceName"/>
          <Data name="/data/geolocation"/>
          <Data name="/data/name"/>
        </ExtendedData>
        <!-- Replace with a polygon to set the zoom level -->
        <Point>
          <coordinates>0,0,0</coordinates>
        </Point>
      </Placemark>
    </Folder>
  </Document>
</kml>

When I try to publish the data using assert id of above created GME vector table, getting
status as "Restart Publisher - Credential was BAD". I dont know what I am doing wrong
here. Please help.

ODK Collect reports XML validation error on line 1 column 2!

Some applications will add the byte order mark when producing a UTF-8 document: https://en.wikipedia.org/wiki/Byte-order_mark

This will cause an xml validation error. 

Want a URL that points to a download of the CSV or KML.

I got it all setup and can get the KML or CSV (CSV has time but KML
does not) and display it in ArcMap but it is a manual process.  Just
wanted a way to grab the KML file from a URL so i can do it easily in
Python like this that I did for EPICollect.  

import  arcpy, os, urllib, sys

# Setup URL
kmlURL = "http://epicollectserver.appspot.com/getMapKML?projectKey=ahBlcGljb2xsZWN0c2VydmVychALEgdQcm9qZWN0GJ-E3AEM"

kmlFile = "c:/temp/epilitter.kml"

# Get html page and store in a local file
urllib.urlretrieve(kmlURL, kmlFile)

Hope that makes sense.  I will submit it online.

Thanks a lot

What steps will reproduce the problem?
1.Log in to aggregate(www.xxx.com/ODKAggregate/) from outside office network.
2.Select the form and click "export submission" from Submission>>Filter Submission
3.Go to "Exported submission" click filename i.e xxxx.csv from "download file"

What is the expected output? What do you see instead?
It should open "save as" window to save the downloaded file. Instead, it goes to the
url www.xxx.com:8080/view and the page is blank. 

What version of the product are you using? On what operating system?
V.1.4.2

Please provide any additional information below.

Define a form with a comment like this at the top:

<!-- This is a test from comment 
    this will blow up: >
blah blah -->
...


This causes ODK Aggregate to insert the load timestamp comment after the first >, inside
the comment, causing future XML parsing to fail.

Can we avoid fetching any of the _bn records for the submissions data page?

I think we can construct the URL for the media without knowing this info. And we know
the media type from the form definition, so we know what treatment to do without getting
the exact contentType.

This would allow ODK Scan surveys to be presented/hosted in AppEngine and not hit the
60-second limit.

We would then only do explicit reads when accessing the media files, the form definition's
media list, and never when accessing the submission rows.

What steps will reproduce the problem?
1. Define a dateTime field.
2. Pull the value of that dateTime field into a later calculate field.

What is the expected output? What do you see instead?

You expect the date and time both to go into the calculate field. Instead, only the
date part is saved.

Please use labels and text to provide additional information.

The issue appears to be in Recalculate.wrapData(), called by Recalculate.apply(). You
are going from a DATETIME data type to a STRING data type, and the default behavior
of wrapData() is to only return DateData((Date)val), as opposed to TimeData((Date)val).
Granted, the data type for the source value is Date in both cases, and so it's a bit
ambiguous. However, it would be easy enough to return TimeData when there is time data
in the Date object.

I am happy to submit a fix for this if there's consensus that it's a good idea.

"we did some tests with forms with multi select fields and are now
wondering about how the stored information should be used, e.g. with
the "Visualize Tools" of ODK Aggregate 1.0.
The problem is that values being selected in a multi select are stored
as combined individual values and thus don't count for the single
values.

Example Code:
<label>Defoliation</label>
             <select ref="/SoilQualitySurvey/Defoliation">
               <label>Select</label>
                <item>
                   <label>Widespread</label>
                   <value>Widespread</value>
               </item>
               <item>
                   <label>None</label>
                   <value>None</value>
               </item>
           </select>


When we select "Widespread" and "None", the diagrams and Spreadsheets
process the data as "Widespread None" and not "Widespread" and "None".
(I would like to attach a Screenshot to make this situation/problem
clearer... I hope you understand what I want to say.)

Or do we make use of the multi-select in a wrong way?"
-- LETSGO CeLeKT



"This is a tricky data representation question.  We punted it in 0.9.x and 1.0 because
a standard solution won't make sense in all situations.

If you have a multi-select with 30 choices, would you want a spreadsheet to be generated
with 30 column headings for this one multi-select, and a "X" under the columns with
matching values?  What if there are 50 choices?   What if the number of choices is
unlimited (this is possible with itemsets drawing their choices from an external database)?
At some point, it makes more sense to produce a spreadsheet with just "N" columns for
a given multi-select (1st choice, 2nd choice, ...), and show up to N choices for that
multiselect (but which N do you show?).

This would all need additional layers of configuration over the simpler version available
in 0.9.x and 1.0.

If you have your data in a MySQL or PostgreSQL database, you may want to look at Tableau
or another data visualization package that can directly work off the native database
tables holding your data."
-- Mitch

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.
 after successful installation when i try and use my username and password is rejected.

JAVA 7, tomcat 6,MySQL community 5.6,ODK Aggregate 1.4

HOW CAN I RESET OR BE ABLE to change the password??