e.g.

error -> default
warn -> -v
info -> -vv
trace -> -vvv

The forms PLACE_TYPE-create.xlsx, PLACE_TYPE-edit.xlsx need to be converted to:

• clinic-create.xml
• clinic-edit.xml
• district_hospital-create.xml
• district_hospital-edit.xml
• health_center-create.xml
• health_center-edit.xml

The only difference between them is that PLACE_TYPE and PLACE_NAME are replaced differently.

The XForm also has some custom edits to work as a place form and create the contact properly:

• Move place to above person (in data model, due to XML ordering mattering, even though it shouldn't: medic/cht-core#1494)
• Move custom place field from init to within place form - so that it isn't saved in place doc
• Move new contact person to first page - move the group inside previous group
• Add space after output field medic/cht-core#3324

The forms at https://github.com/medic/medic-projects/tree/master/standard/forms are fully generated by this convention, so tests could be created based around the xlsx and forms in that project.

The media files listed in an XLSform are not found in the app because the links are incorrect. All images and videos are given a path within an image or video folder by pyxform eg jr://images/positive.png, but on the form doc these media files are in _attachments with no folder.

In the previous convert.sh script these folders were stripped, but we are not (yet) doing this with convert-forms.js. The workaround for now is to edit the XML to strip the folders before running medic-conf with upload-app-forms. For instance, <value form="image">jr://images/positive.png</value> becomes <value form="image">jr://positive.png</value>

Against local couch or pouch server.

We have 10+ Standard projects using the same base configuration that gets updated whenever fixes or new features are added to the Standard config. Typically the different Standard projects have slight modifications to the Standard config, such as locale_outgoing, default_country_code, and gateway_number, and we do not want to override these when pushing an updated Standard config..

Manually maintaining the modifications for Standard projects is very error prone and important changes can get lost with updates. To avoid this situation we have been using a child settings.json which refers to a parent app_settings.json. We have a script that then generates the app_settings for the child project accordingly. We should include a reliable way to do this within medic-config to get rid of one of the last separate config scripts still in use. This is a good time to review and improve the process if possible.

Excerpt of usage in the settings.js script related to the child settings.json:

Structured like app_settings.json, and copied into the parent settings.
Additionally, special fields can be used:
  _parent  path to parent app_settings into which these settings will be merged
  _forms   array of forms to include. All others will be removed. Because the
                merge into the parent settings uses a deep copy it is not possible
                to remove fields. Setting which forms to use allows a future proof
                way to specify the forms to include in the final app_settings.

Here is an child setting file which specifies the forms in use, so that forms for other use cases are not shown in the reports filter*:

{
  "_parent": "../standard/app_settings.json",
  "_forms": [
    "N",
    "P",
    "V",
    "D",
    "F",
    "OFF",
    "ON"
  ],
  "locale_outgoing": "id",
  "default_country_code": "62",
  "gateway_number": "+62812_______",
  "district_admins_access_unallocated_messages": true,
  "outgoing_deny_list": "3636, 1166, 98888, TELKOMSEL"
}

* it could be better to add context to JSON forms so that they can be hidden from the Reports tab filter accordingly.

report.x.csv -> x.report.csv
place.y.csv -> y.place.csv

also allow prefixes, e.g.

arbitrary.person.csv
brbitrary.x.report.csv
crbitrary.y.place.csv

Header content should be redacted.

e.g. with one of:

• https://www.npmjs.com/package/minimist
• https://www.npmjs.com/package/commander
• https://www.npmjs.com/package/yargs

should be delimeter for separating actions from action-args, but it's incorrectly treated as an action

Creating users has become quite complex, as you need...

• a doc in the _users db
• a user-settings doc in the medic db
• a contact to associate with the user (optional but wanted most often)
• a facility to assign the user to (optional but wanted most often)

This is really annoying when creating a bunch of users.

Create a bulk import command so it's easy for an admin to upload a file and have all required documents created on the server.

While there it may make sense to implement a user export, but the export wouldn't be able to have the password, so make sure you handle this case without overwriting existing user's passwords.

The current header naming structure is functional but confusing, especially that there are 3 different formats that work differently. We should update the header format to make it easy to use for tech leads and partners. Below is a proposed format that has a common format for plain headers, those with external references matching a field, and those matching to a row number.

CURRENT STRUCTURES:

1. No external reference, with optional conversion to Type:
   [ {{TYPE}}: ]{{TARGET}}

2. Reference to a CSV row:
   csv:{{doc_type}}: [ id | doc | .source_field ] >target_field_name
   Note: unclear if type conversion is currently supported with this format

3. Match to another doc:
   match={{field_in_other_doc}}:field1=value1&...&fieldN=fieldN: [ id | doc | .source_field ] >target_field_name
   Note: unclear if type conversion is currently supported with this format

PROPOSED STRUCTURE:

[ {{TYPE}}: ] [ {{SOURCE}}: ] {{TARGET}}

TYPE and SOURCE are optional. Without them we have the following:

{{TARGET}}
No type conversion or external reference

{{TYPE}}:{{TARGET}}
No external reference, set the type for conversion of data in the column.

{{SOURCE}}:{{TARGET}}
No conversion done, and the target gets the result of the reference as a string or object

It should be easy enough to differentiate the two 2-part formats since TYPE is one of 6 possible types that don't overlap with the SOURCE types.

TYPE

The type of data for this field, saved as corresponding JSON type:

• string default
• number
• boolean
• object
• date date in ISO8601 format
• timestamp date in MS since epoch. Useful for reported_date

SOURCE

The docs in which to look for the reference:
{{doc_type}}[?match={{field_name}}&{{field_X}}={{value_x}}&{{field_Y}}={{value_y}}]

• The doc_type is the type of the doc you are referencing: [data_record, person, clinic, health_center, district_hospital].
• With match the row's value is matched to the value of the specified column in the source docs. If match is missing the column's value (a number) will be used to find the corresponding row number in the source csv. See Questions section about whether we need to support row number matching.
• Further narrowing can be done with field/value pairs.
• If the source is empty that is ok, it is just not an external reference!

TARGET

Assigning the data to a field name in the resulting JSON:

[{{source_field}}>]{{target_field}} where source_field could be one of the following:

• .field_name for any field on the object
• doc for the whole doc (might not be needed at all, see Questions below)
• id for the _id value (might not be needed at all, see Questions below)

Some examples:

• target_property_name
• doc>target_property_name
• id>target_property_name
• ._id>target_property_name
• ._id>target_property_name._id
• ._name>target_property_name

EXAMPLES

Here are some column headers in 1.6.12 vs proposed structure

Without reference

sex --> ::sex
or sex (no change!)

timestamp:reported_date --> timestamp::reported_date
or timestamp:reported_date (no change!)

Reference by row number

csv:F:id>target_property_name --> string:F:._id>target_property_name._id
or F:._id>target_property_name._id

csv:F:doc>target_property_name --> object:F:target_property_name
or F:target_property_name

csv:F:.some_field>target_property
-->string:F:.some_field>target_property
or F:.some_field>target_property

Reference with match

match=P:X=a&Y=y:id>target_property_name
--> string:F?match=P&X=a&Y=y:._id>target_property_name._id
or F?match=P&X=a&Y=y:._id>target_property_name._id

match=P:X=a&Y=y:doc>target_property_name
--> object:F?match=P&X=a&Y=y:target_property_name
or F?match=P&X=a&Y=y:target_property_name

match=P:X=a&Y=y:.some_field>target_property_name
--> string:F?match=P&X=a&Y=y:.some_field>target_property_name
or F?match=P&X=a&Y=y:.some_field>target_property_name

match=external_id:type=clinic:id>parent
--> string:clinic?match=external_id:._id>parent._id
or clinic?match=external_id:._id>parent._id

match=external_id:type=clinic:doc>parent
--> object:clinic?match=external_id:parent
or clinic?match=external_id:parent

match=external_id:type=person:._id>fields.patient_id
--> string:person?match=external_id:._id>parent_id
or person?match=external_id:._id>parent_id

match=external_id:type=person:doc>contact
--> object:person?match=external_id:contact
or person?match=external_id:contact

QUESTIONS

• Is it necessary to have the doc and id keywords? Seems like we can eliminate them and still get the same result by assigning whatever the value is, either an object or string. Without it it is pretty clear what is going on and avoids needing to learn/understand the keywords.
• Do we need reference by row number? If there is no use case for this perhaps we can remove that to reduce complexity.
• Should we consider allowing type=place for looking at any place type? If so, differentiation between place types can still be done with the field matching eg type=clinic

The file structure that we currently have does not represent well how the content of the files are associated to each other.

example-project/
    ...
    targets.json
        tasks/
            rules.nools.js
            schedules.json

The rules.nools.js is in the tasks folder, but it also generates the targets, so it should be moved out of tasks/. Also, the json files are the definition files for possible tasks and targets, so we should rename them as such. At that point the tasks folder isn't really needed anymore:

example-project/
    ...
    rules.nools.js
    targets.json
    tasks.json

We should find a way to allow setting the default language for Collect forms. Having the default language attribute in our Enketo form breaks them in our app, so we remove it here. Setting a default language is useful however in Collect. This could be resolved in medic-configurer, otherwise we could figure out why the default language attribute is not working in our Enketo forms... and fix that.

https://prettier.io/ - if it's opinion is correct

Consider https://www.npmjs.com/package/js-beautify

Thanks @alxndrsn for making this awesome tool!

We should bring in it as a supported tool at medic at your earliest convenience.

Not very helpful if e.g. you want to upload to a cloned instance, and re-uplaod to a live one (or vice-versa).

type is set from csv filename.

It's a bit distracting, sometimes

https://github.com/medic/medic-projects/blob/master/scripts/download-xlsforms-from-drive.js + https://github.com/medic/medic-projects/blob/master/default-sms-workflow/standard/xforms-on-google-drive.json

Rationale: we need to show this field in the place group only when relevant, so cannot fully replace the place's name field. Having it there full time would mean having a non-relevant field saved with the place, which is a duplicate field and possibly junk.

Currently done crudely at https://github.com/medic/medic-projects/blob/3f6671f79b1763d45e5faab87cea8e4b7ce5aae8/standard/forms/process_place_forms.sh#L82-L86

With the --json flag, we can better control output and presentation of xls2xform output as per https://github.com/XLSForm/pyxform/blob/master/pyxform/xls2xform.py#L64-L85.

Related: #41

Not clear where it comes from, either

If you upload invalid JSON for the app_settings the destination is thankfully not updated, however there is no notice of such in the command line:

medic-conf . https://admin:[password]@[instance].app.medicmobile.org upload-app-settings
INFO Processing config in . for https://admin:****@standard.app.medicmobile.org.
INFO Actions:
     - upload-app-settings
INFO Extra args: undefined
INFO Starting action: upload-app-settings…
INFO upload-app-settings complete.

The actual response when uploading invalid JSON is: {"error":"bad_request","reason":"invalid_json"}
We should report this back to the user.

e.g.

medic-conf --url=http://admin:pass@localhost:5988
medic-conf --instance=asdf-norway.dev

• don't try to use port 5988
• handle port-in-use errors by trying different ports, or at least giving an indication of where the problem was

Not type: 'report'

Most tools with options either display help or wait for input if you just type their name. This one starts doing stuff, probably not correctly (since I haven't specified where I want it to deploy to):

$ medic-conf
INFO Processing config in standard for undefined.

It should probably just display help instead.

The final report can get very big. It would be great if this was instead saved as e.g. upload-to-docs.2017-11-02_${timestamp}.log.json, and a more concise report displayed on the console (e.g. just totals).

json_docs-for-import -> json_docs-imported

Rationale: we want to show the new contact fields below the "Select Primary Contact" question, not on a separate page.

Currently done crudely at https://github.com/medic/medic-projects/blob/3f6671f79b1763d45e5faab87cea8e4b7ce5aae8/standard/forms/process_place_forms.sh#L88-L95

folder: .../standard
instance: standard.app...
confirm: no

folder: .../abc-denmark
instance: def-norway.app...
confirm: yes

Example confirmation:

Are you sure you want to upload abc-denmark configuration to def-norway.app.medicmobile.org?

And add colours around the config and instance names.

2.12 requires:

• embedding _id of other objects
• embedding full objects

We are working to transition from png to svg for all icons in our app. Currently, it's only possible to upload PNGs using medic-conf. We should add support for uploading of SVGs as well. cc @alxndrsn

Add imported_date (ISO date) field after generating IDs

This is not necessary, but people used to old-style URLs might use it by mistake.

currently breaks

Add secondary instance <instance id="contact-summary"/> to every app form via medic-config to get a contact's info via the contact-summary (instead of /data/inputs/contact).

Needed for medic/cht-core#3413

In legacy mode (up to 2.12.x) docs can have somewhat circular references, but limited by the app. For instance the primary contact for a place has doc.parent.contact == doc. In our app it is dealt with by not including the fields that would create infinite nesting eg doc.parent.contact does not further nest parent.

Here is a minimal example that we need to handle.

person.csv:

external_id,match=external_id:type=clinic:doc>parent,name,sex,timestamp:reported_date
place_714_contact,place_714,Person A,female,2012-01-23
place_715_contact,place_715,Person B,female,2012-01-25

place.clinic.csv:

external_id,name,match=external_id:type=person:doc>contact,timestamp:reported_date
place_714,Person A Family,place_714_contact,2012-01-23
place_715,Person B Family,place_715_contact,2012-01-25

Right now running csv-to-docs on these we get

ERROR TypeError: Converting circular structure to JSON
    at Object.stringify (native)
    at pretty ([...]\npm\node_modules\medic-configurer-beta\src\fn\csv-to-docs.js:9:26)
    at fs.recurseFiles.filter.reduce.then.then ([...]\npm\node_modules\medic-configurer-beta\src\fn\csv-to-docs.js:65:64)
    at process._tickCallback (internal/process/next_tick.js:103:7)

And the line in question: const pretty = o => JSON.stringify(o, null, 2);

currently breaks

When creating forms to test an issue or feature I upload the form via medic-conf and then removing it via futon. Now that we have to do a SSH tunnel to access futon it's more of a hassle to remove a single form. Via medic-conf the only option right now is to delete-all-forms and then re-upload them all previous forms. It would be nice if we could delete a single form from an instance instead. For example, using delete-*-forms -- form_name, similar to how we can use upload-*-forms -- form_name.

https://prettier.io/

or something. Or maybe csv: still makes more sense for row references? maybe csv: can be used to refer to a csv filename by row, which would fix #36

In order for the Collect app to download forms from an instance, the collect form properties file must include context.collect: true. To avoid manual changes to the collect properties files, this should be done automatically by medic-configurer when converting collect forms.