rapidftr / tracker Goto Github PK

The existing reports within RapidFTR are created by a scheduler which runs every so often to generate the reports which can be accessed through a user's browser. Some deployments are having issues accessing reports which points to failure in the scheduler. This fix should ensure that the scheduler persists through known failure scenarios especially OS shutdown, etc.

Currently, the APK used with the integration tests has android device administration enabled. This causes problems when running integration tests with Calabash, since calabash cannot by default seamlessly install and uninstall the APK without disabling device administration first. We need to set up a separate profile for tests so that the APK used with the integration tests does not have Device Administration enabled.

We're currently using Solr for all advanced search capabilities. We would like to evaluate ElasticSearch on the following lines:

• How much effort will it take to integrate ElasticSearch into the existing codebase
• Does it support the same level of multi-lingualism (can it properly index accented characters, and match both é and e)
• What pain points does it solve in the existing codebase, and in the long run, is it worth switching

Using Docker to create a one-click installer for Linux OSes. The installer will be stored in Dropbox folder available publicly.

• Register a child with photo
• Click on the small thumbnail of the photo
• You'll see full size image of the child, but that cannot be zoomed/panned

PhotoView is a nice Android widget that allows to pan and zoom images. We can use this when showing child's full size image.

Presently, CouchDB via Chef is installed without HTTPS, and with a default
username/password of rapidftr/rapidftr. This needs to be secured as follows:

Summary:

• Present cookbooks:
  • Uses a global "rapidftr/rapidftr" username for CouchDB.
  • Uses no HTTPS.
• Target:
  • Use a randomly generated username/password instead of a global default.
  • Use HTTPS.

Background:

• rapidftr-infrastructure is the Chef repository. It is linked as a git submodule into the main RapidFTR repository. You can quickly get a production VM up and running by cloning the RapidFTR repository, and running vagrant up prod.
• CouchDB is a separate cookbook under site-cookbooks/couchdb.
• An instance of RapidFTR can be deployed using the site-cookbooks/rapidftr cookbook. There can be multiple instances of RapidFTR running under the same machine. So each one needs to have a separate admin username/password for CouchDB. Each instance of RapidFTR will have a unique hostname. The default hostname is "localhost".
• CouchDB can either be configured using /etc/couchdb/local.ini, or by adding our own configuration files under the /etc/couchdb/local.d folder.

Tasks:

• Enable HTTPS in local.ini in the CouchDB cookbook. This will enable HTTPS globally.
• The system admin must manually feed in the required HTTPS certificates under site-cookbooks/couchdb/files. Easy way is to create self signed certificates. This part of creating the certificate need not be automated, ideally the system administrator would already have SSL certificates. The cookbook should simply copy over the certificates. For example, refer the rapidftr cookbook.
• Then, under the rapidftr cookbook, for each deployment, we need to generate an administrator for CouchDB. The sysadmin need not know this username/password, it can be automatically created. The easiest way is to create a local couchdb config file, called /etc/couchdb/local.d/<hostname>.ini, which will have the admin user configuration for this rapidftr instance. Both username and password can be randomized UUIDs.
• Then, RapidFTR needs a "couchdb.yml" file with this username/password. You'll find a rake task called "db:create_couchdb_yml" in the rapidftr cookbook. Edit this task to use the generated username/password.
• Remove the default rapidftr/rapidftr couchdb user from the "local.ini" master configuration file.

Move to GitHub issues for issue tracking, and Waffle for the board.

Open Issues:

• Documentation on new procedure of picking up and contributing stories
• Story pipeline/flow
• Milestones/Label structure
• Announce in Google group

@Quoin has upgraded RapidFTR and most of its plugins to Rails 4 and Ruby 2. We need to merge those changes into the current codebase. The changes can be viewed here:

https://github.com/quoin/RapidFTR/tree/rails_4.0_upgrade

Use Chef instead of Capistrano for production deployment. Chef has been chosen after evaluation Ansible and Salt. Ansible doesn't support Windows, and Salt is not stable in Vagrant. The idea is to use the same Chef scripts for both Vagrant as well as real Production deployments.

When syncing large photos/media the sync process crashes and the dialog either hangs or disappears on different devices. The proposed fix is to ensure that device only downloads resized photos.

When a user provides a different URL on their mobile device, they're logged into the instance that was provided during initial set up/. This fix will ensure that a user can login to different instances using the same device.

Currently we're not tracking Code Quality metrics in any stage. We need to add that as a rake task so that we can fail the build whenever code quality goes down.

CodeClimate does an excellent job, and that's where we'd like to be. But unfortunately it can't be integrated into the rake tasks - it simply runs after merging, which makes it an afterthought.

We need some tool that can directly run as a rake task, raise an error and fail the build if metrics are going down. ruby-toolbox has a good selection of gems for this purpose.

We're presently storing JSON serialized data in SQLCipher (SQLite-like, but encrypts data on storage). We'd like to evaluate Couchbase Lite for Android for storing data instead, since its already in JSON form.

Challenges:

• It should also support encrypted storage just like the present SQLCipher

This feature shall enable support for multi lingual deployments of RapidFTR. A user should be able to search for data in a preconfigured language using their device. At the moment, any text with accented characters isn't well displayed on the screen and searches using these characters do not return any results.

This is a placeholder story...

Upgrade Android to version 4.0.3 since all devices coming out in the market now are 4.x devices, and its extremely difficult to even procure 2.x devices anymore.

• Move network operations to a background thread to avoid android.os.NetworkOnMainThreadException
• Make sure StrictMode is enabled and has no issues
• Change deprecated APIs
• Upgrade SQLCipher (NOTE: Backwards incompatible)
• Use Holo theme
• Remove backward compatible data encryption

• Move searching to a background thread with a progress dialog
• Use Pagination with a Producer/Consumer pattern. i.e.
  • Have a Producer that takes as input the Cursor (with records filtered using LIKE), and this should loop through the cursor and yield only the matching children
  • A Consumer should pick 10 records from the producer and display it on the screen. When the user scrolls to the end of the page, then it should fetch the next 10 records, and so on.
  • Have a look at RxJava for some examples.

This release of RapidFTR will enable a SYS ADMIN choose whether to use standard set of forms or define/customise their own set of forms during deployment. This feature will include implementation of the steps necessary to achieve this set up.
After using the installer to set up RapidFTR, the admin would access the URL to the server instance and immediately be provided with options to configure the instance.

OkHTTP is an awesome HTTP client for Java. It is optimized for best network use (SPDY, connection pooling, etc) and comes with excellent sane defaults.

We use a class called FluentRequest for all HTTP connections. We don't use Apache HTTPClient directly in the code anywhere. So it should be easy to pick this one class, and replace Apache HTTP client with the OkHTTP client.

For an idea of how to configure the OkHTTP client, please take a look at the spike code. Also take a look at the spike story #9. OkHTTP is very simple and barebones. It delegates all the Multipart form uploads to an addon library called Mimecraft, again by the same developers. So you'll likely end up using both OkHTTP and Mimecraft.

Use one or more of the following suggestions to improve performance in low bandwidth situations:

• SPDY
• GZip
• Connection pooling
• Caching
• Separate text/audio/photo downloads, and use appropriate ones in low bandwidth scenarios if possible

Retrofit is a clever HTTP REST Client library. As part of this spike, we would like to know:

• How easy it is to integrate Retrofit for HTTP requests instead of the current custom code in Android
• Proof of concept working code for any of the requests (e.g. Login request)
• Make sure it does not need any API changes on the server side, and it should support the existing API endpoints
• Check if it supports multiple file/audio upload semantics

Some Default Fields are hard coded in code (e.g. Name, Protection Status, FTR Status, etc). We need to remove these hard coded values to make the Form Builder as customizable as we want it to be.

This feature shall enable support for multi lingual deployments of RapidFTR. A user should be able to search for data in any language. At the moment, searches using these characters do not return any results.

Deployment instructions have changed since switching to Chef. Update documentation and upgrade path.

Presently in Android, we're using plain-ol SQL queries with custom DAO objects. We want to move to using a lightweight data access layer like JDBI.

Goals:

• Proof of concept code for using JDBI in our present DAO layer

Challenges:

• JDBI uses JDBC connections, whereas in Android the database access pattern is a bit different
• We're using SQLCipher, not SQLite. It has the same API as SQLite, but stores data in encrypted format.

Check google groups post: https://groups.google.com/forum/#!topic/rapidftr/6BbrZfr-2Zc

• Upgrade to Rails 4 for security updates
• Upgrade to Ruby 2.0 for security updates
• Upgrade CouchDB and related plugins

When viewing child records on the web, a field worker should be able to sort the records based on the field specified.

The process of setting up Android in user's machine is very long and complicated. You have to install the SDK, various APIs and Machine Images, create AVDs, etc. For a newcomer having a few hours - it takes the whole time to setup the machine, instead of developing valuable code.

We already have Vagrant for the Rails repository, but we want the same for Android as well, with Chef scripts that will automate the task of installing android and other various stuff. Please check the Vagrantfile and the relevant Chef cookbooks for an idea.

Challenges:

• Android emulator is already slow. Running it inside the VM is going to be even slower! If the base image is Ubuntu, then we must have KVM acceleration enabled. Refer the Android emulator hardware acceleration page for more details.
• The emulator will run using Xvfb/Xvnc. If SSH X11 forwarding is supported by Vagrant, then we need to use it. Otherwise we can install Xvfb/Xvnc and then forward the VNC port.

Maven with its multi-module project setup is currently producing a lot of pain. We want to move away from the multi-module setup (related to #11), and also check moving from Maven to Gradle.

Goals:

• Get rid of multi-module project setup
• Spike using Gradle and its android plugin
• Check IDE integration
• Make sure unit tests pass with Gradle
• Make sure custom integration tests can be integrated into Gradle (related to #11)

Idea is to structure the build as follows:

• Travis-CI for running tests on pull requests
• We'll merge code only if Travis-CI passes
• Snap-CI for deploying to dev/qa/test
• Snap-CI for running the optional code quality metrics

The errors displayed when a user logs in or fails to sync aren't informative to enable a user decide the next course of action. This fix addresses the following list of errors displayed in the current version.

1. Failed to fetch form sections
2. Login unsuccessful
3. Sync failed, please try to login again
   The following are possible scenarios to consider for sync and login failure

• server error
• permission denied
• session expired on mobile
• session expired on server
• offline login
• online login
• timeout of sync
• sync single record always says "Successful" even if failed

Currently we're not tracking Code Quality metrics in any stage. We need to add that as a maven task so that we can fail the build whenever code quality goes down.

Tasks:

• Add checkstyle with some default sane metrics
• Add it to maven
• Some parts of the current codebase itself could start failing
• To avoid that, add it to maven as a separate maven profile

Robolectric 1.x is deprecated, we need to move to Robolectric 2.x for Android unit tests.

Add performance tests using Locust or Gatling.

• Need not aim to cover all the pages, focus is on introducing the framework first.
• Start with a simple scenario - e.g. Login > Register Child > List Children

The project has now been simplified to a single-module (single pom.xml) project. Previously it was one parent with two child projects which caused a lot of IDE issues.

This makes it easier to move from Maven to Gradle. We already spiked on Gradle and here are the observations:

• Gradle works very well. Doesn't download the internet, downloads very small amount of dependencies
• The Android plugin works very well. APK builds are much faster
• Major Bottleneck: Ability to run Android unit tests. The Gradle android plugin has no clear way to run the Unit tests alone, and it always runs unit tests as part of the integration tests (and consequently tries to look for emulator and stuff). This is a known issue with Gradle Android plugin, and many (hackish) projects have come up to provide Unit testing support. Most of those don't work well with the IDE. Some don't work well with Robolectric. So if those issues can be made easy, we welcome the switch to Gradle.

Firefox is taking a long time to run the tests in CI. A move to PhantomJS for testing would greatly help development. Capybara has the Poltergeist gem for running tests in PhantomJS.

Objective: Introduce PhantomJS for a few cucumber browser tests. This story is not to port every firefox test into phantomJS, instead please help us with:

• Setup a separate cucumber tag for running tests with PhantomJS. Let other browser tests keep running in Firefox as they are.
• Slowly start marking the firefox scenarios one-by-one to run on PhantomJS. If they don't pass, let them remain in Firefox. Start with the simplest ones
• Add a cucumber rake task to run the phantomJS tests

Slim is a neat and concise templating framework for Ruby. It is miles better than the default ERB templates used in Ruby/Rails. We'd like to use that for our templates since it makes them more readable and easier to maintain.

Goals:

• Start porting ERB templates to Slim slowly, beginning with the Login page. We can merge independent pages as and when they are complete.
• Tests should need no changes, since the actual output HTML/CSS should not change. Minor changes for convenience are fine.
• Use two-space indent

We also evaluated HAML and Jade. Slim is more concise than both.

The application stores encrypted cookies for 20min. This fix should ensure that sync doesn't fail when server and device have different timestamps.

If your device/client time is off by more than 20mins, the server automatically logs user out. The problem is that by the time client/device acquires the cookie, it's already expired.

Presently the integration tests that we're using on Android use Robotium, which is good but getting very difficult to create and maintain. We want to switch to one of these libraries:

• Calabash - Cucumber like integration tests for Android
• Expresso - Google's android testing tools

We're presently preferring Calabash, but we also want the integration tests to be more performant, since it takes a long time to run in the emulator.

We also want to get rid of the Multi-module maven project because of the integration test suite, and switch it to a single-module maven project. This will make IDE integration even simpler.

We need to add LogStash and Kibana to the existing Chef provisioner. Steps are as follows:

• Chef scripts are in the rapidftr-infrastructure repository
• They are also linked as a submodule in the RapidFTR project in the infrastructure folder
• Clone RapidFTR repository
• Make sure Vagrant is installed
• Run vagrant up prod to provision a Production virtual machine. You can then keep changing the cookbooks and run vagrant provision prod to re-provision the VM
• Check the infrastructure/site-cookbooks/rapidftr cookbook for details on where the Log is stored, etc. The logs are usually stored inside /srv/rapidftr/<environment>/shared/log, and we need to expose all logs in this folder
• Make sure Kibana and LogStash are installed as separate cookbooks
• As part of the rapidftr cookbook, just add the relevant configuration files and/or shippers for LogStash
• Make sure Kibana is secured with HTTPS and a default username/password

This fix will ensure that the sync process automatically resumes when data connection breaks or changes connectivity mode/network type i.e. LTE, 3G, EDGE, GPRS, etc without the user having to re-initiate it. Currently, when a field worker is syncing and the data connection on their mobile device either breaks or switches between different network types i.e. LTE, 3G, EDGE, GPRS, etc, the process simply stops/exits/hangs and requires the field worker to re-initiate the sync to proceed. Sometimes they have to do this several times before succeeding.

GIVEN that I am a field worker,
WHEN my mobile device is connected to WIFI
AND I am syncing records with the server
AND I change the connection type to LTE, EDGE, GPRS during the sync process
THEN I should be able to view an error message indicating that sync has broken due to network failure

GIVEN that I am a field worker,
WHEN my sync process fails due to data/connection/network failure
AND I am connected to an active data connection (have connectivity)
THEN the sync process should automatically resume on my device
AND I should be able to view the status/progress

Presently the UI uses a lot of hand-coded CSS, which makes it very brittle to modify and build on top of. We'd like to introduce the Zurb Foundation UI framework to make it easier to achieve the same styles.

• Goal: We want the same present UI, but the underlying HTML/CSS to use Foundation instead. Overhauling the UI/UX will happen later.
• Approach: This story focuses only on the Login page and the header/navigation area. As long as each page looks (almost) the same, we can keep changing pages one by one as part of subsequent stories.
• Use rails-assets.org to pull in the dependencies.
• The colors need to remain same, because these are neutral colors decided with some long thinking behind it. The organizations we work with each have their own color brandings, and we want RapidFTR to be independent and unassociated with those colors, hence the neutral color scheme.
• Smaller stuff like button colors/gradients are left to your goodwill.
• Foundation has been decided upon after spikes in previous code jams.
• This may also involve changing the corresponding cucumber tests (since the CSS may change).

Right now, we're using encrypted cookies to store the last_access_time and then use it to expire the session. The details are explained in #31. But the session is expired only after 20 minutes. As part of this story, we want to expire the session when browser closes as well.

Developer Notes:

• We're using encrypted cookie to store the session ID and last_access_time
• If you set this cookie's expiry to "nil", then it becomes a HTTP session cookie, which gets deleted automatically after the browser closes. But this behavior clashes with the mobile client
• So basically, for any request to the normal Web controllers, the cookie expiry time should be nil (so that it expires after browser closes). For any request to the APIs (/api/*), the cookie should have an expiry time of 1 week, otherwise the mobile will "forget" this cookie when the app closes, leading to all sorts of problems
• We use "ApplicationController" as the base class for all Web requests, and "api/APIController" as the base for all API requests. These two are the right places to change this behavior
• Rails allows you to set only one value for the expiry_time. This is set in "config/session_store.rb". So, you have to set the expiry_time here to "nil", and then somehow override this in the APIController and change this to 1 week only for API controller requests. This will be very tricky, you'll have to look at the Rails code that reads "expiry_time" (do a grep on your $GEM_HOME) and then see how to override this