openworm / movement_cloud Goto Github PK

FeaturesMeans isn't currently linked with Experiments, and what I assumed to be the foreign key in FeaturesMeans (experiment_id) is of a different size (Big20) from the id maintained with Experiments.

Create a document to help guide people maintaining their own database, and running the Django server.

This will make it easier to see.

Right now the server complains:

Not Found: /favicon.ico
[07/May/2017 16:43:37] "GET /favicon.ico HTTP/1.1" 404 2085

The server is running (yay!) and pointed at the dev branch.

http://movement.openworm.org/webworm/

But now @cheelee perhaps you can tell me why the data is not showing up. Thanks

To avoid fully dumping the database, and to try a more surgical approach, try this on your local machine:

mysqldump mrc_db4 exit_flags > exit_flags_old.sql

in remote mysql:

use mrc_db4;
drop table exit_flags;
source exit_flags_old.sql

We'll need to figure out how to do mysqldump to get a baseline version of the schema.

Subsequent changes to the schema should basically be modification statements, because the database already has data so the original code cannot simply be re-run.

It was not like this before. Will need to figure this out.

Then you can PR the crossfilter branch into the dev branch.

Hi @ver228, in the latest version of the database (I am calling it mrc_db4 in our AWS mysql server), you have two tables:

• experiments
• experiments_full

I believe the old database only had "experiments". What is the new table for, and does it render the old table redundant?

In fact I cannot even access this new _full table:

mysql> select count(*) from experiments_full;
ERROR 1449 (HY000): The user specified as a definer ('ajaver'@'localhost') does not exist

Thanks

Records in the database sometimes have null values for certain fields. These are reflected in the tool as "Unknown" or "None" (following a fairly arbitrary assignment scheme based on the nature of the field.)

There needs to be a reverse mapping for when "Unknown" or "None" values are required. It remains to be seen if this is even possible. It means a query on the database requesting for matching a NULL value. If this does not make sense, the tool should not allow "Unknown" or "None" fields to be selected for filtering purposes.

The Experiment name field appears to be constructed in a way that annotates a subset of the experiment properties (e.g. date) as is typical for some choice for file names. This tends to make for a long name that brings no extra benefit to a database search interface, and should be dropped.

Steps to add Avelino

From the dev server

See all users:
cut -d: -f1 /etc/passwd | sort

ubuntu
mysql
mcurrie
cwlee

Create a new user, ajaver:

sudo adduser ajaver (password: mexico)

Note: to change the password later, do this:
sudo passwd ajaver

From Avelino's local machine

From Avelino's local machine: Generate a public/private ssh key pair
ssh-keygen -t rsa

List all Avelino's local machine public keys:
ls ~/.ssh/*.pub

Copy the following public key:
cat ~/.ssh/id_rsa.pub

Avelino please use slack to direct message Michael this public key.

Then Michael will add it to

/home/ubuntu/.ssh/authorized_keys, which is a text file currently containing only the OW_latest_spore_keypair public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCFCqSfNxnGp5WbnXdHZaUw2t80whn+3rjWHgoKLu9TpTUI5A9ZBPitresH7H7cFcBWQIvSg5/NcjOezAg5yANj88pbjcMAlEoliKC+EIeEcQx3ICU84Ss0WcqQSTbi/sprzsx4s2p6SnzYEDFh OW_latest_spore_keypair

This key will let you connect to [email protected], on port 22.

The two tables are now available in the latest database schema. Time to implement the feature accordingly.

I think it would look nice.

You can fix it by restoring the code you deleted, not knowing what it was for...

Now I understand what it was for: if you used the checkboxes "Monday", "Tuesday", etc, then someDay would be selected.

Restore it by looking at commit MichaelCurrie/crossfilterExample@c0c3c6f

Goal: create a site like this: https://square.github.io/crossfilter/

Using this library: https://github.com/crossfilter/crossfilter
and D3.

... That can be interacted with.

Just to show it can be done, at first.

Hi Avelino,

I'm just looking at the two example worms you had sent to me:

Now at the .wcon file: https://www.dropbox.com/s/rvy4vwhwoj4o7l8/asic-1%20%28ok415%29%20on%20food%20L_2010_07_08__11_46_40___7___5.wcon?dl=0

Seems to have 3885 entries for "t" at -0.04, before the time increments normally. Is there a reason for this?

Thanks

Create new extended test database with links to -

1. experiment data files (Zenodo archive)
2. youtube videos that demonstrate key worm movement characteristics. The goal is to embed the videos in the database interface users can view after filtering.

@MichaelCurrie Is it possible to dump a CSV file from the database instead of a .sql file? I'm encountering significant difficulty trying to import the latter into an sqlite3 database we could use as a convenient test database for development purposes.

Currently the results are all projected onto the year 2001, a limitation of the original example I am adapting. Let's make it possible to view multiple years?

Or maybe it's better this way.

As a sanity check, to avoid the server returning thousands of results, perhaps there should be a limit of 50 in the search results page.

This would fix the problem of thousands of youtube videos on one page...

The url download file generated can contain the full list, though.

If I do a search and select all Males but do not narrow the search in any other way, and click the red search button, I get a page that simply says "Service unavailable" in plain text in the upper left corner.

In order to be able to use the new d3 path module, a healthy upgrade to d3 V4 would be a good idea.

Here is an example of the original, redone with V4.

https://bl.ocks.org/micahstubbs/66db7c01723983ff028584b6f304a54a

This might be a good way to clean up some of the junk from the example I originally forked, which was not very well-written.

This will need to be tested.

1. Create a features table containing descriptions of each feature (possibly based on https://github.com/openworm/open-worm-analysis-toolbox/blob/master/open_worm_analysis_toolbox/features/feature_metadata/features_list.csv) (DONE)
2. Add information about a "top 10" features specifically referencing the "default core features" (DONE)
3. Replace https://github.com/openworm/movement_cloud/blob/dev/webworm/defaultCoreFeatures.json with a view on the features table.
4. Have features_means reference this features table as a foreign key
5. Link features_means to experiments (#14)
6. Make a schema change PR! (before I do, changes will be live in the MySQL server without being reflected in the schema tracked by version control!)

In static/webworm/crossfilter_parameters.js and template/webworm/crossfitler-template.html, attempting to remove one of the charts like the following results in none of the charts getting populated:

    "charts": [
        "hour",
        "path_range",
        "iso_date"
    ],

and

  <div id='charts'>
    <div id='chart0' class='chart'>
      <div class='title'></div>
    </div>
    <div id='chart1' class='chart'>
      <div class='title'></div>
    </div>
    <div id='chart2' class='chart'>
      <div class='title'></div>
    </div>
  </div>

From André:

It would make sense to have most of the metadata there. At least the strain, gene and allele (where available), time stamp, sex, and condition (i.e. on food or not) would all be useful to have as a quick summary for anyone looking at the data directly on the page.

https://sandbox.zenodo.org/record/71862#.WP3k8dKLSM8

I have added features that require JSON file processing. requirements.txt reflects this.

For now, we will need to add "pip install simplejson" to the static server software stack environment.

In the future we should consider a setup framework that allows us to automatically take server systems that satisfy basic requirements, and then augment them to satisfy any additional software stack requirement changes due to changes in the repo. The clearest example of this is if we need support for more python modules like matplotlib.

@ver228 I found two problems with the sample WCON files you have given me:

• the "timestamp" format used needs to include the time zone as per https://tools.ietf.org/html/rfc3339
• the frame rate has not been specified. units.t = "s" but it should be "0.04s" for 25fps, for instance.

Currently there is no indication what the values in the results represent:

Right now it's just using a timer.

These warnings are annoying me:

Here's the list of 'core' features with a brief description.

• Andre

length - the length of the worm's midline

area - the area of the worm

midbody_bend_mean_abs - the absolute value of the worm's midbody bend angle

max_amplitude - the maximum distance between the points on the worm body that are farthest from the line connecting the head and the tail

head_tip_speed_abs - the absolute value of the worm's head speed (will capture some contribution from the high frequency 'foraging' motion of the worm's head).

midbody_speed_abs - the absolute value of the worm's midbody speed

path_range - the farthest distance recorded between the worm's staring point and any point on its trajectory during the recording

forward_motion_frequency - the frequency of forward motion bouts

paused_motion_frequency - the frequency of pauses (where the worm doesn't move much)

backward_motion_frequency - the frequency of reversals (where the worm moves backward for some time)

Include instructions on use with screenshots.

Design and extend current interface features for full use-case workflow -

1. User filters experiments of interest
2. Filtered experiments are then used to fetch real experiment movement data (from permanent archives like Zenodo - https://zenodo.org/about)
3. Collected movement data are packaged, and delivered (?) to user.

Based on selected features, generate crossfilter data that can be consumed by the client-side crossfilter javascript program.

https://github.com/crossfilter/crossfilter/wiki/API-Reference

From their example, it looks like it's just a JSON file that would be needed. In this example, the client-side crossfilter function is applied to some hardcoded data, that could easily be derived from a JSON file.

var payments = crossfilter([
  {date: "2011-11-14T16:17:54Z", quantity: 2, total: 190, tip: 100, type: "tab", productIDs:["001"]},
  {date: "2011-11-14T16:20:19Z", quantity: 2, total: 190, tip: 100, type: "tab", productIDs:["001", "005"]},
  {date: "2011-11-14T16:28:54Z", quantity: 1, total: 300, tip: 200, type: "visa", productIDs:["004" ,"005"]},
  {date: "2011-11-14T16:30:43Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["001", "002"]},
  {date: "2011-11-14T16:48:46Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["005"]},
  {date: "2011-11-14T16:53:41Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["001", "004" ,"005"]},
  {date: "2011-11-14T16:54:06Z", quantity: 1, total: 100, tip: 0, type: "cash", productIDs:["001", "002", "003", "004" ,"005"]},
  {date: "2011-11-14T16:58:03Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["001"]},
  {date: "2011-11-14T17:07:21Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["004" ,"005"]},
  {date: "2011-11-14T17:22:59Z", quantity: 2, total: 90, tip: 0, type: "tab", productIDs:["001", "002", "004" ,"005"]},
  {date: "2011-11-14T17:25:45Z", quantity: 2, total: 200, tip: 0, type: "cash", productIDs:["002"]},
  {date: "2011-11-14T17:29:52Z", quantity: 1, total: 200, tip: 100, type: "visa", productIDs:["004"]}
]);

The worm_length chart ends up overlapping with the iso_date chart when changing from:

    "charts": [
        "hour",
        "worm_length",
        "path_range",
        "iso_date"
    ],

to:

    "charts": [
        "hour",
        "iso_date",
        "worm_length",
        "path_range",
    ],

However the following order works as expected:

    "charts": [
        "iso_date",
        "hour",
        "worm_length",
        "path_range",
    ],

This URL documents the various idioms other groups adopt to support multiple split settings for development, and for deployment.

https://code.djangoproject.com/wiki/SplitSettings

We should consider and agree on a way we can be comfortable working with.

This should also address the issues of different developers having a slightly different local setup for their test database. Regardless, it is important to note that Django expects migrations to be packaged as part of the repository.

Rather than starting on the Results tab, if there are no results yet, start on the "Search Tools" tab instead. This will be more intuitive for users.

If you go to movement.openworm.org/webworm, you'll see the below error.

I've confirmed that our mysql server has an mrc_db database, along with a mrc_db4 database, which we should now be using. The latter database has the requisite features_means table referenced here.

However, I have run migrations and restarted the server, so I'm not sure why the reference to mrc_db persists.

python3 manage.py migrate auth --database=mrc_db4_link_for_django
python3 manage.py migrate --database=mrc_db4_link_for_django
python3 manage.py migrate --fake webworm --database=mrc_db4_link_for_django
# We use “fake” to deal with “Alleles” migration nonsense

Then I use htop to identify the server process, sudo kill to kill it, then I run:

nohup sudo python3 manage.py runserver 0.0.0.0:80 & (as ubuntu)

It should be about time we added CI tests to this repo. This will need some research as I don't know how one conducts a test on a GUI. My idea -

1. CI starts the server with a smaller test database.
2. (Research needed) CI triggers a script requesting from the server a specific set of search parameters.
3. (Research needed) CI somehow acquires the HTML output from the results tab.
4. HTML output is compared against a static expected output file.

The condition where Zenodo links are not available in the database is not properly handled by the code for downloading (providing the URLs list for) data.

Progress:

• incorporate sperm animation (https://bl.ocks.org/mbostock/1136236) into chart DONE
• add border to sperm animation DONE
• figure out how to control # of results DOM element. DONE
• vary the number of sperm shown based on how many results are returned DONE
• hide worms, don't just kill them: http://bl.ocks.org/d3noob/5d621a60e2d1d02086bf MOOT
• make sperm look like worms (using https://developer.mozilla.org/en-US/docs/Web/SVG/Tutorial/Paths, e.g. https://codepen.io/Klaujesi/pen/Meajyx) DONE
• add tooltip like in https://github.com/MichaelCurrie/bubble_chart MOOT
• move the results window to the right MOOT
• link to real worm data (may still depend on simplifying the representation as in openworm/tracker-commons#133, or maybe not, if we just export a 5-second clip of each WCON file) DONE

CSS selectors:

.class
#id
*
element e.g. p selects all

elements

Migrate web_code from /home/ubuntu to /srv
Figure out what process started the django server

Do this so multiple users can access the Django server rather than just one (ubuntu) as it is now.