cytoscape / appstore Goto Github PK

Make App Store runnable with one click/command.

• Create Dockerfile for Django app
• Create docker-compose.yml for the entire App Store:
  • MySQL
  • Apache Web Server
  • Django

In order to incentivize app authors to complete as many fields as possible we might consider adding badges with progress bars (e.g., 75% tasks completed to achieve badge X). These badges would also highlight to users which apps are better annotated, etc.

We already have some badges. Some are manually set and some are driven by fields. But we might need a more advanced badge system to manage progress. See for example, https://github.com/jiaaro/django-badges

There are many app names that consist of two or more words squashed together without spaces. Eg: clusterMaker2, EnrichmentMap, ClueGO, StringApp, AutoAnnotate etc. The problem is that users often enter the name with spaces into the search box, eg “enrichment map”. The search feature does not handle this very well, EnrichmentMap is way down the list of results when searched for this way when it should be first. Another example is searching for “string app”, StringApp is like the 100th result.

Over time, these data will get messy. Should curate them annually?

Going for complete replacement of social_auth with Python-social-auth(psa).
Mainly, dealing with the views.py for constructing data about user through the authentication pathway using the pipeline consisting of : Login, after login handling, logout

Great help from: https://github.com/python-social-auth/social-examples
Done for now! Still suspect trouble in future.

Move this document to here and make it up-to-date.

• http://wiki.cytoscape.org/AppStore

From Kei Ono:

Recently, I've got emails (at least from 3 people in Japan, one in
China and one in India) about App Store issues.

If users run Cytoscape on their machines, and access App Store page,
it displays "Install" button to directly install the app to Cytoscape.
However, if they click the "Install" button (NOT the Download one), it
simply downloads JAR file to the download directory and App Store does
not install the App to their CytoscapeConfig dir.

Does anyone have same issue recently? Looks they are all outside of
US, and I cannot reproduce the issue here (I've tested with Time
Warner cable and UCSD networks).

If this is an App store issue, it should be fixed because it's really confusing.

This requires putting download requests onto an async queue so that download stat writes are done by a single process.

via Tim Hull

Need to update backup strategy for Dockernized MySQL.

RemovedInDjango19Warning: Loading the url tag from the future library is deprecated and will be removed in Django 1.9. Use the default url tag instead.

If an app is submitted that exports APIs, we prompt for a pom.xml file in addition to the JAR file. However, if the package in this pom.xml file starts with org.cytoscape, the submission is rejected. This has caused issues for core apps that export APIs (such as OpenCL) - to submit them, we've had to edit the pom and insert an extra space to trick the App Store into accepting our submission.

When you install AgilentLitSearch app for the first time from the App Store, it will prompt you with a license agreement. When you click OK it will go ahead and install the app (great!) and it will also prompt you to download the jar manually (confusing).

It should simply install the app and then return you to the app page.

On this page:
http://apps-dev.cytoscape.org/apps/agilentliteraturesearch

this license link is 404
http://blueoaksoftware.com/Agilent/ALS/license.html

I've removed the view of the global map, but geoip and google API key are still being used to process and generate the data for the map views. We should remove all of this processing and dependencies.

Goal: remove dependencies on geoip and google api key for maps

Barry's note to an app developer:

"It seems to have surfaced a bug in the App Store that has lead to Cytoscape not starting up … everywhere in the world.

The problem is with the version naming of the CoNet app. Your previous apps were 1.0.7.beta, 1.0.6.beta, and so on. This recent version was 1.1.beta. The version name the App Store expects is .... 1.1.beta is illegal because “beta” isn’t a number.

This should have been caught by the App Store, but wasn’t. So, I used the App Store admin console to manually change your app version to 1.1.0.beta (with “.0” added). After this, Cytoscape starts up like it should. I wasn’t able to change the version in the manifest file, so there is still some inconsistency associated with CoNet."

Via Tim Hull:

I recently noticed that if you have the App Store website and Cytoscape open, clicking the "Install" button for an app with a license agreement prompt will download the JAR instead of directly installing in Cytoscape after accepting the license agreement. This works fine for apps with no license agreement displayed.

Alex: Or it should only offer "Download" button. IOW, it needs to do what it offers to do.

We are starting to collect apps that are apparently old and unloved ... meaning they don't have editors that respond and have stopped working (e.g., Semascape).

Rather than kick them out of the store, we should retire them to an Attic where they can still be reached but don't occupy top-of-mind for most store users.

First, we need a policy for what goes into the Attic, and then we need to develop a simple UI for users to rummage.

For now, we have installed a cron job that restarts the App Store once a day. However, we can improve service by having the search index daemon function as designed and avoiding the restart.

From user:
I just wanted to let you know about a bug/error I came across when using the Cytoscape app store. Someone might want to look into it. I just submitted an app and signed out and this is the error I got:

In the "download" module of the App Store code base, there should only be one instance of ReleaseDownloadsByDate for each day, and only one instance of AppDownloadsByGeoLoc for each GeoLoc instance. However, if there are multiple requests made at the same time at the beginning of the day, this can cause more than one instance of ReleaseDownloadsByDate to be created. This scenario causes the _increment_count function in download/views.py to fail, preventing anyone from downloading any app for the entire day.

Short term solution:
Fix _increment_count to not use get_or_create() function. Instead, it will retrieve the first instance (at index 0) of ReleaseDownloadsByDate for a given date. This will cause some download numbers to be lost, but won't lead to more catastrophic errors.

Long term solution:
Create an async queue. Updating download stats are pushed onto the async queue, and the download is forwarded immediately to the user. Any catastrophic problem would be pushed to the async queue, and users would be unaffected.

From Barry:
For the sake of documenting what we think happened to the App Store today, it seems to be a concurrency problem where two “null release” records were generated for 2016-02-01. This could happen in the Pyton get_or_create() function, which tried to get the “null release” record for 2016-02-01 and failed … deciding to create a new “null release” record. Meanwhile, another user came in and did the same thing at the same time. Thus, two “null release” records got created. Bad news.

This was possible because the database table doesn’t have a “unique” constraint for the release_id field:

image001.png

And if it did, one of the threads would have failed on the create, but at least only one record would have been created. As for the thread that failed, the proper recovery would be something like this: http://stackoverflow.com/questions/2235318/how-do-i-deal-with-this-race-condition-in-django

More information on this is here: http://stackoverflow.com/questions/6586552/django-how-to-do-get-or-create-in-a-threadsafe-wa

Users agree that the links provided in the RESOURCES section are useful... but they often don't see these until we point them out!

Looking for ways to make this useful section more apparent...

1. Maybe simply increasing the base font size and icon size for the content in this section would help?
2. Maybe changing the background or icon color scheme?

Let's try a few things and get feedback from the team.

Add another icon (i.e., internal tag) for apps that "support automation", following the same pattern used by existing annotation icons. We would also add a field in RESOURCES for a link to "Automation" documentation. Then, the icon would automatically be added whenever this field is filled in, just like how the "open source" icon works.

See icons in this example

Some observations of correlation by Barry:
"Got another one of these today. So far, I've seen a number of them, and they're loosely correlated with app submissions, though it's difficult to tell, as we're getting a bunch of submissions lately.

Here's what I know (date and e-mail title ... the e-mail body is always the same):

Jan 12 08:14am: Cytoscape App Store - App Repo Deploy (Release API ID: 21)
Jan 14 09:09am: Cytoscape App Store - App Repo Deploy (Release API ID: 22)
Jan 17 08:18am: Cytoscape App Store - App Repo Deploy (Release API ID: 23)
Jan 25 02:50am: Cytoscape App Store - App Repo Deploy (Release API ID: 24)
Jan 26 10:03am: Cytoscape App Store - App Repo Deploy (Release API ID: 25)

I think I found the API ID in the apps_releaseapi table ... as the id column. The corresponding info is:

+----+------------+--------------------------------------+
| id | release_id | pom_xml_file |
+----+------------+--------------------------------------+
| 21 | 477 | diffany/releases/1.0.0/pom_1.xml |
| 22 | 481 | cytransfinder/releases/1.2/pom_2.xml |
| 23 | 482 | cytransfinder/releases/1.3/pom_1.xml |
| 24 | 486 | cytransfinder/releases/1.4/pom_1.xml |
| 25 | 488 | cytransfinder/releases/1.5/pom_1.xml |
+----+------------+--------------------------------------+

So … I have installed both Diffany and cyTransFinder … they seem to run OK.

This problem started with diffany and also triggers on new cytransfinder releases. It does not happen with other releases after diffany: Mclique, CyIsomorphism, ncmine, vital, entropt-layout, or dynet."

https://docs.google.com/document/d/1hmrbPNw1B37isSID5pNKIiff-Zh5gT4E6iwyz5LDYSM

In the example below, it appears the trailing whitespace after PSICQUIC is being trimmed off and the resulting app name (without a space) is then not being recognized as a valid, extant dependency.

Need to find out where in the code this whitespace trim is occurring and squash it...

Manifest-Version: 1.0
Bnd-LastModified: 1461973063033
Build-Jdk: 1.8.0_91
Built-By: thully
Bundle-Activator: org.cytoscape.internal.CyActivator
Bundle-ManifestVersion: 2
Bundle-Name: Core Apps
Bundle-SymbolicName: org.cytoscape.core-apps-meta
Bundle-Version: 3.4.0
Created-By: Apache Maven Bundle Plugin
Cytoscape-App-Dependencies: BioGRID Data Source;3.3.0,Biomart Web Servic
e Client;3.3.0,BioPAX Reader;3.3.0.1,Command Line Implementation;3.3.0,
CyCL;3.3.1,cyREST;3.3.2,JSON Support;3.4.0,Network Merge;3.3.1,NetworkA
nalyzer;3.3.1,OpenCL Prefuse Layout;3.3.1,PSI-MI Reader;3.3.0,PSICQUIC
Web Service Client;3.3.0,SBML Reader;3.3.0,Welcome Screen;3.4.0
Import-Package: org.cytoscape.service.util;version="[3.4,4)",org.osgi.fr
amework;version="[1.5,2)"
Tool: Bnd-1.50.0

I wanted to add a citation to an app on the app store, but it is not possible to reference a DOI in the citation field.

The publication is a preprint on biorxiv
http://biorxiv.org/content/early/2016/07/05/062091
citable via a DOI
http://dx.doi.org/10.1101/062091

It seems the app store allows only PubMed ids as citation, because it does not allow any doi in the citation field.
When I try I get "Whoops! Please fix the fields in red. Once you're done, click Save again."

DOI's should be supported for citation. Often the preprint is available via a DOI for a long period of time while one still has to wait for the publication to get accepted/published.

When a user submits an app for the first time, the "successfully submitted" page (and email) should include a message and link to join the new cytoscape-app-dev google group.

Currently, we provide a Resources link labeled "Cite this App" based on a provided pubmed ID. Users, nonetheless, have requested a link to papers citing the app... which is the same link!

To make this second purpose more obvious, we could add a link at the top (next to "downloads") using the same pubmed URL but labeled as "see citations" or something like that. This way users would identify the link as relevant to their intention.

Since this is an web app, GitHub Flow is a good way to manage branches:

• https://guides.github.com/introduction/flow/

Following instructions here: http://wiki.cytoscape.org/AppStore
on new VM at UCSD.

Notes

• Create jeff account
• Installed LAMP

apt-get install apache2
apt-get install libapache2-mod-wsgi
apt-get install mysql-server
apt-get install php5 libapache2-mod

• Installed Python 2.6 (ended up with 2.7)

add-apt-repository ppa:fkrull/deadsnakes
apt-get update
apt-get install python2.6 python2.6-dev

_Note: Looks like I've got both 2.6 and 2.7 installed... Could be a problem later. Looks like 2.7 is the active one..._

• Installed xapian

apt-get install python-xapian
apt-get install libxapian-dev

• Installed xapian-binding from source

wget http://oligarchy.co.uk/xapian/1.2.13/xapian-bindings-1.2.13.tar.gz
tar -xzvf xapian-bindings-1.2.13.tar.gz
cd xapian-bindings-1.2.13
./configure --with-python

_Error: Python.h not found!_

• Installed regular python-dev to address error (though might be dependent on Python 2.7 now?

apt-get install python-dev

• Second attempt to build xapian-bindings

./configure --with-python

_configure: WARNING: Xapian library is version 1.2.16 but the bindings are version 1.2.13 - we strongly recommend using matching versions._

• Installed image libs

apt-get install libjpeg8
apt-get install libjpeg-dev
apt-get install libjpeg8-dev
apt-get install libpng-dev

• Installed GeoIP

apt-get install php5-geoip
apt-get install libgeoip-dev
pip install geopip
apt-get install libgeoip1

_Tried all of these in vain attempt to generate /usr/lib/libGeoIP.so.1 installed (needed later). In the end, I found it at /usr/lib/x86_64-linux-gnu/libGeoIP.so.1 and I'm not sure which command above generated it._

• Installed python packages listed in wiki docs using pip install

easy_install pip
pip install django
apt-get install libmysqlclient-dev
pip install mysql-python
pip install pillow
pip install django-social-auth
pip install IPython

Code

• Clone code
  git clone http://github.com/cytoscape/appstore .
• Followed instructions for conf dir files. Ran django-admin startproject cyappstore to generate a temporary settings.py with a new SECRET_KEY to add to apikeys.py. Copied all other values from previous instance of CyAppStore.
  _Make sure dbs.py info matches the actual mysql info_
• Installed mvn in /home/jeff for mnv.py

mkdir cyappstore-pkgs
cd cyappstore-pkgs
wget http://www.carfab.com/apachesoftware/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar xzvf apache-maven-3.3.9-bin.tar.gz

• Copied apache config for appstore.conf and updated python dir path to use 2.7. Also made logs dir mkdir /var/www/CyAppStore/logs.
  _NOTE: the 2.6 vs. 2.7 issue is also expressed here_
• Then enabled conf and reloaded

sudo a2ensite appstore.conf
service apache2 reload

• Also instanced and updated maven-app-repo-settings.xml with same info as in previous site
• Followed instructions for geolite dir

_Waiting on creating cronjobs for later..._

• Loaded data from an old mysqldump

create user 'cyappstore'@'localhost' identified by 'appsrule';
grant all on cyappstore.* to 'cyappstore'@'localhost';
create database cyappstore;
use cyappstore;
source cyappstore-160414.sql;

_Will update with latest backup just prior to going live_

Test

• Dependencies
  python external_scripts/test_dependencies.py
  ***FAILS: python version (ok); django version (ok); JAVA_HOME (fix and try again)

apt-get install default-jre
update-alternatives --config java
vi /etc/environment
JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java"

source /etc/environment

• GeoIP
  python manage.py test_geoip
  _We have a problem! importlib call changed in newer version of django which is paired with newer version of python. Press ahead or get old stable versions? Let's try old..._

apt-get install python-virtualenv
virtualenv --python=/usr/bin/python2.6 /var/www/CyAppStore
source bin/activate
pip install Django==1.4.5

And reinstall all the pip's above now in the python2.6 env and repeat the xapian wget and build in this virtualenv as well...

Now both dependency test run OK!

• Temporary domain setting

vi /etc/hosts
132.239.8.177   apps-dev.cytoscape.org
vi /etc/apache2/sites-enabled/appstore.conf
ServerName  localhost
a2dissite 000-default

_UNDO these after domain is properly set_

• Addressing missing sys.path for settings.py

add2virtualenv /var/www/CyAppStore

We should override the dynamic sorting of category tags and pin "collections" to the top.

Setup guide should be updated for new Docker containers.

https://github.com/bitlabstudio/django-review

We need an administrator process for deleting apps that were submitted and accepted accidentally.

Since Python 2.6 is obsolete, need to test the system with the latest Python 2.7.

(3.x migration is a separate issue)

Can't reproduce this on local machine

When submitting an app, the submission confirmation is sent after pressing the Submit button. This seems a bit weird given that you still have to confirm the submission on the next page - it seems like the confirmation should be sent after that.

This would reduce security issues and address Unicode encoding problems in Python 2.