Giter Club home page Giter Club logo

galaxy-ipython's Introduction

Info:

This repository is not maintained anymore, as it has been integrated as core feature into Galaxy. Read more about this in the January 2015 Galaxy NewsBrief and in our Community page.

Galaxy IPython Integration

This projects integrates IPython Notebook, a interactive computational environment, with Galaxy. We hope to make Galaxy more attractive for bioinformaticians and to combine the power of both projects to unlock creativity in data analysis, but also in Next-Generation-Training courses.

IPython in Galaxy

IMAGE ALT TEXT HERE

Requirements

The only requirement is to have Docker installed on your system. For a detailed instruction how to install docker, please look at the docker website.

Installation

Copy the template and config folder in your GALAXY_ROOT/config/plugins/visualizations/ipython folder and restart Galaxy. Alternatively, you can clone the repository with

git clone https://github.com/bgruening/galaxy-ipython.git config/plugins/visualizations/ipython

The IPython visualisation option should be visible next to the usual Charts or Trackster options in your visualisation menue.

Starting IPython in Galaxy

Features

  • run IPython directly in your Galaxy main window or in Galaxy Scratchbook
  • complete encapsulated python environment with matplotlib, pandas and friends installed
  • access to all datasets from your current history via pre-defined IPython function
  • manipulate and plot data as you like and export your new files back into the Galaxy history
  • access your Galaxy instance from IPython with bioblend and redefined functions
  • save the IPython Notebook into your Galaxy history
  • notebook persistence across sessions
  • saved IPython Notebook files can be viewed in HTML and re-opened
  • self-closing and cleaning IPython docker container

How does it work

The mako template from the Galaxy visualisation framework renders the interface and builds all files and commands needed to lunch the docker container. A config file is saved under /import/ inside the docker container. The IPython webpage running in docker will be included in a HTML object and displayed to the user. Depending on your configuration some JavaScript magic is done to add an object to the DOM, which will be loaded by the browser and conditionally handles login. Iside of docker TCP connections are monitored using cron. As soon as the user quits using the notebook, the dropping TCP connections are recognised and the cotainer cleans up after itself and kills itself.

Functions and variables

For your convience, we have added a few pre-defined functions to the IPython profile.

get

The get function will copy a dataset, identified by the history_id, from your current Galaxy into the docker container. It will return the file path to the copied file.

Example:

   with open( get( 44 ), 'r') as handle:
      for line in handle:
         print( line )

put

The put function takes a file path and transfers the file to the current history of your Galaxy session.

Example:

   put( './my_file.tsv', file_type = 'tabular')

HISTORY_ID

Accessing your current history is possible with the pre-defined HISTORY_ID variable.

connect to Galaxy

If you want to communicate with Galaxy you can use get_galaxy_connection() to get an bioblend GalaxyInstance object.

Example:

   from bioblend.galaxy.histories import HistoryClient
   gi = get_galaxy_connection()
   hc = HistoryClient( gi )
   for dataset in hc.show_history( HISTORY_ID , contents=True) ):
      print( dataset['id'] )

Security

By default, no security turned on. To do this, you must adjust two variables, apache_urls and password_auth in the configuration file (config/ipython.conf).

The apache_urls variable should be set to True, and containers should be secured via Apache+SSL for production usage. Please see the setup document for more information.

The password_auth variable should be set to True. It functions by generating a random password which is only shared with the docker, and the user's browser through Galaxy. In theory this should be a secure system (or at least as secure as the weakest link), however if you find a vulnerability in the system we've devised, please contact us and we'll patch it as soon as possible!

Authors

History

  • v0.1: Initial public release (01.08.2014)

Licence (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

galaxy-ipython's People

Contributors

bgruening avatar hexylena avatar hrhotz avatar jmchilton avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

galaxy-ipython's Issues

Supporting different languages inside the IPython

IPython will soon support many different languages, like perl, R and ruby. We need a user-friendly way to set the default interpreter and to use put/get from every language.

ToDo:

  • Language selection interface
  • IPython magic function to put/get datasets
  • enhance the ipython-docker image to ship basic R/perl/ruby environments

Password displayed but login succeeds.

@bgruening are you experiencing this issue too?

Login will succeed but you still see a popup with the password?

  • POST works, receives location: /{port}/ipython/ in response headers
  • GETs /{port}/ipython/ and receives a 404 not found.

HTTP/S needs to be configurable

(03:43:18 PM) jmchilton: erasche2: You are passing the password over clear-text HTTP - so yes I would call this a problem.
(03:43:29 PM) jmchilton: Proxying the traffic though Galaxy should fix that.
(03:43:31 PM) erasche2: We work under the assumption of galaxy being under SSL endpoint
(03:43:44 PM) jmchilton: notebook_access_url = "http://%s/ipython/%s/notebooks/ipython_galaxy_notebook.ipynb" % ( HOST, PORT )
(03:43:59 PM) erasche2: oh crap, you're right. That does need an S.
(03:44:28 PM) erasche2: Thanks for catching that!

get() method not working

Hi,
This extension is really cool! There seems to be some problem with reading the input data. Please check the attached screen-shot.

screenshot from 2017-07-11 16-07-01

Lack of example history

In order to showcase the full powers of our package we should build an example history that users can view and import (dunno how that'd work across galaxies).

It should have some example tabular/sam datasets and then we should have a couple notebooks for each of the different major graphics packages we have.

Allow for launching docker on remote hosts

A few possibilities:

  • SSH to a specified remote host in the conf file, launch there
  • Run via the native galaxy Job system.

This of course raises the issue of having to keep port mappings somewhere, so this may add a dependency, for this configuration option, of having a database available somewhere to talk to.

(01:33:00 PM) Unode_: if you can somehow channel that via the current job handling system would be perfect I'd say
(01:33:07 PM) erasche: we then randomly test to find an open one, and launch docker with that port
(01:33:11 PM) erasche: on a remote host...that's tougher.
(01:33:23 PM) erasche: hmmm @ via current job handling system.
(01:33:27 PM) erasche: I can see the argument for it
(01:33:33 PM) erasche: but the integration would be ... interesting.
(01:33:42 PM) erasche: Hmmm. I'll have to think on that.
(01:33:51 PM) erasche: I'll add it to our bug for remote hosts
(01:33:52 PM) Unode_: no, I mean the request to start a new docker. That would go via job handler, with all it's queues and rules on where the job should run.
(01:34:03 PM) erasche: yes.
(01:34:15 PM) guerler_ [[email protected]] entered the room.
(01:34:23 PM) erasche: i.e., it'd just submit a job as the user to the queue
(01:34:30 PM) erasche: and whenever it got executed it'd load up a galaxy instance.
(01:34:30 PM) Unode_: yup
(01:34:33 PM) erasche: docker/ipy*
(01:34:35 PM) erasche: not galaxy
(01:34:55 PM) erasche: I can't tell whether the sys admin in me is terrified or really excited at that idea
(01:34:57 PM) erasche: haha
(01:35:01 PM) Unode_: :D
(01:35:25 PM) Unode_: The crazy part is to then connect from the frontend to the docker machine.
(01:35:51 PM) erasche: yeah.....
(01:36:01 PM) erasche: the apache magic we do would get a fair bit more complicated
(01:36:21 PM) Unode_: I don't even know how that would work...
(01:36:21 PM) erasche: I think I remembere that you can read from a csv file in apache....we could write a list of host/port mapping...oh gods what am I doing

Unable to connect via bioblend

Hey guys,

I was trying to set it up for me (Ubuntu 14.04),
it was necessary to change the apache conf a bit from what you recommended:
I had to add "Allow from 127.0.0.1" to the ipython location,
and strange enough, the rewrite rule has to target 127.0.0.1, not localhost ... .
Anyway, with

apache_urls = True
password_auth = True
ssl = False

I can now start an ipython notebook, but I can't get history items, nor save to the history, or
save the notebook, and occasionally apache seems to crash and return a 404.
I think not being able to save/retrieve history items is because the communication using bioblend is not working.
The settings in conf.yaml are correct, and I can copy and paste the settings to another
ipython session on my Desktop, and it connects fine.
I can ping my host from within the ipython session, so it shouldn't be a network issue.
This is my call to bioblend:

import bioblend
gi=bioblend.galaxy.GalaxyInstance(url='http://lbcd41.snv.jussieu.fr/galaxy', key=mykey)
hl = gi.histories.get_histories()

it returns:ConnectionError: Unexpected HTTP status code: 404: None
Any idea what to look at now?

Blank screen when container starts too slowly

The notebook isn't up yet, so of course we cannot request that page. May need some JS logic in order to poll the page to see if it's up yet, alternatively just a second delay or so?

login is not working anymore with latest JS restructuring

With a complete fresh galaxy installation using localhost:8080 without apache the login procedure is not working anymore with authentication enabled. The worst part is, that the password login page is displayed but without any password hint. This happens because the AJAX call is successful for some reason but the cookie is not set. Getting the key manually from firebug and entering it is working and is setting the correct cookie.

async = false probably needs to be set, but it is also not solving this issue.

Imho it boils down, that we can't set a cookie via the ajax login procedure, why this is not possible and why it is still in a successful state ... I'm not sure. Also no clue why it broke at all.

Trouble with docker-in-docker proxy (?)

Hello @bgruening and @jmchilton ... I'm trying to set up a galaxy server that runs IPython. I'm having a hard time getting things going, and was hoping for some advice on how to get moving.

I built the docker image with this Dockerfile and its friends in the same gist: https://gist.github.com/mlovci/4bc35649bde8e6795a72

host$ cd docker_files/minimal
host$ docker build -t minimal .
.... this builds in 3 minutes
host$ docker run -it -P --privileged -v ${HOME}/instance_store/galaxy/:/export/ minimal bash
root@<container>$ /usr/bin/startup
.... this takes few moments to start and I'm able to view the galaxy cluster at <IP>:<PORT>, as expected. So far so good.
Then, I log in as admin and upload the test tab-delimited file I have in my gist. After it uploads I can click through Visualize -> IPython and I get a white screen on the browser.

Meanwhile, back in the terminal, I can watch the logs at /home/galaxy/uwsgi.log and it tells me that it's pulling bgruening/docker-ipython-notebook and after a bit, the new image shows with root@<container>$ docker images

For 60 seconds after the image finishes pulling, I can see that an ipython notebook process is running fine. And, if I expose the port that it is running on to the host, I can see the notebook dashboard.... it just doesn't display within the frame in galaxy.

I suspect there's something wrong with the way I set up the proxy handling, but I don't know enough about apache2 and galaxy at the same time to begin to fix this.

Can you bestow your knowledge?

Thank you,
Mike

Container telemetry for intelligent decisions

I'd like to see us have some sort of opt-in telemetry framework (this might be a separate git project), so that we can make intelligent decisions regarding the containers.

Ideally on first-run they'd get a screen, like with firefox, asking if they'd be willing to anonymously share usage information. If they answered positively, we'd cache their response (somehow... probably via /import/).

I'd like to see some of the following metrics:

  • lifetime of containers
  • dataset number + hashed username

These two would let us tell if people were accidentally closing containers, and if we should bump the crontab schedule to allow for reopening a recently closed container. We don't want to waste resources, but we also don't want to close prematurely if this is a user issue.

  • packages imported
  • packages installed (pip, apt-get)

These would let us determine if we should ship containers with more software, and which pieces of software we could safely remove because they were unused. (This may be conflated with a lack of advertising, see #11)

Is this reasonable? If this is, let's spec out a plan for how we might do this

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.