mfitzp / pathomx Goto Github PK

Running 3.0.0.rc4 on OSX 10.10 Yosemite on a late 2013 macbook pro;

I can import a couple/few Bruker NMR successfully, and load up some processing tools (Peak picking, baseline correction, etc), and then randomly the software crashes.

I've pastebin'd the crash report.

V3.0.0.rc3
OS X 10.10 Yosemite (Beta 6).
Retina Macbook Pro (Late 2013).

If you delete a tool from the workspace, the workspace section itself shifts to the right irreversibly. Only closing and restarting seems to fix the problem.

In attached screenshots the issue is demonstrated. By two-finger tap (right click) on the text input, and selecting delete, the issue arrises.

Seems to happen for various input files, or for removing any tool.

EDIT: If you two-finger tap on a remaining tool in the workspace, it seems to shift everything back to normal. However this isnt an ideal fix, for example if the other tools are now off-screen.

Recent commits that you have in Files ui.py and others have python2 print statements.
e.g.,

File "/home/akshay_v/workspace/pathomx/pathomx/ui.py", line 1759
    print "Dummy lock"

Implement a lock for MATLAB, R etc. to prevent multiple tools trying to run code simultaneously.

Both MATLAB and R tools are single-threaded, single instance access so to share between tools a lock is needed. Each tool must wait for the first tool to complete before it can start it's own processing.

Integration with the Figshare online figure/data sharing platform. This will need quite a bit of back-end integration e.g. for the user to set their Figshare key.

I'm trying to install from the Github repo on OS X, following the developer installation instructions. I've got all of the dependencies installed including IPython, but when I run pathomx I get:

DEBUG:root:Loading globals.py
DEBUG:root:Loading settings...
DEBUG:root:Setting up managers...
DEBUG:yapsy:PluginManagerSingleton initialised
DEBUG:root:Loading ui.py
DEBUG:root:Loading data.py
DEBUG:root:Loading views.py
DEBUG:root:Loading translate.py
Traceback (most recent call last):
  File "Pathomx.py", line 7, in <module>
    from pathomx import Pathomx
  File "/Users/robin/code/pathomx/pathomx/Pathomx.py", line 50, in <module>
    from . import ui
  File "/Users/robin/code/pathomx/pathomx/ui.py", line 40, in <module>
    from IPython.nbconvert.filters.markdown import markdown2html_mistune
ImportError: cannot import name markdown2html_mistune

I'm running Python 2.7 and IPython 2.4.0. Interestingly, when I try and import that module I get the following results:

>>> import IPython.nbconvert.filters.markdown as m
>>> dir(m)
['BytesIO', 'ConversionException', 'NodeJSMissing', 'TextIOWrapper', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_node', '_verify_node', 'cast_bytes', 'check_version', 'get_output_error_code', 'markdown2html', 'markdown2html_marked', 'markdown2html_pandoc', 'markdown2latex', 'markdown2rst', 'marked', 'os', 'pandoc', 'print_function', 'subprocess', 'warnings']

It seems like lots of the other markdown conversion functions are there, but not the one that pathomx uses.

Any ideas?

Download/import data through the Galaxy (a remote batch processing tool) API.

Some long-running calculations would be nice to 'skip' using a local cache of processed data. The cache should hash the inputs (data) and config (settings) for a given tool, which taken together should predict the output. The runner can check for the existence of a cached result before initiating the run and return that data instead.

Implement cache as a store of pickled objects.

Tools can opt-out of caching.

Currently the graphs in Pathomx are very unconfigurable. Simple things like being able to change scales/etc. and add error bars would be very helpful.

The task is to define a set of basic options that are supported on multiple matplotlib graph types and create a ConfigPanel for them and add the config hooks to the views. In order to support the config by default as long as the panel is in place, the views should either function without the settings (preferable?) or set defaults on the config (bit hacky).

The config handler has hooks into the QtWidgets so you can define a config option 'show_error_bars' and set the widget as a handler and everything is automated - updating the widget refreshes the graph without intervention so this should be fairly straightforward to implement. One option here is to have the config panels add default settings options for any setting they can control (assuming it's not already set).

This would mean a tool can simple add the panel and forget.

Hi,
I am a new potential user of Pathomx but I was not able to install it on my mac. I'm running OSX 10.7.5 - I realize that might be the problem right there, updating to 10.9 could solve the problem. Nevertheless, FYI, when I click on the dmg, drag the unpacked Pathomx to my Applications folder and try to open it, this immediately appears in the crash report (see below).

Any advice is appreciated!
Katrine
http://www.whiteson.org/katrine/

Process: launchd [1167]
Path: /Applications/Pathomx.app/Contents/MacOS/Pathomx
Identifier: launchd
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: launchd [222]

Date/Time: 2014-04-15 13:42:22.405 -0700
OS Version: Mac OS X 10.7.5 (11G63b)
Report Version: 9

Crashed Thread: Unknown

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00007fff5fc01028

Backtrace not available

Unknown thread crashed with X86 Thread State (64-bit):
rax: 0x0000000000000055 rbx: 0x0000000000000000 rcx: 0x0000000000000000 rdx: 0x0000000000000000
rdi: 0x0000000000000000 rsi: 0x0000000000000000 rbp: 0x0000000000000000 rsp: 0x0000000000000000
r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x0000000000000000 r11: 0x0000000000000000
r12: 0x0000000000000000 r13: 0x0000000000000000 r14: 0x0000000000000000 r15: 0x0000000000000000
rip: 0x00007fff5fc01028 rfl: 0x0000000000010203 cr2: 0x00007fff5fc01028
Logical CPU: 0

Binary images description not available

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 1
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 1966
thread_create: 1
thread_set_state: 0

Model: MacBookPro8,1, BootROM MBP81.0047.B26, 2 processors, Intel Core i7, 2.8 GHz, 8 GB, SMC 1.68f98
...

Hi,
I am using a 2009 Mid Macbook Pro, along with Pathomx 3.0.2.
After installation (by dragging the application to the application folder)
It wouldn't open properly and crashes instantly.
I have pasted the error message as followed: http://pastebin.com/q1SjCBqw. Would you kindly tell me what I have done wrong? Thanks!

Hi, I have successfully set up the dependencies required to set up the development environment. Thanks for your updated documentation in http://docs.pathomx.org/en/latest/dev_install.html#macos-x
Now, I forked the project and cloned into my local computer. When I run python -m pathomx.Pathomx I get the following error: /Library/Python/2.7/site-packages/pandas/io/excel.py:626: UserWarning: Installed openpyxl is not supported at this time. Use >=1.6.1 and <2.0.0. .format(openpyxl_compat.start_ver, openpyxl_compat.stop_ver)) Traceback (most recent call last): ................. .................. from .version import VERSION_STRING ImportError: No module named version
Any ideas?

Pathomx allows connection of multiple simple tools into complex workflows. But analysis can be limited by the availability of a defined tool for a given task. As Pathomx ships with the entire numpy/scipy stack it would be nice to have this available for users.

This enhancement requires development of a code editor that can take inputs from the defined tool input ports and output the results to the outputs (customiseable) views. The structure of the tool generator code is such that this is relatively simple to achieve.

Interface requirements: PyQt Python code editor (an implementation of this is available) and a UI for input and output port definition (the former will require some method for limiting accepted inputs; but this may be left for future development).

Internal code requirements: The code will be run with Python eval while importing a number of useful tools including numpy, scipy, nmrglue, etc. This is of course horribly unsafe and so the code will follow the example of remote-service tools and default to auto-pause and warnings.

This is a bitesize task for developers interested in joining Pathomx development.

Linux support would be a lot easier if Pathomx was Python 3 compatible. Some dependent libraries (e.g. nmrglue) do not yet support Python 3. We can either help them convert, or accept that some tools won't function on these builds.

Unsatisfyable dependency? You mean >= not >? Pip only has v1.1.2 available at
https://pypi.python.org/pypi/PySide

$ sudo pip install metapath
Downloading/unpacking metapath
  Running setup.py egg_info for package metapath

Downloading/unpacking PySide>1.1.2 (from metapath)
  Could not find a version that satisfies the requirement PySide>1.1.2 (from metapath) (from versions: )
No distributions matching the version for PySide>1.1.2 (from metapath)
Storing complete log in /home/ben/.pip/pip.log

Currently it's not possible to install Pathomx on Linux. This is largely due to the fact that most Linux distributions ship PyQt5 for Python3 only. Switching to Python3 is planned however there are a number of dependent libraries that don't currently support it (e.g. nmrglue).

It would be easier to support this if we were using a plugin system (e.g. distutils) that didn't require us to ship the dependencies of our plugins with our core code.

Currently all dependencies for plugins need to be shipped with Pathomx itself. This limits plugins to using packages that we provide, which shouldn't be the case. We can support custom package install via pip e.g.

import pip
pip.main(['install','scipy','--upgrade'])

We can use this to auto-upgrade packages as plugins depend on them. Define these dependencies in the .pathomx-plugin definition file and auto-run the upgrade-script on startup for all activated plugins (needs to happen before any tools are dragged to the workspace or may error) and any activated after.

We can run the installation code on the kernel to keep the UI free.

A matplotlib driven difference graph is available in the views, but not currently supported as an independent tool. Add a new tool to the basic_graphs plugin to support this.

The matplotlib code is mostly concentrated in the views.py module. Each of these views handle taking in data, manipulating it and then outputting it via a matplotlib graph. The simplest way to add a new basic graph is through the basic_graphs plugin ( plugins/basic_graphs/basic_graphs.py )

If you open basic_graphs.py you'll see two basic tools defined. Tools are what you use to manipulate/visualise data - and the things you can drag around in the Pathomx interface. The init sets up the toolbars, inputs and ouput ports (appear as circles on the editor), defines the type of data the tool accepts and then adds a view component. The view is the thing that you'll be creating (a new type of graph).

Below this there are two functions generate and prerender. In this case 'generate' does nothing except take in data from the input port and puts it out the output port (labelled 'View'). In more complex tools this is where the most of the processing occurs and it's multi-threaded. The 'prerender' function takes the output of the generate function and adapts it read for the view. This allows multiple tools to use the same basic view code, regardless of their outputs. Of course all this could happen in the 'generate' function but since it doesn't affect down-stream tools, we move it out of the way. This allows the next tool in line to start processing before we've finished rendering the graph.

The 'prerender' function is also multi-threaded, but the actual rendering is not (code in views.py currently occurs in the GUI thread) so the goal is to do as much as possible before passing the data on. In BarTool for example we generate a mean set of data, and add stddev to the statistics for the dataset. This allows the plot to plot the mean + error bars.

The output of this function is passed to the view. In this case it's the one defined by: self.views.addView(MplCategoryBarView(self), 'View') i.e. it's MplCategoryBarView (Mpl is a convention to identify it as a matplotlib graph view). If you open views.py on line 1016 you'll find the definition for MplCategoryBarView. As you can see it takes the one input output by the 'prerender' then turns it into a graph. The code for this is quite complicated for axis grouping, etc. so we'll skip that for now.

If you work your way back up views.py to MplDifferenceView you'll see a much simpler view. This takes two dso (data set objects) holding data, then plots them highlighting the difference between them. Simple as that.

So your first challenge (should you choose to accept it) is to create a new Tool in the basic_graphs.py plugin. To do this I recommend you copy-paste the SpectraTool and amend it to support two inputs (input1, input2) and pass them to a MplDifferenceView.

The editor is nice in many ways, but a bit clunky and dry in others. It would be nice to get some useful visual enhancements in there to make it even more intuitive.

• Inputs and ouputs are represented by lines rather than individual ports - making it impossible to select the destination of data. Changing these to connectors for each I/O would be far better.
• Auto-highlight acceptable port targets when dragging an output
• Allow overriding of the consume-check when dragging (force connection)
• Annotations, names and regions could be applied to help visually clarify the workflow
• Snap to grid on drag drop functions

In release v3.0.0.rc5 for OS X, there seems to be some missing/broken graphics.

For example, when using the example thp1_1d_nmrlab_metabolab.csv dataset provided, the spectra plot differently (worse) than in an earlier release.

Also it appears that the markers for a PCA scores plot are missing, but based on the shapes of the circle outlines for the clusters, it must be plotting it correctly behind-the-scenes.

Add a correlation plot for data under basic graphs.

Plot should offer configuration of what input variables (labels, entities) to plot onto a correlation plot, together with correlation testing and statistics.

Some mechanism (specific plots or other) of filtering data visually would be great. The simplest implementation would be the ability to create regions in a scatter plot that are then selected and output as a particular dataset (output port). In this way it would be possible, e.g. to use a PCA to filter data (cf. FACS data filtering).

How this would work on bar or line plots is another thing - or how to interpret the selection in a sensible way.

Integrating with existing MATLAB tools would be a real bonus.

The simplest cross-platform method for integration would probably be using pymatlab-bridge but other method's are available.

Passing data into MATLAB and running a script/function (probably definable within the tool) might be the easy part. Implementing a useable interface for pulling apart Pathomx dataset objects and reassembling the results might be a bit more tricky! However, once an interface for this is worked out it can form the basis of other linkages (e.g. [R]).

Trying to use the metabohunter plug-in results in an error stating metabohunter does not exists.
I'm running the lastest windows installer

I just installed Pathomx 2.3.0 (I had 2.1.0 working before this), but when I want to start the program I get the following error message:

Some system for assigning graph markers and colours for both classes (groups in data) and scales (ranges) is needed. It would probably make sense to define this centrally attached to the workflow, so that you can set them once and all figures will obey the same theme.

I'm sort of envisaging some kind of filter system on class markers. So, for example if you have 4 classes M1, M2, DC, Mo. You could define classes assigning by filter "m" diamonds to M1, M2, Mo; "dc" blue circles; "1" green"; "2" red; "o" purple. The end result being the M1 is a green diamond.

This needs a nice logical interface to work together with the appropriate lookup functions for use in the views.

Add options to export figures from all views either to a specific folder, to the source data folder, or other. Figures should be output on calculation completion.

Implementing this will probably also add a foundation from automated image export from specific tools.

Add support for PLS-Regression using either the MATLAB script or (preferable) a native implementation. All steps, including e.g. variable selection, may need to be broken down into multiple tools if they're possibly applicable to other tools.

Import of IPython notebooks to (multiple) custom script tools. Auto-generate the connections through variable references.

Export of workflows to runnable notebooks should also be possible. It may be worth considering using the IPython notebook as the backend format for Pathomx generally, depending if a sensible way to manage inter-cell dependencies is possible.

Sometimes it is nice to automatically run an analysis based on external triggers. Pathomx should support 'automator' tools that can trigger the execution of other tools.

Automators are tool-like but do not contain editable code. The output is a config dict which can be connected into another tool to automatically change it's configuration.

Using this setup implementation of specific automators can be achieved. In each case the config update will trigger the target.

• timer trigger; emit config containing {'timestamp':<>}
• file changed trigger; emit config containing {'filename':<>}
• folder changed trigger; emit config containing {'folder':<>}

Together with IPython notebook import this would allow import of QtIPy tool settings to offer a more 'useful' "data automator" tool.