Error in projectLoop batch (in be_loop_model) where order of reshaped array after loop projection is reversed from what it should be.

Need a custom memory use calculation for SignalFilter. Taking the FFT of the input data converts it to complex values which increases the initial size.

/io/translators/df_utils/parse_dm3.py in parse_dm_tag_data(f, outdata)
227 else:
228 _delim, header_len, data_type = get_from_file(f, "> 4s l l")
--> 229 assert(_delim == "%%%%")
230 ret, header = dm_typesdata_type
231 assert(header + 1 == header_len)

Also implement SHO fitting.

┆Issue is synchronized with this Asana task

All main datasets should have appropriate labels and units attributes to tell visualizers / plotters about the data.
This applies to all translators.
The processing and analysis classes should either take copy these attributes from the source dataset or create new attributes as appropriate

┆Issue is synchronized with this Asana task

Remove any redundant functions.
Effectively almost all functions consist of a set of line plots or image plots.
Make some general functions where appropriate - see plotLoops as a good example
Its OK to have quick plotting functions for specific use cases. However these functions shouldn't have to reinvent the wheel.

┆Issue is synchronized with this Asana task

in Image windowing

Things from BEPS and image cleaning notebooks that would be useful for other applications should be generalized.

Colorbars needed for visualization of spatial maps, as well as export ("Save Figure") button for each visualizer.

All translators should attempt to extend the newly updated Translator class.

makeMain should be moved to hdf_utils

┆Issue is synchronized with this Asana task

When reading a large dataset in chunks, the get_data method updates the start and end indices by itself. This would cause serious problems when fitting since the indices would have moved to the next chunk even before the get_guess method can be called. Need a better book-keeping method to keep track of indices of the current chunk

┆Issue is synchronized with this Asana task

I setup a sphinx conf.py and built the documentation (sphinx-apidoc) . I also setup a webhooks for the repo to communicate with readthedocs.io for automatic update of docs.
All of the above works. But the package structure (due to imports) needs to be updated first.
Anyway checkout the master-branch.
Help needed!

Should take any MAIN dataset regardless of the dimensionality. Core components implemented in viz.jupyter_utils

┆Issue is synchronized with this Asana task

just play with num_comps:

num_comps = 7
px.plot_utils.plot_loops(np.arange(9), np.random.rand(num_comps,9))

Some plot attributes are being hard coded rather than pulled from the datasets. Ex: Field names in visualize_sho_results.

Need ability to view the spatial maps of the Loop Fit parameters (V+, V-, Work of switching, etc.). Currently only allows visualization of the parameters of the loop fit function (a1,a2,b1,b2, etc.).

The BE_SHO_model does not. Other models should be checked as well.

pycroscopy\io\translators\igor_ibw.py in _read_parms(ibw_wave)
173 Dictionary containing parameters
174 """
--> 175 parm_string = ibw_wave.get('note').decode('utf-8')
176 parm_string = parm_string.rstrip('\r')
177 parm_list = parm_string.split('\r')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 2200: invalid start byte

Adapt the PySPM guess strategy to the same format as the ones in the guess methods. This should be trivial.

┆Issue is synchronized with this Asana task

Sometimes corrupts parms file during translation.

This class will be very similar to the model class, except that there is no optimization. Loop projection would be the first class that extends this. Other examples include FFT filtering.

┆Issue is synchronized with this Asana task

ioHDF5 simplifies the writing significantly but we don't have a class that simplifies reading. One could envision a class that would take a file handle and create some sort of a tree structure where the only nodes are the groups and the only leaves are the main datasets. all ancillary datasets would have to be accessed as attributes. This class should be able to request an N dimensional form of the dataset and it should be easy to request the spectroscopic vector for plots regardless of the dimension. This class would not read the entire file into memory - rather it would make operations a lot simpler. This class would aid the visualizer (and ORS) significantly.

┆Issue is synchronized with this Asana task

Use region references instead of separate groups and datasets.

AttributeError Traceback (most recent call last)
in ()
----> 1 input_file_path = px.io.io_utils.uiGetFile(caption='Select translated .h5 file or raw experiment data',
2 filter='Parameters for raw BE data (*.txt *.mat *xls .xlsx);;
3 Translated file (.h5)')
4
5 (data_dir, data_name) = os.path.split(input_file_path)

AttributeError: module 'pycroscopy.io.io_utils' has no attribute 'uiGetFile'

The io_utils is missing the uiGetFile

AIC / BIC

Look at arguments for Kmeans such as n_jobs and init to speed up K-means
We may need to look for anther package that speeds up SVD.

Currently attributes are named as "attribute_1_unit" for those with units and "attribute_2" for those without units. A parser looking for certain attributes would currently absorb "unit" and "2" as the units. We should be naming the attributes to something like "attribute_1__unit" or "attribute_1-unit" to avoid confusion

At least explore other alternatives that make better use of CPU / GPU resources and are possibly more memory efficient. Something that computes with limited memory on PCs would be great.

The win_slices are all floats on Mac. Reported by Artem. uint+int is returning a float.

Need to check if running 32-bit system and adjust memory usage accordingly.

This is necessary to figure out if we need to take the complex conjugate on the data

/Users/nl7/anaconda2/lib/python2.7/site-packages/pycroscopy/io/hdf_utils.py in ()
10 from warnings import warn
11 import numpy as np
---> 12 from .microdata import MicroDataset
13
14 all = ['getDataSet', 'getH5DsetRefs', 'getH5RegRefIndices', 'get_dimensionality', 'get_sort_order',

ImportError: No module named microdata

┆Issue is synchronized with this Asana task

The idea is to provide an alternate framework for code to scale to multi CPU systems. The same scientific analysis and processing functions should be usable in the current multiprocessing implementation and the future MPI implementations. This should ensure that scientists can continue to write / test / execute simple functions and minimal effort needs to be applied to scale the computation to a large number of cores / CPUs

All image plots need colorbars with units. Consistency across notebooks for this is also something we should work on.

Check phase of raw data.
You will have to account for phase wrapping.

┆Issue is synchronized with this Asana task

Add pycroscopy and underlying package (eg - scipy) versions to the datasets for any operation. These should be added to datasets and not datagroups to encompass the situation such as the Guess operation coming from a different source (eg. instrumentation software) and the fit coming either from pycroscopy or BEAM etc.

This applies to the current BEPSndf, BEodf, relaxation, and GmodeLine translators

┆Issue is synchronized with this Asana task

Port the tree class from PySPM to pcroscopy

┆Issue is synchronized with this Asana task

I was wondering if you were interested in uploading the package on conda-forge, which, as you probably know, is a widespread and robust package managing system. If you agree, I can prepare the feedstock recipe.