Clean up and update util scripts.

┆Issue is synchronized with this Asana task

Description:

As Noah and I have discussed, it would be nice to allow installation of plantCV into a user/system library of packages via the standard python mechanism. This will free users from having to add the package source to their $PYTHONPATH, and enable use of the help() function, now that all (?) of the plantCV functions have PEP8 doc strings.
Cf. the Image Harvest installation instructions.

Context

I create this issue just as a public reminder---I think we are committed to doing this for version 2.0, and so I wanted it to be in the list for that milestone. Please tag this issue as an enhancement, if you like.

Further suggestions

The setuptools package (rather than the legacy distutils package) is recommended for creating source distributions. I will keep reading through the documentation at setuptools.readthedocs.io and packaging.python.org as time allows.

Maybe it would be worth uploading the project to PyPI at some point, but that can wait.

As noted, "Scientific software tends to have more complex dependencies than most", and plantCV is no exception, primarily because it requires OpenCV to be installed. One could imagine automatic dependency handling to enable a user to install both plantCV and OpenCV in one shot, but it is not entirely obvious how to do this. So we should probably just update the docs to show that homebrew makes it relatively easy to install OpenCV on Macs, and otherwise defer worrying about this.

Using plantcv-pipeline.py to specify an image mask does not currently pass the information to the jobs it creates. For reference, the error I received was

OpenCV Error: Assertion failed ((scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F)) in cvtColor, file /build/opencv-SviWsf/opencv-2.4.9.1+dfsg/modules/imgproc/src/color.cpp, line 3959
Traceback (most recent call last):
 File "/home/jeffrey/Desktop/plantcv_test.py", line 118, in <module>
   main()
 File "/home/jeffrey/Desktop/plantcv_test.py", line 51, in main
   device, brass_mask1 = pcv.rgb2gray_hsv(brass_mask, 'v', device, args.debug)
 File "/home/jeffrey/plantcv/lib/plantcv/rgb2gray_hsv.py", line 14, in rgb2gray_hsv
cv2.error: /build/opencv-SviWsf/opencv-2.4.9.1+dfsg/modules/imgproc/src/color.cpp:3959: error: (-215) (scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F) in function cvtColor

If the mask is specified in the analysis script via default=my_mask.png, the pipeline does work.

Alternative parallelization schema for PlantCV that will allow the selection of some metadata (angle etc).

Thoughts on converting to an MIT license instead of GPLv2?

┆Issue is synchronized with this Asana task

Add scripts to allow plotting from database so you can select images to plot based on more detailed metadata.

Starting use-case: @jshoyer (and others) is using Raspberry Pi cameras to do top-down imaging of Arabidopsis thaliana plants grown in flats (3 x 6). @ArashAb is developing a function to automatically segment and analyze each plant separately.

Related issue: #10

In image_analysis.pl, image file name parsing is not working when the measurement label has an "_" character in it.

• Windows
• OS X
• Linux (revised)

┆Issue is synchronized with this Asana task

These things in instructions seem to be outdated (reported by someone via email).

git clone https://github.com/nfahlgren/plantcv.git
export PYTHONPATH=$PYTHONPATH:$HOME/lib/opencv/lib/python2.6/site-packages:$HOME/plantcv

A method that uses localized contour shape for segmenting out regions of high contrast within contours of plant masks. This method has shown some initial success with Setaria viridis for tracking discrete regions within plants during tiller exertion and with further development it may become a more flexible tool that can be broadly applied to other systems and distinct organ types (grass inflorescences, complex dicot leaves, etc...).

Add ability to have images created in debug mode to print to the screen. This will enable images to be posted to the output captured by Jupyter for real-time workflow debugging.

Multi-Plant Region of Interest:

Will be adding a function specifically to isolate multiple-plants in a field-of-view. We are using it for welled-plates but will write the function so it generally works for multiple plants per plate or pots.

(Per discussion on Tuesday Dec 1 with @nfahlgren)

Add normalization and correction steps to the PlantCV pipeline so that the Clowder PlantCV extractor (extract.sh) produces phenotypes that could be used in a GWAS analysis.

This is the first step required for terraref/computing-pipeline#31

I tried to run the fill command, copying the example in the docummentation page:

device, binary_img = pcv.median_blur(img, 5, device, debug=True)
device, mask = pcv.median_blur(img, 5, device, debug=True)
device, fill_image = pcv.fill(binary_img, mask, 200, device, debug=True)

The median_blur works fine, but I get the following error from the fill

Traceback (most recent call last):
  File "./test_fill.py", line 39, in <module>
    main()
  File "./test_fill.py", line 37, in main
    device, fill_image = pcv.fill(binary_img, mask, 200, device, debug=True)
  File "/home/sur/lib/plantcv/lib/plantcv/fill.py", line 15, in fill
    ix,iy= np.shape(img)
ValueError: too many values to unpack

I'm not a python expert, but I understand this means some problem with assigning variable types.
Thanks

Bug for crop mask function, ValueError: negative dimensions are not allowed

A few suggestions for a version 13 of https://figshare.com/articles/DDPSC_Phenotyping_Manuscript_1_Files/1272859

• Add DOI for the relevent publication:
  http://doi.org/10.1016/j.molp.2015.06.005
• Add link to plantCV v1 source code archive:
  http://doi.org/10.6084/m9.figshare.1447311
  • Details about which script produced which output file would be nice.
    Even better would be a Makefile to regenerate the ouput on demand.
  • Aside: the figshare page
    for the archived source tree
    is really terrible:
    no DOI and no link to GitHub commit on that page.
    Could archiving on Zenodo be any better?
• Add a ReadMe file
  • I will try to find time to draft one.

Hi,

Do you have a plan to upgrade plantcv to support opencv 3.x which has quite a few API changes? We tried to deploy plantcv on some HPC clusters which have GPU/CUDA 7.0 deployed. But opencv 2.x is too old for the cuda api version supported.

Or opencv 3.x is already supported by a branch?

Thanks,
Yan

Add instructions and style-guide on how to create or edit the documentation, which is now embedded in the git repository itself and can be updated on a commit-level basis where necessary.

The analyze_color docs and docstring need to be updated to remove the cslice_type argument (no longer used).

Documentation is missing for the dilate function.

When you run image_analysis.pl without the -c flag and the database does not exist it causes an error. We already check for the existence of the database to warn for overwrite, but need to check for the opposite.

The call to plot_image in erode should include the cmap='gray' argument.

In plantcv source code, I tested the image_analysis scripts to handle three types of input images from the LemnatecIndoor image collection from http://figshare.com/articles/DDPSC_Phenotyping_Manuscript_1_Files/1272859 : NIR_SV, VIS_SV, VIS_TV. However, when i randomly pick three images from the sample data in box, one from each type, only VIS_SV worked with tweaking. The other two types crashed opencv.

Data is retrieved from https://uofi.app.box.com/terraref-box-shared/1/4678953842

The error I encountered is:
OpenCV Error: Assertion failed ((scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F)) in cvtColor, file /home/ubuntu/software/opencv/modules/imgproc/src/color.cpp, line 3959 Traceback (most recent call last): File "/home/ubuntu/plantcv/scripts/image_analysis/vis_tv/vis_tv_z3000_L1.py", line 118, in <module> main() File "/home/ubuntu/plantcv/scripts/image_analysis/vis_tv/vis_tv_z3000_L1.py", line 61, in main device, brass_mask1 = pcv.rgb2gray_hsv(brass_mask, 'v', device, args.debug) File "/home/ubuntu/plantcv/lib/plantcv/rgb2gray_hsv.py", line 14, in rgb2gray_hsv hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) cv2.error: /home/ubuntu/software/opencv/modules/imgproc/src/color.cpp:3959: error: (-215) (scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F) in function cvtColor
The command I ran is:
~/plantcv/scripts/image_analysis/vis_tv/vis_tv_z3000_L1.py -i ./VIS_TV_z3000_329934.jpg -o output

See: terraref/computing-pipeline#76

Any reason why we should not add the -1 flag in the cv2.imread call to read images in their existing format instead of converting to color (default behavior)? Otherwise plantcv.readimage is only useful for color images.

Need a page (or something) describing the outputs of the analysis functions. Units, meaning, etc.

Probably just a simple fix of adding in the correct links to the readthedocs version.

See here:
https://github.com/danforthcenter/plantcv/blame/master/README.md#L30-L32

I am working on converting our existing documentation to readthedocs and will release a v1.1 with the updated documentation. Once I'm done with that, we will add the v1.1 documentation to the dev branch and then people can start collaboratively developing the documentation for v2.0.

We think matplotlib plotting functions are limiting our ability to run PlantCV over 14 cpu. To overcome this we are trying to remove matplotlib functions (or make output optional) wherever possible.

Different versions of the Perl module Time::Local output epoch time in either second precision (integer) or millisecond precision (float). This causes a problem with the downstream database queries. I need to explicitly recast seconds as an integer.

Normalize color between images with inconsistent lighting, would require a color standard in the image.

When running PlantCV parallel pipeline script (plantcv-pipeline.py) if sqlite db already exists it prompts for an overwrite.

If on a non-interactive system, such as a queuing system it will then either hang or exit as if all went normally.

Maybe reverse it and create database by default or add an option to auto overwrite.

The documentation for these functions suggests that they accept kernels, but they now only accept odd integers and build the kernels internally. This might be fine and we can just update the documentation. Is there a use case where it would be useful to supply a custom kernel? We only build symmetric kernels with the supplied input, is it useful to make other types?

The snapshot_id assigned by image_analysis.pl uses a local index to group images into snapshots. This fails to assign the correct snapshot_id to VIS TV images because they are processed separately from VIS SV images. I am realizing now that the datetime field could be used as the snapshot ID instead since it is already unique for images from the same snapshot. I propose removing snapshot_id from the database.

I have been going through the 'vis tutorial' again.

The individual steps are clear and nicely illustrated, but I have no idea how one should build a brand new pipeline or adapt/optimize an existing one. The method seems to be to copy-and-paste an existing pipeline then iteratively adjust the threshold values and eyeball the results. I find this harder to think about once the pipeline diverges (should I put more effort into optimizing saturation or blue-yellow?), partly because the downstream files (the ones that do not have a threshold value baked into their name) get overwritten when I (foolishly?) rerun a pipeline in the same directory. Perhaps I just need to experiment further and/or read more.

A hypothetical new tutorial might start with some species with a simpler shape. Alternatively, it might be illustrative to start with a more challenging background.

Semi-related note 1: specifying an output directory with the -o flag has no effect for me. The intermediate images created with in debug mode always show up in my current directory. Perhaps I have some shell or python variable configured oddly.

Semi-related note 2: is the 'parallelization' perl script deprecated in favor of the python one? If so, it would be nice to update that tutorial at some point.

Boundary line tool doesn't show up properly for images that are not the same size as lemnatec captured ones.

We should update the paths to the python executable to be more generic. Instead of

!/usr/bin/python

we should use

!/bin/env python

In the current plantcv version, the function analyze_color() is changed to have 11 parameters, while all the scripts in scripts/image_analysis still use 12. Please make it consistent.

Calculate "0" threshold for Fmin images from avg Fdark image.

I think these templates are useful, see: https://github.com/terraref/computing-pipeline/tree/master/.github

┆Issue is synchronized with this Asana task

Develop a queuing system/batch processing option to add on in addition to the current local/multiprocessing capabilities.

Currently a user can decide whether or not to output analysis images by modifying an image analysis script. Instead we propose to make this a standard option switch.