dynaslum / satsense Goto Github PK

The function scipy.misc.imread has been removed from scipy since version 1.3, but we use it in the function satsense.util.mask.load_mask_from_file.

As a Remote sensing expert
I want to Train a machine learning algorithm using the calculated features
because I can then apply the algorithm to detect slums.

Estimate: 3

We have our own implementation of the HoG feature, but there is also an implementation available in scikit-image:
https://scikit-image.org/docs/stable/api/skimage.feature.html#hog

Maybe we could use that instead.

As a agent based model researcher
I want to Validate my model using a complete set of slums
because I want to make sure my model is correct

Update the satsense.util.mask module so it works with the new design

• Make it possible to install satsense with pip. Upload from travis?
• Create a package on conda-forge too?

The skimage.feature.canny function was changed in version 0.15 of scikit-image. We need to update our unit tests and check that it still works as expected before we can update scikit-image.

please solve issue argent

As a remote sensing expert
I want to Calculate the features in satsense on a satellite image.
because This allows me to use them to detect slums

Estimate: 2

As a Remote sensing user
I would like to be able to restore a saved FeatureVector instance from file (netcdf or geotif) for Texton and Sift features.
because then I do not have to recalculate the initialization of these features to calculate them.

In case you are experiencing NetCDF: HDF errors after installation, see rasterio/rasterio#1574.

We need a careful look at whether or not the canny edge and texton descriptor images are acceptably similar when computed on smaller chunks of the image instead of on the entire image.

It would also be good to generalize this, so it is easy to add image types that require access to information about the entire image.

As a Satsense developer
I would like to add a windows build to the automated build system
so that I can be sure that satsense works correctly on windows systems

Windows on appveyor (Boatswain has appeveyor config as an example)

As a user,
I want to run satsense over a few sentinel-2 tiles, and osm shapefile as my ground label
because I want to do landcover classification.

Work on this is done in #14

• Prevent copying a large numpy arrays between processes
• Use python slices instead of windows
• Check speed of rasterio windows against loading the image in bands (-> this was too slow to use for window generation)

Partly done in the develop branch.

We should make sure the version is set correctly

One idea was to have a test based on area + location of the classified result.

As a Satsense developer
I would like to add a mac build to the automated build system
so that I can be sure that satsense works correctly on mac systems

As a Remote sensing expert
I want to Calculate the texton and sift features on a large satellite image on a computer with limited memory
because I do not have access to a computer with enough memory

Estimate: 5

List of features that need tests:

• hog
• lacunarity
• ndxi
• pantex
• rid (do we believe this feature is worth the effort?)
• sift
• texton

The minimum test that should be available is a regression test. More is better, but maybe not feasible in the limited amount of time we have.

We may also consider comparing with the features as computed by spfeas.

• Make a version 0.8 release as is
• Create a doi on zotero
• Add to https://www.research-software.nl

We need a strategy to handle masked image pixels values in the feature computations. This is probably different per feature if you want to do something meaningful with masked image data. The simplest and safest strategy is probably to return a masked or fill value instead of the feature value(s) if the input image contains any masked value.

Write unit tests for the satsense.image.Image class

When loading computed features at a later time, the mask is not properly loaded...
This cost me quite some time to figure out what was going on with my features (all my statistics were messed up, +infinity for the means, etc)

I was loading features from .tif files.
I did look at the feature loading/saving code but it seemed like there is code for handling the mask...

This is the temporary fix I used (which assumes the mask fill_value = 1e+20):

x_test_mask = np.ma.masked_values(x_test, 1e+20)
mask = x_test_mask.mask

vector_mask = np.any(mask, axis=1)

index_vector = ~vector_mask
x_test = x_test[index_vector]
y_test = y_test[index_vector]

There is some suspicion that the generated windows are slightly shifted wrt to the edge of the image.

As a Remote sensing QGIS user
I would like to Use the feature extraction capabilities of satsense from QGIS
because I would like to use these features, but do not have experience in programming python.

Check if it is feasible to make it possible to use satsense as a QGIS plugin and do so if the resulting installation procedure is not too complicated.

As a Remote sensing expert user of satsense
I would like to Have an example notebook showing how to use satsense
because I can use this as a starting point for my own research.

Most notebooks are outdated. Update or remove once we have a more stable API.

This notebook is most informative and relatively up to date:
satsense/notebooks/demo_notebooks/Classification/Simple Example.ipynb

As a Remote sensing expert
I want to Apply the trained machine learning algorithm on a satellite image to detect slums
because I can then give this result as a validation set to the model builder

NEEDS REFINEMENT

From travis:

cv2.error: OpenCV(3.4.3) /io/opencv_contrib/modules/xfeatures2d/src/sift.cpp:1207: error: (-213:The function/feature is not implemented) This algorithm is patented and is excluded in this configuration; Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function 'create'

As a remote sensing user of satsense
I would like to be able to look up how to use the relevant parts of satsense in the documentation
because without documentation using satsense will be very hard

• Check public interface for docstrings
• Example Jupyter Notebooks #17
• Write User Guide
• Readthedocs documentation (sphinx)

Estimate: 8 (timeboxed to 5)

The satsense.features.FeatureSet class should provide a convenient way for users to combine satsense.image.FeatureVector or satsense.features.Feature objects into an array that can be used for machine learning tasks.

The set should allow;

• easy addition and removal of features or feature vectors
• automatic reshaping from a number of feature vectors of shape (width, height, window_number, feature_length) into (n_samples, combined_feature_size)
• support computing and loading from disk or mix of these operations
• train/test/validation split

The feature extraction example uses texton to (partly) describe the pantex example output, both in the text as in the subcaptions of the figure.

https://satsense.readthedocs.io/en/latest/notebooks/feature_extraction.html#texton

As a user
I want to read examples in the documentation that work
because I will have trouble using satsense if they don't

We should probably enable doctests, this can be done by adding --doctest-modules under addopts in setup.cfg.

We should add more netcdf conventions to the writing of files.

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch05s06.html

There is a checker for nc files here:
http://cfconventions.org/compliance-checker.html

Maybe it would be nice to use parametrized fixtures or use pytest.mark.parametrize to reduce the amount of duplicated code in the test functions.