Giter Club home page Giter Club logo

pyoints's Introduction

Pyoints

Pyoints is a python package to conveniently process and analyze point cloud data, voxels and raster images. It is intended to be used to support the development of advanced algorithms for geo-data processing.

General concept

The key idea of Pyoints is to provide unified data structures to handle points, voxels and rasters in the same manner. It is assumed that the data can be interpreted as a two or three dimensional point cloud. Thus we have a collection of geo-objects, here called GeoRecords, which are characterized by two or three dimensional coordinates coords, a spatial reference proj and a transformation matrix t. The spatial reference and transformation matrix are required to define the global location of the geo-objects. Next to the origin, the transformation matrix also stores the scale and rotation of the local coordinate system.

The unified data structures simplify the joint use of point clouds, voxels and rasters significantly, while keeping their natural characteristics. To ease the processing and analysis, each point, voxel or raster cell are stored in the commonly used numpy record array according to its natural structure. For example, a three dimensional point cloud represents a set of points, each characterized by a coordinate and none to many additional attributes. So the points are stored in a one dimensional record array. Conversely raster images are stored in a two dimensional record array, which allows for the commonly used index access of raster cells. Still, each cell is characterized by a coordinate and none to many additional attributes. Three dimensional voxels are accordingly represented by three dimensional record arrays.

Since all objects of a voxel or raster are also interpreted as a point cloud, spatial neighborhood queries, like nearest neighbors or distance search, can be performed with unified functions. Thus, each extension of GeoRecords provides a spatial index IndexKD to conveniently perform efficient neighborhood queries. The class IndexKD is a wrapper of different spatial indices, to handle k-dimensional spatial queries of different kinds. The spatial indices are always only initialized on demand for performance reasons, but cached for future queries. For example, an instance geoRecords of GeoRecords representing three dimensional points allows to create a three dimensional spatial index by calling geoRecords.indexKD(). If you are not interested in the third dimension, you can call geoRecords.indexKD(2) to perform two dimensional spatial queries.

If you just want to handle coordinates without additional attributes, the class Coords might interest you. This class also provides the IndexKD feature, but waives to use of a projection and transformation matrix.

Is Pyoints suitable for me?

Based on the general concept of Pyoints presented above, a bunch of algorithms, functions and filters have been implemented, to process geo-data with low programming efforts.

Of course, one might wonder whether Python is adequate to analyze point cloud data since other languages languages are much more efficient. This is obviously true, but in the experience of the author, Python is very useful to implement and test new algorithms very quickly. Algorithms for point cloud analysis and processing often rely on spatial neighborhood queries. Since Pyoints takes advantage of very efficient Python libraries, which are basically wrappers for binaries written in more efficient languages, the performance loss is limited. Additionally, Pyoints takes care of selecting and caching appropiate spatial indices for you, so you can focus on the core of your algorithm idea.

So, if you have an algorithm idea and you want to implement it quickly and play around with different settings, Pyoints is made for you. After finding an approiate algorithm it can be implemented in a more efficient language if you like. Thus Pyoints is particulary designed for scientists and developers of new geo-spatial algorithms.

Installation

Conda Installation

Conda installation is recommended. To install Pyoints with Conda run:

conda install -c laempy pyoints

Currently only Linux (64 bit) has been packaged with Python 3.5, 3.6 and 3.7.

To enable full LAS support (spatial reference system definition), you need to install liblas and its python bindings manually.

pip install liblas

Installation from source

We recommend to use a virtual environment (like Conda) to install Pyoints.

Installation of the external dependencies:

conda install gdal pyproj "numpy>=1.15" rtree

Cloning of the repository:

git clone https://github.com/laempy/pyoints.git
cd pyoints

Installation of the python dependencies via pip.

pip install -e .

Conda Build

If your system is not supported yet, you might build and install it using the conda reciepe specifying your desired Python version.

Build

conda build /path/to/pyoints/conda --python=3.x

Installation

Create a virtual environment pyoints_env and add the newly build Pyoints package. The package path can be found with conda build . --output.

conda create -n pyoints_env /path/to/builded/package/pyoints-*-py*_*.tar.bz2 --use-local

Finally install the package:

conda activate pyoints_env
conda install pyoints

Tests

Test the package by running the provided doctests:

python /path/to/pyoints/tests/test_pyoints.py

Documentation

You can find the documentation at docs. Here you can find some introducing tutorials and docstring examples for each module.

License

Copyright (c) 2018, Sebastian Lamprecht, Trier University, [email protected]

Pyoints is free software made available under the GNU General Public License v3 or later (GPLv3+). See LICENSE for details.

Citing

Please cite Pyoints as follows:

S. Lamprecht (2019): "Pyoints: A Python package for point cloud, voxel and raster processing". DOI

Contributing

Any conribution to the pyoints project is welcome. See CONTRIBUTING for details.

Acknowledgements

Funding

This work has been supported by the European Commission under the grant agreement number 774571 Project PANTHEON.

Datasets

For the tutorials the "Stanford Bunny" dataset has been used:

Stanford Computer Graphics Laboratory (1994): "Stanford Bunny", URL: https://graphics.stanford.edu/data/3Dscanrep/, (Accessed: 17 January 2019)

Python Depencencies

This software dynamically binds the following Python packages (distributed via pip). Many thanks to the authors for providing such great open source software.

cylinder_fitting

Xingjie Pan

dill

Mike McKerns

pygdal

Frank Warmerdam

laspy

Grant Brown and Howard Butler

liblas

Romulo Goncalves

networkx

NetworkX Developers

numpy

Travis E. Oliphant et al.

opencv-python

Olli-Pekka Heinisuo

pandas

The PyData Development Team

plyfile

Darsh Ranjan

pyproj

Jeff Whitaker

scikit-learn

Andreas Mueller

scipy

SciPy Developers

rtree

Howard Butler

External Depencencies

Pyoints binds following external libraries as required by some of the pip packages.

GDAL

Frank Warmerdam

Liblas

Howard Butler, Mateusz Loskot et. al.

libspatialindex

Marios Hadjieleftheriou

pyoints's People

Contributors

danielskatz avatar laempy avatar reedanders avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

pyoints's Issues

[JOSS-REVIEW] Citations in paper.md

In relation to openjournals/joss-reviews#990

In text citation for various software packages are done in an incorrect manner. Instead of

Libraries, like (Rusu & Cousins, 2011), (Zhou, Park, & Koltun, 2018) and (PDAL contributors, 2018) focus on 3D point cloud processing.

write

Libraries, like PCL (Rusu & Cousins, 2011), Open3D (Zhou, Park, & Koltun, 2018) and PDAL (PDAL contributors, 2018) focus on 3D point cloud processing.

See here for more examples of correct use of in-text citations.

JOSS Review - examples / getting started

It would be really helpful in the documentation to include a first example to get new users up and running and doing something with pyoints. The documentation is really focussed on the API. Having a Getting Started,User Guide, or Examples section with a simple example that produces a plot can help motivate users to explore what they can do with the package. For inspiration, here are a few docs that were recently accepted into JOSS and are well-done:

For the JOSS submission (openjournals/joss-reviews#990), I think clear pointers to a single example from the README or the docs would suffice.

about [python] Test

First and foremost, great development work.
I was just wondering if failing 2 tests from test_pyoints.py still needed me to take further action to make the pyoints module work well. I am running Ubuntu 20.04 LTS and Python 3.8.3 (default, Jul 2 2020, 16:21:59)

...............Run doctests for Pyoints 0.2.0
...................../home/andrew/pyoints/pyoints/georecords.py:216: UserWarning: 'proj' not set, so I assume '+proj=latlong +datum=WGS84 +to +proj=latlong +datum=WGS84 +units=m +no_defs'
  warnings.warn("'proj' not set, so I assume '%s'" % proj.proj4)
.........................................../home/andrew/anaconda3/lib/python3.8/site-packages/numpy/core/numeric.py:207: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  a = empty(shape, dtype, order)
...FF............................................................
======================================================================
FAIL: GeoTransform (pyoints.projection)
Doctest: pyoints.projection.GeoTransform
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andrew/anaconda3/lib/python3.8/doctest.py", line 2204, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyoints.projection.GeoTransform
  File "/home/andrew/pyoints/pyoints/projection.py", line 161, in GeoTransform

----------------------------------------------------------------------
File "/home/andrew/pyoints/pyoints/projection.py", line 187, in pyoints.projection.GeoTransform
Failed example:
    print_rounded(tCoords, 3)
Expected:
    [[ 2560446.801  5537522.386]
     [ 2560808.009  5537303.984]
     [ 2564724.211  5539797.116]
     [ 2566007.32   5552049.646]]
Got:
    [[ 2560396.394  5537394.098]
     [ 2560757.556  5537175.718]
     [ 2564673.297  5539668.53 ]
     [ 2565956.354  5551919.643]]


======================================================================
FAIL: Proj (pyoints.projection)
Doctest: pyoints.projection.Proj
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/andrew/anaconda3/lib/python3.8/doctest.py", line 2204, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyoints.projection.Proj
  File "/home/andrew/pyoints/pyoints/projection.py", line 38, in Proj

----------------------------------------------------------------------
File "/home/andrew/pyoints/pyoints/projection.py", line 63, in pyoints.projection.Proj
Failed example:
    print('AUTHORITY["EPSG","4326"]' in proj.wkt)
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/home/andrew/pyoints/pyoints/projection.py", line 73, in pyoints.projection.Proj
Failed example:
    print('AUTHORITY["EPSG","4326"]' in proj.wkt)
Expected:
    True
Got:
    False


----------------------------------------------------------------------
Ran 144 tests in 3.743s

FAILED (failures=2)

Once again, great work on this module. I hope to use it on my research and will gladly cite you if so.
Regards,
Andrew

pygdal version

Pip installation fails because pygdal doesn't recognise the installed GDAL Version.

As an workaround pip install pygdal==$(gdal-config --version).* is used.

[JOSS-REVIEW] Community guidelines missing

In relation to openjournals/joss-reviews#990

The community guidelines currently consists of these few words under the section "Contact":

In case of questions, suggestions for improvement or software issues, please use 	
the issue board or contact Sebastian Lamprecht, [email protected]. 	
Contribution to the software is always welcome.

While this touches the bare minimum I would appreciate a separate section called "Community" or "Contributing", describing in a bit more words how to contribute to the project either with code or bug reports. GitHub has a nice feature for this in contributing.md files.

JOSS review - conda installation error

Related to #2 and openjournals/joss-reviews#990, I was also unsuccessful in trying to install from conda (on my local machine: osx and on binder - e.g. try jupyter, which runs linux 64bit). Here is the error log I got:

Solving environment: failed

CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/leampy/noarch/repodata.json>
Elapsed: 00:00.148486
CF-RAY: 4968e3cded3fc56a-ORD

The remote server could not find the noarch directory for the
requested channel with url: https://conda.anaconda.org/leampy

As of conda 4.3, a valid channel must contain a `noarch/repodata.json` and
associated `noarch/repodata.json.bz2` file, even if `noarch/repodata.json` is
empty. please request that the channel administrator create
`noarch/repodata.json` and associated `noarch/repodata.json.bz2` files.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state.
Further configuration help can be found at <https://conda.io/docs/config.html>.

For the JOSS submission, you do not necessarily have to have this resolved, but I would suggest that in the README, you

  • describe the machine specs for which the conda install should work
  • provide alternate installation instructions for installing from source (as @kbevers described in #2):
    • git clone https://github.com/laempy/pyoints.git
    • pip install gdal # is there a reason this is not listed in your requirements.txt?
    • conda install gdal, pyproj, rtree
    • pip install -e .

[JOSS-REVIEW] Installation instructions

In relation to openjournals/joss-reviews#990

The installation instructions are only written for Ubuntu, although in the text it is somewhat hinted that Windows is also supported.

A clear description of which systems this package is intended for would be nice. Since this is a Python package it (theoretically) should work on at least Linux, OS X and Windows.

I recommend adding a conda environment file and basing the installation instructions on that instead. This will improve the chances of a successful installation immensely.

As a side note, I happen to work on a Mac and am having issues getting started with the installation. This package is not at fault, a mix of homebrew several GDAL and Python installations are complicating things a bit here. This could be avoided with a conda setup.

Tolerances too narrow in some tests

While running the test-suite I found that three tests failed. The test results are correct but the test fail since the output in the unittest is not reproduced 100% as seen in the output below. We are talking about minor numerical instabilities here. Usually this sort of problem is dealt with by specifying a tolerance for which the requested value is to lie within. Here it can be done by only printing a set amount of decimals for the test output which should truncate the floating point errors seen below.

$ python tests/test_pyoints.py
Run doctests for Pyoints
..................................../Users/kevers/dev/pyoints/pyoints/georecords.py:200: UserWarning: 'proj' not set, so I assume '+proj=latlong +datum=WGS84 +to +proj=latlong +datum=WGS84 +units=m +no_defs'
  warnings.warn("'proj' not set, so I assume '%s'" % proj.proj4)
......F................F................................................F.............................Warning 1: Setting nodata to nan on band 2, but band 1 has nodata at nan. The TIFFTAG_GDAL_NODATA only support one value per dataset. This value of nan will be used for all bands on re-opening
Warning 1: Setting nodata to nan on band 3, but band 1 has nodata at nan. The TIFFTAG_GDAL_NODATA only support one value per dataset. This value of nan will be used for all bands on re-opening
..Warning 1: Setting nodata to nan on band 2, but band 1 has nodata at nan. The TIFFTAG_GDAL_NODATA only support one value per dataset. This value of nan will be used for all bands on re-opening
Warning 1: Setting nodata to nan on band 3, but band 1 has nodata at nan. The TIFFTAG_GDAL_NODATA only support one value per dataset. This value of nan will be used for all bands on re-opening
..
======================================================================
FAIL: from_corners (pyoints.grid.grid.Grid)
Doctest: pyoints.grid.grid.Grid.from_corners
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/pyoints/lib/python3.7/doctest.py", line 2198, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyoints.grid.grid.Grid.from_corners
  File "/Users/kevers/dev/pyoints/pyoints/grid/grid.py", line 188, in from_corners

----------------------------------------------------------------------
File "/Users/kevers/dev/pyoints/pyoints/grid/grid.py", line 210, in pyoints.grid.grid.Grid.from_corners
Failed example:
    print(raster.shape)
Expected:
    (2, 4)
Got:
    (2, 3)


======================================================================
FAIL: PolynomInterpolator (pyoints.interpolate)
Doctest: pyoints.interpolate.PolynomInterpolator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/pyoints/lib/python3.7/doctest.py", line 2198, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyoints.interpolate.PolynomInterpolator
  File "/Users/kevers/dev/pyoints/pyoints/interpolate.py", line 296, in PolynomInterpolator

----------------------------------------------------------------------
File "/Users/kevers/dev/pyoints/pyoints/interpolate.py", line 327, in pyoints.interpolate.PolynomInterpolator
Failed example:
    print(interpolator([0, 1]))
Expected:
    1.5
Got:
    1.4999999999999998


======================================================================
FAIL: Plane (pyoints.vector)
Doctest: pyoints.vector.Plane
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/pyoints/lib/python3.7/doctest.py", line 2198, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for pyoints.vector.Plane
  File "/Users/kevers/dev/pyoints/pyoints/vector.py", line 909, in Plane

----------------------------------------------------------------------
File "/Users/kevers/dev/pyoints/pyoints/vector.py", line 1033, in pyoints.vector.Plane
Failed example:
    print(plane.t.to_global(local_coords))
Expected:
    [[ 1.  2.]
     [ 4.  6.]
     [ 0.  1.]
     [ 7. 10.]]
Got:
    [[ 1.00000000e+00  2.00000000e+00]
     [ 4.00000000e+00  6.00000000e+00]
     [-2.77555756e-17  1.00000000e+00]
     [ 7.00000000e+00  1.00000000e+01]]


----------------------------------------------------------------------
Ran 142 tests in 2.893s

FAILED (failures=3)

add GeoTIFF LAS support

The GeoTIFF spatial reference definition of LAS is currently handled with the optional dependency liblas. So, a way to convert GeoTIFF tags to WKT and vice versa would be nice.

JOSS review - paper citations

Just a couple minor items with respect to the paper:

  • for the Bradski citation, it there a url that could be included to point to the source

  • is the consortium, P citation correct? I imagine it is intended to be cited as the "Pantheon Consortium" or similar?

  • in the GitHub citation - I would recommend either

    • (preferred): set up zenodo with this repository so that the code is archived with a doi and cite that (you will need to create a doi to finalize the JOSS submission anyways)
    • or if you would like to use the github url, then I would suggest that rather than citing the main github url that you cite the link to your repository.
  • For the Rusu & Cousins citation: is there a doi for the paper that could be included?

related to openjournals/joss-reviews#990

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.