jay-zeng / geospark Goto Github PK

GeoSpark is listed as Infrastructure Project on Apache Spark Official Third Party Project Page (http://spark.apache.org/third-party-projects.html)

GeoSpark is a cluster computing system for processing large-scale spatial data. GeoSpark extends Apache Spark with a set of out-of-the-box Spatial Resilient Distributed Datasets (SRDDs) that efficiently load, process, and analyze large-scale spatial data across machines. GeoSpark provides APIs for Apache Spark programmer to easily develop their spatial analysis programs with Spatial Resilient Distributed Datasets (SRDDs) which have in house support for geometrical and Spatial Queries (Range, K Nearest Neighbors, Join).

GeoSpark artifacts are hosted in Maven Central. You can add a Maven dependency with the following coordinates:

The following version supports Apache Spark 2.X versions:

groupId: org.datasyslab
artifactId: geospark
version: 0.5.0

The following version supports Apache Spark 1.X versions:

groupId: org.datasyslab
artifactId: geospark
version: 0.5.0-spark-1.x

##News

• GeoSpark Gitter Chat is now online! Chat with our GeoSpark users and ask questions!
• Babylon Visualization Framework on GeoSpark is now available! Babylon is a large-scale in-memory geospatial visualization system. More details are available here: Babylon GeoSpatial Visualization

1. Apache Spark 2.X releases (Apache Spark 1.X releases support available in GeoSpark for Spark 1.X branch)
2. JDK 1.7 or later
3. You might need to modify the dependencies in "POM.xml" and make it consistent with your environment.

Note: GeoSpark Master branch supports Apache Spark 2.X releases and GeoSpark for Spark 1.X branch supports Apache Spark 1.X releases. Please refer to the proper branch you need.

1. Have your Spark cluster ready.
2. Download pre-compiled GeoSpark jar under "Release" tag.
3. Run Spark shell with GeoSpark as a dependency.

./bin/spark-shell --jars GeoSpark_COMPILED.jar

3. You can now call GeoSpark APIs directly in your Spark shell!

1. Create your own Apache Spark project in Scala or Java
2. Add GeoSpark Maven coordinates into your project dependencies.
3. You can now use GeoSpark APIs in your Spark program!
4. Use spark-submit to submit your compiled self-contained Spark program.

GeoSpark Scala Example for GeoSpark 0.4 or later

GeoSpark Scala Example for GeoSpark 0.3.x

Test Data

GeoSpark Java Example

Test Data

Note: Scala can call Java APIs seamlessly. That means GeoSpark Scala users use the same APIs with GeoSpark Java users.

Please refer to GeoSpark Scala and Java API Usage

GeoSpark extends RDDs to form Spatial RDDs (SRDDs) and efficiently partitions SRDD data elements across machines and introduces novel parallelized spatial (geometric operations that follows the Open Geosptial Consortium (OGC) standard) transformations and actions (for SRDD) that provide a more intuitive interface for users to write spatial data analytics programs. Moreover, GeoSpark extends the SRDD layer to execute spatial queries (e.g., Range query, KNN query, and Join query) on large-scale spatial datasets. After geometrical objects are retrieved in the Spatial RDD layer, users can invoke spatial query processing operations provided in the Spatial Query Processing Layer of GeoSpark which runs over the in-memory cluster, decides how spatial object-relational tuples could be stored, indexed, and accessed using SRDDs, and returns the spatial query results required by user.

Supported Spatial RDDs: PointRDD, RectangleRDD, PolygonRDD, LineStringRDD

Native input format support

Comma-Separated Values (FileDataSplitter.CSV), Tab-separated values (FileDataSplitter.TSV), Well-Known Text (FileDataSplitter.WKT), and GeoJSON (FileDataSplitter.GeoJSON) as the input formats. Users only need to specify input format as Splitter and the start and end offset (if necessary) of spatial fields in one row when call Constructors.

User-supplied input format mapper

Examples: user-supplied input format mapper

You can load your mapper like this:

mySpatialRDD = PointRDD(sparkContext, InputLocation, userSuppliedMapper);
mySpatialRDD = PointRDD(sparkContext, InputLocation, PartitionNum, userSuppliedMapper);

mySpatialRDD = LineStringRDD(sparkContext, InputLocation, userSuppliedMapper);
mySpatialRDD = LineStringRDD(sparkContext, InputLocation, PartitionNum, userSuppliedMapper);
...

GeoSpark supports R-Tree (GridType.RTREE) and Voronoi diagram (GridType.VORONOI) spatial partitioning methods. Spatial partitioning is to repartition RDD according to objects' spatial locations. Spatial join on spatial paritioned RDD will be very fast.

GeoSpark supports two Spatial Indexes, Quad-Tree (IndexType.QUADTREE) and R-Tree (IndexType.RTREE). Quad-Tree doesn't support Spatial K Nearest Neighbors query.

GeoSpark currently provides native support for Inside, Overlap, DatasetBoundary, Minimum Bounding Rectangle and Polygon Union in SRDDS following Open Geospatial Consortium (OGC) standard.

GeoSpark so far provides Spatial Range Query, Spatial Join Query, and Spatial K Nearest Neighbors Query.

#Babylon Visualization Framework on GeoSpark Babylon is a large-scale in-memory geospatial visualization system.

Babylon provides native support for general cartographic design by extending GeoSpark to process large-scale spatial data. It can visulize Spatial RDD and Spatial Queries and render super high resolution image in parallel.

Babylon and GeoSpark are integrated together. You just need to import GeoSpark and enjoy them! More details are available here: Babylon GeoSpatial Visualization

Jia Yu, Jinxuan Wu, Mohamed Sarwat. "A Demonstration of GeoSpark: A Cluster Computing Framework for Processing Big Spatial Data". (demo paper) In Proceeding of IEEE International Conference on Data Engineering ICDE 2016, Helsinki, FI, May 2016

Jia Yu, Jinxuan Wu, Mohamed Sarwat. "GeoSpark: A Cluster Computing Framework for Processing Large-Scale Spatial Data". (short paper) In Proceeding of the ACM International Conference on Advances in Geographic Information Systems ACM SIGSPATIAL GIS 2015, Seattle, WA, USA November 2015

GeoSpark makes use of JTS Plus (An extended JTS Topology Suite Version 1.14) for some geometrical computations.

Please refer to JTS Topology Suite website and JTS Plus for more details.

• Please join

• Email us!

• Jia Yu (Email: [email protected])

• Jinxuan Wu (Email: [email protected])

• Mohamed Sarwat (Email: [email protected])

Please visit GeoSpark project wesbite for latest news and releases.

GeoSpark is one of the projects under Data Systems Lab at Arizona State University. The mission of Data Systems Lab is designing and developing experimental data management systems (e.g., database systems).

We appreciate the help and suggestions from the following GeoSpark users (The list is growing..):

• @gaufung
• @lrojas94
• @mdespriee
• @sabman
• @samchorlton
• @Tsarazin
• @TBuc
• ...