esri / spatial-framework-for-hadoop Goto Github PK

The Spatial Framework for Hadoop allows developers and data scientists to use the Hadoop data processing system for spatial data analysis.

License: Apache License 2.0

Java 100.00%

data-management spatial-analysis

spatial-framework-for-hadoop's Introduction

spatial-framework-for-hadoop

The Spatial Framework for Hadoop allows developers and data scientists to use the Hadoop data processing system for spatial data analysis.

For tools, samples, and tutorials that use this framework, head over to GIS Tools for Hadoop.

What's New

Hive v4 compatibility (as of Y2023).
ST_Centroid now returns the geometry centroid rather than the center of its envelope (as of v2.1).

Features

JSON Utilities - Utilities for interacting with JSON exported from ArcGIS
- Javadoc
Hive Spatial - User-Defined Functions and SerDes for spatial analysis in Hive and SparkSQL
- UDF Documentation
- JSON SerDe

Getting Started

Pre-Built

Up-to-date releases may be available on Github but may not be on Maven Central.

Maven

Build as you would any other Mavenized repository. All dependencies are pulled automatically.

Ant

Ant build files are also available, but are considered legacy, and may likely be removed in a future release.

At the root level of this repository, you can build a single jar with everything in the framework using Apache Ant. Alternatively, you can build a jar at the root level of each framework component (i.e., hive/build.xml).

The build files use Maven Ant Tasks for dependency management. You will need the jar in a place Ant can find it (i.e., ~/.ant/lib/maven-ant-tasks-2.1.3.jar).

Dependencies

Esri Geometry API for Java - Java geometry library for spatial data processing.

Requirements

Geometry v2.2
Hive v1 and above or SparkSQL (compatibility details)
Hadoop v2 and above
Workflows calling MapReduce jobs require the location of the custom job to be run.
Custom MapReduce jobs that use the Esri Geometry API require that the developer has authored the job, (referencing the com.esri.geometry.* classes), and deployed the job Jar file to the Hadoop system, prior to the ArcGIS user submitting the workflow file.
System administrators should always deploy an appropriate version of dependencies into production, that include important and especially security patches, even if a JAR file has been built with a lower version.

Resources

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing.

Licensing

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

spatial-framework-for-hadoop's People

Contributors

Stargazers

Watchers

Forkers

jwills jinxingster weishandong jayant-s tlpinney 909652048 iopenstack jayantshekhar strategist922 johnrnavarro nangal ddkaiser wyingquan vsingh58 risdenk alexliu68 andrewcottam zhouning ebradshaw80013 parker20121 parkhyunji mrflip ccsgdzh charles-cai flaviofalcao wallsky zengqiang2006 dreamerseverus andradejeff spacecurve sidahmed mrecho rishabhwadhawan lanxy88 ejde shjgiser wjwang4300 aguha-github vivovip xdlwd086 andr7430 lexie123 broderickwang gischen mainak24 likehua mpetcu-dg copoo komamitsu hskimsky david62243 hogwartsrico justvarshney colin-sgrc willtemperley tamerico krishna2020 hellocomrade ricebeans jxiang du1970 ee08b397 waynedou lvvi allenlu2008 coleferrier chaosmail hsjung6 geyfadam gisfromscratch modulexcite coolbole xuyongmin gcoandchao 383747787 cyqllfh manfredpaul npopa catacavaco nubiofs knsg huiyao351 leiyazhen fzn0268 painbug solidjerryc mastertony vovoma leusonmario mladkov gzqlovemxq spatialball nbuchanan thtfkai coderfi robertus100 alexandertolstoy jangnum sudeepdestiny zewdie2010

spatial-framework-for-hadoop's Issues

Multipolygon not parsed correctly

When we have an ESRI JSON geometry built from multiple polygons, then, when parsed through the ESRI JSON SerDe, the boundaryshape binary comes out as NULL.

This is the contents of our JSON, which is a single geometry made by 3 polygons:

[
    {
      "geometry": {
        "spatialReference": {
          "wkid": 4326
        },
        "rings": [
          [
            [
              [
                10.038311,
                54.360541
              ],
              [
                10.042831,
                54.360229
              ],
              [
                10.045357,
                54.360038
              ],
              [
                10.048616,
                54.359562
              ],
              [
                10.051387,
                54.359276
              ],
              [
                10.054239,
                54.359085
              ],
              [
                10.056521,
                54.359131
              ],
              [
                10.058885,
                54.359273
              ],
              [
                10.060923,
                54.359699
              ],
              [
                10.063043,
                54.360316
              ],
              [
                10.065408,
                54.361454
              ],
              [
                10.066795,
                54.362213
              ],
              [
                10.068915,
                54.363542
              ],
              [
                10.068917,
                54.363827
              ],
              [
                10.067857,
                54.36378
              ],
              [
                10.069979,
                54.365488
              ],
              [
                10.06998,
                54.3662
              ],
              [
                10.069003,
                54.366676
              ],
              [
                10.070719,
                54.369477
              ],
              [
                10.070151,
                54.370474
              ],
              [
                10.070803,
                54.371234
              ],
              [
                10.072108,
                54.371518
              ],
              [
                10.076349,
                54.371848
              ],
              [
                10.078385,
                54.371372
              ],
              [
                10.080343,
                54.370312
              ],
              [
                10.081887,
                54.369422
              ],
              [
                10.083754,
                54.36926
              ],
              [
                10.086474,
                54.369502
              ],
              [
                10.090448,
                54.370351
              ],
              [
                10.094423,
                54.370958
              ],
              [
                10.097977,
                54.371077
              ],
              [
                10.101323,
                54.37083
              ],
              [
                10.105294,
                54.370218
              ],
              [
                10.107385,
                54.370094
              ],
              [
                10.110521,
                54.369604
              ],
              [
                10.113447,
                54.368992
              ],
              [
                10.116165,
                54.368624
              ],
              [
                10.119717,
                54.368012
              ],
              [
                10.119718,
                54.368012
              ],
              [
                10.122017,
                54.368131
              ],
              [
                10.123479,
                54.367521
              ],
              [
                10.126616,
                54.367031
              ],
              [
                10.129958,
                54.36654
              ],
              [
                10.134765,
                54.365926
              ],
              [
                10.136857,
                54.36568
              ],
              [
                10.139157,
                54.365555
              ],
              [
                10.144382,
                54.36494
              ],
              [
                10.144493,
                54.364503
              ],
              [
                10.151799,
                54.363343
              ],
              [
                10.151698,
                54.362825
              ],
              [
                10.147551,
                54.363578
              ],
              [
                10.145868,
                54.361911
              ],
              [
                10.149912,
                54.360352
              ],
              [
                10.150203,
                54.359202
              ],
              [
                10.145469,
                54.361163
              ],
              [
                10.143881,
                54.359094
              ],
              [
                10.141897,
                54.356506
              ],
              [
                10.141297,
                54.354377
              ],
              [
                10.140997,
                54.353226
              ],
              [
                10.14198,
                54.352016
              ],
              [
                10.144543,
                54.350633
              ],
              [
                10.146417,
                54.349685
              ],
              [
                10.151247,
                54.347459
              ],
              [
                10.154298,
                54.344865
              ],
              [
                10.15646,
                54.34164
              ],
              [
                10.156947,
                54.340258
              ],
              [
                10.157232,
                54.337149
              ],
              [
                10.156439,
                54.336401
              ],
              [
                10.155251,
                54.33554
              ],
              [
                10.153273,
                54.334333
              ],
              [
                10.151786,
                54.332552
              ],
              [
                10.1501,
                54.330712
              ],
              [
                10.147823,
                54.328585
              ],
              [
                10.147624,
                54.327779
              ],
              [
                10.147226,
                54.327204
              ],
              [
                10.146138,
                54.326342
              ],
              [
                10.14663,
                54.325708
              ],
              [
                10.14564,
                54.324903
              ],
              [
                10.144847,
                54.324215
              ],
              [
                10.144448,
                54.322949
              ],
              [
                10.143062,
                54.321454
              ],
              [
                10.141577,
                54.320247
              ],
              [
                10.139996,
                54.319385
              ],
              [
                10.135747,
                54.317663
              ],
              [
                10.133692,
                54.314402
              ],
              [
                10.131781,
                54.311394
              ],
              [
                10.13385,
                54.310816
              ],
              [
                10.136727,
                54.315475
              ],
              [
                10.142557,
                54.318001
              ],
              [
                10.143938,
                54.317884
              ],
              [
                10.147201,
                54.319779
              ],
              [
                10.148483,
                54.319951
              ],
              [
                10.14898,
                54.320526
              ],
              [
                10.150165,
                54.320754
              ],
              [
                10.151347,
                54.320293
              ],
              [
                10.154804,
                54.321036
              ],
              [
                10.156581,
                54.321495
              ],
              [
                10.155402,
                54.322475
              ],
              [
                10.15896,
                54.324025
              ],
              [
                10.160435,
                54.322756
              ],
              [
                10.164091,
                54.324018
              ],
              [
                10.163009,
                54.325055
              ],
              [
                10.17003,
                54.328614
              ],
              [
                10.169149,
                54.330226
              ],
              [
                10.166387,
                54.330576
              ],
              [
                10.164013,
                54.32914
              ],
              [
                10.161341,
                54.327359
              ],
              [
                10.160154,
                54.32667
              ],
              [
                10.158968,
                54.326268
              ],
              [
                10.15897,
                54.326786
              ],
              [
                10.162035,
                54.328337
              ],
              [
                10.166192,
                54.331267
              ],
              [
                10.167476,
                54.331207
              ],
              [
                10.170137,
                54.330686
              ],
              [
                10.170332,
                54.329649
              ],
              [
                10.171119,
                54.329073
              ],
              [
                10.173192,
                54.329242
              ],
              [
                10.174376,
                54.329183
              ],
              [
                10.175468,
                54.33062
              ],
              [
                10.173991,
                54.331141
              ],
              [
                10.173007,
                54.332005
              ],
              [
                10.172117,
                54.332064
              ],
              [
                10.170346,
                54.33316
              ],
              [
                10.170647,
                54.334426
              ],
              [
                10.172422,
                54.333848
              ],
              [
                10.173904,
                54.334249
              ],
              [
                10.174006,
                54.335169
              ],
              [
                10.175976,
                54.334016
              ],
              [
                10.177065,
                54.334648
              ],
              [
                10.172734,
                54.337703
              ],
              [
                10.174018,
                54.338047
              ],
              [
                10.175304,
                54.338909
              ],
              [
                10.175315,
                54.341269
              ],
              [
                10.175123,
                54.34242
              ],
              [
                10.175328,
                54.344203
              ],
              [
                10.17451,
                54.3459
              ],
              [
                10.176603,
                54.346384
              ],
              [
                10.180158,
                54.34711
              ],
              [
                10.181407,
                54.34589
              ],
              [
                10.182871,
                54.346253
              ],
              [
                10.184746,
                54.344789
              ],
              [
                10.186002,
                54.345152
              ],
              [
                10.186831,
                54.343568
              ],
              [
                10.184948,
                54.343205
              ],
              [
                10.185774,
                54.341255
              ],
              [
                10.187661,
                54.342348
              ],
              [
                10.188083,
                54.343322
              ],
              [
                10.191637,
                54.343439
              ],
              [
                10.193724,
                54.342948
              ],
              [
                10.194347,
                54.342094
              ],
              [
                10.196227,
                54.341726
              ],
              [
                10.199569,
                54.341477
              ],
              [
                10.201448,
                54.341108
              ],
              [
                10.20659,
                54.340917
              ],
              [
                10.20531,
                54.340422
              ],
              [
                10.205519,
                54.339676
              ],
              [
                10.204451,
                54.339057
              ],
              [
                10.206363,
                54.337936
              ],
              [
                10.206784,
                54.337065
              ],
              [
                10.205716,
                54.336446
              ],
              [
                10.205923,
                54.335204
              ],
              [
                10.206772,
                54.334457
              ],
              [
                10.206552,
                54.332967
              ],
              [
                10.204422,
                54.33297
              ],
              [
                10.205482,
                54.332099
              ],
              [
                10.205051,
                54.330857
              ],
              [
                10.203338,
                54.329493
              ],
              [
                10.201847,
                54.329247
              ],
              [
                10.201627,
                54.327881
              ],
              [
                10.199281,
                54.327264
              ],
              [
                10.199063,
                54.326396
              ],
              [
                10.200765,
                54.325772
              ],
              [
                10.201826,
                54.325149
              ],
              [
                10.202676,
                54.32465
              ],
              [
                10.204378,
                54.324026
              ],
              [
                10.205661,
                54.325018
              ],
              [
                10.209292,
                54.327124
              ],
              [
                10.212066,
                54.328112
              ],
              [
                10.214627,
                54.329226
              ],
              [
                10.217613,
                54.329593
              ],
              [
                10.218035,
                54.328847
              ],
              [
                10.217176,
                54.327483
              ],
              [
                10.217594,
                54.325992
              ],
              [
                10.218225,
                54.324499
              ],
              [
                10.215875,
                54.323013
              ],
              [
                10.214589,
                54.321648
              ],
              [
                10.216926,
                54.320402
              ],
              [
                10.215641,
                54.319038
              ],
              [
                10.213933,
                54.318296
              ],
              [
                10.212226,
                54.317554
              ],
              [
                10.210305,
                54.316936
              ],
              [
                10.209246,
                54.318118
              ],
              [
                10.208401,
                54.319236
              ],
              [
                10.205842,
                54.318495
              ],
              [
                10.202856,
                54.317756
              ],
              [
                10.200509,
                54.316766
              ],
              [
                10.19838,
                54.316894
              ],
              [
                10.197101,
                54.316896
              ],
              [
                10.196671,
                54.315902
              ],
              [
                10.195182,
                54.316153
              ],
              [
                10.194113,
                54.315161
              ],
              [
                10.19241,
                54.315661
              ],
              [
                10.190279,
                54.315043
              ],
              [
                10.190696,
                54.313054
              ],
              [
                10.190476,
                54.311688
              ],
              [
                10.189405,
                54.310323
              ],
              [
                10.188764,
                54.309455
              ],
              [
                10.189824,
                54.308584
              ],
              [
                10.189819,
                54.30759
              ],
              [
                10.191094,
                54.306967
              ],
              [
                10.192151,
                54.30535
              ],
              [
                10.192355,
                54.303362
              ],
              [
                10.191286,
                54.302494
              ],
              [
                10.19149,
                54.300506
              ],
              [
                10.189786,
                54.30026
              ],
              [
                10.189989,
                54.298398
              ],
              [
                10.189771,
                54.297031
              ],
              [
                10.191473,
                54.297028
              ],
              [
                10.191769,
                54.296761
              ],
              [
                10.192322,
                54.296282
              ],
              [
                10.195089,
                54.296153
              ],
              [
                10.197639,
                54.29528
              ],
              [
                10.19721,
                54.294659
              ],
              [
                10.19891,
                54.294035
              ],
              [
                10.198055,
                54.293291
              ],
              [
                10.201033,
                54.292913
              ],
              [
                10.202306,
                54.292166
              ],
              [
                10.201453,
                54.291671
              ],
              [
                10.202725,
                54.290551
              ],
              [
                10.205701,
                54.289925
              ],
              [
                10.205049,
                54.287068
              ],
              [
                10.204548,
                54.286955
              ],
              [
                10.202941,
                54.286255
              ],
              [
                10.20414,
                54.285433
              ],
              [
                10.204335,
                54.284496
              ],
              [
                10.206741,
                54.284024
              ],
              [
                10.209547,
                54.283668
              ],
              [
                10.208743,
                54.283084
              ],
              [
                10.209441,
                54.282381
              ],
              [
                10.210033,
                54.280271
              ],
              [
                10.209227,
                54.27957
              ],
              [
                10.207613,
                54.277816
              ],
              [
                10.205999,
                54.276062
              ],
              [
                10.203993,
                54.275597
              ],
              [
                10.202191,
                54.27642
              ],
              [
                10.203404,
                54.278409
              ],
              [
                10.202805,
                54.278878
              ],
              [
                10.201603,
                54.279114
              ],
              [
                10.198788,
                54.277948
              ],
              [
                10.199583,
                54.27619
              ],
              [
                10.19898,
                54.276074
              ],
              [
                10.199171,
                54.274082
              ],
              [
                10.197967,
                54.273617
              ],
              [
                10.197361,
                54.272798
              ],
              [
                10.197957,
                54.271509
              ],
              [
                10.198954,
                54.270804
              ],
              [
                10.200556,
                54.269982
              ],
              [
                10.202358,
                54.269394
              ],
              [
                10.201544,
                54.267053
              ],
              [
                10.199731,
                54.265416
              ],
              [
                10.196515,
                54.263548
              ],
              [
                10.195711,
                54.263198
              ],
              [
                10.194911,
                54.263551
              ],
              [
                10.192902,
                54.262617
              ],
              [
                10.192901,
                54.262266
              ],
              [
                10.190894,
                54.261917
              ],
              [
                10.188684,
                54.261218
              ],
              [
                10.190686,
                54.260395
              ],
              [
                10.186874,
                54.259581
              ],
              [
                10.187067,
                54.257941
              ],
              [
                10.186052,
                54.25525
              ],
              [
                10.184042,
                54.253848
              ],
              [
                10.183837,
                54.253028
              ],
              [
                10.182434,
                54.253265
              ],
              [
                10.180828,
                54.252564
              ],
              [
                10.179024,
                54.252684
              ],
              [
                10.175817,
                54.252806
              ],
              [
                10.173524,
                54.255385
              ],
              [
                10.172323,
                54.255621
              ],
              [
                10.173129,
                54.256791
              ],
              [
                10.172328,
                54.257026
              ],
              [
                10.170923,
                54.25656
              ],
              [
                10.168917,
                54.256445
              ],
              [
                10.167115,
                54.256799
              ],
              [
                10.16471,
                54.257037
              ],
              [
                10.163711,
                54.257741
              ],
              [
                10.162106,
                54.257274
              ],
              [
                10.161109,
                54.258681
              ],
              [
                10.159103,
                54.258918
              ],
              [
                10.157499,
                54.258569
              ],
              [
                10.156698,
                54.258804
              ],
              [
                10.154291,
                54.258221
              ],
              [
                10.153483,
                54.2567
              ],
              [
                10.153678,
                54.255177
              ],
              [
                10.153468,
                54.252953
              ],
              [
                10.153264,
                54.251548
              ],
              [
                10.151656,
                54.250613
              ],
              [
                10.148856,
                54.252022
              ],
              [
                10.147453,
                54.252024
              ],
              [
                10.145051,
                54.25308
              ],
              [
                10.144247,
                54.252847
              ],
              [
                10.141045,
                54.254138
              ],
              [
                10.142051,
                54.255074
              ],
              [
                10.137046,
                54.256954
              ],
              [
                10.135245,
                54.257893
              ],
              [
                10.131432,
                54.256843
              ],
              [
                10.129427,
                54.256493
              ],
              [
                10.126618,
                54.256145
              ],
              [
                10.124012,
                54.255796
              ],
              [
                10.123817,
                54.258022
              ],
              [
                10.12001,
                54.258728
              ],
              [
                10.117805,
                54.25873
              ],
              [
                10.116002,
                54.258966
              ],
              [
                10.115004,
                54.260373
              ],
              [
                10.111999,
                54.261078
              ],
              [
                10.108992,
                54.261197
              ],
              [
                10.106384,
                54.261434
              ],
              [
                10.103581,
                54.262022
              ],
              [
                10.099973,
                54.262727
              ],
              [
                10.096566,
                54.263198
              ],
              [
                10.094761,
                54.263083
              ],
              [
                10.092558,
                54.264256
              ],
              [
                10.089757,
                54.266952
              ],
              [
                10.088153,
                54.266953
              ],
              [
                10.083544,
                54.268244
              ],
              [
                10.08515,
                54.268945
              ],
              [
                10.086154,
                54.269413
              ],
              [
                10.085755,
                54.270701
              ],
              [
                10.085757,
                54.271521
              ],
              [
                10.08656,
                54.272691
              ],
              [
                10.087564,
                54.273393
              ],
              [
                10.091177,
                54.273976
              ],
              [
                10.092381,
                54.274795
              ],
              [
                10.092382,
                54.275849
              ],
              [
                10.092987,
                54.276669
              ],
              [
                10.092387,
                54.277723
              ],
              [
                10.091588,
                54.27866
              ],
              [
                10.091789,
                54.279831
              ],
              [
                10.092795,
                54.281235
              ],
              [
                10.092598,
                54.28229
              ],
              [
                10.093605,
                54.284397
              ],
              [
                10.094212,
                54.286856
              ],
              [
                10.091209,
                54.289082
              ],
              [
                10.089404,
                54.289552
              ],
              [
                10.085994,
                54.289906
              ],
              [
                10.083987,
                54.289673
              ],
              [
                10.08198,
                54.28862
              ],
              [
                10.07616,
                54.287804
              ],
              [
                10.075357,
                54.287219
              ],
              [
                10.076157,
                54.286633
              ],
              [
                10.069335,
                54.284882
              ],
              [
                10.067128,
                54.284883
              ],
              [
                10.064925,
                54.287225
              ],
              [
                10.060912,
                54.287344
              ],
              [
                10.058106,
                54.288166
              ],
              [
                10.058711,
                54.289922
              ],
              [
                10.05771,
                54.291914
              ],
              [
                10.058715,
                54.293318
              ],
              [
                10.05992,
                54.294254
              ],
              [
                10.059921,
                54.295073
              ],
              [
                10.05912,
                54.296245
              ],
              [
                10.058119,
                54.297651
              ],
              [
                10.054507,
                54.297418
              ],
              [
                10.051095,
                54.297186
              ],
              [
                10.049691,
                54.297538
              ],
              [
                10.048287,
                54.298358
              ],
              [
                10.049495,
                54.300934
              ],
              [
                10.04809,
                54.301989
              ],
              [
                10.048294,
                54.303861
              ],
              [
                10.046689,
                54.304682
              ],
              [
                10.045685,
                54.304682
              ],
              [
                10.044885,
                54.306439
              ],
              [
                10.045488,
                54.308196
              ],
              [
                10.046493,
                54.308664
              ],
              [
                10.047095,
                54.308546
              ],
              [
                10.048099,
                54.309483
              ],
              [
                10.051111,
                54.309833
              ],
              [
                10.052716,
                54.309364
              ],
              [
                10.05372,
                54.30948
              ],
              [
                10.054724,
                54.310066
              ],
              [
                10.042281,
                54.311709
              ],
              [
                10.043288,
                54.315691
              ],
              [
                10.043493,
                54.318618
              ],
              [
                10.043096,
                54.323185
              ],
              [
                10.0441,
                54.324824
              ],
              [
                10.04591,
                54.325877
              ],
              [
                10.046715,
                54.327283
              ],
              [
                10.049126,
                54.32857
              ],
              [
                10.05033,
                54.328921
              ],
              [
                10.052948,
                54.334306
              ],
              [
                10.055733,
                54.33375
              ],
              [
                10.05857,
                54.333132
              ],
              [
                10.058165,
                54.331375
              ],
              [
                10.059169,
                54.330321
              ],
              [
                10.060975,
                54.32915
              ],
              [
                10.062379,
                54.328681
              ],
              [
                10.064588,
                54.328562
              ],
              [
                10.066195,
                54.328913
              ],
              [
                10.066997,
                54.328327
              ],
              [
                10.069603,
                54.325281
              ],
              [
                10.072213,
                54.325162
              ],
              [
                10.074021,
                54.325395
              ],
              [
                10.075226,
                54.32598
              ],
              [
                10.075829,
                54.326917
              ],
              [
                10.08507,
                54.32855
              ],
              [
                10.087474,
                54.326089
              ],
              [
                10.094908,
                54.328425
              ],
              [
                10.095313,
                54.329128
              ],
              [
                10.097324,
                54.33018
              ],
              [
                10.09853,
                54.330882
              ],
              [
                10.100138,
                54.331465
              ],
              [
                10.101146,
                54.33287
              ],
              [
                10.102754,
                54.333688
              ],
              [
                10.104162,
                54.334273
              ],
              [
                10.104567,
                54.335677
              ],
              [
                10.106579,
                54.337198
              ],
              [
                10.10297,
                54.339661
              ],
              [
                10.103578,
                54.341767
              ],
              [
                10.103381,
                54.343407
              ],
              [
                10.101777,
                54.345048
              ],
              [
                10.10178,
                54.345985
              ],
              [
                10.100376,
                54.346923
              ],
              [
                10.101382,
                54.347625
              ],
              [
                10.097371,
                54.35032
              ],
              [
                10.09355,
                54.349035
              ],
              [
                10.096357,
                54.346223
              ],
              [
                10.09314,
                54.34564
              ],
              [
                10.091131,
                54.345408
              ],
              [
                10.088319,
                54.345995
              ],
              [
                10.084302,
                54.345647
              ],
              [
                10.078072,
                54.344363
              ],
              [
                10.076869,
                54.345417
              ],
              [
                10.074657,
                54.344716
              ],
              [
                10.074656,
                54.344131
              ],
              [
                10.072446,
                54.343546
              ],
              [
                10.070438,
                54.344016
              ],
              [
                10.067626,
                54.34472
              ],
              [
                10.06803,
                54.345657
              ],
              [
                10.069838,
                54.346007
              ],
              [
                10.070842,
                54.345889
              ],
              [
                10.07205,
                54.34706
              ],
              [
                10.069841,
                54.347763
              ],
              [
                10.068235,
                54.348584
              ],
              [
                10.063012,
                54.348469
              ],
              [
                10.061604,
                54.347884
              ],
              [
                10.057587,
                54.347652
              ],
              [
                10.052966,
                54.34742
              ],
              [
                10.052763,
                54.34625
              ],
              [
                10.050955,
                54.346368
              ],
              [
                10.04995,
                54.345666
              ],
              [
                10.048142,
                54.345901
              ],
              [
                10.044928,
                54.346136
              ],
              [
                10.042517,
                54.345786
              ],
              [
                10.041713,
                54.346137
              ],
              [
                10.036893,
                54.346958
              ],
              [
                10.035087,
                54.349183
              ],
              [
                10.033079,
                54.350004
              ],
              [
                10.033685,
                54.353517
              ],
              [
                10.034489,
                54.355389
              ],
              [
                10.034489,
                54.356795
              ],
              [
                10.034491,
                54.358083
              ],
              [
                10.035899,
                54.358668
              ],
              [
                10.035498,
                54.359839
              ],
              [
                10.037909,
                54.359604
              ],
              [
                10.038311,
                54.360541
              ]
            ]
          ],
          [
            [
              [
                10.115549,
                54.37064
              ],
              [
                10.117205,
                54.371
              ],
              [
                10.118859,
                54.371119
              ],
              [
                10.11969,
                54.371961
              ],
              [
                10.121135,
                54.371357
              ],
              [
                10.120927,
                54.370635
              ],
              [
                10.121959,
                54.370393
              ],
              [
                10.122791,
                54.371597
              ],
              [
                10.123828,
                54.37256
              ],
              [
                10.126106,
                54.37328
              ],
              [
                10.128589,
                54.37376
              ],
              [
                10.130658,
                54.373999
              ],
              [
                10.130456,
                54.375204
              ],
              [
                10.130458,
                54.375927
              ],
              [
                10.131496,
                54.377009
              ],
              [
                10.132527,
                54.376407
              ],
              [
                10.132739,
                54.377731
              ],
              [
                10.13357,
                54.378935
              ],
              [
                10.135024,
                54.380741
              ],
              [
                10.134197,
                54.381103
              ],
              [
                10.136273,
                54.38315
              ],
              [
                10.140654,
                54.384474
              ],
              [
                10.141654,
                54.386456
              ],
              [
                10.14195,
                54.390051
              ],
              [
                10.138337,
                54.391914
              ],
              [
                10.137278,
                54.393486
              ],
              [
                10.137488,
                54.394953
              ],
              [
                10.137492,
                54.396197
              ],
              [
                10.135383,
                54.400564
              ],
              [
                10.134494,
                54.405469
              ],
              [
                10.135577,
                54.408533
              ],
              [
                10.136509,
                54.4103
              ],
              [
                10.13711,
                54.411172
              ],
              [
                10.13794,
                54.412234
              ],
              [
                10.138711,
                54.413449
              ],
              [
                10.139477,
                54.413146
              ],
              [
                10.139474,
                54.412304
              ],
              [
                10.145887,
                54.41085
              ],
              [
                10.147237,
                54.411993
              ],
              [
                10.148692,
                54.413677
              ],
              [
                10.150149,
                54.415724
              ],
              [
                10.151192,
                54.418012
              ],
              [
                10.150575,
                54.418977
              ],
              [
                10.151202,
                54.420663
              ],
              [
                10.152445,
                54.421023
              ],
              [
                10.151827,
                54.421626
              ],
              [
                10.15266,
                54.42295
              ],
              [
                10.154116,
                54.424876
              ],
              [
                10.155981,
                54.425356
              ],
              [
                10.157852,
                54.426799
              ],
              [
                10.157652,
                54.428607
              ],
              [
                10.16056,
                54.430893
              ],
              [
                10.163671,
                54.431851
              ],
              [
                10.168015,
                54.430642
              ],
              [
                10.167609,
                54.43281
              ],
              [
                10.16906,
                54.432688
              ],
              [
                10.169469,
                54.431482
              ],
              [
                10.169879,
                54.430398
              ],
              [
                10.170082,
                54.429434
              ],
              [
                10.171526,
                54.428106
              ],
              [
                10.171936,
                54.427022
              ],
              [
                10.172966,
                54.425815
              ],
              [
                10.173998,
                54.42485
              ],
              [
                10.176062,
                54.42316
              ],
              [
                10.177507,
                54.422195
              ],
              [
                10.178329,
                54.420747
              ],
              [
                10.179771,
                54.418938
              ],
              [
                10.181834,
                54.417368
              ],
              [
                10.183279,
                54.416161
              ],
              [
                10.184929,
                54.414713
              ],
              [
                10.186787,
                54.413506
              ],
              [
                10.188228,
                54.411937
              ],
              [
                10.189254,
                54.409646
              ],
              [
                10.190281,
                54.407837
              ],
              [
                10.191515,
                54.406028
              ],
              [
                10.191304,
                54.405186
              ],
              [
                10.192126,
                54.403859
              ],
              [
                10.192325,
                54.402051
              ],
              [
                10.191701,
                54.401329
              ],
              [
                10.190656,
                54.399282
              ],
              [
                10.190652,
                54.398318
              ],
              [
                10.189401,
                54.396393
              ],
              [
                10.189396,
                54.395188
              ],
              [
                10.18918,
                54.39326
              ],
              [
                10.189173,
                54.391935
              ],
              [
                10.190204,
                54.390969
              ],
              [
                10.191648,
                54.389883
              ],
              [
                10.190405,
                54.389644
              ],
              [
                10.188748,
                54.389646
              ],
              [
                10.186061,
                54.390132
              ],
              [
                10.184407,
                54.390376
              ],
              [
                10.183577,
                54.389895
              ],
              [
                10.180268,
                54.39002
              ],
              [
                10.176128,
                54.389906
              ],
              [
                10.174678,
                54.389426
              ],
              [
                10.174468,
                54.388703
              ],
              [
                10.171775,
                54.387984
              ],
              [
                10.170948,
                54.388226
              ],
              [
                10.168668,
                54.387265
              ],
              [
                10.164939,
                54.386428
              ],
              [
                10.161419,
                54.385589
              ],
              [
                10.160588,
                54.384506
              ],
              [
                10.16141,
                54.383179
              ],
              [
                10.162025,
                54.381371
              ],
              [
                10.163468,
                54.380284
              ],
              [
                10.164917,
                54.380523
              ],
              [
                10.166976,
                54.377991
              ],
              [
                10.164905,
                54.377512
              ],
              [
                10.163041,
                54.376791
              ],
              [
                10.161177,
                54.376312
              ],
              [
                10.159727,
                54.375591
              ],
              [
                10.158894,
                54.374146
              ],
              [
                10.157854,
                54.373063
              ],
              [
                10.157024,
                54.371859
              ],
              [
                10.155571,
                54.370535
              ],
              [
                10.154532,
                54.369211
              ],
              [
                10.152461,
                54.368732
              ],
              [
                10.149568,
                54.369096
              ],
              [
                10.147292,
                54.36922
              ],
              [
                10.14419,
                54.369344
              ],
              [
                10.14109,
                54.369709
              ],
              [
                10.138197,
                54.370435
              ],
              [
                10.13654,
                54.369714
              ],
              [
                10.134675,
                54.368632
              ],
              [
                10.131572,
                54.368876
              ],
              [
                10.128057,
                54.369
              ],
              [
                10.122887,
                54.369126
              ],
              [
                10.121026,
                54.369369
              ],
              [
                10.11834,
                54.369974
              ],
              [
                10.115549,
                54.37064
              ]
            ]
          ],
          [
            [
              [
                10.138393,
                54.367423
              ],
              [
                10.143359,
                54.3679
              ],
              [
                10.145013,
                54.367777
              ],
              [
                10.14832,
                54.367412
              ],
              [
                10.148731,
                54.366568
              ],
              [
                10.147902,
                54.366087
              ],
              [
                10.145421,
                54.36609
              ],
              [
                10.143559,
                54.366092
              ],
              [
                10.139838,
                54.366578
              ],
              [
                10.138393,
                54.367423
              ]
            ]
          ]
        ]
      },
      "attributes": {
        "Id": "01002000",
        "Name": "Kiel",
        "area": 109.773753263,
        "ID_long": 1002000,
        "ID_text": "01002000"
      }
    }
  ]

ST_AsText Outputs Null on a Polygon Containing a Pole

On polygons containing the North or South Pole, ST_AsText outputs NULL, instead of WKT. (The result is the same whether the constructor used is ST_Polygon or ST_GeomFromText.)

Contrast: ST_AsText on PostgreSQL outputs WKT; ST_AsJSON on Hive outputs REST JSON.

hive> select ST_AsText(ST_SetSRID(ST_Polygon('polygon((0 -88, 45 -88, 90 -88, 135 -88, 180 -88, -135 -88, -90 -88, -45 -88, 0 -88))'), 4326)) from onerow;
NULL
hive> select ST_AsText(ST_SetSRID(ST_Polygon('polygon((0 88, 45 88, 90 88, 135 88, 180 88, -135 88, -90 88, -45 88, 0 88))'), 4326)) from onerow;
NULL
hive> select ST_AsJson(ST_SetSRID(ST_Polygon('polygon((0 88, 45 88, 90 88, 135 88, 180 88, -135 88, -90 88, -45 88, 0 88))'), 4326)) from onerow;
{"rings":[[[0.0,88.0],[-45.0,88.0],[-90.0,88.0],[-135.0,88.0],[180.0,88.0],[135.0,88.0],[90.0,88.0],[45.0,88.0],[0.0,88.0]]],"spatialReference":{"wkid":4326}}
hive> select ST_AsJson(ST_SetSRID(ST_Polygon('polygon((0 -88, 45 -88, 90 -88, 135 -88, 180 -88, -135 -88, -90 -88, -45 -88, 0 -88))'), 4326)) from onerow;
{"rings":[[[0.0,-88.0],[-45.0,-88.0],[-90.0,-88.0],[-135.0,-88.0],[180.0,-88.0],[135.0,-88.0],[90.0,-88.0],[45.0,-88.0],[0.0,-88.0]]],"spatialReference":{"wkid":4326}}

move gp tools to their own repo

Update Wiki Pages

Add the following pages to wiki

Hadoop Tools
Geoprocessing Tools
JSON Formats
Sample Workflows

Add JSON storage handler

Simplify JSON table create statements so that...

CREATE TABLE tablename  (...)                              
ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde'              
STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

can be simply written as...

CREATE TABLE tablename   (...)                         
STORED AS 'com.esri.hadoop.hive.JsonStorageHandler'

Retest ST_Intersection

Retest ST_Intersection once we have a new build of geometry-api, with the following test cases.

select ST_Relate(ST_Polygon('polygon((1 0, 3 0, 1 2, 1 0))'), ST_Polygon('polygon((0 1, 2 1, 0 3, 0 1))'), '212111212') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((1 0, 3 0, 1 2, 1 0))'), ST_Polygon('polygon((0 1, 2 1, 0 3, 0 1))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((3 0, 3 3, 0 3, 3 0))'), ST_Polygon('polygon((2 2, 5 2, 2 5, 2 2))'), '212101212') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((3 0, 3 3, 0 3, 3 0))'), ST_Polygon('polygon((2 2, 5 2, 2 5, 2 2))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((2 0, 2 2, 0 2, 2 0))'), ST_Polygon('polygon((1 1, 3 1, 1 3, 1 1))'), '212101212') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((2 0, 2 2, 0 2, 2 0))'), ST_Polygon('polygon((1 1, 3 1, 1 3, 1 1))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((0 0, 2 0, 0 2, 0 0))'), ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'), '212F11FF2') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((0 0, 2 0, 0 2, 0 0))'), ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((0 0, 3 0, 0 3, 0 0))'), ST_Polygon('polygon((1 1, 2 1, 1 2, 1 1))'), '212F11FF2') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((0 0, 3 0, 0 3, 0 0))'), ST_Polygon('polygon((1 1, 2 1, 1 2, 1 1))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((0 0, 2 0, 0 2, 0 0))'), ST_Polygon('polygon((1 1, 1 0, 0 1, 1 1))'), '212F01FF2') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((0 0, 2 0, 0 2, 0 0))'), ST_Polygon('polygon((1 1, 1 0, 0 1, 1 1))'))) from onerow;
select ST_Relate(ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'), ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'), '2FFF1FFF2') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'), ST_Polygon('polygon((0 0, 1 0, 0 1, 0 0))'))) from onerow;

select ST_Relate(ST_Polygon('polygon((0 0, 3 0, 0 3, 0 0))'), ST_Polygon('polygon((2 2, 2 0, 3 0, 3 3, 0 3, 0 2, 2 2))'), '212111212') from onerow;

select ST_AsText(ST_Intersection(ST_LineString(0,2, 2,3), ST_Polygon(1,1, 1,4, 4,4, 4,1))) from onerow;
select ST_Relate(ST_LineString(0,2, 2,3), ST_Polygon(1,1, 1,4, 4,4, 4,1), '10T0F1T02') from onerow;
select ST_AsText(ST_Intersection(ST_Polygon(1,1, 1,4, 4,4, 4,1), ST_LineString(1,1, 1,4))) from onerow;
select ST_Relate(ST_Polygon(1,1, 1,4, 4,4, 4,1), ST_LineString(1,1, 1,4), 'FF2101FF2') from onerow;
select ST_Relate(ST_Polygon(1,1, 1,4, 4,4, 4,1), ST_LineString(1,3, 1,4, 0,4), 'FF2101102') from onerow;

select ST_AsText(ST_Intersection(ST_LineString(0,2, 0,4), ST_LineString(0,2, 0,4))) from onerow;
select ST_AsText(ST_Intersection(ST_LineString(0,2, 0,4), ST_LineString(0,3, 0,5))) from onerow;
select ST_AsText(ST_Intersection(ST_LineString(0,2, 0,5), ST_LineString(1,3, 0,3, 0,4, 1,4))) from onerow;

select ST_AsText(ST_Intersection(ST_LineString(0,2, 0,4), ST_Point(0,3))) from onerow;

select ST_AsText(ST_Intersection(ST_Point(1,1), ST_Point(1,1))) from onerow;

UTM coordinates format to Lat / Lang

Hello,

Actually I'm using this function to get distance between two points:

ST_GeodesicLengthWGS84(ST_SetSRID(ST_LineString(centre[0], centre[1], table.coordinate[0], table.coordinate[1]), 4326))

coordinates[0] and coordinates[1] fields are in Latitude / Longitude format and this works.

Now i have a new table with coordinates in UTM format and i don't know if it is possible to parse this format to Lat / Lang. I haven't found any mention in documentation to this format.

Any response would be appreciated, thanks!

String.equals versus Object== in UDFs for Hive

Several places in the UDFs, when comparing geometry-type strings, identical-object operator == is used, where String.equals should be used.

GeometryUtils and about a dozen other files.

ST_GeodesicLengthWGS84 require WKID?

Hi,

I am looking to use the Hive Geospatial UDFs to filter queries based on the spherical distance between LON/LAT points. ST_GeodesicLengthWGS84 seems perfect for this. However, I tried the following example (from https://github.com/Esri/spatial-framework-for-hadoop/wiki/UDF-Accessors#st_geodesiclengthwgs84):

SELECT ST_GeodesicLengthWGS84(ST_Linestring(0.0,0.0, 0.3,0.4)) FROM src LIMIT 1;

and this returns "null" rather than the expected value. Looking into the hive log and the UDF code, this is apparently because the geometry passed to GeodesicLengthWGS84 doesn't have a WKID set to 4326. Running instead:

SELECT ST_GeodesicLengthWGS84(ST_GeomFromText('LineString(0 0, 0.3 0.4)', 4326)) FROM src LIMIT 1;

works as expected, returning 55421.4085[..]. Shouldn't the first (much simpler) example be possible?

Issue with ST_Area (again) and ST_Multipolygon

Hello!
When I try to call ST_AREA on one of my polygon I didn't get the good result...

SELECT
ST_AREA(ST_POLYGON('
 MULTIPOLYGON (((1001200 2432900, 1001420 2432691, 1001250 2432388, 001498 2432325, 1001100 2432100, 1001500 2431900, 1002044 2431764, 1002059 2432120, 1002182 2432003, 1002400 2432300, 1002650 2432150, 1002610 2432323, 1002772 2432434, 1002410 2432821, 1002700 2433000, 1001824 2432866, 1001600 2433150, 1001200 2432900)), ((1000393 2433983, 1000914 2434018, 1000933 2433817, 1000568 2433834, 1000580 2433584, 1000700 2433750, 1000800 2433650, 1000700 2433450, 1000600 2433550, 1000200 2433350, 1000100 2433900, 1000393 2433983)), ((1001200 2432900, 1000878 2432891, 1000900 2433300, 1001659 2433509, 1001600 2433150, 1001200 2432900)), ((1002450 2431650, 1002300 2431650, 1002300 2431900, 1002500 2432100, 1002600 2431800, 1002450 2431800, 1002450 2431650)), ((999750 2433550, 999850 2433600, 999900 2433350, 999780 2433433, 999750 2433550)), ((1002950 2432050, 1003005 2431932, 1002850 2432250, 1002928 2432210, 1002950 2432050)), ((1002600 2431750, 1002642 2431882, 1002750 2431900, 1002750 2431750, 1002600 2431750)), ((1002950 2431750, 1003050 2431650, 1002968 2431609, 1002950 2431750)))'))
FROM dual;

Return 0,5km2 but I should have 2km2 (calculate with QGIS)...

Somebody have an hint?

Kelevra

Hive 13 compatibility

JsonSerde fails to initialize because TypeInfoFactory.binaryTypeInfo changed from TypeInfo to PrimitiveTypeInfo.

Develop one or two more sample workflows

ST_AsText Reported to Return Null on a Multi-Part Geometry

Comment on issue# 35 (point 2) refers to ST_AsText returning NULL.

The filer of issue #35 has provided info via email, that indicates that in certain queries for which the intermediate result is a multi-polygon geometry, the geometry appears be treated as a single-part polygon instead of MultiPolygon, with the following effects:

ST_GeometryType returns ST_POLYGON instead of ST_MULTIPOLYGON
ST_AsText returns NULL (due to exception thrown in Geometry-API)

So far, the developers of Spatial Framework for Hadoop have been unable to reproduce those reported results.

ST_Aggr_Union null returns, spatial-reference, & trace log

Existing ST_Aggr_Union:

logs a trace at error level;
ignores the spatial-reference of the geometries;
returns null in some tests on some valid input, including the following:

  hive> select ST_AsText(ST_Aggr_Union(ST_LineString(geom))) from com_esri_hadoop_hive_aggr_line;
  NULL

  hive> select geom from com_esri_hadoop_hive_aggr_line;                                         
  linestring (0 0, 0 3)
  linestring (0 1, 0 4)
  linestring (0 2, 0 5)

These may be deferred until a version of geometry-api-java is available that enhances support for aggregate operations on geometries from a stream from file.

ST_GeomFromGeoJSON - changes with geometry-api

Several regressions in ST_GeomFromGeoJSON with today's build of geometry-api - some but not all inputs.

--- tests-text.ref  2013-03-14 13:01:18.519121479 -0700
+++ tests-text.out  2013-03-14 12:52:03.009122073 -0700
@@ -71,19 +72,19 @@
 hive> select ST_AsText(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4]}}')) from onerow;
-POINT (1.2 2.4)
+NULL
 hive> select ST_AsJson(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4]}}')) from onerow;
-{"x":1.2,"y":2.4,"spatialReference":{"wkid":4326}}
+NULL
 hive> select ST_AsText(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4, 3.3]}}')) from onerow;
-POINT Z (1.2 2.4 3.3)
+NULL
 hive> select ST_Z(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4, 3.3]}}')) from onerow;
-3.3
+NULL
 hive> select ST_AsText(ST_GeomFromGeoJson('{"geometry":{"type":"LineString", "coordinates":[[1,2], [3,4]]}}')) from onerow;
-LINESTRING (1 2, 3 4)
+NULL
 hive> select ST_MinY(ST_GeomFromGeoJson('{"geometry":{"type":"LineString", "coordinates":[[1,2], [3,4]]}}')) from onerow;
-2.0
+NULL
 hive> select ST_AsText(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4]}}, "crs":{"type":"name", "properties":{"name":"urn:ogc:def:crs:OGC:1.3:CRS84"}}')) from onerow;
-POINT (1.2 2.4)
+NULL
 hive> select ST_AsText(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4]}}, "crs":{null}')) from onerow;
-POINT (1.2 2.4)
+NULL
 hive> select ST_AsJson(ST_GeomFromGeoJson('{"geometry":{"type":"Point", "coordinates":[1.2, 2.4]}}, "crs":{null}')) from onerow;
-{"x":1.2,"y":2.4,"spatialReference":{"wkid":4326}}
+NULL
 hive> select ST_AsJson(ST_GeomFromJson('{"x":1.2,"y":2.4,"spatialReference":{"wkid":4326}}')) from onerow;
{"x":1.2,"y":2.4,"spatialReference":{"wkid":4326}}
hive> select ST_AsJson(ST_GeomFromJson('{"x":1.2,"y":2.4,"spatialReference":{"wkid": 0}}')) from onerow;
{"x":1.2,"y":2.4}

JsonSerde.java serialization and deserialization skips the spatial reference information

Add inline help to GP tools

ST_GeometryN - null on MultiPoint of one point

With a prior build of the geometry-api:

$ /usr/local/hive/bin/hive -S -i ~/.hgeocf
hive> select ST_GeometryType(ST_GeomFromText('multipoint ((1 1))')) from onerow;
ST_MULTIPOINT
hive> select ST_AsText(ST_GeometryN(ST_GeomFromText('multipoint ((1 1))'), 1)) from onerow;
POINT (1 1)
hive> select ST_GeometryType(ST_GeometryN(ST_GeomFromText('multipoint ((1 1))'), 1)) from onerow;
ST_POINT

With current build of geometry-api:

$ /usr/local/hive/bin/hive -S
hive> select ST_AsText(ST_GeometryN(ST_GeomFromText('multipoint (10 40, 40 30, 20 20, 30 10)'), 3)) from onerow;
POINT (20 20)
hive> select ST_GeometryType(ST_GeomFromText('multipoint ((1 1))')) from onerow;
ST_MULTIPOINT
hive> select ST_AsText(ST_GeometryN(ST_GeomFromText('multipoint ((1 1))'), 1)) from onerow;
NULL
-- logs exception:
ST_GeometryN: java.lang.ClassCastException: com.esri.core.geometry.ogc.OGCPoint cannot be cast to com.esri.core.geometry.ogc.OGCMultiPoint.

NullPointerException in JsonSerde

Caused by: java.lang.NullPointerException
at com.esri.hadoop.hive.GeometryUtils.getWKID(GeometryUtils.java:126)
at com.esri.hadoop.hive.GeometryUtils.geometryFromEsriShape(GeometryUtils.java:78)
at com.esri.hadoop.hive.serde.JsonSerde.serialize(JsonSerde.java:356)
...

(Reported to me by email.)

It would appear that bytesWritable is null at line 356 of JsonSerde, when this exception is thrown.

Create spatial index R-tree

I would like to know how you can create a spatial R-tree index type or other spatial index in hive... and how it works. I can not find examples for hive. I know examples for other databases, but in this case... no

Someone can help me?

thanks

Add serialize to hive JsonSerde

Curruently, the Hive JsonSerde only supports deserialize (JSON -> row/column). Needs serialize functionality (row/column -> JSON)

JsonSerde & data type in JSON for numerical columns

When a Hive table is created with
row format serde 'com.esri.hadoop.hive.serde.JsonSerde'
and populated with insert overwrite ... , the values in JSON are of string type, whether the table column type is string or numerical.
With numerical column types, it would be better for the JSON to contain numerical values rather than string values.

[Bug] Relationship

If Geometry 1 located in orthogonal Geometry 2 such as ST_LineString or any polygon, return value is false.
But other GIS system returns true.

[ST_Crosses]
SELECT ST_Crosses(st_linestring(1,0, 1,2), st_linestring(0,1, 2,1)) FROM onerow limit 1;
false

SELECT ST_Crosses (st_linestring(2,0, 2,3), st_polygon(1,1, 1,4, 4,4, 4,1, 1,1)) FROM onerow limit 1;
false

Relationship methods caused to same result.
For example, ST_Equals, ST_Intersects, ST_Overlaps.

Move Hive version compatibility code into a shims

Issue with ST_Area multi-call

When you call twice or more ST_Area in the same request you get always the same result.
ie :
select
ST_Area(ST_Polygon(1,1, 1,4, 4,4, 4,1))
from source.dual;
return 9.0

select
ST_Area(ST_Polygon(1,1, 1,2, 2,2, 2,1))
from source.dual;
return 1.0

but
select
ST_Area(ST_Polygon(1,1, 1,4, 4,4, 4,1)),
ST_Area(ST_Polygon(1,1, 1,2, 2,2, 2,1))
from source.dual;
return 1.0 1.0 instead of 9.0 1.0

Pig UDFs

I've been working on using the Esri/geometry-api-java to be a spatial framework for Pig -- have just hit the milestone of being able to do a spatial join by splitting on tiles at the mapper then joining two bags of geometries using a quadtree on the reducer.

The work currently lies within my fork of datafu for no other reason than it has a good unit test harness and I'm familiar with it. Since then I've also learned about pigeon.

This repo is probably the obvious home for the code, though. Is that interesting? Do you have feedback on the approach I'm taking?

Migrate InputFormats from hive-spatial repo

Support Array of Geometry and Geometry as Struct

If Multi-part geometry could be returned as a hive Array of single-part geometries, the array could be used with explode() - this may provide one possible solution to the objectives behind issue #35 .

While struct is mentioned in issue #9 in the context of import from JSON, structs could be useful in a more general context.

Point could be a structure of x, y [,z] [,m] .
Structure for Polygon would take more thought.

ST_GeomFromJson, ST_GeomFromGeoJson - input JSON in struct vs string

The constructors from Esri JSON & from GeoJSON support parsing JSON from string.
Add support to construct from JSON that is represented as a struct in Hive QL.

Create spatial index R-tree

Someone can help me?

thanks

UnenclosedJsonRecordReader needs to respect file split start and length

UnenclosedJsonRecordReader currently scans an entire file for each split it gets. This is incorrect as FileSplits define a byte offset begin and end. The record reader must read records only within these ranges.

Remove build output jars from repository

We should only have source and library files in the repository. Build output can be hosted elsewhere

ST_Geometry constructors throw exception on CSV header row

With a Hive table defined on a CSV file that erroneously contains a header row, the number-pair constructors throw exception. Should return null instead of throwing exception.

Remove dependency on Hive PDK as it is not supported in Hive 0.11+

See https://issues.apache.org/jira/browse/HIVE-4304.

PDK tests can be commented out and used as a reference for expected output.

Create earthquake workflow sample using Hive

Recreate the MapReduce earthquake aggregation sample workflow in Hive

ST_AsText(ST_Polygon(1,1, 4,1, 1,4)) -- null

hive> select ST_AsText(ST_Polygon(1,1, 4,1, 1,4)) from onerow;
NULL

cf:

hive> select ST_Polygon(1,1, 4,1, 1,4) is null from onerow;
false
hive> select ST_GeometryType(ST_Polygon(1,1, 4,1, 1,4)) from onerow;
ST_POLYGON
hive> select ST_AsJson(ST_Polygon(1,1, 4,1, 1,4)) from onerow;
{"rings":[[[1.0,1.0],[4.0,1.0],[1.0,4.0],[1.0,1.0]]]}
hive> select ST_AsText(ST_Polygon('polygon ((1 1, 4 1, 1 4))')) from onerow;
POLYGON ((1 1, 4 1, 1 4, 1 1))

Also with ST_ExteriorRing, null result varies by which constructor:

hive> select ST_ExteriorRing(ST_Polygon(1,1, 4,1, 1,4)) is null from onerow;
true
hive> select ST_ExteriorRing(ST_Polygon('polygon ((1 1, 4 1, 1 4))')) is null from onerow;
false

Anyway to search in hive by distance from point?

Sorry about this issue because it isn't a real issue, but i can't find in documentation the way to search points within a distance from center point.

I searched in google for this, but few people use this library and there are few examples.

Please provide me a solution here or at: [email protected]

Thank you for your incredible job!!

Build out earthquake aggregation sample workflow

Develop sample workflow for earthquake aggregation using California counties

Develop MapReduce sample
Add Oozie workflow execution
Clean up instructions to run

Create spatial index R-tree

Someone can help me?

thanks

ST_Relate on LineString-Polygon

Expected

hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '1***') from onerow;
true
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '_2****') from onerow;
false
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '**1') from onerow;
false
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '_**2*') from onerow;
true

Actual:

hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '1***') from onerow;
false
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '_2****') from onerow;
true
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '**1') from onerow;
true
hive> select ST_Relate(ST_LineString('linestring(0 2, 2 3)'), ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), '_**2*') from onerow;
false

Comparison:

hive> select ST_Relate(ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), ST_LineString('linestring(0 2, 2 3)'), '1***') from onerow;
false
hive> select ST_Relate(ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), ST_LineString('linestring(0 2, 2 3)'), '_2****') from onerow;
true
hive> select ST_Relate(ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), ST_LineString('linestring(0 2, 2 3)'), '**1') from onerow;
true
hive> select ST_Relate(ST_Polygon('polygon((1 1, 4 1, 4 4, 1 4, 1 1))'), ST_LineString('linestring(0 2, 2 3)'), '_**2*') from onerow;
false

This will be fixed in geometry-api.

Implement Hadoop output formats for (Un)enclosed JSON

Should implement output formats for JSON formats that correspond with the input formats

EnclosedJsonOutputFormat
EnclosedJsonRecordWriter
UnenclosedJsonOutputFormat
UnenclosedJsonRecordWriter

Convert Hive UDFs to GenericUDF

UDFs that extend GenericUDF have more control over how the objects passed to them are handled. Here are a few of the benefits of switching to generic - I'll use ST_Contains as an example.

Simplified expressions

ST_Contains takes two arguments and checks to see if the first geometry contains the geometry argument.

ST_Contains(ST_GeomFromText('POLYGON (( ... ))'), ST_Point(longitude, latitude))

Using GenericUDF we can support different data types and simplify the previous expression to

ST_Contains('POLYGON (( ... ))', ST_Point(longitude, latitude))

Optimizing for constants

In the last expression, 'POLYGON (( ... ))' is a literal constant. We can use this property to optimize at least two different aspects of the operator.

only create the OGCGeometry once per instance of the GenericUDF class
if the first argument to ST_Contains is constant, we can accelerate the corresponding OGCGeometry.

Eliminate unneeded (de)serialization between UDFs

This isn't necessarily related to GenericUDF conversion, but can happen at the same time.

Example:

ST_Contains('POLYGON (...)', ST_Point(lon,lat))

In this case, ST_Point serializes the OGCPoint to bytes and then ST_Contains deserializes those bytes back to an OGCPoint. This is expected behavior when values are passed across nodes, but in the case of this example, each call to ST_Contains almost certainly happens on the same node as the corresponding call to ST_Point. This is not a huge performance hit for points, but for larger polygons it will be.

Aggregating Average in Earthquake Sample

In the earthquake-reducer, the calculation for aggregate average should be a weighted average of the prior aggregate and the additional data.

@@ -26,7 +26,7 @@ public class ReducerClass extends Reducer<Text, EarthquakeDataWritable, Text, Ea
            float maxMag = data.getMax();

            agg_cnt += cnt;
-           agg_avgMag = (agg_avgMag == 0) ? avgMag : (agg_avgMag + avgMag) / 2;
+           agg_avgMag = (agg_cnt == 0) ? avgMag : (agg_avgMag * agg_cnt + avgMag * cnt) / (agg_cnt + cnt);

            agg_minMag = Math.min(agg_minMag, minMag);
            agg_maxMag = Math.max(agg_maxMag, maxMag);

Missing dependency for javax.jdo:jdo2-api:jar:2.3-ec

When building with Ant 1.8.4.

ST_Union can return non-simplified geometry

hive> SELECT ST_AsText(ST_Union(ST_Polygon(1, 1, 1, 4, 4, 4, 4, 1), ST_Polygon(4, 1, 4, 4, 4, 8, 8, 1))) FROM onerow;
MULTIPOLYGON (((1 1, 4 1, 8 1, 4 8, 4 4, 1 4, 1 1)))

Note the (1 1, 4 1, 8 1) in the result.

Proposed fix will to be to use geometry-collection with geometry.ogc.OGCGeometry.union .

Develop Hive UDAFs ST_Aggr_ConvexHull and ST_Aggr_Intersection

ST_AsGeoJSON working implementation.

Once export to GeoJSON is supported in geometry-api, ST_AsGeoJSON needs to be tested, and can be streamlined.

Spatial Index

JsonSerde - ClassCastException

Got ClassCastException with LazyString in JsonSerde with queries like the following:

create external table trips_by_origin_cell(leftlon double, botlat double, rightlon double, toplat double, totnum int, samedest int, pct double, destlhs double, destbot double, destrhs double, desttop double)
row format delimited fields terminated by '\t'
location '/user/rwhitman/out-trips-by-origin-cell';
create external table trip_origin_json  (totnum int, samedest int, pct double, destlhs double, destbot double, destrhs double, desttop double, shape binary)
row format serde 'com.esri.hadoop.hive.serde.JsonSerde'
stored as inputformat 'com.esri.json.hadoop.UnenclosedJsonInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location '/user/rwhitman/trip-origin-json';
insert overwrite table trip_origin_json select totnum, samedest, pct, destlhs, destbot, destrhs, desttop,
  ST_Polygon(leftlon,botlat, rightlon,botlat, rightlon,toplat, leftlon,toplat) from trips_by_origin_cell;

More detail of the exception:

Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.LazyString cannot be cast to org.apache.hadoop.io.Writable
        at com.esri.hadoop.hive.serde.JsonSerde.serialize(Unknown Source)

Candidate patch:

                Writable writable;
                //   = (Writable)fieldWritables.get(i);  // usually Text; exception on LazyString
                Object tmpObj = fieldWritables.get(i);  // usually Text; exception on LazyString
                if (tmpObj instanceof LazyPrimitive) {
                    writable = ((LazyPrimitive)(tmpObj)).getWritableObject();
                } else {
                    writable = (Writable)tmpObj;
                }
                //Float dummy = (Float)tmpObj;  // temporary test

                jsonGen.writeObjectField(columnNames.get(i), writable.toString());

Requesting the Multi part to Single part support.

I am working on dissolve functionality and I used ST_Aggr_Union to achieve the dissolve result. I am getting the result as multi-part result.

I am looking for some function which may provide single-part result or may convert into single-part polygon from multi-part polygon.

Please let me know if further information is needed.

esri / spatial-framework-for-hadoop Goto Github PK

spatial-framework-for-hadoop's Introduction

spatial-framework-for-hadoop

What's New

Features

Getting Started

Pre-Built

Maven

Ant

Dependencies

Requirements

Resources

Issues

Contributing

Licensing

See Also

spatial-framework-for-hadoop's People

Contributors

Stargazers

Watchers

Forkers

spatial-framework-for-hadoop's Issues

Recommend Projects

Recommend Topics

Recommend Org