martinfleis / efficient-geopandas-workshop Goto Github PK

Writing up here, and with some questions (can push when I am sure you are not doing that to avoid conflicts in the notebook)

Distance operations

Pairwise distances

QUESTION: How to calculate the distance between all points of two GeoDataFrames?

points1 = geopandas.points_from_xy(numpy.random.uniform(0, 1000, 1_000), numpy.random.uniform(0, 1000, 1_000))
gdf1 = geopandas.GeoDataFrame(geometry=points1)
points2 = geopandas.points_from_xy(numpy.random.uniform(0, 1000, 1_000), numpy.random.uniform(0, 1000, 1_000))
gdf2 = geopandas.GeoDataFrame(geometry=points2)

(can maybe just use one dataframe and do pairwise with itself to simplify)

%%time
# do not use this code
gdf1.geometry.apply(lambda g: gdf2.distance(g))

-> ca 750ms

We can use the broadcasting of the shapely ufunc by reshaping as 2D arrays (a (N, 1) and (1, N) gives (N, N) result):

%%time
shapely.distance(np.asarray(gdf1.geometry)[:, np.newaxis], np.asarray(gdf2.geometry)[np.newaxis, :])

-> ca 500ms

Note: so this doesn't give that much of a speedup nowadays, since both are using the fast ufunc under the hood, and distance is a relatively expensive function, and so you are only avoiding the inner loop for one dimension.

So it's not that big of an improvement.

I was think that we could show as a follow-up to use an strtree: what if you only want the distances for points that are less than xx m away from each other?

tree = shapely.STRtree(gdf1.geometry)

%%time
idx2, idx1 = tree.query(gdf2.geometry, predicate="dwithin", distance=100)
shapely.distance(np.asarray(gdf1.geometry)[idx1], np.asarray(gdf2.geometry)[idx2])

-> ca 80ms

We can't directly use query_nearest here, unless you of course are fine with just getting the one nearest within a search radius, instead of all within a search radius.

I suppose we will already show nearest as well in another example, or can combine it with this one?