Help debugging a OOM issue when the search population increases about spark-lucenerdd HOT 3 OPEN

@zouzias Do you have any idea?
I keep getting this error and changing the partitioning does not help

from spark-lucenerdd.

Can you try with linker method linker.method = "collectbroadcast"?

I think you should be able to broadcast the search base to all executors since they are only 2M.

See: https://github.com/zouzias/spark-lucenerdd/blob/master/src/main/resources/reference.conf#L19

Also, if you have any fields that you do not use for linkage, you should remove them from the DataFrame, i.e.

val fields = Array("usefulCol1", "usefulCol2", ...)
val projectedSearchBase = searchBase.select(fields)

and then

val results = index.link(searchBase.rdd, Linker.queryBuilderLinker, topK).filter(_._2.nonEmpty) 

from spark-lucenerdd.

Thank you for your response @zouzias

Both dataframes contain only one field. It is a string field containing addresses so they are not that long too.

I can't use linker.method = "collectbroadcast" because I am using the custom build discussed on #162 . I am using relatively complex queries and they are very hard to build using Strings (and hard to manage overall) . Query builders do a better job here.

Do you have any idea about what could be the source of the OOM error? It seems that executors are allocating too much much memory , but the executors are relatively big and sizes are relatively small.

I tried with even smaller datasets and it is still failing (size > 100K). It works sometimes with bigger datasets , but it is not reliable.

from spark-lucenerdd.