Improve logging about spark-lucenerdd HOT 2 OPEN

How about an info message when the linkage action takes place since linkage can take a lot of time?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian

This is related to #190 .

from spark-lucenerdd.

How about an info message when the linkage action takes place since linkage can take a lot of time?

Could you please clarify this?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

Sadly I can't do this as I need to use the Query Builders with the "link" method and it is not supported as we discussed in pull request #162

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 13) had a not serializable result: org.apache.lucene.search.BooleanQuery
Serialization stack:
	- object not serializable (class: org.apache.lucene.search.BooleanQuery, value: (home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5)
	- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
	- object (class scala.Tuple2, (0,(home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5))
	- element of array (index: 0)

from spark-lucenerdd.