[SUPPORT] java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch about hudi HOT 8 OPEN

It looks like a known issue reported in: #9305

from hudi.

Hi @danny0405, yes it does look similar. However, the table was already running with Spark 3.3.2 and hudi 0.13.1 without errors. The only changes here were we upgraded the Hudi version to 0.14.1 and turned on metadata.

The casting also seems to be in the opposite direction and the first run with Hudi 0.14.1 did not have metadata and succeeded. Do you think the issue is related to how the metadata table is saved? In other words, is metadata not supported with Spark 3.3.2?

Thanks for the help!

from hudi.

Do you think the issue is related to how the metadata table is saved? In other words, is metadata not supported with Spark 3.3.2?

It is supported, can you share you config options related with metadata table?

from hudi.

Hi @danny0405 , we are using defaults only. All hudi configs specified are listed above. Is there something we should configure specifically?

from hudi.

I'm pretty sure it is a jar conflict, can you check the jar that involves the reported class?

from hudi.

@vicuna96 How many columns are there in your dataset? If its more than 100, did you tried setting spark.sql.codegen.maxFields

from hudi.

Hi @danny0405 , this seems to be in the spark-catalyst_2.12-3.3.2.jar package. but org.apache.spark.sql.catalyst.expressions.UnsafeRow does not extend org.apache.spark.sql.vectorized.ColumnarBatch. Is this expected in different versions?

Hi @ad1happy2go , I can give it a try but the table should have less than 100 columns and also this seems like a spark property rather than hudi property and the spark version has not changed. I will update once I get a chance to test it.

from hudi.

@vicuna96 Did you get a chance to test out.

from hudi.