autodeployai / pmml4s-spark Goto Github PK

Our team is utilizing pmml4s-spark for the past couple of years. We wanted to update our current version 0.9.3 to 0.9.16 and I wanted to know what are the added features and enhancements made.

Hi! I'm trying to use AWS EMR to score a Dataframe using a pmml model, but got error that is not explicit to trace. Any help would be appredicated.

Read the dataframe:
val df = spark.read.parquet("myDataframe.parquet")

df.show() works fine.

Read the pmml model:
val model = ScoreModel.fromFile("myxgboostModel.pmml")

Then run the following code：
model.transform(df).show()

Gave an error message:
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

Environment: Spark 2.4.4, Zeppelin 0.8.2.

However, I didn't meet this error when I was coding with Intellij on my macbook.

The method java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names() is introduced in spark 2.4.3 .so it is not compatible with lower version of spark.

Receiving below error with spark version 2.3

scala> val score=model.transform(df)
java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names()[Ljava/lang/String;
at org.pmml4s.spark.ScoreModel.transform(ScoreModel.scala:56)
... 51 elided

Also, need to know what all dependent jars should be imported to score a model in spark-scala?