hibayesian / spark-fm Goto Github PK

Hi ,hibayesian !Thanks a lot for spark-fm sharing.I have checked the sample code which uses model=train.fit(trainData) and model.transform(testData) for prediction.
My concern is that how can I save the trained model in local txt and load it in future use.
I only found the related weight of the model but the weight can not be reloaded directly as model.
Looking forward to your concern and feedback.Thanks.

I find FMClassifier in spark3, but I don't know what formate my featuresCol should be. I used my gbdt feature, but the result AUC is bad.

some of my code:
val classifier = new FMClassifier()
.setLabelCol("click")
.setFeaturesCol("gbdtFeature")
.setPredictionCol("predictClick")
.setProbabilityCol("probabilitys")

http://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/classification/FMClassifier.html

thank in advance!

Q1. compare with the https://github.com/Intel-bigdata/imllib-spark，the auc performance of this spark-fm is very bad in my ctr prediction task, i have try mini-batch SGD、L-BFGS、parallel SGD and parallel FTRL, all auc < 0.7，but the same dataset and parameters on imllib-spark is 0.88，is there some fault in the implementation?
Q2. in my opinion, optimizer FTRL is not a good method in FM optimize task，because ftrl will lead to the matrix V is sparse，which will recduce the effect of cross features.

1. Bug1
   in FactorizationMachinesUpdater, need to use the below code, otherwise, SGD will exist with only 1 iteration.
   val weightsNew = weightsOld.copy.toArray

2. Bug2
   in FactorizationMachinesGradient, change the code below, for the loss calculation.
   // case Algo.BinaryClassification => -Math.log(1 + 1 / (1 + Math.exp(-p * label)))
   case Algo.BinaryClassification => Math.log(1 + Math.exp(-p * label))
   @hibayesian