feedzai / feedzai-openml-java Goto Github PK

Implementations for Feedzai's OpenML APIs to allow for usage of machine learning models in the Java programming language.

Home Page: https://www.feedzai.com

License: Apache License 2.0

Shell 0.19% Java 98.23% Python 1.59%

feedzai-openml-java's People

Contributors

Stargazers

Watchers

Forkers

krisztinaknagy machado117 albertoeaf sofg0mes vinaymenon tiagomssantos hugomiguelabreu andrefcruz joaoleveiga artpdr renato2000

feedzai-openml-java's Issues

H2O Training has a data-race on super-csv

The following problem is happening sometimes when we train concurrently 2 or more H2O models in the same JVM:

java.lang.ArrayIndexOutOfBoundsException: -6 at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:623) at java.lang.StringBuilder.append(StringBuilder.java:202) at org.supercsv.encoder.DefaultCsvEncoder.encode(DefaultCsvEncoder.java:81) at org.supercsv.io.AbstractCsvWriter.escapeString(AbstractCsvWriter.java:102) at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:196) at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:146) at org.supercsv.io.CsvListWriter.write(CsvListWriter.java:71) at com.feedzai.openml.h2o.H2OUtils.lambda$writeDatasetToDisk$2(H2OUtils.java:90) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.feedzai.openml.h2o.H2OUtils.writeDatasetToDisk(H2OUtils.java:88) at com.feedzai.openml.h2o.H2OModelCreator.fit(H2OModelCreator.java:174) at com.feedzai.openml.h2o.H2OModelCreator.fit(H2OModelCreator.java:64)

The problem was diagnosed to be in the H2OUtils class where we are using:

new CsvPreference.Builder(CsvPreference.STANDARD_PREFERENCE).build()

While that seems to do a copy of that STANDARD_PREFERENCE static object, it actually reuses internally a StringBuilder for one of the objects, and for some reason super-csv mutates that later during CSV writes.

Remove @since tags introduced in the FairGBM commit

The latest commit introduced some @since tags to master:

00e3ab4

which apparently is no longer part of our code guidelines. @mlobofeedzai asked to revert those @since tags.

bug in tests: Java surefire forking is causing LightGBM to fail with serious C++ issues

As of #121 LightGBM tests can still fail after the first run with a serious C++ error from the LightGBM side:

[ERROR] munmap_chunk(): invalid pointer
[ERROR] Aborted (core dumped)

After that the following messages will show up:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Openml LightGBM lib 3.0.1-with_model_locale_fix_for_java_and_streaming SUCCESS [  1.659 s]
[INFO] openml-lightgbm-module 1.0.12-SNAPSHOT ............. SUCCESS [  0.378 s]
[INFO] openml-lightgbm 1.0.12-SNAPSHOT .................... FAILURE [  9.691 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12.312 s
[INFO] Finished at: 2022-05-27T14:48:38+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project openml-lightgbm: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider && /home/alberto.ferreira/.sdkman/candidates/java/8.0.302-open/jre/bin/java -javaagent:/home/alberto.ferreira/.m2/repository/org/jacoco/org.jacoco.agent/0.8.4/org.jacoco.agent-0.8.4-runtime.jar=destfile=/home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/jacoco.exec -jar /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/surefire/surefirebooter2472734679044173114.jar /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/surefire 2022-05-27T14-48-29_842-jvmRun1 surefire8810960277057352224tmp surefire_07484658466683880775tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests:
[ERROR] com.feedzai.openml.provider.lightgbm.LightGBMBinaryClassificationModelTrainerTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider && /home/alberto.ferreira/.sdkman/candidates/java/8.0.302-open/jre/bin/java -javaagent:/home/alberto.ferreira/.m2/repository/org/jacoco/org.jacoco.agent/0.8.4/org.jacoco.agent-0.8.4-runtime.jar=destfile=/home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/jacoco.exec -jar /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/surefire/surefirebooter2472734679044173114.jar /home/alberto.ferreira/code/external/feedzai-openml-java/openml-lightgbm/lightgbm-provider/target/surefire 2022-05-27T14-48-29_842-jvmRun1 surefire8810960277057352224tmp surefire_07484658466683880775tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests:
[ERROR] com.feedzai.openml.provider.lightgbm.LightGBMBinaryClassificationModelTrainerTest
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

How to reproduce

# Go to repo root and run:
mvn clean install -DskipTests
cd openml-lightgbm  # Go to provider directory to run its tests.

Tests will always run with:

mvn clean test

However, they will always fail (at least after the first run) with:

mvn test

Conclusion

surefire settings: How to have `mvn test` work all the time

By simply changing maven-surefire-plugin's forking settings (see [1]) it's possible to have mvn test work all the time too.

This is the observed behaviour regarding mvn test:

default settings fail (forkCount=1, reuseForks=true)
disabling forks works (forkCount=0)
not reusing forks works (reuseForks=false)

Update: All fork settings can still fail after all. The failure odds change though. The only command that works all the time is still mvn clean test. This means the approaches described below don't fix the issue.

Analysis

Given that disabling forks altogether vs running the tests in a forked process (default) causes the issues, the problem seems caused by Java's test stack and not by the LightGBM C++ side (which always works with mvn clean test anyway).

Note: Upgrading to the latest versions of jacoco 0.8.6 and maven-surefire-plugin to 3.0.0-M6 didn't fix the issue either.

Possible solutions

Now, we can disable forks altogether, instead of forking the process before starting the tests once (default, which is failing); or we can set tests not to reuse forks. The latter option is slower as it forks the process for every test class.

Are there any downsides to doing these approaches? And which of those approaches is best?

References

1. maven-surefire-plugin forking docs

Ownership issue extracting the openmp tar building the lightgbm docker image

Simply running the command

mvn clean install -DskipTests

I get the following errors

The command '/bin/sh -c curl -sL https://releases.llvm.org/$CLANG_VER/clang%2bllvm-$CLANG_VER-x86_64-linux-gnu-ubuntu-14.04.tar.tar: openmp-9.0.0.src/libomptarget/deviceRTLs/nvptx/test: Cannot change ownership to uid 107418, gid 89939: Invalid argxz -o clang.tar.xz  && tar -C /usr/local -xf clang.tar.xz --strip 1  && curl -sL https://releases.llvm.org/$CLANG_VER/openmp-$CLument
tar: openmp-9.0.0.src/libomptarget/deviceRTLs/nvptx/docs/ReductionDesign.txt: Cannot change ownership to uid 107418, giANG_VER.src.tar.xz -o openmp.tar.xz  && tar -xf openmp.tar.xz  && cd openmp-$CLANG_VER.src  && mkdir build  && cd build  && cmakd 89939: Invalid argument
e -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang ..  && make  && make install  && echo /usr/local/lib > /etc/ld.so.conf.dtar: openmp-9.0.0.src/libomptarget/deviceRTLs/nvptx/docs: Cannot change ownership to uid 107418, gid 89939: Invalid argument
/openmp.conf  && ldconfig  && cd ../..  && rm clang.tar.xz  && rm openmp.tar.xz  && rm -rf openmp-$CLANG_VER.src' returned a nontar: openmp-9.0.0.src/libomptarget/deviceRTLs/nvptx/CMakeLists.txt: Cannot change ownership to uid 107418, gid 89939: I-zero code: 2
nvalid argument

I worked around the issue by simply adding the --no-same-owner to the tar command in the docker file projects-open-source/feedzai-openml-java/openml-lightgbm/lightgbm-builder/make-lightgbm/docker/lightgbm-ci-build-env/Dockerfile

like this: tar -xf openmp.tar.xz --no-same-owner

Environment:

Docker version 20.10.14, build a224086
Host OS: Ubuntu 20.04.4 LTS (64-bit)
Host Kernel: 5.13.0-44-generic

org.codehaus.mojo:exec-maven-plugin version is missing.

This error appeared on release build.
I suppose that this affected our releases.

[WARNING] 'build.plugins.plugin.version' for org.codehaus.mojo:exec-maven-plugin is missing. @ com.feedzai:openml-lightgbm:[unknown-version], /home/travis/build/feedzai/feedzai-openml-java/openml-lightgbm/pom.xml, line 103, column 21

As shown on below image:

Rename OpenML LGBM & FGBM provider & algorithm names

With the introduction of FairGBM now we have LightGBM and FairGBM support in Pulse:

However, in the UI they show up as completely different things, which doesn't give the best UX:

Microsoft LightGBM - LightGBM Binary Classifier
Feedzai GBM - FairGBM (LightGBM with fairness constraints)

We could also use the name Feedzai GBM for the LightGBM provider, and at most rename its algorithm to Microsoft LightGBM Binary Classifier, turning into something more organized:

Feedzai GBM - Microsoft LightGBM Binary Classifier
Feedzai GBM - FairGBM (LightGBM with fairness constraints)

I also have doubts if FairGBM should also be renamed Binary Classifier, for now it only allows that.

FairGBM: add tests to ensure FairGBM improves fairness on a small test dataset

Originally posted by @AlbertoEAF in #116 (comment):

We have no tests ensuring what we want to achieve - fairness :p

Yes, we can be using FGBM in the C++ part but let's say we have some bug in the provider and we don't pass the adequate parameters to it. It'll behave as LGBM. It'll happen and we have no clue.

We need to have a test ensuring we achieve a higher fairness than LGBM and sacrifice recall for it.

[refactor] LightGBM: Use try-with-resources on autocloseable classes that manage memory

Originally posted by @gandola in #116 (comment):

LightGBMBinaryClassificationModel is a autoclosable. this means that we should make use of it.

1- LightGBMBinaryClassificationModel::close() should be responsible to release all the resources related with the instance.
2- Use Try...With...Resources (similar to python With clause) to ensure that resources are released

Example:

try(final LightGBMBinaryClassificationModel model = fit(...)){
   return getClassScores(dataset, model, maxInstancesToScore);
}
// You don't need to explisitly call the close() `Try...With...Resources` will do that for you.

We should do that for all Autoclosable resources.

LightGBM make.sh using wrong version

LightGBM make.sh is using v2.3.0 instead of v2.3.1.

@shenggwang at least we're still using the build artifacts from the internal project or were they rebuilt as well?

tests: Unify LGBM test datasets

Issue

Originally posted by @AndreFCruz in #116 (comment):

It was generated with the following ipython notebook. Should we commit this?
generate-fairgbm-sensitive-attribute.zip

Request

To test fairgbm the test datasets were modified. Documentation of the new column generated by André should be added to the file https://raw.githubusercontent.com/feedzai/feedzai-openml-java/master/openml-lightgbm/lightgbm-provider/src/test/resources/test_data/stats.org, which declares how features were generated.

Also, since @shenggwang introduced tests for the explanations/contributions, he added another whole set of test sets based on those initial test datasets that I had created, meaning we now have two similar but different datasets.

I suggest unification of all those test resources to avoid the redundant test payloads in the repo. Given that Sheng had the work to explain how to generate the new test sets in Python, with code that can be executed in the future, and André also used Python code to generate updated test set, I suggest getting rid of the older datasets I generated (and described in stats.org, for they use excel formulas instead), regenerating Sheng's datasets with the new python code from André and refactoring the tests to use the new test sets. Also, add that updated python code info to the README.