levyfan / sentencepiece-jni Goto Github PK

How to use sentencepiece in java，
Is there any maven dependency？

Thanks a lot!!

java.lang.UnsatisfiedLinkError: File /sentencepiece_jni.dll was not found inside JAR.

at com.github.google.sentencepiece.SentencePieceJNI.<clinit>(SentencePieceJNI.java:11)
at com.github.google.sentencepiece.SentencePieceProcessor.<init>(SentencePieceProcessor.java:13)
at com.github.google.sentencepiece.SentencePieceProcessorTest.setUp(SentencePieceProcessorTest.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

java.lang.NullPointerException
at com.github.google.sentencepiece.SentencePieceProcessorTest.tearDown(SentencePieceProcessorTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

Thanks for this work, @levyfan! Would you entertain supporting sentencepiece 0.1.9+? I have the changes necessary to support this release set and would be happy to make a PR.

Hi, I am wandering if this JNI could be used on Android device and any example of the process?

Great work, thank you.

Hello,
Firstly thank you for your great work.
When i want to implement your sentencepiece JNI in Android studio, i have this error: ninja: error: 'sentencepiece/src/sentencepiece-build/src/libsentencepiece.a' is needed to build libsentencepiece.so .

It seems like that i need to add the file 'libsentencepiece.a'. Therefore could you please tell me how to get this file? I suppose that i should use the command 'g++ -c hello.cc' to obtain 'hello.o' then use 'ar cr libsentencepiece.a hello.o' to obtain 'libsentencepiece.a'. But i don't kown the 'hello.cc' correspond which code.cc in the github sentencepiece/google

Thanks a lot for your reply!

Not able to generate build on Mac 12.4 (Monterey)

Failed after configuration step

Logs
50%] Performing configure step for 'sentencepiece'
loading initial cache file /sentencepiece-jni/target/sentencepiece/tmp/sentencepiece-cache-Release.cmake
-- VERSION: 0.1.92
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to: /sentencepiece-jni/target/sentencepiece/src/sentencepiece-build
[ 60%] Performing build step for 'sentencepiece'
[ 0%] Building CXX object src/CMakeFiles/sentencepiece.dir//third_party/protobuf-lite/arena.cc.o
[ERROR] Command execution failed.
[ 0%] Building CXX object src/CMakeFiles/sentencepiece.dir//third_party/protobuf-lite/arena.cc.o
In file included from sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:37:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/limits:105:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/type_traits:420:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstddef:37:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:37:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/limits:105:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/type_traits:421:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:37:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/limits:121:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:39:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/utility:228:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:51:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/typeinfo:62:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstdint:144:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/stdint.h:123:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include/stdint.h:52:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/stdint.h:33:9: error: unknown type name 'uint8_t'
typedef uint8_t uint_least8_t;
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/stdint.h:44:9: error: unknown type name 'uint8_t'
typedef uint8_t uint_fast8_t;
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:51:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/typeinfo:63:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/exception:83:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstdlib:85:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/stdlib.h:93:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/stdlib.h:66:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/wait.h:110:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:202:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:216:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:236:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:258:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:289:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/resource.h:328:2: error: unknown type name 'uint8_t'
uint8_t ri_uuid[16];
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:51:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/typeinfo:63:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/exception:85:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:582:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__thread/poll_with_backoff.h:13:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/chrono:833:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:587:
/sentencepiece-jni/target/sentencepiece/src/sentencepiece/version:1:1: error: expected unqualified-id
0.1.92
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:590:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__threading_support:457:44: error: use of undeclared identifier '__error'
while (nanosleep(&__ts, &__ts) == -1 && errno == EINTR);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/errno.h:81:17: note: expanded from macro 'errno'
#define errno (__error())
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:1066:73: error: reference to unresolved using declaration
_Tp __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstddef:49:1: note: using declaration annotated with 'using_if_exists' here
using ::ptrdiff_t _LIBCPP_USING_IF_EXISTS;
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:1071:65: error: reference to unresolved using declaration
_Tp* __cxx_atomic_fetch_add(__cxx_atomic_base_impl<_Tp*> * __a, ptrdiff_t __delta, memory_order __order) _NOEXCEPT {
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstddef:49:1: note: using declaration annotated with 'using_if_exists' here
using ::ptrdiff_t _LIBCPP_USING_IF_EXISTS;
^
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/third_party/protobuf-lite/arena.cc:31:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena.h:55:
In file included from /sentencepiece-jni/target/sentencepiece/src/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/arena_impl.h:36:
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/atomic:1087:73: error: reference to unresolved using declaration
_Tp* __cxx_atomic_fetch_sub(__cxx_atomic_base_impl<_Tp*> volatile* __a, ptrdiff_t __delta, memory_order order) _NOEXCEPT {
^
/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/cstddef:49:1: note: using declaration annotated with 'using_if_exists' here
using ::ptrdiff_t _LIBCPP_USING_IF_EXISTS;
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[5]: *** [src/CMakeFiles/sentencepiece.dir//third_party/protobuf-lite/arena.cc.o] Error 1
make[4]: *** [src/CMakeFiles/sentencepiece.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [sentencepiece/src/sentencepiece-stamp/sentencepiece-build] Error 2
make[1]: *** [CMakeFiles/sentencepiece.dir/all] Error 2
make: *** [all] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.089 s
[INFO] Finished at: 2022-07-25T17:59:03+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (make) on project sentencepiece: Command execution failed.: Process exited with an error: 2 (Exit value: 2) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (make) on project sentencepiece: Command execution failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:326)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

I check C++ compiler version using gcc --version
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Please help out @levyfan @erip

Hi @levyfan , the current version of sentencepiece used in this project is 0.1.96. The latest version available is 0.2.0. I tried upgrading the version in the CMakeLists.txt which lead to many errors because the 0.2.0 sentencepiece version required cxx 17 leading to few data type mismatches (std::basic_string).

Is there any possibility of upgrading the latest version in this repo as well ?

Thanks for the great repo! For a couple of reasons, this repo doesn't work on Windows.

First reason: make not generally available on Windows, so this fails to build. This can be replaced by native cmake install per here.

Second reason: Sentencepiece cannot link shared libraries on Windows, so this will need to be conditionally changed (see here in the CMakeLists file.

There may be some other small issues, but these are the big blockers.

I git cloned the repo and ran the SentencePieceProcessorTest. It throws the error

java.lang.UnsatisfiedLinkError: File /libsentencepiece_jni.dylib was not found inside JAR.

	at com.github.google.sentencepiece.SentencePieceJNI.<clinit>(SentencePieceJNI.java:11)
	at com.github.google.sentencepiece.SentencePieceProcessor.<init>(SentencePieceProcessor.java:13)
	at com.github.google.sentencepiece.SentencePieceProcessorTest.setUp(SentencePieceProcessorTest.java:21)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)


java.lang.NullPointerException
	at com.github.google.sentencepiece.SentencePieceProcessorTest.tearDown(SentencePieceProcessorTest.java:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

Did someone have similar issue before? Thanks!

Hi @levyfan would like to whether this sentence piece works in a Mac systems

Hello there,
I have cloned https://github.com/levyfan/sentencepiece-jni/releases/tag/v0.0.1 and built it on Ubuntu and windows.
Then I tested both jars with the same input and same model.
Surprisingly I am getting different results for the same input and same model.

The input sentences is Necesita más discusiones
Output on linux jar: [▁, Necesita, ▁más, ▁disc, us, iones]
Output on windows jar: [?, Necesita, ?m, �, �, s, ?disc, us, iones]
Python output: ['▁', 'Necesita', '▁más', '▁disc', 'us', 'iones']

I noticed, If I pass non-diacritics sentence, Both windows jar and linux jar are producing the same result
The input sentences: Luego
Output on Linux jar: [▁Luego]
Output on windows jar: [?Luego]
Python output: ['▁Luego']

Java code snippet

import com.github.google.sentencepiece.SentencePieceProcessor;

public class Main {

    public static void main(String[] args)  throws InterruptedException{
    SentencePieceProcessor ss = new SentencePieceProcessor();
    ss.load("tokenizer.model");
    System.out.println(ss.encodeAsPieces("Necesita más discusiones"));

    }
}

I am using hugging face model
You can use this model in python to cross-check the result

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
print(tokenizer.tokenize("Luego"))

For your reference, I have attached those two jars and the model below
Windows jar
Ubuntu
Model

Please help out @levyfan @erip

Anyone help me, plz?
My log is bellow:

https://drive.google.com/file/d/1Jjk_1G99aXmDY5WRlOrpa6aLhFuaTwxO/view?usp=sharinghttps://drive.google.com/file/d/1Jjk_1G99aXmDY5WRlOrpa6aLhFuaTwxO/view?usp=sharing

https://drive.google.com/file/d/1kjjKsx6mwaZF6WdLygMk4krGUmNVLmKk/view?usp=sharing

With the newly merged work in #12 and #14, it's possible to publish cross-platform artifacts to Maven central or even GitHub Packages for broad consumption. I think there may be a few other things to consider before doing this (e.g., should the sentencepiece version == package version? should it be a classifier?), but this issue tracks that.

    @Test
    public void testJvmCrashLoad() {
        Assert.assertEquals(null, sp.idToPiece(38067));
    }

This test will crash jvm.

I have created the Linux jar on an ubuntu machine. Then I moved the ubuntu jar to a CentOS machine and tried to tokenise a text. It threw me an error.

Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/libsentencepiece_jni4849969498159746745.so: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /tmp/libsentencepiece_jni4849969498159746745.so)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
        at java.lang.Runtime.load0(Runtime.java:810)
        at java.lang.System.load(System.java:1088)
        at com.github.google.sentencepiece.SentencePieceJNI.<clinit>(SentencePieceJNI.java:9)
        at com.github.google.sentencepiece.SentencePieceProcessor.<init>(SentencePieceProcessor.java:13)

I cloned the sentencepiece-jni repo in the centos machine and created a jar. Then I replaced the ubuntu jar with the newly created centos jar and run the code. It worked.
I have attached the ubuntu Jar file and the centos jar file for your reference.
https://drive.google.com/drive/folders/1ubpmParRAoMIs5Zj0Ehah9Sd_BBwgyEm?usp=sharing

Ubuntu OS details

NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

CentOS details

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Please help out @levyfan @erip
Thanks in advance.

I am writing to inform you about some library dependency issues that I have encountered, specifically a GLIBC version conflict on the Linux operating system.

I compiled the repository on a machine which has 'GLIBC 2.35'. However, when I tried to use the compiled jar on another Linux machine which has 'GLIBC 2.17', I received the following error:
java.lang.UnsatisfiedLinkError: /tmp/libsentencepiece_jni98487993849071377.so: /lib64/libm.so.6: version 'GLIBC_2.29' not found (required by /tmp/libsentencepiece_jni98487993849071377.so)

To resolve this issue, I compiled the repository on the lowest GLIBC machine. The compiled jar now works fine on both machines. However, I am wondering if there are any dependencies that need to be met on Windows and Linux?

@levyfan @erip