Support for OS400 about jffi HOT 79 OPEN

I neglected to mention that malloc is supported on OS400.

I see malloc.h is optionally included in configure if _MSC_VER is specified. I also see alloca has the same parameter list/types as malloc. My understanding is that alloca releases memory upon a function going out of scope/returning and malloc requires a manual call to free(ptr) to accomplish the same (Microsoft malloc docs).

I tried altering configure to be as follows (note addition of || _OS400) without success and it produced the same error**.

# ifdef _MSC_VER || _OS400
#  include <malloc.h>
#  define alloca _alloca
# else
. . .

**[exec] /home/aaron/git/jffi/jni/jffi/LongDouble.c:82:11: error: incompatible implicit declaration of built-in function 'alloca' [-Werror]

from jffi.

This is the right place to report bugs with jffi!

We can put your good, working changes into a PR any time.

alloca allocates memory from the calling function's stack. When the called function returns, the stack pointer moves back to the caller's position, so the memory is deallocated automatically. This is equivalent to having "char[256] x" in your code.

Because of the stack effects and automatic deallocation, alloca can't directly be emulated with malloc. You're running into problems because the code calling alloca doesn't actually pay attention to the configure results.

I'm not sure how to proceed. We could modify all code that uses alloca, but it seems really unfortunate to cripple other platforms for OS/400. I'm poking around to see if there's a blessed alternative to alloca on OS/400. Maybe you have better developer docs?

from jffi.

I think it may be possible to transplant the magic ifdef alloca block from jni/libffi/include/ffi_common.h to jni/jffi/jffi.h. Can you try that?

from jffi.

What compiler does OS/400 use? Does it support C99? If so, you could just try to replace alloca() with variable-length arrays.

from jffi.

If I read the linked compiler documentation correctly, it does support C99.

So you should be able to replace the alloca call with something like this:

diff --git a/jni/jffi/LongDouble.c b/jni/jffi/LongDouble.c
index cb8f243..b6031a7 100644
--- a/jni/jffi/LongDouble.c
+++ b/jni/jffi/LongDouble.c
@@ -32,9 +32,6 @@

 #include <stdio.h>
 #include <stdlib.h>
-#ifdef __sun
-# include <alloca.h>
-#endif
 #include <stdint.h>
 #include <stdbool.h>
 #include <jni.h>
@@ -75,11 +72,10 @@ Java_com_kenai_jffi_Foreign_longDoubleFromString(JNIEnv *env, jobject self, jstr
   jbyteArray array, jint arrayOffset, jint arrayLength)
 {
     long double ld;
-    char* tmp;
     jsize len;

     len = (*env)->GetStringUTFLength(env, str);
-    tmp = alloca(len + 1);
+    char tmp[len + 1];
     (*env)->GetStringUTFRegion(env, str, 0, len, tmp);
     ld = strtold(tmp, NULL);
     jffi_encodeLongDouble(env, ld, array, arrayOffset, arrayLength);

from jffi.

I am using the gcc compiler**. I will work on testing the two suggestions and get back to you.

**

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/QOpenSys/opt/freeware/bin/../libexec/gcc/powerpc-ibm-aix6.1.0.0/4.6.2/lto-wrapper
Target: powerpc-ibm-aix6.1.0.0
Configured with: ../gcc-4.6.2/configure --with-as=/usr/bin/as --with-ld=/usr/bin/ld --enable-languages=c,c++,fortran --prefix=/opt/freeware --mandir=/opt/freeware/man --infodir=/opt/freeware/info --enable-threads --enable-version-specific-runtime-libs --disable-nls --enable-decimal-float=dpd --host=powerpc-ibm-aix6.1.0.0
Thread model: aix
gcc version 4.6.2 (GCC)

from jffi.

@aaronbartell That's pretty recent GCC version. It should support VLAs so if the pragma magic @headius suggested doesn't work, please try the above. You probably need to convert more code, of course, but it should be straight-forward.

from jffi.

@penberg, it made it past LongDouble.c and is now choking on FastNumericInvoker.c so it would seem this is viable. Now I will try suggestion from @headius.

See log here

from jffi.

I think it may be possible to transplant the magic ifdef alloca block from jni/libffi/include/ffi_common.h to jni/jffi/jffi.h. Can you try that?

Here is the log (failed). Below are the diffs so I can make sure I made the correct changes.

jffi.h diff

ffi_common.h diff

from jffi.

@aaronbartell I meant to copy the pragmas into jffi.h. libffi and the jffi JNI binding get built separately, and will both need the pragmas. Since libffi built fine without the pragmas and failed (with alloca warnings and related errors) when they were removed, it would seem we're on the right track.

from jffi.

I now have that ifdef code in both areas. I did an ant clean and then ant jar with the shell results found here. Here is the config.log in case that helps.

Let me know if you'd like me to run it again with either -bloadmap or -bnoquiet to obtain more information.

from jffi.

That's looking much closer. Something wrong with the final linking of libjffi, but everything seems to have compiled. Perhaps there's a gcc flag needed here to indicate this is a shared library and not an executable?

from jffi.

@headius Yes, -shared. It looks like it's missing from the log @aaronbartell posted.

from jffi.

I believe we have a successful build! See log here. The issue is I had the operating system upper-cased on this line.

I think one thing needing to be cleaned up is the name of the resulting .jar file because it contains a slash (/) in it:

[zip] Building zip: /home/aaron/git/jffi/dist/jffi-ppc-OS/400.jar

I will work to find how to get rid of the slash in the name.

from jffi.

I am now running through the ant test motions. I first changed libtest/GNUmakefile to have a section for os400 by copying the aix equivalent:

diff --git a/libtest/GNUmakefile b/libtest/GNUmakefile
index 9e70664..65c3097 100644
--- a/libtest/GNUmakefile
+++ b/libtest/GNUmakefile
@@ -153,6 +153,13 @@ ifneq ($(findstring mingw, $(OS)),)
   LIBEXT = dll
   PICFLAGS=
 endif
+
+ifeq ($(OS), os400)
+  LIBEXT = a
+  SOFLAGS = -shared -static-libgcc
+  PICFLAGS += -pthread
+endif
+
 ifeq ($(CPU), sparcv9)
   MODEL = 64
 endif

That caused some pthread errors to show up during ant test - see log.

Looking through the libtest/GNUmakefile it doesn't appear PICFLAGS (where -pthread is added) is used on the compiles so I made the following change to see if it would further the test, and it did.

diff --git a/libtest/GNUmakefile b/libtest/GNUmakefile
index 9e70664..65c3097 100644
--- a/libtest/GNUmakefile
+++ b/libtest/GNUmakefile
 $(LIBTEST):  $(TEST_OBJS)
-       $(CC) -o $@ $(LDFLAGS) $(TEST_OBJS) -lm
+       $(CC) -pthread -o $@ $(LDFLAGS) $(TEST_OBJS) -lm

 clean::
        # nothing to do - ant will delete the build dir

Then I run ant test again (note I didn't do ant clean) and seemingly get further, though all of the tests fail (see log here)

Thoughts on what direction I should take these tests?

from jffi.

Does anything in jffi/build/test/results show what the errors are? With all tests failing I'd guess it's not able to load libjffi.so.

from jffi.

Great progress btw...I'm sure we'll iron this out soon :-)

from jffi.

@headius, there was a file for each failure in jffi/build/test (sorry I missed those). Here is an example of one and they are all conveying basically the same error of java.lang.UnsatisfiedLinkError.

Concerning libjffi.so, here is what a find produces:

-bash-4.2$ pwd
/home/aaron/git/jffi
-bash-4.2$ find . -name libjffi*
./build/jni/libjffi-1.2.a

from jffi.

Doh! If I would have looked further into the aforementioned stack trace I would have seen this:

Caused by: java.lang.RuntimeException: cannot determine operating system

I will do some digging to see what I can find.

from jffi.

I have made it a bit further and am now stuck with errors in this full stack trace.

Here's an excerpt:

java.lang.UnsatisfiedLinkError: /home/aaron/git/jffi/build/jni/jffi-1.2.srvpgm ( 0509-022 Cannot load module /home/aaron/git/jffi/build/jni/libjffi-1.2.srvpgm.so. 0509-026 System error: A file or directory in the path name does not exist.)

It is true the above link/file doesn't exist. What portion of the build should have created it? I ran ant jar and then ant test.

In case you need to know, here are my changes to StubLoader.java to account for OS400:

diff --git a/src/main/java/com/kenai/jffi/internal/StubLoader.java b/src/main/java/com/kenai/jffi/internal/StubLoader.java
index 9a1d842..1e2e14d 100644
--- a/src/main/java/com/kenai/jffi/internal/StubLoader.java
+++ b/src/main/java/com/kenai/jffi/internal/StubLoader.java
@@ -82,6 +82,8 @@ public class StubLoader {
         WINDOWS,
         /** IBM AIX */
         AIX,
+        /** IBM OS400 */
+        OS400,
         /** IBM zOS **/
         ZLINUX,

@@ -136,7 +138,9 @@ public class StubLoader {
         } else if (startsWithIgnoreCase(osName, "sunos") || startsWithIgnoreCase(osName, "solaris")) {
             return OS.SOLARIS;
         } else if (startsWithIgnoreCase(osName, "aix")) {
-            return OS.AIX;
+            return OS.AIX;
+        } else if (startsWithIgnoreCase(osName, "OS/400")) {
+            return OS.OS400;
         } else if (startsWithIgnoreCase(osName, "openbsd")) {
             return OS.OPENBSD;
         } else if (startsWithIgnoreCase(osName, "freebsd")) {
@@ -208,8 +212,10 @@ public class StubLoader {
         if (getOS().equals(OS.DARWIN)) {
             return "Darwin";
         }
+        if (getOS().equals(OS.OS400)) {
+            return "OS400";
+        }

-
         String osName = System.getProperty("os.name").split(" ")[0];
         return getCPU().name().toLowerCase(LOCALE) + "-" + osName;
     }

from jffi.

The StubLoader changes look good.

The libjffi.so would be created by the native part of the build. It occurs to me now you may need to manually copy the it to the archive directory (working from memory here since my machine is busted). Then I believe the jar task will incorporate it into the jffi jar file.

from jffi.

Bummer about your machine.

Is this the file you are talking about? Does it matter it is an archive library vs. shared object?

-bash-4.2$ pwd
/home/aaron/git/jffi
-bash-4.2$ find . -name "*libjffi*"
./build/jni/libjffi-1.2.a

from jffi.

No, the file is jffi-SOMETHING.jar, and the build will put it in dist:

-build-platform-jar:
     [echo] platform=Darwin
      [zip] Building zip: /Users/headius/projects/jffi/dist/jffi-Darwin.jar

Copy that to archive/ and rebuild, and I believe it will get included into the final jffi jar. The build does not copy to archive/ because the files there are known to git.

from jffi.

BTW...the final, working OS/400 native jar, copied to archive/, is what you'd PR (ideally as a separate commit from the changes needed to build it).

from jffi.

Here's the steps I've taken:

1 - rm archive/jffi-ppc-OS400.jar

2 - ant clean

3 - ant jar

4 - mv dist/jffi-ppc-OS400.jar archive/

5 - ant jar <---- This didn't update the archive/jffi-ppc-OS400.jar's timestamp so I figured to continue with another ant clean. I left archive/jffi-ppc-OS400.jar in place from the previous ant jar.

6 - ant clean

7 - ant jar

8 - ant test This produced the previous errors:

java.lang.UnsatisfiedLinkError: could not locate stub library in jar file. Tried [jni/OS400/jffi-1.2.srvpgm, /jni/OS400/jffi-1.2.srvpgm]

and

java.lang.UnsatisfiedLinkError: /home/aaron/git/jffi/build/jni/jffi-1.2.srvpgm ( 0509-022 Cannot load module /home/aaron/git/jffi/build/jni/libjffi-1.2.srvpgm.so.

9 - cp dist/jffi-ppc-OS400.jar archive/

10 - ant test <---- running this produced the same errors as in step 8.

I am guessing I did something wrong. What should I have done?

from jffi.

Your steps have a lot of duplication, but the end result should have been correct if the library's getting built right. Once you've built the native archive and copied the jar into archive/ you can just do normal builds from then on. The archive/ jar will not be updated unless you copy a new one in place, but it is where the build gets artfacts to insert into the jffi jar.

Check that the jffi jar has either the OS400 native jar or the files that would be inside it (the native lib itself). Check that the path/filename jffi uses to load that library actually matches the file (I see .so, and you mentioned .a, so there's a lead).

It looks like jffi may not have the right heuristics for building a shared lib filename on OS/400, and so it can't locate the built library.

from jffi.

Here's the archive/jffi-ppc-OS400.jar contents:

The call to System.mapLibraryName(stubLibraryName) produces jffi-1.2.srvpgm. I am not familiar with the .srvpgm extension and am wondering if this is an OS400 thing or if it is a Java thing. OS400 does have something called "service programs" but they have to do with a different/native runtime environment on the machine.

So then I tried manually changing the path to jni/ppc-OS400/libjffi-1.2.a and it still isn't finding it. Other ideas?

from jffi.

A little more info... it seems System.load in StubLoader.java always appends .so to the end of the name. Now I am curious how the AIX folks are getting this accomplished as they also have an archive (i.e. libjffi-1.2.a).

from jffi.

I wonder if System.loadLibrary is a way to route around this. I've never spent a lot of time looking at how those paths work.

I've put out a call for help on Twitter. If System.load is actually appending an invalid extension (and System.mapLibraryName is adding a bogus or atypical extension) that would seem to be a bug in IBM's JVM. That seems unlikely to me.

Do you have an IBM iSeries or J9 support person you can talk to about this? I'm running out of ideas and don't have an iSeries machine here to play around on :-)

from jffi.

I tried swapping in System.loadLibrary (replacing System.load) and have the same results.

that would seem to be a bug in IBM's JVM. That seems unlikely to me.

Agreed. Unlikely, though I don't know how common it is to load C/C++ archive files into Java on this platform.

Do you have an IBM iSeries or J9 support person you can talk to about this?

I do have a person at IBM I can reach out to but I wanted to ask one more question first. Do you know where the loadJar() process/method is looking for the jar file? Here is what I have for jars in the entire git clone:

-bash-4.2$ find . -name "*.jar"
./archive/jffi-Darwin.jar
./archive/jffi-arm-Linux.jar
./archive/jffi-i386-FreeBSD.jar
./archive/jffi-i386-Linux.jar
./archive/jffi-i386-OpenBSD.jar
./archive/jffi-i386-SunOS.jar
./archive/jffi-i386-Windows.jar
./archive/jffi-ppc-AIX.jar
./archive/jffi-ppc-Linux.jar
./archive/jffi-ppc64-Linux.jar
./archive/jffi-s390x-Linux.jar
./archive/jffi-sparc-SunOS.jar
./archive/jffi-sparcv9-SunOS.jar
./archive/jffi-x86_64-FreeBSD.jar
./archive/jffi-x86_64-Linux.jar
./archive/jffi-x86_64-OpenBSD.jar
./archive/jffi-x86_64-SunOS.jar
./archive/jffi-x86_64-Windows.jar
./archive/jffi-ppc-OS400.jar    <----------------------
./lib/junit/junit-3.8.2.jar
./lib/junit_4/junit-4.5-src.jar
./lib/junit_4/junit-4.5.jar
./build/native.jar
./dist/jffi.jar
./dist/jffi-ppc-OS400.jar      <----------------------------
./dist/jffi-complete.jar

I'm running out of ideas and don't have an iSeries machine here to play around on :-)

My hope is to get a chroot environment setup on a community IBM i so CI (like travis-ci.org) can be run. The price points for "IBM i in the cloud" are coming down** but still a decent amount of work to setup. If I get one setup I will let this group know.

** $100/month from iDevCloud.com for a dedicated virtual machine used for educational/non-commercial purposes

from jffi.

Do you know where the loadJar() process/method is looking for the jar file?

The contents of all the archives are re-packed directly into the eventual jffi jar, I believe. So if you look in the jffi jar, you should see the native libs for all platforms.

If I get one setup I will let this group know.

That would be great :-) I also poked a few friends of mine at IBM who might be able to assist us getting this library built and loading properly.

from jffi.

So if you look in the jffi jar, you should see the native libs for all platforms.

Here is what I have in the dist/ directory after running a build:

$ ls dist/
jffi-complete.jar   jffi-ppc-OS400.jar  jffi.jar

The following lists the contents of each .jar file. As you can see, only jffi-complete.jar and jffi-ppc-OS400.jar have the .so and .a files for each platform. Should jffi.jar also have them?

dist/jffi-complete.jar

dist/jffi-ppc-OS400.jar

dist/jffi.jar

from jffi.

FYI, we'd prefer if you used Gist for long pastes.

Right right, sorry...I meant the jffi-complete jar. If you were using jffi.jar directly, you could try using jffi-complete.jar instead (maybe you already did).

Let me confirm I can run ant test here and then I'll confirm it's using the complete jar. If it is, we still have some work to do on loading the OS/400 binary.

from jffi.

Ok...so ant test works for me just fine on OS X, after removing -Werror from libtest's makefile.

This line in build.xml indicates where it's looking for the native libs:

<sysproperty key="java.library.path" value="${build.native.dir}"/>

build.native.dir here is /jni. If you have the appropriate library there, the test should work...if it's able to load the file correctly.

I wonder if you could try renaming the library to the extension jffi is looking for...at least then we could tell if the library works, and focus on getting it to load properly.

from jffi.

FYI, we'd prefer if you used Gist for long pastes.

Will do. I've updated my previous post with gist entries to clean it up for historical purposes.

I wonder if you could try renaming the library to the extension jffi is looking for...at least then we could tell if the library works, and focus on getting it to load properly.

I renamed build/jni/libjffi-1.2.a to build/jni/libjffi-1.2.srvpgm.so. Then the error said it was looking for ./test.srvpgm so I renamed ./libtest.a to ./test.srvpgm (it wasn't looking for a .so extension). Renaming those got us past the loading issue and produced this java dump and this file: javacore.20140909.151537.67913.0002.txt.

It appears line 982 is where it first references the Java side of jffi.

Snap.20140909.151537.67913.0003.trc (binary, let me know if I should jextract it)

core.20140909.151537.67913.0001.dmp (2.2GB bytes in size so I didn't create a gist)

The rename obviously worked for loading the files, but do these errors tell us the original .a files weren't built correctly or is it telling us we need to build them initially as .so?

from jffi.

Wow that's a cool error... illegal instruction!

My first thought was that it's related to some assembly-generation logic in JNR that tries to make faster-than-jni stubs on the native side, but that logic all lives in jnr-ffi.

So then the real answer seems to be that .a isn't the output format we want to use. I poked around a bit, and at least on Linux a .a file is intended to be used as a statically-bound library, not a dynamically-bound library. I feel like there's make/cc flags missing still.

from jffi.

FYI - my iSeries contacts have confirmed that iSeries JVM can recognize and load ".so", ".a" & ".srvpgm" files.

They showed a test case that loads all three kinds of libraries, provided the native library is on the java.library.path (which from the native stacktrace in the javacore, it looks like it is).

from jffi.

I've done some more looking and found the JNIEXT and LIBEXT settings in jni/GNUmakefile and libtest/GNUmakefile respectively. I altered them, as shown below, to not resort to .a extension and instead .so. That resulted in this successful build with .so as extension but the unit tests still fail with the illegal instruction (same as when I manually renamed from .a to .so.

Just now saw response from @DanHeidinga. Can we see the example code IBM showed or was it done under an NDA?

diff --git a/jni/GNUmakefile b/jni/GNUmakefile
index bdf2fe1..b200332 100755
--- a/jni/GNUmakefile
+++ b/jni/GNUmakefile
@@ -223,7 +223,7 @@ ifeq ($(OS), os400)
   SOFLAGS = -shared -static-libgcc
   CFLAGS += -pthread
   LDFLAGS += -pthread
-  JNIEXT = a
+  #JNIEXT = a
   STRIP = strip
 endif

diff --git a/libtest/GNUmakefile b/libtest/GNUmakefile
index 9e70664..0e60019 100644
--- a/libtest/GNUmakefile
+++ b/libtest/GNUmakefile
@@ -130,6 +130,12 @@ ifeq ($(OS), solaris)
   SOFLAGS = -shared -static-libgcc
 endif

+ifeq ($(OS), os400)
+  #LIBEXT = a
+  SOFLAGS = -shared -static-libgcc
+  PICFLAGS += -pthread
+endif
 $(LIBTEST):  $(TEST_OBJS)
-       $(CC) -o $@ $(LDFLAGS) $(TEST_OBJS) -lm
+       $(CC) -pthread -o $@ $(LDFLAGS) $(TEST_OBJS) -lm

from jffi.

@DanHeidinga Can we get an example of how to properly make a JNI library on iSeries? cc command line, etc?

from jffi.

@aaronbartell Could you look to see whether libraries under JAVA_HOME are .a or .so? Several of them would be JNI extensions; if we don't see .a files then we may be missing something about how to build this stuff.

from jffi.

@headius I ran the below find and here is the output.

bash-4.2$ find /QOpenSys/QIBM/ProdData/JavaVM/jdk60/32bit -name "*.a" -o -name "*.so"

I started digging through IBM documentation for JNI calls to C/C++ and found Getting started with Java native methods. Then I found this page with a full example of JNI to C.

Here are the example source files:
PaseExample1.h
PaseExample1.c
PaseExample1.java

I then used the following to get the example running:

javac PaseExample1.java
javah -jni PaseExample1
export LIBPATH=/home/aaron
gcc -c -fPIC PaseExample1.c -o PaseExample1.o -I/QOpenSys/QIBM/ProdData/JavaVM/jdk60/32bit/include/
gcc -c PaseExample1.c -o libPaseExample1.so -I/QOpenSys/QIBM/ProdData/JavaVM/jdk60/32bit/include/
gcc PaseExample1.o -shared -o libPaseExample1.so
java PaseExample1.java

Output

-bash-4.2$ java PaseExample1
Value of str is 'String for PaseExample1'

Thoughts?

For historical purposes I've learned why it was looking for .srvpgm extension on this page.

from jffi.

We use xlC rather than gcc on iSeries so the commandlines may not be that helpful. That being said, here's the commandline used to build one of the internal jni argument test libraries:

xlC_r -O3 -DIPv6_FUNCTION_SUPPORT -q32 -q mbcs -qlanglvl=extended -qarch=ppc -qinfo=pro -qalias=noansi -qxflag=LTOL:LTOL0 -qsuppress=1506-1108 -D_XOPEN_SOURCE_EXTENDED=1 -D_ALL_SOURCE -DRS6000 -DAIXPPC -D_LARGE_FILES -qtbtable=full -qlist -qsource -DJ9OS_I5 -DJ9OS_I5_V7R2 -I..//iseries -qhalt=w -I. -I../include -I../include_core -I../oti -I../nls -qtbtable=full -qlist -qsource -DJ9OS_I5 -I/osxpf/v7r2m0f.cuep/bld/cmvc/pase.pgm/p5.cuep/include -I/osxpf/v7r2m0.cuep/bld/cmvc/pase.pgm/p5.cuep/include -I/afs/rchland.ibm.com/usr8/j9/src/rebuilt/jdk70/32bit27/sr1fp1.v7r2/jre/lib/ppc/default/iseries -DUT_DIRECT_TRACE_REGISTRATION -c -o cdefs.o cdefs.c

rm -f ../libjniargtests.so

ld -b32 -G -bnoentry -bernotok -bmap:jniargtests.map -bE:jniargtests.exp -bI:..//iseries/i5exports.exp
-bI:/osxpf/v7r2m0.cuep/bld/cmvc/pase.pgm/p5.cuep/lib/as400_libc.exp -bI:/afs/rchland.ibm.com/usr8/j9/src/rebuilt/jdk70/32bit27/sr1fp1.v7r2/jre/lib/ppc/default/iseries/i5exports.exp -L. -L../ -L../lib/ -o ../libjniargtests.so
copyright.o args_01.o args_02.o args_03.o args_04.o args_05.o args_06.o args_07.o args_08.o args_09.o cdefs.o
-lc_r -lC_r -lm -lpthread

from jffi.

@aaronbartell Were you able to make any more progress based on @DanHeidinga's response? We're so close!

Your find output seems to indicate that we probably do need .so for the JNI libraries, since e.g. libhprof is also a JNI library.

from jffi.

I've cleaned up a few more things so my build/test wasn't such a hack and better adheres to how OS400 loads .a files. Latest changes here for anyone that is interested.

Going back to to @DanHeidinga's input response.

Here is the IBM xlc compiler options list (expand left nav to see options).
Here is the Option mappings for gxlc and gxlc++ utilities - XL C/C++ Enterprise Edition V8.0 for AIX which we can use to learn the gcc to xlc option mappings.

The majority of the mappings don't have a direct translation. My talents are lacking regarding what options are significant. Could anybody else give input on that front? Here is the latest successful build results. Here is the latest ant test Java dump. Note that libjffi-1.2.a is referenced but it doesn't appear to give a line number. Is there a gcc setting that would provide better errors?

from jffi.

Hi @aaronbartell. I tried to help you, but this work is totally outside my area of ​​expertise. I hope you will come to complete this task. I am extremely grateful to you for this library that will dramatically improve my use of JRuby.

from jffi.

I did some compilation tests. I do not understand why you absolutely want to use xlc. Originally there is no gcc or xlc installed on the machine. What makes you force the use of one rather than another?

from jffi.

@pierrickrouxel , the xlc compiler entered the picture as a way to learn what options we may need to specify on gcc. I would much prefer that gcc was used so others can also easily compile JRuby without the need to purchase the xlc compiler (not an in-expensive purchase).

I did find that perzl.org has a libffi port for AIX and the rpm installs successfully on IBM i. I am digging into that to see if I can simply use the compiled libffi files from there and plop them into .../jffi/build/jni/libffi-ppc-os400/.libs/

from jffi.

Oh ! Ok I misunderstood. I had not seen that the tests did not pass. Sorry.

from jffi.

@aaronbartell Any recent progress? I just released 1.2.8, but it would sure be nice to have OS/400 support in 1.3.0 (which will require rebuilds of a number of other native libs anyway).

from jffi.

@headius
Status: I am working on this again. I've started documenting the entire IBM i build process on my jffi fork in hopes others can help, though I am also working to get an IBM i machine provisioned so you (or others) can ssh in and poke around to speed things up. Stay tuned.

Progress:
I've done an upstream merge into my fork and the current build is yielding these results. In short, the error is as follows:

     [exec] In file included from /home/abartell/git/jffi/build/jni/libffi-ppc-os400/include/ffi.h:67:0,
     [exec]                  from /home/abartell/git/jffi/jni/jffi/jffi.h:45,
     [exec]                  from /home/abartell/git/jffi/jni/jffi/Array.c:39:
     [exec] /home/abartell/git/jffi/build/jni/libffi-ppc-os400/include/ffitarget.h:159:5: error: "_CALL_ELF" is not defined [-Werror=undef]
     [exec]  #if _CALL_ELF == 2

Here is file ffitarget.h. Note, I have not encountered this _CALL_ELF error before and am now looking into it.

UPDATE 2015-04-16 12:29pm CST
I commented out that section of ffitarget.h and now it is building successfully and I have file dist/jffi-ppc-OS400.jar. I will go back through this conversation on what to do next.

UPDATE 2015-04-16 3:40pm CST
I am now getting the following error when running ant -v test (full log). I don't know which class it is looking for. Is it looking for the unit test files in build/test/classes/com/kenai/jffi?, because those do exist. Is it looking for jffi-ppc-OS400.jar?, because I copied it to archive/ after the most recently successful ant jar.

    [mkdir] Created dir: /home/abartell/git/jffi/build/test/results
    [junit] Exception in thread "main" java.lang.NoClassDefFoundError:
    [junit] Caused by: java.lang.ClassNotFoundException:
    [junit]     at java.net.URLClassLoader.findClass(URLClassLoader.java:666)
    [junit]     at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:942)
    [junit]     at java.lang.ClassLoader.loadClass(ClassLoader.java:869)
    [junit]     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:336)
    [junit]     at java.lang.ClassLoader.loadClass(ClassLoader.java:847)
    [junit] Running com.kenai.jffi.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)

Here's the contents of build/test/results/IGNORETHIS.txt - the only file in the results/ directory. Note the line Testcase: null which leads me to believe it isn't find the build/test/classes/com/kenai/jffi files. Thoughts?

Testsuite: com.kenai.jffi.Batch-With-Multiple-Tests
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec

Testcase: null took 0.007 sec
    Caused an ERROR
Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.

from jffi.

It does look like it's having some strange problem finding the test classes, and ending up trying to run a null test case. But this seems like an ant problem rather than a jffi problem. The other possibility is that the CNFE is caused by something else, like the native library not being found or some problem during linking. I would think those would show up in a verbose output or in the test results, though, and usually they'd show a LinkageError somewhere.

Next step would probably be to add some logic to jffi's static initializers to see how far it's getting in booting during tests. If it's not booting at all, there may be something wrong with ant or our test target when run on OS/400.

from jffi.

UPDATE
Changed test.fork to no in build.xml, as shown below, and it now invokes the tests successfully.

<property name="test.fork" value="no"/>

The only test failure is returnDefaultF128HighPrecision in NumberTest. I put in the following debugging statements:

    private void returnF128HighPrecision(InvokerType type) {
        LibNumberTest lib = UnitHelper.loadTestLibrary(LibNumberTest.class, type);
        BigDecimal param = new BigDecimal("1.234567890123456789");
        BigDecimal result = lib.ret_f128(param);
        BigDecimal delta = param.subtract(result).abs();
System.out.println("param         :" + param.toPlainString());
System.out.println("result        :" + result.toPlainString());
System.out.println("delta         :" + delta.toPlainString());
System.out.println("new BigDecimal:" + (new BigDecimal("0.0000000000000000001")).toPlainString());
System.out.println("delta.compareTo1:" + delta.compareTo(new BigDecimal("0.0000000000000000001")) );
System.out.println("delta.compareTo2:" + (delta.compareTo(new BigDecimal("0.0000000000000000001")) < 0));
        assertTrue(delta.compareTo(new BigDecimal("0.0000000000000000001")) < 0);
    }

And that produces the following results:

Testsuite: com.kenai.jffi.NumberTest
Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.243 sec
------------- Standard Output ---------------
param         :1.234567890123456789
result        :1.2345678901234567
delta         :0.000000000000000089
new BigDecimal:0.0000000000000000001
delta.compareTo1:1
delta.compareTo2:false
------------- ---------------- ---------------

Testcase: returnDefaultF128HighPrecision took 0.058 sec
    FAILED
null
junit.framework.AssertionFailedError
    at com.kenai.jffi.NumberTest.returnF128HighPrecision(Unknown Source)
    at com.kenai.jffi.NumberTest.returnDefaultF128HighPrecision(Unknown Source)

I will continue to look into this to determine what might be going on, but if anyone knows what might be wrong please let me know.

UPDATE 2015-04-30 15:25CST
I am remembering I commented out some code in ffitarget.h to get past the earlier _CALL_ELF issue. Would that have something to do with it? I do not know the purpose of FFI_TRAMPOLINE_SIZE. I believe FFI_TRAMPOLINE_SIZE resolves to 40. I tried changing it to 32 but it had no affect.

UPDATE 2015-05-06 10:08CST
Discovered I missed LibraryTest erroring out (I was only looking at failures). I've fixed it with the following diff. I will now go back to the com.kenai.jffi.NumberTest.returnF128HighPrecision error.

diff --git a/src/test/java/com/kenai/jffi/UnitHelper.java b/src/test/java/com/kenai/jffi/UnitHelper.java
index cc241b3..cb7b33f 100644
--- a/src/test/java/com/kenai/jffi/UnitHelper.java
+++ b/src/test/java/com/kenai/jffi/UnitHelper.java
@@ -68,6 +68,8 @@ public class UnitHelper {
                 } else {
                     return "libc.a(shr_64.o)";
                 }
+            case OS400:
+                return "libc.a(shr.o)";
             default:
                 return "libc.so";
         }

UPDATE 2015-05-06 17:00CST
More info. Changing this line in NumberTest.java to be two less digits on the right of the decimal allows the test to pass (see below for diff). The call to lib.ret_f128(param) returns a 16 scale BigDecimal and param contains an 18 scale BigDecimal, which is why the subsequent delta.compareTo fails.

diff --git a/src/test/java/com/kenai/jffi/NumberTest.java b/src/test/java/com/kenai/jffi/NumberTest.java
index f4aa3b8..13c4699 100644
--- a/src/test/java/com/kenai/jffi/NumberTest.java
+++ b/src/test/java/com/kenai/jffi/NumberTest.java
@@ -261,9 +261,19 @@ public class NumberTest {

     private void returnF128HighPrecision(InvokerType type) {
         LibNumberTest lib = UnitHelper.loadTestLibrary(LibNumberTest.class, type);
-        BigDecimal param = new BigDecimal("1.234567890123456789");
+        BigDecimal param = new BigDecimal("1.2345678901234567");
         BigDecimal result = lib.ret_f128(param);

As I understand it, the call to lib.ret_f128(param) loads libtest.a (libtest.so on other platforms) which in turn invokes the following line in Number.c (I might be wrong in my assumption of what is getting invoked). Does anybody know if BigDecimal scale could be lost (scale of 18 down to 16) in the below line of code?

#define pack_f128(buf, v) do { *(long double *)(buf) = v; } while(0)

from jffi.

@headius, IBM was kind enough to respond in private email with the following. I would guess this means the test will never pass as-is. How would you like me to handle this? Should I deliver jffi-ppc-OS400.jar to you?

The double value of param is got through the method ---- BigDecimal.doubleValue(), if so, the result you got is an expected value when using BigDecimal("1.234567890123456789"), the result will be 1.2345678901234567. Because the representation for float/double value in Java is IEEE-754 standard. the value "1.234567890123456789" exceed the scope which IEEE-754 standard represents.

from jffi.

Thank you for your job @aaronbartell. I look forward to it!

from jffi.

Can you integrate this feature please?

from jffi.

could those .jar file be uploaded in jffi/archive file section also?

from jffi.

I do have the .jar file for OS/400 but at this point it is out of date (Oct 2014). To make this right we'd really need a fresh fork and implement the changes in my commits.

Some things to note:

• I had to hardcode this line because the OS wasn't being retrieved as expected.
• I included full IBM i (aka OS/400) wiki build instructions on fork.

I am prepping for two conferences in Chicago so am short on time until after the first week of Dec. If somebody else wanted to build this please feel free.

from jffi.

Hello @aaronbartell, thanks for bringing it this far. I tried to follow up on your last comment and merged your commits into a fresh fork of this repo. In order to get a successful build I only had to change:

--- jni/libffi/src/powerpc/ffitarget.h  (date 1449494021000)
+++ jni/libffi/src/powerpc/ffitarget.h  (revision )
@@ -156,7 +156,7 @@
 #define FFI_V2_TYPE_DOUBLE_HOMOG   (FFI_TYPE_LAST + 2)
 #define FFI_V2_TYPE_SMALL_STRUCT   (FFI_TYPE_LAST + 3)

-#if _CALL_ELF == 2
+#if defined(_CALL_ELF) && _CALL_ELF == 2
 # define FFI_TRAMPOLINE_SIZE 32
 #else
 # if defined(POWERPC64) || defined(POWERPC_AIX)

Unfortunately, ant test fails and the error message is mangled. Maybe someone can help me out here? Is there a way to get a proper error message? Here is the transcript for ant -v runtest -Dtest=PointerTest. I'm sorry, but the machine i'm using is set up to use Italian and I don't have a clue on how to change that.

from jffi.

@boskowski See this post for a unit test property change you need to make to get past the java.lang.ClassNotFoundException error.

from jffi.

@aaronbartell Thanks, I definitely have to catch up with the backlog. Now I'm stuck with the OS complaining the library (build/jni/libjffi-1.2.a) has a wrong signature (i..e an invalid magic number).
Update: Here's the log. Error 0509-103 is in Italian, translated: "The module has an invalid magic number." The build log is here.

from jffi.

@boskowski Please include a gist of stack traces so I can look further into errors. Note, you will need to hard code the OS to not have a slash in it (i.e. os/400 vs os400) because the OS is used in the file names and obvious a slash will cause issues.

from jffi.

@aaronbartell I just updated the previous message with the log.

from jffi.

@boskowski, so you've got all my changes including this one where I comment out the default JNIEXT to not be .a?

from jffi.

Trying not to overdo it, I switched JVM from Java 8 64bit to IBM J9 VM (build 2.6, JRE 1.6.0 OS/400 ppc-32 jvmap3260_26sr8fp4-20150414_02 (JIT enabled, AOT enabled). Building against this codebase plus the diffs below resulted in all tests running successfully except for NumberTest, which fails regardless of the length of the input.

--- src/main/java/com/kenai/jffi/Platform.java  (revision a803d182d8cdbad1eb71fd9a759d764113b9bb11)
+++ src/main/java/com/kenai/jffi/Platform.java  (revision )
@@ -415,7 +415,7 @@

         @Override
         public String mapLibraryName(String libName) {
-            return "lib" + libName + ".a";
+            return "lib" + libName + ".so";
         }

         @Override

--- src/test/java/com/kenai/jffi/UnitHelper.java    (revision a803d182d8cdbad1eb71fd9a759d764113b9bb11)
+++ src/test/java/com/kenai/jffi/UnitHelper.java    (revision )
@@ -63,6 +63,7 @@
             case WINDOWS:
                 return "msvcrt.dll";
             case AIX:
+            case OS400:
                 if (Platform.getPlatform().addressSize() == 32){
                     return "libc.a(shr.o)";
                 } else {

NumberTest yields

java.lang.NumberFormatException: Not a valid char constructor input: 1,2345678901234567
    at java.math.BigDecimal.bad(BigDecimal.java:1859)

this is the test output file (here I shortened the number as already tried by @aaronbartell, but the result is the same as with the original number).

What looks strange is the comma, which isn't contained in com/kenai/jffi/NumberTest.java:264. So I tried this:

$ jruby -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on IBM J9 VM jvmap3260_26sr8fp4-20150414_022.6 +jit [OS/400-PowerPC]
$ irb
io/console on JRuby shells out to stty for most operations
irb(main):001:0> p = java.math.BigDecimal.new("1.234567890123456789")
=> #<Java::JavaMath::BigDecimal:0x7b6666da>
irb(main):002:0> p = java.math.BigDecimal.new("1,234567890123456789")
Java::JavaLang::NumberFormatException: Not a valid char constructor input: 1,234567890123456789
        from java.math.BigDecimal.bad(java/math/BigDecimal.java:1859)
...

Does anyone have an idea what magic managed to conjure up the comma replacement?

from jffi.

Does anyone have an idea what magic managed to conjure up the comma replacement?

What does DSPSYSVAL QCCSID convey?

% system "DSPSYSVAL QCCSID"
                                                   System Values                                                        Page     1
5770SS1 V7R2M0  140418                                                                            LITMIS1   12/08/15  16:36:45 UTC
                Current                         Shipped
 Name           value                           value                           Description
 QCCSID      >  37                              65535                           Coded character set identifier
     Note:  > means current value is different from the shipped value
                                         * * * * *   E N D  O F  L I S T I N G   * * * * *

from jffi.

System-wide CCSID is 65535, I changed it to 37 for my user profile, but I guess this is not enough... I'll check with the sysadmin if a system-wide change is feasible. Thanks, @aaronbartell.

Update 2015-12-09: Changing the system-wide CCSID of the target machine is a no go.

However, looking at the Java properties, it seems that my CCSID (037) is inherited by the job as can be seen in the properties ant dumps when executing the test, e.g.

os400.job.file.encoding=Cp037
os400.file.encoding.ccsid=00819
ibm.system.encoding=ISO8859-1

I also wonder how any encoding issue can cause a change in the contents of a string constant, turning a dot into a comma, as the error appears before the string is converted into a number.

Update: I didn't notice that the error is caused by ret_f128 as it quite obviously receives the BigDecimal in the wrong format.

from jffi.

Hey folks, where do we stand today?

from jffi.

@headius I need your input on this question

Then I would need to re-fork (to get latest) and re-implement changes (assuming the un-passing test is ok).

SIDE NOTE: I am creating an online service for having easy access to the IBM i operating system. It's called Litmis Spaces and currently only supports Node.js and Ruby development with Python, Java, and PHP coming soon.

My hope is to automate builds of JRuby-like things so IBM i folks become first class citizens in open source.

from jffi.

@aaronbartell Thanks, sorry I missed the question. Yes, sounds like we should go forward with the binary assume that is a bad test, and if possible mask out or patch the test to work appropriately on AIX.

WRT getting regular testing on i-series...that would obviously be great :-) Let me know if I can do anything to help make that happen.

from jffi.

@aaronbartell I'll bump this to next release so you have a chance to re-pull and re-base your changes.

from jffi.

@headius You released 1.2.12 but I don't see the OS400 support. Why?

from jffi.

@pierrickrouxel Well, I don't see a PR for it anywhere and I'm a little confused at this point what to merge. It's an easy matter to spin a new release, so can someone sort out what we need to do?

from jffi.

@aaronbartell Still out there? If you're happy with what you have now, can we get a (rebased) PR?

from jffi.

@headius @pierrickrouxel

I can do a rebased PR but not until after I get done with two international travels (Toronto this week and Stockholm in June). Earliest I could accomplish this task is June 25th.

Side note: Bummed I can't make the RUM meetup tonight as I see JRuby is the topic (I live in Mankato, MN). Have fun!

from jffi.

@aaronbartell Ok, whenever you can get to it is great, or if someone else here wants to try to do it that's great too.

Yeah, the RUM thing was a last-minute decision...but I'm hoping to get to RUM more regularly.

from jffi.

All: This is one of the only outstanding issues in the JFFI tracker, and I'd like to resolve it. Do we have anything current that could go into 1.2.13 or a big bang 1.3 release?

from jffi.

Could you change
BigDecimal param = new BigDecimal("1.234567890123456789");
to:
BigDecimal param = new BigDecimal("1.234567890123456789", MathContext.DECIMAL128); ?

from jffi.

Pinging this one last time to see if someone can contribute a binary or provide access to an environment where we can build for IBM i-series.

from jffi.

I have access to IBMi 720, but we are in the process of upgrading to a Power 9 IBMi next week and would prefer to do it after we upgrade to the newer machine. It would also but built on the most current i hardware and software environment. Thanks.

from jffi.