Trouble Compiling with Visual Studio and Intel Fortran about sundials HOT 11 CLOSED

Thank you for your replies @drreynolds and balos1.

-The reason I turned to Cvode from Intel ode solver was that FCvode (version 2003) was thread safe. It means a lot that Pr. Reynolds confirmed that Fcvode (version 2003) is thread safe. Unfortunately the past month I didn't succeed to parallelize my code through openmp. I am using threadprivate allocatable arrays to feed data to Fcvode inside openmp regions and I get inconsistent results with lots of crashes due to either Floating point exceptions or Memory access violations. My code works perfectly in serial compilation (absence of any openmp directives). So Its either the latest Intel OneApi Fortran compiler has made small changes in the latest openmp implementation that I am not aware of or using Fcvode forced my compiler to use F2003 Fortran that may be incompatible with existing code.

-At the moment I am trying to produce a simple example that clearly exhibits inconsistent results between serial and openmp version of my code. If I have something concrete I 'll create a new issue in hope for resolution.
-I replied in the current thread cause I am not quite sure the compilation process for producing Fcvode is absolutely correct. I worry that internal optimizations of fcvode or Cvode may spawn extra threads inside openmp regions and create unexpected results. Though this shouldn't be the case.

Best Regards

from sundials.

At least, does anyone tried to compile CVode with Visual studio (2013 version or later) and encountered similar problems where Fortran objects are not compiled because the Fortran and C objects must be in different projects?

from sundials.

There was a previous user report on the sundials email list about the issue with Visual Studio and Fortran projects. It seems that you have found a resolution for that though. I don't quite follow what you have done to allow it to compile, but If you had to change code in SUNDIALS, posting the diff or a pull request would be appreciated.

As far as the run-time issue in Debug mode... I believe this is because the FN_VGetArrayPointer function calls c_f_pointer with a shape of [1] here. In Debug mode there is likely a bounds check that does not like this.

You can try to update the FN_VGetArrayPointer function to:

function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
type(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function

If this works, then we can try to incorporate it into the next SUNDIALS release.

from sundials.

Dear Balos,

Yep we have a winner!
After 1.5 month of efforts CVode on Fortran runs!

The code you proposed fixed the problem with out of boundaries exception.
After updating the "FN_VGetArrayPointer" Function the example ("examples/cvode/F2003_serial/cv_analytic_sys_dns_f2003.f90") ran properly and after doing some debug the "yvec" is an array with size of 3 elements as it was supposed to be.

A couple of things to note:

1. The fix you proposed has a typo:

function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
type(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function

The line "type(C_INT64_T) :: length " is supposed to be "integer(C_INT64_T) :: length " else it won't compile, so the correct update would be:

function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
integer(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function

2. The subroutine "c_f_pointer" is used in a similar way elsewhere is the code and may need an update/fix too.
   Please have a look:

function FN_VGetDeviceArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 

farg1 = c_loc(v)
fresult = swigc_FN_VGetDeviceArrayPointer(farg1)
call c_f_pointer(fresult, swig_result, [1])
end function

Best regards,
Bill

from sundials.

Dear Balos,

Regarding the steps I followed to compile under Visual Studio 2013 and 2019 unfortunately I didn't update any code. I just did some manual changes on the visual studio project that the Cmake-gui had generated. This means that every time the generate button is pressed on Cmake all the changes I did, must be redone. For the a simple version of Cvode (no examples or tests) this requires about 50 mins of copy paste. To fix properly Cvode, extensive changes on Cmake files are needed, so that the C and Fortran files are not anymore in the same project and instead are separated in different projects. The compilation order should be, first build the Fortran project and then the C targets.

Since the Sundials Cmake files seemed to me very complicated (so many targets with different options)

1. I just created a simple Fortran project and I added all .f90 files I could find in Cvode. I duplicated the extension of Fortran file from .f90 to .f90.f90.
2. I imported the Fortran project in the existing solution the Cmake-gui produced.
3. I found all targets that pointed to a .f90.obj file and I changed the location of where the linker should search to find them. So for example in the solution explorer (right top hand side of visual studio) there is target (project) named "sundials_fcvode_mod_static" and if you expand it (expand the object folder too) this contains many objects like "fcvode_mod.f90.obj", fnvector_serial_mod.f90.obj and so on. If you look at the properties of these objects (right hand side of screen, under solution explorer) you will see a field called "Relative Path". This path contains the path where the linker should look to locate these objects. If you change this path to the "debug" or "release" folder of the Fortran project created above, then all will be ok. The linker will find the fcvode_mod.f90.obj objects and the "All_Build" will compile properly.
4. To build the "Install" target I just created a new folder named "Fortran_Static" in the "build_dir" directory and inserted inside it all the .mod files created from the Fortran project above. Then hit build the "Install" target and everything worked.

I hope this helped a bit on the process I followed to compile Cvode with Visual Studio. If there is a need for more clarification let me know. It took me some time compile Cvode and I hope this is useful somehow.

Cheers,
Bill

from sundials.

@Bill-gitt Thanks for the detailed writeup that others can refer to for a workaround. We will try to fix both issues (the Visual Studio build, and FN_VGetArrayPointer problem) in a future SUNDIALS release.

from sundials.

Dear Balos,

It seems I have encountered another issue with the FCvode. It is related to its thread-safety, it seems independent threads access shared memory when executing memory release functions (FCVodeFree or FSUNMatDestroy etc...) and the program crashes. So, should I create a new issue in github or continue here in the current thread (it may be compile related)?

Regards,
Bill

from sundials.

from sundials.

@drreynolds is correct that FCVODE (the FORTRAN 77 interface to CVODE) is not theadsafe, but I think you are referring to the FCVode function from the sundials_fcvode_mod module (i.e. the F2003 interfaces), correct @Bill-gitt?

If you are indeed referring to the FCVode function from the sundials_fcvode_mod module, please start a new issue for it.

from sundials.

Dear balos1,

After lots of effort, I finally managed to make Fcvode work properly inside openmp regions. This is a success I believe (unless something else comes up later).
To give a bit of backstory so others may benefit from it, this is what happened. After trying to run simple Fcvode examples in parallel yielded random results. Most of the time everything worked out well but one out ten runs 1/10 the program could give erroneous results. Thusly I decided to use Intel inspector on my code. What I found out was eye opening, more than 25 warnings where revealed inside Fcvode libraries (irrelevant of the code I have written) that indicated failure of thread safely. I knew though that Fcvode should be thread safe and thusly the reason for failing should be in compilation. Searching a bit further and testing lots of different compilation flags I found out what was missing. The directive "qopenmp" was missing from the compilation of Fcvode interface. Searching online in Intel I found this

(1) “Static local data” is shared by each thread, i.e. each thread may access the same data location! This is potentially unsafe.
(2) “Automatic data” is independent between each thread; it has its own, independent copy on its own stack.
With Intel Fortran when compiled and running serially, the defaults are local scalar variables are automatic and local arrays are static. But when compiling with -qopenmp, local arrays are automatic by default. It is the same as compiling with -auto. This may require an increase in the maximum stack size. If the stack size is too small, there will be a segmentation fault at runtime.

Which means that when not using qopenmp local arrays are not thread safe...

After using qopenmp and running intel inspector again the 25 warnings were now only 2 which seemed nothing serious.
So for anyone that wants to run Fcvode inside openmp regions make sure that you compile code with qopenmp.

That's it for now. I have managed to compile Fcvode with Intel Visual studio and use its thread safety features, successfully.

Best regards

from sundials.

@Bill-gitt Thanks for posting your findings.

from sundials.