This would allow us to have the full driver, from matrix and multivector creation to solving with those.

Description
Complete documented toolset and workflow for semi-automated interface, documentation and testing capabilities, leveraging Trilinos C++

Plan

• Draft list of tools needed to build ForTrilinos
• Draft list of capabilities provided ForTrilinos
• Write formal report

Criteria
Formal report.

@maherou @kevans32

Potential alternatives:

• Github's gh-pages
• ReadTheDocs
• Wiki (github?)

This issues is to track status of Jenkins CI testing.

• Create a Docker container with Trilinos TPL similar to how DTK does it
• Make sure tests pass in the container
• Hook up Jenkins CI with Github
• Configure Jenkins to launch CI on PR

Should it be this the standard header for Trilinos

!*********************************************************************
! ForTrilinos: Object-Oriented Fortran 2003 interface to Trilinos
!                Copyright 2010 Sandia Corporation
!
! Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
! the U.S. Government retains certain rights in this software.
!
! Redistribution and use in source and binary forms, with or without
! modification, are permitted provided that the following conditions are met:
!
! 1. Redistributions of source code must retain the above copyright
!    notice, this list of conditions and the following disclaimer.
!
! 2. Redistributions in binary form must reproduce the above copyright
!    notice, this list of conditions and the following disclaimer in the
!    documentation and/or other materials provided with the distribution.
!
! 3. Neither the name of the Corporation nor the names of the
!    contributors may be used to endorse or promote products derived from
!    this software without specific prior written permission.
!
! THIS SOFTWARE IS PROVIDED BY SANDIA CORPORATION "AS IS" AND ANY
! EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
! IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
! PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SANDIA CORPORATION OR THE
! CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
! EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
! PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
! PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
! LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
! NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
! SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
!
! Questions? Contact 
!                    Andrey Prokopenko <email>
!                    Seth Johnson <email>
!                    Mitch Young <email>
!*********************************************************************

(and similar for C++ proxy?)

A meta-issue to collect all Tpetra wrapper issues.

• Export Trilinos Doxygen as XML
• Convert XML to SWIG .i file with doxy2swig
• Modify .i file as needed
• (in SWIG fortran) add support for %feature("autodoc")

This will be a little more difficult than the linear solvers that are currently in the TrilinosHandle class, since there is no abstraction layer like Stratimikos sitting on top of Anasazi. For a user to be able to select the type of SolverManager to use, we will need some scaffolding to handle that.

I guess another question would be how this will interact with the existing TrilinosHandle code. Should we modify TrilinosHandle to have Eigen solver capability, or should we make a separate class to handle Eigensolvers?

I tried to symlink these two repos into my trilinos/packages directory and added the -D Trilinos_ENABLE_CTrilinos=ON and -D Trilinos_ENABLE_ForTrilinos=ON to my configure script but the variables were not used by cmake.
Where can I find instructions on how to build these two packages?

Status:

• Blocked by swig-fortran/swig#34
• Blocked by #71 (ArrayView treatment)
• May want #83

Description
implement and test the new features planned for inclusion within Phase 2 ForTrilinos

Plan

• Implement test of IOC features
• Implement test of memory transfer

Criteria
Code that successfully demonstrates all the mechanisms required for full Phase 2 implementation

Status:

• Blocked by swig-fortran/swig#34
• Blocked by #71 (ArrayView treatment; ArrayRCP)
• May want #83
• May want #84
• May want #85

Description
implement and test the new features planned for inclusion within Phase 2 ForTrilinos

Plan

• Implement test of IOC features
• Implement test of memory transfer

Criteria
Complete informal summary of memory transfer mechanisms for Fortran/C++, share with project members

Nico Schlomer has a Mikado repo that provides a simple interface to Trilinos. I wonder if we can leverage some of that for simple drivers in ForTrilinos.

Description
Design the plan to allow the codes to have custom routines within Fortran to be called by the Trilinos C++ code

Plan

• Display and document what the Fortran codes have done in the past to perform custom callbacks from C++ interfaces to the Fortran
• Outline range of options to enable an automated IOC via ForTrilinos
• Decide and document the plan

Criteria
Code that successfully demonstrates the use of IOC on the target code within the ForTrilinos repository

Description
Assess performance of key kernels such as sparse matrix multiplication, multigrid smoothers on pre-Exascale test platforms.

Plan

• Assess the performance of the interface between Fortran and Trilinos
• Investigate bottlenecks associated with data transfers
• Run studies on Summit/Sierra platforms

Criteria
Informal report characterizing the performance, developing a strategy for improving the performance

Description

Plan

• Write necessary code to expand initial results from previous tests
• Apply tests to target code and verify they work

Criteria
Code committed to ForTrilinos master repository, write up phase 1 results for proceedings/journal

@sethrj @youngmit @bscollin

I did a quick attempt at using flang with ForTrilinos. It compiles the source just fine, but is getting stuck in tests with the following errors all over the place:

F90-S-0285-Source line too long

It seems that Fortran 2003 standard limits the number of characters in a line to 132 in the free form (and to 72 in fixed). It would be fine with continuation lines, but I think the macros in the test (like EXPECT_EQ) are assembled in a single line that exceeds the length.

My question is: is using long macros standard-compliant? Or is it just a feature that ifort/gfortran support?

To test #8, the driver should

• Assemble a simple linear system (say, 5-point or 7-point stencil) and rhs
• Pass it to the wrapper function
• Check the resulting solution

Depends on #10, #11.

@bmpersc @jwillenbring @trilinos/framework

In the ForTrilinos repo we have old branches that may have been migrated during the move to github but have nothing to do with the ForTrilinos repo:

• tribits_github_snapshot
• panzer_stk_conversion

Can we remove those?

The Fortran module files (.mod, for gfortran) are not being copied into the install path when using 'make install'. They should be copied into the include/ directory.

It is a good idea to have a set of different mini-codes (whatever we call them, proxy apps/mini-apps/micro-apps/examples) to test the functionality without requiring a full-blown codebase (like CAM/VERA/etc).

We can probably take some of the already written C++ mini-apps and convert them to Fortran. Some ideas:

• Tpetra tutorial examples
• Other examples in Trilinos (maybe Ifpack2? Belos?)
• Mini-app based on VERA
  Take output matrices from VERA to create a mini app of the eigenvalue solver. They have this already in C to test the trilinos solver before putting into VERA. The dumped matrix is available.
• Shallow water matrix from CAM to use for linear and nonlinear solver, IOC or use C++ examples from Albany CISM and CAM to compare kernels
• HPCG

Goal: unit testing and to look at performance of mini-app with Trilinos versus Fortrilinos example.

Description
Based on the needs of the target Fortran codes, determine the tools and processes necessary to create the capabilities within Phase 1 ForTrilinos

Plan

• Decide on C++ intrinsics that are needed to support the Fortran interface (e.g. functions, classes, inheritance, templates)
• Determine which Trilinos packages and classes are needed for initial ForTrilinos to support MPACT and CAM
• Assess feasibility of using PyTrilinos SWIG wrappers for construction of ForTrilinos SWIG interface files

Completion
Prioritized list of functionalities to provide as part of Phase 1 and 2

1. What is ForTrilinos license?
2. Are all files licensed the same?
3. Should ForTrilinos version 2 have the same license?

• Create v1 version branch
• Make clear in documentation and on front page README that the old v1 version is unstable and the master branch is for development
• Document the current testing status
• Make sure the test pass so that we are able to enable CI

Description
Develop, test, and release phase 3 of ForTrilinos, with the capability to maintain data structures across the application and Trilinos

Plan

• Develop initial features to be included in phase 3
• apply tests to target code
• finalize and release phase 3 of For Trilinos

Criteria
Release of ForTrilinos tag with phase 3 features to master

During CSE17 I was told that Tpetra is going away from the dynamic graph allocations. I think that for ForTrilinos this means that we should not port any functionality that does that. This would definitely include Tpetra::CrsGraph and Tpetra::CrsMatrix. Possibly something else. This would make the Tpetra's implementation quite lean. However, it is unclear to me at this point how we are going to provide access to Kokkos::CrsMatrix and such.

Right now, forteuchos.i looks something like this:

%module forteuchos
%include "file1.i"
%include "file2.i"

The problem with this is directives in file1.i are also applied to files2.i. For example, a general %ignore remove; in file1.i would also ignore all remove in file2.i.

This behaviour is unexpected, and wrong. We need to figure out the right way to approach things.

Currently, in developer mode, make will call swig to regenerate the wrapper files only if the file it has as SOURCE is updated. Otherwise, it does not.

To give a specific example, lets say we have forteuchos.i with the following content:

%module forteuchos
%include Teuchos_RCP.i

and our CMakeLists.txt has

  MAKE_SWIG(MODULE forteuchos
    SOURCE forteuchos.i
    EXTRASRC ${EXTRA_SOURCE}
    )

Any changes in Teuchos_RCP.i in this case will not be automatically rebuilt until forteuchos.i is touched.

Variants so far:

1. @sethrj started with DBCF
2. ForTrilinos v1 has unit test framework that reminds general Trilinos.

I'm leaning towards picking up pieces of v1 framework

Related to #70.

• Teuchos::Array
• Teuchos::ArrayRCP
• Teuchos::ArrayView

https://github.com/lanl/SNAP/tree/master/ports/snap-kf

While the eventual goal is to expose a significant portion of Trilinos functionality to Fortran applications, it is a good idea to expose the potential applications to the benefits of Trilinos as early as possible.

The idea is to provide a minimal interface (not existing in Trilinos proper at this point) which would solve a linear systems given few Fortran arrays with data corresponding to matrix and vector, a communicator, and a parameter list. Something like this (on C++ side):

int solve_linear_system(
  MPI_Comm comm,
  const int * row_ptrs, int num_rows,
  const int * col_inds, int num_values 
  const double * vals, int num_values,
  const double * rhs, int size,
  double * sol, int size,
  const ParameterList* list);

Not all sizes are necessary, but it makes it easier to convert from Fortran.

On the C++ side we wrap the data in the proper Trilinos classes (and constructing missing ones like Map), and pass it to Stratimikos.

Meta-issue to collect all Teuchos issues.

Description
Assess needs and plans for ForTrilinos

Plan

Completion
Create informal summary of plans for Phase 1, share with project members

Use DTK as an example.

So far, we've only been running tests within ForTrilinos build. We need to check that the result for make install and an external driver works. This will also provide a good example of how to link against ForTrilinos.

At the moment, the wrapper files in ForTrilinos source are regenerated only if .i file have changed. However, this breaks down if we simultaneously develop SWIG which updates the .swg typemaps and such. In my case, my ForTrilinos is configured to depend on swig in ~/local/opt/swig. I install it there manually through running make install in the swig directory. However, newly installed files are not picked up automatically thus wrapper files are not regenerated.

One workaround I can think of is to configure CMake to point to the sources of SWIG somehow.

@sethrj Can you share your workflow? I noticed that you did not run make install yourself.

Description

Plan

• Extend SWIG to create Fortran wrappers for C++ code with production-level reliability
• Create initial SWIG interface files to Trilinos packages
• Create basic suite of functions and tools to access Trilinos with Fortran calls
• Implement and test within MPACT
• Release as part of github repo of ForTrilinos

Criteria
Phase 1 is successfully implemented within a test case of the MPACT code. New ForTrilinos code shipped with the main Trilinos code. Pull request of SWIG/Fortran submitted to mainline SWIG development.

Find linear solver fortran code that we can target, perhaps the UTEP or MFIX guys have a code we could use. Note: COBRA, also an NE code we have access to, could be used, because it uses Futility interface to access trilinos linear solvers

The Inversion of Control in Futility is organized as follows.

User create some subroutine, for example

    SUBROUTINE testcalcResid() BIND(C)
      !Do some matvec
    ENDSUBROUTINE testcalcResid

This function is passed through a proxy by using C_FUNLOC:

call JFNK_Init(C_FUNLOC(testcalcResid), ...)

JFNK_Init has the following prototype:

    subroutine JFNK_Init(fptr,...) bind(C,NAME="JFNK_Init")
      IMPORT :: C_FUNPTR
      TYPE(C_FUNPTR),INTENT(IN),VALUE :: fptr
      ...
    endsubroutine

with a corresponding C interfaces:

extern "C" void JFNK_Init(int &id, void (*funptr)(), const int idx,
                          const int idF) {
    id = jfnk->new_data(funptr, evec->get_vec(idx), evec->get_vec(idF));
}

eventually ending in

class ModelEvaluator_JFNK : public NOX::Epetra::Interface::Required {
public:
ModelEvaluator_JFNK(void (*functionptr)(),Teuchos::RCP<Epetra_Vector> x, Teuchos::RCP<Epetra_Vector> F){
    fptr=functionptr;
    xloc=x;
    Floc=F;
}

...

bool computeF(const Epetra_Vector& x,
              Epetra_Vector& f,
              NOX::Epetra::Interface::Required::FillType)
    {
      // Residual calculation
      *xloc = x;
      fptr();
      f = *Floc;
      return true;
    }
private:
    void (*fptr)() = NULL;
    ...
};

We need to store generated files in the ForTrilinos directory, so that
the users don't need to build them themselves (they may not have
access to SWIG).

• Should we make it optional? Like a CMake switch between generating
  files, and storing them?
• MAKE_SWIG is useful for generating, but is not useful for storing
• How can we make sure that the updated files are being regenerated?

Would like to coordinate with PyTrilinos.