ParLinearForm does not match with local vector and HypreParMatrix about mfem HOT 4 CLOSED

The differences you're encountering are because of the differences between "local vectors" (L-vectors) and "true vectors" (T-vectors).

A (Par)GridFunction is a "local vector" — it contains all the degrees of freedom for all the elements in the mesh partition. This means that degrees of freedom that lie on the boundary of the mesh partitions are shared (duplicated) among the MPI ranks of those partitions.

"True vectors" are vectors that contain only true DOFs (TDOFs). These are de-duplicated: every true DOF belongs to exactly one MPI rank.

When you call FormSystemMatrix, the resulting matrix will act on true DOFs, so the dimensions will be less than the dimensions of the grid functions or linear forms.

To get a true DOF vector corresponding to a ParLinearForm, call the function ParLinearForm::ParallelAssemble.

To extract a true DOF vector corresponding to a ParGridFunction, call the function ParGridFunction::ParallelProject.

To covert a true DOF vector into a ParGridFunction, call the function ParGridFunction::Distribute (or ParGridFunction::SetFromTrueDofs — they are synonymous).

This diagram might help explain things:

There is also some documentation here.

from mfem.

It's clear now. Thank you @pazner .

from mfem.

Thank you @pazner, my current source is correct now with ParallelAssemble(). I have 2 last questions.

1. Is there something special about HypreParVector? I assume it's same as a Vector with true dofs size.
2. ParallelProject and ParallelAssemble calls
   pfes->GetRestrictionMatrix()->Mult() and pfes->GetProlongationMatrix()->Mult()Transpose, respectively. Are they
   equivalent?

from mfem.

1. Is there something special about HypreParVector? I assume it's same as a Vector with true dofs size.

HypreParVector is a Vector with size equal to the number of true DOFs. It also wraps a hypre_ParVector, which is an object from the hypre library that allows it to easily interface with hypre matrices and preconditioners. In most MFEM code, you can use Vector instead of HypreParVector.

2. ParallelProject and ParallelAssemble calls
   pfes->GetRestrictionMatrix()->Mult() and pfes->GetProlongationMatrix()->Mult()Transpose, respectively. Are they
   equivalent?

ParallelAssemble will perform assembly, in other words it will add contributions from shared processors. This is the right thing to do for dual vectors like linear forms. ParallelProject will restrict the vector, meaning it will just take one of the shared values (if the code is consistent, all of the shared values will be equal, and it won't matter which one you take).

from mfem.