fmihpc / vlsv Goto Github PK

Currently reader does not care what the integer value is, only 0 and 1 are legal.

It would be nice to have a tool plotting the memory layout of vlsv files. Since everything is already in an xml tree, I suppose that marching through it with some python xml functions would be easy, retrieving the start bytes of all fields and plotting them in order.

The use case we would have now, in Vlasiator, would be to check easily how much space is used for real-space fields vs. velocity space saved every n cells, and estimate how the file size would grow when going to n/2 spacing. We can do it by hand looking at data size * vector size * entries, but a quick graphical tool would be cool.

Issue submitted for the record, no hurry at all.

In the current form, the multiRead with offsets I had tried to deploy along with Vlasiator still only does one multiReadUnit at a time, and that works though defeating the point of trying to mash the units into one read.

https://github.com/fmihpc/vlsv/blob/master/vlsv_reader_parallel.cpp#L380-L394

Here I think there is a mixup of pointers to the ->array members of the multi read units which point to the buffer allocated in user code to get the data read, and pointer offsets that should map to the offsets we compute at great length in the user code. I don't think that the datatype we make with MPI_Type_create_struct should be getting pointers to the destination buffers. multiReadOffsetPointer should be passed for the destination buffers at https://github.com/fmihpc/vlsv/blob/master/vlsv_reader_parallel.cpp#L399, that I agree with.

Posting this for posterity, we won't address this right away in relation to Vlasiator.

I get this when compiling vlsv with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn):

In file included from vlsv_common.cpp:23:
./vlsv_common.h:204:99: error: redefinition of default argument
   void convertValue(T& value,const char* const buffer,datatype::type dt,int dataSize,const bool& swapEndianness=false) {
                                                                                                  ^              ~~~~~
./vlsv_common.h:140:117: note: previous definition is here
  ...T> void convertValue(T& value,const char* const ptr,datatype::type dt,int dataSize,const bool& swapEndianness=false);
                                                                                                    ^              ~~~~~
1 error generated.
make: *** [vlsv_common.o] Error 1

Even though both arguments are the same I think only one of them should have a default.

Excuse me, with the super computer, when running this code with a small number (less than 40) of cores (the number of 'np' following the command 'mpirun'), everything is working fine. However, once I use a larger number of cores, the program reports insufficient memory error.

In large simulations close() is quite slow, on Hornet it used 45% of total write time (2.9GB vlsv file, 200 nodes).

It appears that MPI reads and writes can fail due to integer overflows, i.e., when the number of bytes read/written by process exceed numeric_limits::max(), which is about 2.15 GB.

Specifically, I tested on armv7l (raspberryPi 3), and got:

mpic++ -O3 -std=c++0x -Wall -fPIC  -c vlsv_common_mpi.cpp
vlsv_common_mpi.cpp: In function ‘ompi_datatype_t* vlsv::getMPIDatatype(datatype::type, uint64_t)’:
vlsv_common_mpi.cpp:159:11: error: duplicate case value
  159 |           case (sizeof(long double)):
      |           ^~~~
vlsv_common_mpi.cpp:155:11: note: previously used here
  155 |           case (sizeof(double)):
      |           ^~~~
make: *** [Makefile:67: vlsv_common_mpi.o] Error 1

Not sure if this is something that needs a specific fix though. This issue is mostly to document it, if anybody stumbles into this in the future. Most likely, it can be worked around by simply commenting out the long double case, as it is not used for any actual application AFAIK.

I tried to get a spreadsheet of the numerical values of a subset of cells. (Say variable is CellID, and I add a threshold(fSaved == 1) and a box selection operators.)

With a vlsv file I only get some values. With a silo files I get the proper list. So there is some form of problem guess?

Corsair restarts were failing because multi IO units in VLSV were returning incorrect byte size for particles.

Correct me if I am wrong, but if the buffer size is set to the maximum of 2G, then this line will overflow and misbehave if the buffer is just below the limit and if bufferTop + size exceeds 2G, as all of these are int.
https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L788

Isn't it?

I was trying to run with TAU and got crashes in MPI_Comm_free with invalid communicator. I found two here:
https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L60 (destructor)
https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L196 (close())
and disabling the latter allows it to run.

What's better style? Dropping this one or the destructor one? Or should we dig deeper to make sure there's no other obscure issues involved?

After a few cases of waiting for Godot in production runs where a restart file was not complete, I'd suggest that some simple test be added to check that the xml footer is present at the end of the file. When trying to open a truncated file with Vlasiator, it just hangs. I suppose that just looking whether the file is long enough or whether the last bytes correspond to the end of the xml footer is not too difficult to add?

@sandroos, if you don't have time to implement it, let us know if you have hints and we can look where/how to implement that.

When setBuffer() is not being called, outputBuffer is a NULL pointer as initialised in the constructor. https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L53 It seems that this situation leads to malloc probblems/OOM crashes in Vlasiator in some cases when it tries to empty the buffer before writing at https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L549.

When setBuffer() is called with an argument of 0, which should behave similarly as the default behaviour when the function is not called when looking at the rest of the code, what happens is that outputBuffer is set as a well-behaving zero-length char array https://github.com/fmihpc/vlsv/blob/master/vlsv_writer.cpp#L830. This does not lead to the crashes otherwise detected.

Now looking at the top of setBuffer(), if the user calls setBuffer() a second time and there is data left in the buffer, it will be silently deallocated and a new one will be allocated. Should this call emptyBuffer() first? Or should the user simply be prevented from calling setBuffer() multiple times, and then the delete is not needed any more?

Until this is solved we recommend to call setBuffer(0) or with the desired buffer size once and only once before any writing sequence.

Anyone else having issues running Visit VLSV plugin on Ubuntu 16.04.1 LTS?

After plugin compilation launching Visit (versions tried: 2.8.2, 2.10.2 and 2.12.0) results in the error message:

Error: The metadata server on host localhost could not be launched or it could not connect back to your local computer. This can happen for a variety of reasons.

The specific error was: "The reason for the exception was not described"

Study if the performance of VLSV can be improved for post-processing. Currently ~5-10 GB VLSV files are not feasible to be analyzed on laptop computers. This may not be due to a RAM memory limit but could be more an issue with the performance of the VLSV reader and the VLSV Visit plugin. For example, could stored cells be sorted for faster access by a separate post-processing tool?

Reader opens the input file in formatted mode in
https://github.com/fmihpc/vlsv/blob/master/vlsv_reader.cpp#L178
which seems to work just fine in every Linux system VLSV has been run on.

I encountered weird issues on Windows 10, however, and I was able to trace it here. Apparently Visual Studio really performs character formatting (unlike Linux) and file can get messed up. The solution is to open the file in binary (unformatted) mode.

filein.open(fnameWithoutPath.c_str(), fstream::in | fstream::binary);

I have in recent months tried to (re)compile the VLSV plugin with newer VisIt versions (2.12.1 at the moment). It generally works (although I sometimes have to link some system MPI libraries as it won't find them properly). However VisIt's mdserver dies when launching VisIt, which it doesn't do when I remove the Vlsv plugin files from the corresponding directory.

I have two theories, either the compatibility of the plugin with 16.04 linuxes is gone, or the VisIt plugin interface has been somewhat modified.

I have next to no clue about the VisIt plugin things, that's why I post it here. @ursg you have more experience so maybe you can look at some point. I understand that @sandroos you don't have a linux available for testing at the moment, do you? In any case, hints are welcome as well.

It is not a big hindrance as I mostly use the plugin on remote engines (taito.csc.fi) and there it works properly, somehow...

I get these when compiling vlsv_reader_parallel.cpp with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn):

In file included from vlsv_reader_parallel.cpp:24:
./vlsv_reader_parallel.h:38:12: warning: 'vlsv::ParallelReader::getArrayInfo' hides overloaded virtual function [-Woverloaded-virtual]
      bool getArrayInfo(const std::string& tagName,const std::list<std::pair<std::string,std::string> >& attribs,
           ^
./vlsv_reader.h:40:20: note: hidden overloaded virtual function 'vlsv::Reader::getArrayInfo' declared here: different qualifiers
      (const vs none)
      virtual bool getArrayInfo(const std::string& tagName,const std::list<std::pair<std::string,std::string> >& attribs,
                   ^
In file included from vlsv_reader_parallel.cpp:24:
./vlsv_reader_parallel.h:45:12: warning: 'vlsv::ParallelReader::open' hides overloaded virtual function [-Woverloaded-virtual]
      bool open(const std::string& fname,MPI_Comm comm,const int& masterRank,MPI_Info mpiInfo=MPI_INFO_NULL);
           ^
./vlsv_reader.h:45:20: note: hidden overloaded virtual function 'vlsv::Reader::open' declared here: different number of parameters
      (1 vs 4)
      virtual bool open(const std::string& fname);
                   ^
In file included from vlsv_reader_parallel.cpp:24:
./vlsv_reader_parallel.h:74:12: warning: 'vlsv::ParallelReader::getArrayInfo' hides overloaded virtual function [-Woverloaded-virtual]
      bool getArrayInfo(const std::string& tagName,const std::list<std::pair<std::string,std::string> >& attribs);
           ^
./vlsv_reader.h:40:20: note: hidden overloaded virtual function 'vlsv::Reader::getArrayInfo' declared here: different number of
      parameters (6 vs 2)
      virtual bool getArrayInfo(const std::string& tagName,const std::list<std::pair<std::string,std::string> >& attribs,
                   ^

I'm not sure how big of a problem these could be but there's probably a reason why clang warns about them.

Many vlsv functions take string parameters, for example, which are not tested for consistency currently. It is possible that these variables have different values at different processes due to user errors. One possibility to (almost) verify string parameters is to compare their hashes.