When running large simulations, writing the data into plotfiles
is extremely fast, but reading the data, either particle data or grid data, can be time-consuming. One reason is that the data is spread in multiple boxes, to be able to account for mesh refinement. For many application, even when the simulation runs with mesh refinement, it would be useful to have a lower-resolution output using the OpenPMD standard, and if possible the hdf5 format, that would be easier and faster to read.
Furthermore, plotfiles
can be massive as all grid quantities and all particles of all species are written to disk. Having the possibility to chose the data written to plotfiles
would also be helpful.
At the end of a simulation, two folders would be created: plotfiles
and diags
(this name is arbitrary). Here are the details of what this issue proposes for each of these output:
plotfiles output
Currently, all grid data and all particle data are dumped, so that plotfiles
contain grid quantities Ex
, Ey
, Ez
, Bx
, By
, Bz
, jx
, jy
, jz
and particle quantities particle_Ex
, particle_Ey
, particle_Ez
, particle_Bx
, particle_By
, particle_Bz
, particle_momentum_x
, particle_momentum_y
, particle_momentum_z
, particle_position_x
, particle_position_y
, particle_position_z
, particle_cpu
, particle_id
, particle_xold
, particle_yold
, particle_zold
, particle_uxold
, particle_uyold
, particle_uzold
and particle_weight
for all species. It would be useful if the user could specifies the species to dump and which particle and field quantities to dump, e.g. with
amr.plot_fields = Ex, Ez, By, jz
amr.plot_species = electron ion
amr.plot_particles = particle_position_z, particle_momentum_uz
Output compliant with the OpenPMD standard
The simulation directory would contain a second output directory, say diags
, that contains a smaller amount of pre-processed OpenPMD-compliant output. Two types of output are considered below for a 3d simulation: sub-sampled arbitrary-dimension grid data and filtered particle data. The user would specify the name of each diagnostics with
warpx.diagnostics = dump3d line particle_dump
dump3d.type = grid_diagnostics
line.type = grid_diagnostics
particle_dump.type = particle_diagnostics
so that a user could define several grid_diagnostics
objects, etc. Here is a more detailed description of each of these:
grid_diagnostics
The user specifies the fields and the output number of points as well as the boundaries of the data to dump, for instance
dump3d.fields = Ex, Ez, By
dump3d.n_cell = 128 64 64 # write 128 points in x and 64 in y and z.
dump3d.lo = -5.e-6 -5.e-5 -30.e-6
dump3d.hi = 5.e-6 5.e-5 120.e-6
for a 3d grid or
line.fields = Ex, Ez, By
line.n_cell = 1 1 64 # write 128 points in x and 64 in y and z.
line.lo = 0 0 -30.e-6
line.hi = 0 0 120.e-6
for a 1d line. The same syntax would also allow 2d slices.
Particle filters
The user specifies the species and can chose a simple filter, for instance
particle_dump.species = beam
particle_dump.filter = "uz > 10."
Note that the specific options to enter in an input file are still to be determined, as they should comply with the PICMI standard, so the choice proposed above is not permanent.
The user may want to dump small data with high frequency (for instance retrieving a line-out at every time), which will probably require buffering in order not to ruin the simulation performances.
Future improvements
- Allow
grid_diagnostics
to return data on a grid of arbitrary dimension that is not aligned with the simulation box. For instance, a user may want to return the Ex
field on a line that goes through (0, 0, 0) with direction (1, 0, 1).
- Allow
grid_diagnostics
to return data on a set of points that are not aligned, for instance on a sphere or the arc of a circle.