cryotools / osaris Goto Github PK

Cut the input topo DEM to the maximum x vs y extent of the input SAR scenes.

Add a module that sends minimal 'ping' batch jobs to the queue to wake up sleeping nodes.

Implement unit testing, e.g. with shUnit2:
https://github.com/kward/shunit2

Delete files used solely during processing. Option to also delete extracted S1 scenes.

Implement an alternative processing concept that makes interferograms for vv and vh polarizations of the same scene.

Whole interferograms are occasionally marked as invalid owing to full-scene shifts in unwrapping of either fwd or rev pairs. To avoid this, set both fwd and rev interferograms to zero at PSI coords when these exist.

Add option to cleanup the folders to save disk space: - remove Processing/raw/ - remove Processing/orig/

Templates are outdated :/

Goal: determine regions of (a) general high stability, (b) general instability, and (c) stable regions with instable events.

• Create pairwise coherence diffs
• Calculate mean coherence and variance throughout the timeseries for each pixel
• From variance, identify region

Combine analysis of coherence/coherence diffs with l.o.s. displacement (and maybe others?).
Extract subareas (e.g. glaciers + buffer) and analyze these in detail.

The current concept in which all config files are stored in one directory can be very inconvenient when working on multiple projects at the same time. Specifically, module config files need to be changed each time when a new run for a different study area is started, even though the configuration for individual modules will remain the same for each study area in most cases.
Therefore, the module config files should be saved to subdirectories of $OSARIS_PATH/config and the path to this module config directory should be a parameter in the main config file of a project.
This will require to modify the check for config files in all modules.

After interferogram generation, all swaths should be merged, so that only one set of files per scene pair is stored in the output directory.

GMTSAR's merge_swath should do the job:

USAGE: merge_swath inputlist output [stem]
inputlist example:
F1/intf/2015036_2015060/S1A_20150609.PRM:F1/intf/2015036_2015060/phasefilt.grd
F2/intf/2015036_2015060/S1A_20150609.PRM:F2/intf/2015036_2015060/phasefilt.grd
F3/intf/2015036_2015060/S1A_20150609.PRM:F3/intf/2015036_2015060/phasefilt.grd
note: use the slave PRM which contains the shift information
output: output.grd [stem.PRM]
note: please put the files to stem.in in the order of swath numbers.
make sure all images have same num_rng_bin

Subsequently, the output grids should be cut to an AOI defined in the config file to avoid huge files.

Add other sensors. Minimum TSX, optimum all from GMTSAR website.

In the current concept, UCM detects only locations where coherence dropped from high to low values. However, it may also be interesting to detect locations where coherence substantially increased. For instance, material displaced by a rockfall event would lead to a substantial increase in coherence if deposited in a place where low coherence prevailed to this point, such as a forest.

This may be optionally activated in the config file. Increasing coherence may be indicated by negative values in the resulting file, so that UCM results would have a value range from -1 to +1.

Process coherence differences in parallel.

Use GMT to crop output files by lat/lon min/max coordinates.

Since the last updates cohrerence diff results consist only of 0 and NaN values. The shape and projection is, however, still correct.

This will require a modification of the Processing and Output directory structure.

The orbit and slice of each scene can be obtained from its manifest.safe:
[...] <safe:relativeOrbitNumber type="start">107</safe:relativeOrbitNumber> [...] <s1sarl1:sliceNumber>4</s1sarl1:sliceNumber> [...]

It should be possible to remove bursts which are not relevant for processing. In the config file a range of bursts to consider in processing should be included, parallel to the selection of swaths.

Incorporate pyAPS or, if possible, a native GMTSAR solution.

Make the output directory structure more convenient.

/Pairs-forward

• Amplitudes
  • preview (png, kml, ps)
• Coherences
  • preview (png, kml, ps)
• Interferograms
  • preview (png, kml, ps)
• Interferograms-unwrapped
  • preview (png, kml, ps)
• LOS-displacements
  • preview (png, kml, ps)
• ...

/Harmonized-interferograms
/Stable-ground-point
...

Radar coordinate data does not need to be copied to the output directory.

In the prepare-data.sh routine, the swath relevant for processing the area of interest should be determined automatically. This will render the manual input in the config file obsolete.
Envisioned work flow:

• Read the orbit direction from S1 annotation XML file:
  grep 'pass' s1a___.xml | tr '<' ' ' | tr '>' ' ' | awk '{print $2}'
• Check if range values are out beyond range range (i.e. < 0, > max) using SAT_llt2rat for each swath
• Add all swaths necessary to process the whole AOI

Add a new processing mode that uses only one scene as master and calculates a time series of deviations from the master scene. Identify potential to resolve permanent scatteres.

Add the option to create a summary file after finishing the processing which summarizes the results. The summary file should contain for each pair: - Intf preview - Cohernece preview - Unwrapping forward/reverse difference [opt, when processed] - Days between scenes - Some basic statistics

These info should be arranged horizontally, one row for each pair.

In addition, coherence differences between two pairs may be added between two rows if these were processed as well.

Basic output should be HTML, optionally convert to PDF.

Grid harmonization must be applied at several stages and to different datasets in many workflows, e.g. when using the GACOS module. There, it would be more convenient to transform the module into a generic script that takes input and output directory as well as reference point location as input.

Create a generic script to mask GMT grid files by a polygon provided in a .gmt or .shp file.
Parameters:

• Input grd file
• Polygon file
• Mask inside / mask outside
• Output grd file
• Buffer (optional, default 0)

Use GMT grdmask to create a mask grid. Subsequently, use GMT grdmath with the 'INSIDE' operator to set data points inside/outside the mask area to NaN.
Shapefiles should be feasible input polygon files as well and automatically converted to .gmt files with to GDAL ogr2ogr (cf. routine in module 'Create Summary Report').

The current orbit matching routine may return false positives when the orbit library is incomplete. An additional check regarding the end date of the orbit file is required to verify that the scene is actually covered by the orbit data.

Add hooks to include modules after each processing step.
Prepare a common design/template for modules that facilitates as much automation of module integration as possible.
In the config file, the activation of a module should be limited to one parameter, e.g. 'acitvate_change_analysis=1".

Add a generic script to mask grid files by a threshold.
Parameters should be

• Input file
• Threshold value
• Operator (GE, GT, EQ, LE, LT)
• Output file

Use GMT grdmath to set all values higher/equal/lower than the threshold to NaN.

All output grid filenames shall have the dates in the first part, followed by a sequence of identifiers for processing steps applied to it in the form
YYYYMMDD--YYYYMMDD-corr_ll-module1-module2.grd

The following SBAS call ...
xdim=$(gmt grdinfo -C unwrap.grd | awk '{print $10}') ydim=$(gmt grdinfo -C unwrap.grd | awk '{print $11}') n_int=$(wc -l < intf.tab) n_scn=$(wc -l < scene.tab) sbas intf.tab scene.tab $n_int $n_scn $xdim $ydim -smooth 1.0 -wavelength 0.0554658 -incidence 30 -range 800184.946186 -rms -dem
results in this error:

mode --> 2

compute RMS misfit
compute DEM error

setting smoothing to 1.000
setting radar wavelength to 0.055 m
setting radar incidence angle to 30.000 degree
setting range to 800184.946 m
Memory Allocation Successful...
read table file ...
number of SAR scenes is 4
number of interferograms is 4
read phase and correlation grids ...
*** buffer overflow detected ***: sbas terminated
======= Backtrace: =========
/usr/lib64/libc.so.6(__fortify_fail+0x37)[0x2ac567943077]
/usr/lib64/libc.so.6(+0x10d230)[0x2ac567941230]
/usr/lib64/libgmt.so.5(gmt_grd_get_format+0x4d8)[0x2ac5639e57b8]
/usr/lib64/libgmt.so.5(gmtlib_read_grd_info+0x1d)[0x2ac5639e5a9d]
/usr/lib64/libgmt.so.5(+0x4ecfc)[0x2ac5639bacfc]
/usr/lib64/libgmt.so.5(GMT_Read_Data+0x203)[0x2ac5639bb513]
sbas[0x404c4e]
sbas[0x401cba]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac567855b35]
sbas[0x4016d9]
======= Memory map: ========
[...]

This will require a list of available orbits from which the files actually needed can be determined. Only these orbits shall then be downloaded and used for further processing.

Will substantially reduce downloads and required HD capacity in cases where only few specific scenes are processed.

Make a SLURM job for data preparation instead if processing on the login node. - Add specific SLURM configuration options (1 job, short processing, much ressources) - Use check_queue.sh to monitor job status - Start subsequent processing steps when finished

Envisioned work flow:

• All scenes for the area and time frame of interest are downloaded (no restriction by relative orbit in config)
• All scenes are extracted to Processing/orig
• The prepare-data.sh routine identifies the pass direction and creates lists of asc/dsc files
• Files for each pass direction are processed separately (raw-asc, raw-dsc)
• Results are brought back together in the Output directory where each pair is identified by date

In the current development version 0.7.3aplha, processing jobs that cover multiple swaths show offsets in the geocoded result datasets.