noaa-owp / inundation-mapping Goto Github PK

There exist two branches that previously explored different configurations and input datasets especially how they relate to the initial stream definitions that are hydro-enforced and whether or not they are seeded to re-derive a new network. This task involves evaluating the value of bringing these branches forward to be compatible with the latest dev commit.

1. foss-fim-nwm-streams: Hydro-enforces (burns) NWM channels and not the NHD+HR subset. For demDerived_reaches.shp, the NWM headwater points are directly seeded instead of the current points that are NWM points snapped to nearest NHD+HR line. This may have value because it reduces the dependency on NHD+HR channel lines and the VAA. So less input data storage requirements and less processing requirements to subset these files to the NWM stream density.
2. foss-fim-no-seeding-config: Still hydro-enforces (burns) the NHD+HR subset but does not seed so does not use streamnet. Involves less processing and less intermediary data. Streamnet is currently the memory bottleneck due to it's watershed feature which is not required.

There may also be value in considering a third branch that is like foss-fim-nwm-streams where it burns the NWM channels but then doesn't seed.

If value is seen in bringing these branches/configuration methods forward, we can include in testing.

Large pixel catchments with outlet points much further downstream than expected causing False Negatives (misses) in resultant mapping. In HUC 111303 (NHDPlusBurnLineEvent_subset: NHDPlusID = 21000300092604), it has been observed that there are large pixel catchments which have outlet locations that appear to excessively further downstream than they should be. This causes the reference elevation (lowest elevation) used in the HAND/REM creation for each pixel catchment to be lower and ultimately reducing the potential mapping due to high HAND/REM values. FIM 2 incorporates an AGREE stream burn in method that includes a shallow then steep slope. FIM 2 pixel catchments appear to have more realistic outlet points along the stream which ultimately is reflected in more realistic mapping. Incorporation of the AGREE (or similar) method may help improve mapping by allowing pixel outlet points to occur in more realistic locations along a stream.

Pixel catchment boundaries (FIM 2 vs FIM 3) note where the outlet points are

Here is the FIM 3 mapping compared to benchmark mapping

For reference output inundation mapping FIM 2 compared to FIM 3

Create deny and allow template files within config dir that allows users to remove or keep specific fim_run outputs. A new module should be written for this and located within fim_run.sh after the GNU parallel steps.

Explore solutions to account for known issue of missing channel bathymetry. This may include options to incorporate observed bathy datasets and/or estimating the channel geometry parameters that can help approximate the bathymetric component not represented in the DEM.

• Establish priorities for implementing bathy solution (observed vs. approximation)
• Take inventory of existing channel geometry datasets and estimation techniques
• Chart out solution pathways: merge obs bathy, estimation - burn DEM, estimation - modify rating curve
• Create mockup of prototype approach(s)
• Develop new code function in run_by_unit/add_crosswalk.py workflow to integrate bathy estimation technique

The headwater stream features currently outputted start further upstream when compared to FIM 2.X FPRiver layer. This is likely due to the behavior in subset_vector_layers.py and specifically how the NHD Headwaters are derived on lines 74-89. Please use the best available eval data before committing to a change. For the 121002 and 120903 test cases, this was seen as one of the significant sources of FPs (over-prediction).

Rasterio can now perform all of the functions of raster.py. This goal of this task is to replace usage of raster.py in the FIM workflow with rasterio methods.

Because leveed areas are now included in the masked area, it is necessary to recache metrics for previous versions of FIM so version-on-version performance changes are better tracked.

Looking at RnR FIM along Pearl River upstream of Bogalusa, Louisiana as seen in screenshot. Although RnR uses MS configuration, looks like FIM is being restricted in parts due to small catchments along Pearl River. Wondering why MS configuration in this area still generates small catchments.

Current behavior

Small catchments along Pearl River even though MS configuration is implemented.

Expected behavior

Larger catchments along Pearl River to facilitate flooding over larger area.

Steps to replicate behavior (include URLs)

RnR FIM Inundation: rfc/rfc_replace_route_5daymax_inundation service on prod GIS Server
FIM v2.X MS Catchments: reference/rnr_catchment_boundary service on dev GIS Server

Screenshots

Current behavior

REM creation in FIM 3 appears to assign the minimum elevation within a pixel catchment as the reference elevation. It does not appear to have a check to ensure that the minimum elevation is along the stream centerline. One result is that inundation maps can show channels as dry despite other areas in the catchment being shown as inundated.

Example: It has been observed in HUC 111303 where a reference elevation is picked that is far away from the stream centerline and the REM value along the stream centerline is 40 m.

Expected behavior

The FIM2 HAND reference elevations are along channels. Areas away from the channel that may be lower than the reference elevation are assigned a 0 HAND value. FIM maps should include the river as part of the inundated area.

Steps to replicate behavior

1. run fim_run.sh

(Will add Google slides presentation)

Include docstrings in code logic in rem.py. Docstrings should explain rationale behind code blocks that perform hydrologic conditioning.

Three HUC8's: 04240001 (Alberta, Canada), 10170104 (Ontario, Canada), and 09040003 (South Dakota & Nebraska border, USA) where non-zero exit codes are observed due to no NHDPlusHR burn lines. HUC8 boundaries in red and the burnlines for the corresponding HUC4's shown in black below. Completion of this issue ensures that a FIM domain run obtains no non-zero exit codes as of dev commit ID 891993b.

Current behavior

No check is done for complete NHDPlusHR dataset availability. If the URL for the HUC4 is available, then that data is processed and added to included_huc*.lst files.

Expected behavior

Some geometry check or attribute query should be down in the WBD preprocessing step to ensure complete NHDPlusHR data availability. A simpler and less elegant method would be to hard-code these three HUCs out of the WBD_National.gpkg and included_huc*.lst files at preprocessing.

Steps to replicate behavior (include URLs)

1. acquire_and_preprocess.py the relevant HUC4s
2. Run fim_run.sh for the three HUC8s

Screenshots

Dry-Run Repo Changes and Transition to Github
Dry-run the repo changes and transition to github
Changes will be published to a new repo named "cahaba" with a TBD description.

** Actions to Script For FIM Transition to GitHub **

• filter out all foss_fim dirs from entire history
  • applies to HAND, fldpln, and VlabRepoQuickTips.txt, and .gitmodules.
  • there maybe others or the above may have been renamed but ive only seen this in branches to be deleted
  • leave .gitignore and .gitattributes
• move contents of foss_fim dir to root level of repo
  • remove empty foss_fim dir
• delete the following branches
  • master, dev, foss-fim-master
• change foss-fim-development to dev
  • set as default branch
• change all remaining branches to dev-<existing_branch_name>
  • exclude any reference to foss-fim-development, foss-fim, or ffd in the <existing_branch_name>
  • example: ffd-inundation -> dev-inundation
  • example: foss-fim-batch-processing -> dev-batch-processing
  • example: MSflows -> dev-MSflows
• change info and email (sorry, just trying to scrub my personal stuff)
  • username: fernandoa123 -> fernando-aristizabal
  • username: fernando.aristizabal -> fernando-aristizabal
  • email: [email protected] -> [email protected]
  • https://docs.github.com/en/github/using-git/changing-author-info
• make dev a protected branch (and main too if we really need one now)
• user permissions
  • read = all members of owp org
  • write = all members of dev team (brad, brian, trevor, ryan)
  • maintain = fernando s
  • admin = fernando a and Nick C

Provide analysis and solutions for possible biases in the reach averaged parameters that feed in manning's equation. Currently the curve is biased down/left.

New prod image with only fim_run dependencies.

In order to test the performance of levee handling in fim_run.sh, a levees inclusion area is required in that "additional_layers" directory for each relevant test_case.

A -l option will be added to run_test_case.py so that users can specify metrics to be produced at leveed areas.

Subtask to #24

Investigate that the thalweg creation in tauDEM flowdircond tool is using correct input channel geometry. The NHDPlusBurnLineEvent_subset layer, which has data gaps (#13) is used to create the flows_grid_boolean.tif (Rasterize Reach Boolean step in run_by_unit.sh). The flows_grid_boolean.tif is then used to create the flowdir_d8_burned_filled_flows.tif (Mask Burned DEM for Streams Only step in run_by_unit.sh). The flowdir_d8_burned_filled_flows.tif is then used in the tauDEM flowdircond tool (Flow Condition Streams step in run_by_unit.sh), which creates a monotonically decreasing thalweg, where it outputs a dem called dem_thalwegCond.tif.

As the input into the flowdircond tool is ultimately based on the NHDPlusBurnLineEvent_subset, which has gaps, investigate that the monotonically decreasing thalweg is behaving as desired. If not, investigate using an alternative stream layer, such as the DEM derived streams.

For example a screen shot of flowdir_d8_burned_filled_flows.tif in huc 111303 (near NHDPlusID 21000300138039)

I noticed references to the pysheds package in several documents/slide decks. I could see some benefit in consolidating code to python, but this would require some testing, perhaps some PR to the project, and modifications to run_by_unit.sh (most likely a port from BASH to python.)

Occasionally gdalbuildvrt is overriding data pixels with nodata values during VRT mosaicing. Happens during fim_run aggregation and inundation aggregation (currently disabled)

NHD+HR elev_cm.tif and the WBD's extend into the ocean (at the very least they do in Puerto Rico). We should consider finding a coastline poly (might be in NHD+HR already) and intersecting that with the job's WBD to use for clipping the rem,catchments, reaches, etc in run_by_unit.sh. This would avoid generating inundation in ocean areas that are in catchments corresponding to NWM forecasts.

In support of FIM v4.X looking for a real example of overlapping depth grids from RnR FIM forecast along MS and NWM FIM forecast. Thinking one of the HUCs in the FL panhandle would be a good example in the area we were assessing with WPOD.

Would it be possible to generate this within a few days?

Please send output to @fernando-aristizabal and @RyanMcCarthy-NOAA.

Current behavior

Expected behavior

Steps to replicate behavior (include URLs)

Screenshots

Incorporate flow divides caused by levees. Consider levee datasets and/or existing DEM's.

run_by_unit.sh is well structured for modularity, however there is an opportunity to improve the documentation of each step in the process. For example, documentation may include an explanation of 1) What inputs go into each step and why, 2) What processes each step undergoes and why, 3) The outputs from each step, why they are required for the next step, whether or not they are crucial to save long-term or if they are intermediary outputs.

Enable MaaS framework to work with FIM 3+

This is a subtask of #4. The boundaries of the WBD dataset do not represent the extent of the rem, slope, and catchment layers.

Current behavior

Catchment layers on the outer boundary of the unit HUC are being clipped out in run_by_unit.sh. This creates gaps between HUCs of the same unit size. This also causes inconsistency in area between different HUC units.

Expected behavior

The catchments can be subset using unique HydroIDs instead and the resulting catchment boundary can be used to define the raster extents.

Steps to replicate behavior (include URLs)

Screenshots

Considering this repo will be open to the public, I think PEP 8 styling would be appropriate. Black makes this easy. There are some pitfalls when using git blame after the change, but there are workarounds.

This is simple to do, but it would touch all python code in the project. So it has a pretty big impact to the repo.

Missing features in NHDPlusHR burnline inputs possibly causing areas with no inundation (FNs). Investigate and resolve by possibly changing the input source. Correct for new input within preprocessing module.

Integrate Dockerhub with github to build new images when a certain action is completed.

Manually tune existing parameters in config/params_template.env to find optimal combinations for minimum version improvement.

Investigate utility of JTTI manning's n estimates and incorporate support for this dataset if useful.

Add functionality to ignore levee protected areas in the evaluation metrics

##########################################################################
Clip WBD8 ##########################################################################
Thu Aug 13 12:45:10 UTC 2020
Warning 1: A geometry of type POLYGON is inserted into layer WBDHU8 of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.

Explore options and implement a system to version artifacts from the various end-user functions.

Something like this open-source workflow management system would be really nice:
https://dvc.org/

Combination of FR and MS configuration datasets enable composite FIMs and mitigate impact of catchment boundary issues along river segments downstream of RFC forecast points.

Current behavior

Workflow enables FIM along all NWM modeled river segments

Expected behavior

User specifies in config or at run time whether workflow should generate FIM datasets for all NWM modeled river segments (i.e. FR), river segments downstream of RFC forecast points (i.e. MS) or both. Catchments generated from MS configuration generally larger than FR equivalent and enables inundation across larger area. Geometry of river segments for FR and MS configurations downstream of RFC forecast points are the same.

More of an idea than an issue. I've noticed a few data pulls from S3. It might be worth it to read archived files directly from S3 using s3fs and py7zr. For example:

import s3fs
import py7zr

fs = s3fs.S3FileSystem(anon=True)
s3_path = 's3://prd-tnm/StagedProducts/Hydrography/NHDPlusHR/Beta/GDB/NHDPLUS_H_1209_HU4_RASTER.7z'

with py7zr.SevenZipFile(fs.open(s3_path), 'r') as zip:
    zip.extract(targets=['HRNHDPlusRasters1209/elev_cm.tif'])

Another option would be to set a mount point to the bucket at the os-level with s3fs-fuse.

For zip archives you should be able to chain gdal's /vsis3/ and /vsizip/ handlers.

Investigate how the HUC 4,6,8 parameterization affects the fim_run outputs especially the inundation inputs: rem_.tif, gw_, and hydro-table. These affect inundation maps and subsequent scoring. Example: fim_run outputs for 120903 and 12090301 are mostly the same if not identical except for the rating curves. This affects the maps and the scoring with 12090301 getting worse results.

Explore options for using NLD levee area polygon to mask inundation output

Job level exit codes are not being caught by the GNU parallel job log feature currently written out as "outputs//logs/summary.log". This maybe caused by piping both stdout and stderr to tee. Unfortunately, the root problem maybe that the print outputs of fim_run.sh are being mis-classified as either stdout and stderr. These issues limit the use of retry/resume features in GNU parallel for failed jobs. They also may cause issues in the new aggregation command at the end of fim_run.sh since a failed job may have none or part of the three required aggregation files. This maybe a good time to review other parallelization tools and if they can handle all the desired features like resource limiting, logging, stdout/stderr printing, progress bars, summary logs, retrying/resuming, etc.

Setup a git hook that initializes fim_run.sh and regression testing for all available test cases for the feature branch use case prior to merging with dev.

Presently, we use NWM v21 lakes layer to mask out lake areas. These lake areas are removed prior to calculating contingency metrics. In the current workflow, FIM 2 does not produce mapping for any catchment designated as a NWM Lake. In many cases these catchments (e.g. lack of mapping) extend beyond the boundary of the NWM Lakes layer. This is different from FIM 1 and FIM 3 where an inundation map is generated in a NWM lake area and it is later masked out. Ultimately, FIM 2 has many areas that are classified (as FN) and evaluated near NWM lakes even though FIM 2 cannot produce mapping in these areas. Additionally a buffer around the NWM lakes may be desirable as for larger flood events the inundation associated with a lake area may extend well beyond the NWM lakes layer.

Using data from a test performed last month, implementing a buffer of NWM v21 lakes can cause FIM 2 CSI scores to increase much more than FIM 3. In some areas, a buffer will cause FIM 3 to flip from an improvement to a regression when compared to FIM 2. How to deal with lakes and potential buffer needs to be finalized.

Please see this slideshow for more information.
https://docs.google.com/presentation/d/1T-7eUzIUnjSilyvooUWeG_YpsESHJHgNmUZT4nko5Fg/edit?usp=sharing

After running HUC 12090301 with FIM 3.0, we found several small tributaries in the NHDPlusBurnLineEvent_subset that we think should not be there based on the workflow. The subset should only contain nhdlines that originates from the nhd_headwater_points_subset. Please see the attached screenshot (tributaries within the red circle). There are no head water points to these small tributaries but for reason are included in the subset.

Subtask for #24 : Modify REM creation to look to the stream for the reference elevation.

Currently REM uses for a reference elevation the minimum value within a pixel catchment. This means that areas that are far away from the channel could be used as the reference elevation.

FIM 2 creates an Euclidian distance and allocation grid to create a 50m buffer around the stream centerline. A zonal minimum elevation within the 50m buffer is determined using the zones from the allocation grid. The zonal minimum elevations are assigned to the stream centerline. These zonal minimum elevations along the stream centerline then undergo a further adjustment to ensure a monotonically decreasing thalweg in the downstream direction. Next, the HAND grid is calculated using the adjusted elevations found along the stream centerline. Finally, the HAND grid values that are negative are reassigned a value of 0.

Examples of this can be found in #24 where REM = 0 is found in areas outside of the stream centerline and in some cases very high HAND values found along the stream centerline.

Most Python scripts in the cahaba repository have unused imported modules.