Comments (8)
I believe we were going to test doing a conversion before accumulation on these variables as well. I think it will be similar to what you did to correct the units on erosion, but we need to go from mg to kg, to 10^6 instead of 10^3.
Also, if there is already a conversion at the end, we'll need to drop that.
Note: My comments are about the pollutants. I suspect your previous unit correction will have fixed the erosion issue.
from qnspect.
With a small 99x100 cell (cell size 30 m) artificial data set, I do NOT get the error at all. I do get an error about no NODATA values to fill. Note that I think I'm not really using good R-Factor values here since I'm getting 9,745 kg/year of soil from one 30x30m cell. That seems a lot.
With real data for Hawaii and Puerto Rico I DO get the error, consistently at the same spot. I'm getting new data to test later.
With HI data, get error here:
Generating local sediments raster ...
GDAL command:
gdal_calc.bat --overwrite --calc "A * B * 907.18474" --format GTiff --type Float32 --NoDataValue -999999.0 -A C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/35d078bbe5f1406f9317da5d9429f351/OUTPUT.tif --A_band 1 -B C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/bfc9ac4f95fa4b55a07db626327684af/OUTPUT.tif --B_band 1 --overwrite --outfile "E:\GIS\QNSPECT\Testing\base05142022\Sediment Local.tif"
GDAL command output:
:1: RuntimeWarning: overflow encountered in multiply
C:\OSGeo4W\apps\Python39\lib\site-packages\GDAL-3.4.2-py3.9-win-amd64.egg\osgeo_utils\gdal_calc.py:522: RuntimeWarning: invalid value encountered in multiply
myResult = ((1 * (myNDVs == 0)) * myResult) + (myOutNDV * myNDVs)
FOR PUERTO RICO DATA error occurs same spot:
Generating local sediments raster ...
GDAL command:
gdal_calc.bat --overwrite --calc "A * B * 907.18474" --format GTiff --type Float32 --NoDataValue -999999.0 -A C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/0e9fa30bbaa44f1eb2869fa2ca003009/OUTPUT.tif --A_band 1 -B C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/ae84be7b31cb43ebb684934f488db742/OUTPUT.tif --B_band 1 --overwrite --outfile "E:\GIS\QNSPECT\Testing\PR_clipped_base\Sediment Local.tif"
GDAL command output:
:1: RuntimeWarning: overflow encountered in multiply
C:\OSGeo4W\apps\Python39\lib\site-packages\GDAL-3.4.2-py3.9-win-amd64.egg\osgeo_utils\gdal_calc.py:522: RuntimeWarning: invalid value encountered in multiply
myResult = ((1 * (myNDVs == 0)) * myResult) + (myOutNDV * myNDVs)
WITH ARTIFICAL DATA, NO ERROR:
Generating local sediments raster ...
GDAL command:
gdal_calc.bat --overwrite --calc "A * B * 907.18474" --format GTiff --type Float32 --NoDataValue -999999.0 -A C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/d0aba6b299b34237b1d81a7376282039/OUTPUT.tif --A_band 1 -B C:/Users/davee/AppData/Local/Temp/processing_MbaJAK/e161a91c2dc8432cb16aac10c0b29a76/OUTPUT.tif --B_band 1 --overwrite --outfile "E:\GIS\QNSPECT\Testing\100x99x30_base\Sediment Local.tif"
GDAL command output:
0.. 20.. 40.. 60.. 80.. 100 - Done
Process completed successfully
from qnspect.
After checking and fixing #29, this is not because of it.
It is because of raster not able to store the resultant value which is large enough.
Possible solutions:
- Change dtype to float64 (but there maybe cases when data is bigger than float 64, this will also mean that it would be unnecessarily float64 in some cases when it does not need to be)
- Make dtype a user input (advanced parameter)
- Change units of calculations and use bigger units.
More details:
https://stackoverflow.com/a/7559950/11428410
from qnspect.
@DaveEslinger @NSPECT do you remember what we settled for as a solution for this one?
from qnspect.
After investigating, it turns out that this is not an overflow issue but GDAL/NUMPY calculation issue. #61 is also caused by GDAL/NUMPY calculation.
The raster values are nowhere close to overflow limit for float32
. The error can be replicated outside of QGIS as well.
A temporary fix has been put in place in 97ae3f1 that may or may not work for a different test area.
A ticket has been filed with GDAL OSGeo/gdal#5609
Alternative tools were considered, like the QGIS processing raster calculator, but the idea was dropped because several bugs exist in the QGIS repo for the QGIS processing raster calculator.
It is important to note that these messages are warnings, and doesn't seem to impact results
from qnspect.
Tested results of GDAL calculator with and without warnings and they are exactly the same. Thus changing the label from bug to warnings.
from qnspect.
@DaveEslinger when you get a chance can you please confirm if you are consistently getting this warning message at the same place in erosion modules for different test datasets.
from qnspect.
from qnspect.
Related Issues (20)
- Division by zero error in comparison tools HOT 7
- Restructure files and folders to make it easy to zip and ship plugin
- Add tests
- Change pollutant selection gui
- Comparison Percent Calculation Incorrect? HOT 6
- Remove alphabetical sort on land cover classes
- Reorder input fields on GUIs to be as consistent as possible HOT 1
- Modify Land Cover (NLCD/C-CAP) is failing to execute.
- Erosion Analysis HOT 4
- Add raster alignment option to Rasterize Soils tool
- Validate user input for Null Values in Lookup Table and return user friendly error HOT 5
- Change the name for the โDesired Outputs [optional]โ group in Pollution Analysis tool HOT 4
- Warn user if `analysis` inputs are temporary layers
- Limit displayed files to common raster types when loading rasters
- Raster colors not loading consistently HOT 3
- Change most "land use" terms to land cover
- It should not be, but is, possible to reclass from CCAP to NLCD and vice-versa HOT 3
- Limit pollutant and erosion calculations to a selected polygon area HOT 5
- Change NoData value, at least in comparison results
- Concentration error introduced with SHA: 18fe2433f31
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qnspect.