Validate user input for Null Values in Lookup Table and return user friendly error about qnspect HOT 5 OPEN

@jamie-carter I was not able to reproduce this issue.

1. I changed the CSV file and didn't provide most values for nitrogen, yet the tool didn't fail; it assumed 0 for those Land Cover classes.
2. I did the same in QGIS, created a lookup table using Create Lookup Table Template and changed nitrogen values to Null. The tool worked as in the previous case, assuming 0 value for them.
   

from qnspect.

@ar-siddiqui I can recreate the problem when using a coefficient table that is saved to a CSV file. If using either a temporary table (edits saved or unsaved) or a table in a GeoPackage (just tested with edits saved) the program works with Nulls.

It fails using a CSV table if the table contains either NULLs, or if the cells are completely empty.

Note that deleting the contents of a temporary or gpkg table immediately fills the cell with NULL. Deleting a CSV cell leaves it empty. You need to manually type in NULL to fill with a NULL. Clearly QGIS is handling these tables differently. All the above comments refer to modifying the table from within QGIS by opening the Attribute Table.

Bad CSV file and log of failing run attached.
ccap_w_nulls.csv
ccap_w_empty_cells.csv
ccap_w_nulls_log.txt
ccap_w_empty_cells_log.txt

from qnspect.

@DaveEslinger, thanks for the detailed feedback. I have looked into it and can recreate the issue.

After investigating, I think the user should not have empty/null values in the lookup table. In some of the cases, depending on the file format QGIS would assume null/empty values to be zeros and not in other cases and would fail (for example csv). This might be by design where certain file formats have their default null fallback values defined, while csv does not have it.

I think we should close this issue and ask users not to leave cells empty. To be more specific about where the issue is occurring I have added a print statement for each process so that the user can easily debug where the error is originating from.

from qnspect.

I agree users should not have null values in their tables. However, I think we need to have a check for that and send an error message and stop processing if that occurs. Just telling them to use good input is unlikely to actually work. Input validation is needed.

from qnspect.

3 options discussed:

1. Leave as-is. We guide users towards using CSVs with 0s and not nulls in documentation an our template. We accept some users will still introduce nulls.
2. Check for CSV inputs only (not all supported formats) and then check for null values. Show specific error message and cancel the run. We probably need to check behavior of all supported formats when they contain nulls.
3. Check inputs for nulls regardless of format, show error message on null and cancel the run. This would cancel runs that otherwise would have succeeded since some formats e.g. geopackage work with nulls values.

Decision for now is to stick with 1, until other must-have defects are addressed then revisit if there's time.

from qnspect.