esri / raster-functions Goto Github PK
View Code? Open in Web Editor NEWA curated set of lightweight but powerful tools for on-the-fly image processing and raster analysis in ArcGIS.
License: Apache License 2.0
A curated set of lightweight but powerful tools for on-the-fly image processing and raster analysis in ArcGIS.
License: Apache License 2.0
To do: Add a section to the Wiki on writing image processing algorithms to process collections of imagery using the Group Items By capability in the RFT properties.
We should develop a cloud filtering algorithm that uses FMask. There are some sensor specific capabilities out there that use FMask, such as sentinel2-cloud-detector
I've been trying out the Hillshade function on mosaic datasets containing bathymetric data. I really like how the Python version looks vs. the built-in mosaic function -- it's much smoother-looking. However, I'm running into a problem when the viewing window crosses the antimeridian. The drawing performance gets severely degraded, and it may take several minutes for the image to display. This becomes an issue when the mosaic is published as an image service to be used in a map viewer, and the user zooms into an area surrounding 180 longitude in the Pacific.
I tried tracing what was going on by using print statements in the Hillshade.py code. I can see that, instead of the updatePixels() function getting called once, it gets called dozens or hundreds of times in a row, with what seems to be tiny windows of pixels.
Here are some steps to replicate the problem:
Not sure if this is a problem with the mosaic dataset code in ArcGIS, or with the Python function. I'm using ArcMap 10.4.1.
Thanks,
Jesse
If a function, with different user specified parameters, is called more than one time inside the same map document, all previous results are replaced by the last one.
As a user, I want to specify Cell Size Threshold in mercator meters and not in the units of the coordinates of the image request. The raster function must internally reproject to request SRS to compare request and threshold cell sizes.
Currently the python raster function assumes that all arrays are c-contiguous. Need to add support for non-c contiguous arrays.
Updated properties are not transmitted to the client by an image service when key metadata raster function is applied on the back-end mosaic dataset (either on top of the mosaic function or as part of a processing template).
cc: @amitjain27, @jdrisdelle, @wju, @rsjiezhang
For thoroughness, we should add try-except
to the pixel percentile calculations
Hi, I'm seeing if you think this is possible. I want to create a raster function where I pass in a polygon and it returns the zonal stats average value of a value raster. I want the output raster but just the min or max of the output zone raster. I want this to function in ImageServer and I use the JS API to pass in a polygon feature set and it returns the resulting summary value. I currently have a raster function that successfully chains Rasterize Features and then Zonal Statistics functions to create the correct zone raster but not sure about the rest. Do I need a python raster function to get the output value?
The hyperlinks for MultidirectionalHillshade.pyd and the accompanying MultidirectionalHillshade.rft.xml raster function template do not seem to be working:
https://github.com/Esri/raster-functions/blob/master/README.md#multidirectional-hillshade
Provide an example of python raster function that's supplies or modifies the Xform object to the raster's geometry.
I am looking at using the Elevation Fill Void function (I believe it is called Fill Raster under the function folder) as an alternative to the Spatial Analyst Fill function. When I run the function in ArcGIS Pro, the result I get are not what I expect and analysis run on the result of the raster function are incorrect.
I looked at the code under the function folder here. There is no inline documentation in the code. I am having trouble following what it does and how it relates the raster tool in ArcGIS Pro.
In the documentation on the wiki page of this repo in the section updateRasterInfo() there are some information about the keyword "resampling". I've been playing a little with this parameter while trying to to get a python raster function to behave as we want. I've a few questions regarding this parameter.
kwargs['output_info']['Resampling'] = False
has some effect but not kwargs['output_info']['resampling'] = False
On the wiki page I also founds some information regarding selectRasters(). But I can't trigger this to run. Is it possilbe to get it to run on a single raster dataset?
The following link is dead:
I am try to reference many Rasters to other rasters (Warb) using python code, But I feild, I need help.
import arcpy
i = 0
while i <= 1:
inRasterworkspace = r'C:\Users\AgmnDesk\Desktop\GooleImage\GI_R'
#inRasterworkspace = arcpy.GetParameterAsText(0)
arcpy.env.workspace = inRasterworkspace
inRasters = arcpy.ListRasters()
inRaster = inRasters[i]
SoTop=arcpy.GetRasterProperties_management(inRaster , "TOP")
SoLeft=arcpy.GetRasterProperties_management(inRaster , "LEFT")
SoRight=arcpy.GetRasterProperties_management(inRaster , "RIGHT")
SoBottom=arcpy.GetRasterProperties_management(inRaster , "BOTTOM")
Source_pnt = '''+ str(SoLeft) + ' ' + str(SoTop) + '' ;' + str(SoRight) + ' ' + str(SoTop) + '' ;' + str(SoLeft) + ' ' + str(SoBottom)+'''
taRasterworkspace = r'C:\Users\AgmnDesk\Desktop\GooleImage\GI2018'
#taRasterworkspace = arcpy.GetParameterAsText(1)
arcpy.env.workspace = taRasterworkspace
taRasters = arcpy.ListRasters('*', 'jpg')
taRaster = taRasters[i]
TaTop=arcpy.GetRasterProperties_management(taRaster , "TOP")
TaLeft=arcpy.GetRasterProperties_management(taRaster , "LEFT")
TaRight=arcpy.GetRasterProperties_management(taRaster , "RIGHT")
TaBottom=arcpy.GetRasterProperties_management(taRaster , "BOTTOM")
Target_pnt = '''+ str(TaLeft) + ' ' + str(TaTop) + '' ;' + str(TaRight) + ' ' + str(TaTop) + '' ;' + str(TaLeft) + ' ' + str(TaBottom)+'''
arcpy.env.workspace = r'C:\Users\AgmnDesk\Desktop\GooleImage\GI2018'
arcpy.env.overwriteOutput = True
arcpy.Warp_management(str(taRaster), Source_pnt, Target_pnt, str(taRaster[:-4])+'tif', "POLYORDER2", "BILINEAR")
i = i + 1
MultiDirectional Hill shade function add support for nodata handling
Hello,
I was wondering how I would be able to set customized slope values in for my results? S instead of the last one being 40% I would like Greater than 30 DEGREES slope.
Thanks,
Ryan
Hi,
I made a simple Python RFT chained with another RFT template. I use a Rasterize Feature to access a value inside a feature table. Then I am able to apply my python RFT to a mosaic. But It is not optimal for me.
I would like to directly use this numeric value stored in a field of my mosaic (this value can change from one raster to another inside the mosaic).
I made lot of test but unsuccessful.
Does someone has an idea on how to do that ?
thanks for your help.
Hello,
I successfully managed to develop a small python raster function. I created a new custom category in C:\Program Files\ArcGIS\Pro\Resources\Raster\Functions\Custom\
and in C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\Resources\Raster\Functions\Custom\
I copied the python script and the xml raster function in these two folders. it should be possible to use my python raster function on the server and arcgis Online. It works on Arcgis pro on the server but, impossible to to apply the function on mosaic service on Arcgis Online.
Thanks for your help.
We want users to be able to view weather data in the context of how it will affect their activities. Using the REMAP function, and a set of thresholds, we can remap weather values into “Favourable”, “Marginal” and “Unfavourable”, relative to the activity. The thresholds at which the values are remapped would be something like;
• Favourable is FROM wind speeds of 0 m/s TO 3 m/s
• Marginal is FROM wind speeds of 3 m/s TO 7 m/s
• Unfavourable is FROM wind speeds of 7 m/s TO 2000 m/s
Storing these ranges of values within each of the raster functions means that they are somewhat inaccessible for editing / checking / updating / re-use. However:
• At present the values are stored within the raster functions themselves because the REMAP function will not allow the use of an external table with remap ranges.
o But even if we could use a table for the remap ranges, nevertheless every time a user wanted to look at the impact of a weather variable on a new activity (e.g. the effect of temperature on running performance) they would need to create a new table – even though they are really just creating new data under a pre-defined schema.
• What we would prefer is for all the activities, weather variables and threshold values to be held in a single table so that users can add / edit / delete thresholds all in one place. For this to occur any one raster function would need to be able to query against that table using both activity and weather variable in the WHERE clause to pull out the correct REMAP values. The table columns would be something like: [ACTIVITY, WEATHERVAR, FROMVAL, TOVAL, UNITS, REMAPVAL], where REMAPVAL would represent ‘Suitability’, and a typical set of values might be: [“Running”, “Temperature”, -30, -20, “Centigrade”, -2], where -2 might represent ‘Unfavourable’.
@jdrisdelle I think the Wiki should reference how to do this in ArcGIS Pro. That's what I am using and some other people that I work with are using. Also, we submit RA jobs from ArcGIS Pro, so it would be beneficial to have the instructions written from ArcGIS Pro.
When I run 'python setup.py install' to install the extensions, do I want
C:\Python27\ArcGIS10.3\python.exe or
C:\Python27\ArcGISx6410.3\python.exe
in my path variable?
The padding configuration provided for the input rasters pads only the left and top side of the pixel blocks, causing issues after manipulation of pixel values.
Hi Guys & Girls,
I wrote a python raster function (slightly changed a given function)
and after using it in ArcMap/Image Analysis, I got a monochrome result raster. After calculating a histogram in order to classify the Image, the result is fine. However, I want to include that histogram computation in my script, so that there is a classified result raster.
Is that possible?
Thats the script:
import numpy as np
class Arithmetic():
def init(self):
self.name = "Arithmetic Function"
self.description = "Performs simple arithmetic operations on two rasters."
self.op = None
self.applyScaling = True
self.applyColormap = False
def getParameterInfo(self):
return [
{
'name': 'r1',
'dataType': 'raster',
'value': None,
'required': True,
'displayName': "Early NDVI",
'description': ""
},
{
'name': 'r2',
'dataType': 'raster',
'value': None,
'required': True,
'displayName': "Late NDVI",
'description': ""
},
{
'name': 'op',
'dataType': 'string',
'value': 'Add',
'required': False,
'domain': ('Add', 'Subtract'),
'displayName': "Operation",
'description': ""
},
{
'name': 'method',
'dataType': 'string',
'value': 'Colormap',
'required': False,
'domain': ('Raw', 'Grayscale', 'Colormap'),
'displayName': "Output Image Type",
'description': ("The type of output expected from this function. Specify Raw for scientific analysis. "
"Pick Grayscale or Colomap for visualization.")
},
]
def getConfiguration(self, **scalars):
return {
'inheritProperties': 2 | 4 | 8,
'invalidateProperties': 2 | 4 | 8,
}
def updateRasterInfo(self, **kwargs):
method = kwargs.get('method', 'Colormap').lower()
self.applyColormap = method == 'colormap'
self.applyScaling = self.applyColormap or method == 'grayscale'
maximumValue = 2.0
minimumValue = -2.0
if self.applyScaling: # maximum output value depends on whether we are scaling
maximumValue = 200.0
colormap = ()
pixelType = 'f4'
if self.applyColormap:
pixelType = 'u1'
colormap = (np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255], dtype='int32'),
np.array([36, 36, 36, 36, 245, 245, 245, 245, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 247, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 250, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 252, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 253, 250, 244, 238, 234, 231, 223, 217, 211, 205, 200, 195, 189, 184, 180, 174, 169, 163, 160, 154, 148, 143, 138, 134, 130, 126, 117, 115, 112, 106, 100, 94, 92, 90, 81, 75, 71, 66, 62, 56, 51, 51, 51, 50, 50, 50, 50, 49, 49, 49, 48, 48, 48, 48, 48, 48, 48, 48, 47, 47, 47, 47, 46, 46, 46, 46, 45, 45, 45, 45, 44, 44, 44, 43, 43, 43, 43, 43, 43, 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41, 40, 40, 40, 40, 40, 39, 39, 39, 39, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38], dtype='uint8'),
np.array([0, 0, 0, 0, 20, 24, 29, 31, 33, 33, 37, 41, 41, 41, 45, 45, 47, 49, 49, 54, 54, 56, 58, 58, 62, 62, 62, 67, 67, 67, 69, 71, 71, 75, 75, 78, 79, 79, 79, 81, 83, 83, 87, 87, 90, 92, 93, 93, 97, 97, 97, 97, 101, 101, 101, 101, 105, 105, 107, 109, 109, 113, 118, 119, 121, 126, 132, 133, 135, 141, 144, 150, 152, 153, 159, 163, 165, 168, 174, 176, 181, 183, 186, 191, 197, 201, 203, 205, 209, 214, 216, 218, 224, 228, 234, 236, 238, 243, 248, 252, 252, 252, 250, 247, 246, 245, 240, 237, 235, 233, 230, 227, 224, 222, 220, 217, 214, 212, 210, 207, 204, 201, 199, 197, 194, 191, 189, 186, 184, 181, 179, 176, 174, 173, 168, 166, 163, 160, 158, 156, 153, 153, 153, 150, 150, 150, 150, 148, 148, 148, 145, 145, 145, 145, 143, 143, 143, 143, 140, 140, 140, 140, 138, 138, 138, 138, 135, 135, 135, 135, 133, 133, 133, 130, 130, 130, 130, 130, 130, 128, 128, 128, 125, 125, 125, 125, 122, 122, 122, 122, 120, 120, 120, 120, 120, 117, 117, 117, 117, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115], dtype='uint8'),
np.array([255, 255, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 20, 23, 25, 33, 38, 40, 43, 48, 54, 59, 61, 64, 69, 77, 79, 82, 87, 92, 97, 99, 102, 107, 115, 120, 123, 125, 130, 138, 141, 143, 150, 156, 163, 165, 168, 173, 181, 186, 186, 187, 180, 176, 173, 169, 163, 157, 150, 146, 142, 136, 132, 126, 123, 119, 114, 108, 105, 101, 96, 93, 88, 84, 81, 77, 70, 68, 64, 60, 55, 49, 47, 45, 37, 33, 28, 24, 21, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype='uint8'))
m = kwargs.get('op', 'Add').lower()
if m == 'add': self.op = np.add
elif m == 'subtract': self.op = np.subtract
kwargs['output_info']['statistics'] = ({'minimum': minimumValue, 'maximum': maximumValue}, ) # we know something about the stats of the outgoing NDVI raster.
kwargs['output_info']['histogram'] = () # we know nothing about the histogram of the outgoing raster.
kwargs['output_info']['colormap'] = colormap # optional colormap if requesting for an color image
return kwargs
def updatePixels(self, tlc, shape, props, **pixelBlocks):
r1 = np.array(pixelBlocks['r1_pixels'], dtype='f4', copy=False)
r2 = np.array(pixelBlocks['r2_pixels'], dtype='f4', copy=False)
np.seterr(divide='ignore')
pixelBlocks['output_pixels'] = self.op(r1, r2).astype(props['pixelType'], copy=False)
return pixelBlocks
def updateKeyMetadata(self, names, bandIndex, **keyMetadata):
if bandIndex == -1:
keyMetadata['datatype'] = 'Processed' # outgoing raster is now 'Processed'
elif bandIndex == 0:
keyMetadata['wavelengthmin'] = None # reset inapplicable band-specific key metadata
keyMetadata['wavelengthmax'] = None
return keyMetadata
Please add documentation for Group Items in Processing Template to wiki.
When looking at RTF and python scripts alone (without the actual data), it is not clear how I can reference different raster datasets inside the raster mosaic as variables.
How do I need to prepare the data/ raster mosaic table? How do I reference the different rasters in the template afterwards?
I haven't been able to get the Hillshade function to work in ArcGIS Pro (2.2.0). I just get a blank image as a result, with no error message. Other functions do work as expected, such as AspectSlope.py. Has anyone else had success with Hillshade.py in Pro?
Note: I didn't install the separate Python extension packages, since the required scipy and numpy packages are already available in the ArcGIS Pro environment.
I'm using ArcGIS Pro 1.4 and want to create my own raster functions with scikit-image. ArcGIS Pro 1.4 is using python 3.5. I downloaded every library from this page in version 3.5 and have it installed, but it is not working.
http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython
Has somebody any idea what my fault is or maybe it's not working with python 3.5?
Hello
I'm not used to use python scripts. How would I implement the code into my ArcMap project?
If I try to generate a new toolbox the script gives no error but also does not produce any output...
Thanks!
Hey!
We're unable to add our rft.xml file with a python raster function. We are referencing this path in the rft file:
[Install]\Server\framework\runtime\ArcGIS\Resources\Raster\Functions\Custom\subfolder\OurRasterFunction.py
Everything seems to work fine in pro where we can update the parameters we want to update and add the rft in our mosaic processing template.
But when we try and publish an image service with the rft file it throws an error telling us the template is invalid.
From the server logs:
Method failed.HRESULT = 0x80043010 : This is a FACILITY_ITF error that is specific to the interface that returned this error. See the documentation of the interface that returned this error for information about this HRESULT.
We were able to publish from arcmap 10.8.1, but then we dont get the capability of changing the parameter in the image service. Its stuck to the default value and we need to be able to deliver this to a customer that uses Pro.
It seems this is a compatibility issue between arcgis server 10.9.1 and the latest pro version when it comes to the template xml structure.
Do we have to upgrade to arcgis server 11 to make this work? Or is it any workaround that dont require to use Arcmap?
I can't share too much information as this is some sensitive data im working with. But i can answer any questions about the setup.
We want to enable the python raster function to specify a scaleFactor
in .getConfiguration()
indicating the difference in input raster resolution w.r.t. the current request resolution.
From Cristelle:
Bug 1: White Patches/Areas appear at zoomed out levels, on application of Aspect-Slope Raster Function on a “Mosaic Dataset” in ArcMap
Version tested on: 10.5
Data Location : \spider\Swap\Cristelle\Zikang\Aspect_Slope\ArcMap\WhiteAreas_Mosaic
Bug 2: Only Pixel borders appear colorized, below scale levels of 20000, on application of Aspect-Slope Raster Function on an Image Service in ArcMap
Version tested on: 10.5
Data Location : https://elevation.arcgis.com/arcgis/rest/services/WorldElevation/Terrain/ImageServer
\spider\Swap\Cristelle\Zikang\Aspect_Slope\ArcMap\PixelBorders_Service
Bug 3: Legend/Colormap doesn’t get applied , on application of Aspect-Slope Raster Function on a Raster Dataset, Mosaic or Image Service in ArcGIS Pro
Version tested on: ArcGIS Pro 1.4
Data Location : \spider\Swap\Cristelle\Zikang\Aspect_Slope\Pro\NoColorMap
https://elevation.arcgis.com/arcgis/rest/services/WorldElevation/Terrain/ImageServer
I'm writing a script that needs to clip a multidimensional raster (Gridmet temperature data) to a smaller area. Poking around the documentation this is the workflow that I came up with
temp_max_file = r"D:\r3_analyst_work\Climate_Change_Modeling\climate_envelope\GridMetData_2020\tmmx_2020.nc" max_temp_rast = arcpy.Raster(temp_max_file,True) clip_frame= r"D:\r3_analyst_work\GIS_work\test_working_netcdf_climate_data\Default.gdb\small_area" max_temp_clip_raster = arcpy.sa.Foreach(max_temp_rast,"Clip",{"ClippingGeometry":clip_frame, "ClippingType":1})
When I run this I get
**RuntimeError Traceback (most recent call last)
In [39]:
Line 1: max_temp_clip_raster = arcpy.sa.Foreach(max_temp_rast,"Clip",{"ClippingGeometry":clip_frame, "ClippingType":1})
File C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\ia_ia.py, in Foreach:
Line 114: return arcgisscripting._ia.Foreach(in_raster=in_raster,
RuntimeError: Failed to apply Raster Function: 'Clip' (The parameter is incorrect. )**
There is no indication what parameter is incorrect.
I had to extract the parameter name by exporting the Clip raster function and then reading the XML document.
Using the ArcPro interface the raster function works correctly on those data sets.
From what I can tell, my code is exactly the same as the NDVI example provided in the Foreach documentation.
Hi,
I have read the RasterFrames supervised machine learning on the page of "https://rasterframes.io/supervised-learning.html", now I would like to implement Deep Learning similar to “supervised machine learning”, such as "YOLO object detection" and "U-Net semantic segmentation".
However, I am so confused about how to write YOLO or U-Net Deep Learning code which is integrated with RasterFrames? So, would you pls help to give me some suggestions? Many Thanks!
Hello,
I'm looking for help trouble shooting a custom raster function. The heart of the rf is an XGBoost model, but something goes wrong when I try to load the model. I've used Pickel to help narrow down the issue, and verified that the xgboost library IS loading and that the filepath for the model is correct. For whatever reason, the model is not loading. Using a notebook in ArcPro, I could not replicate the issue (everything runs as it should). It seems to be an issue specifically with the raster function. I'm really at a loss as to how to fix this and any insight would be much appreciated!
Other notes: Everything up to the loading of the model is working (verified this using Pickel). The rf on a whole runs, but provides no output (because the model is not loading).
The code around the issue:
def updatePixels(self, tlc, shape, props, **pixelBlocks):
# Preprocessing: Create mask(s)(vector and raster) as needed
# Get bands
image_array = np.array(pixelBlocks['NAIP_ras_pixels'], 'float32')
red = image_array[0,:,:]
green = image_array[1,:,:]
blue = image_array[2,:,:]
nir = image_array[3,:,:]
def calc_rdvi(nir_values, red_values):
nom = nir_values - red_values
denom = np.sqrt(nir_values + red_values)
return (nom/denom)
def calc_osavi(nir_values, red_values):
nom = nir_values - red_values
denom = nir_values + red_values + 0.16
return (nom/denom)
def calc_evi(nir_values, red_values, blue_values, G=2.5, C1=6, C2=7.5, L=1):
nom = nir_values + red_values
denom = nir_values + (C1 * red_values) - (C2 * (blue_values + L))
evi = G*(nom/denom)
evi = np.nan_to_num(evi, posinf=0, neginf=0)
return evi
def calc_vari(green_values, red_values, blue_values):
return (green_values - red_values)/(green_values + red_values - blue_values)
def calc_msavi(nir_values, red_values):
term1a = np.square((2*nir_values) + 1)
term1b = 8*(nir_values-red_values)
term1 = np.sqrt(term1a-term1b)
term2a = (2*nir_values)+1
term2 = term2a - term1
return term2/2
def calc_gli(green_values, red_values, blue_values):
nom = (green_values - red_values) + (green_values - blue_values)
denom = (2*green_values) + red_values + blue_values
return (nom/denom)
rdvi = calc_rdvi(nir, red)
osavi = calc_osavi(nir, red)
evi = calc_evi(nir, red, blue)
vari = calc_vari(green, red, blue)
msavi = calc_msavi(nir, red)
gli = calc_gli(green, red, blue)
array_shape = rdvi.shape
vi_array = np.array([rdvi.flatten(), osavi.flatten(), evi.flatten(), vari.flatten(), msavi.flatten(), gli.flatten()])
del (red, green, blue, nir, rdvi, osavi, evi, vari, msavi, gli)
# Fix any nan or inf that result from VI calculation (problems in nodata areas)
vi_array = np.nan_to_num(vi_array, posinf=0, neginf=0)
# Step 2: Create the dataframe
def generate_vi_df(two_dim_vi_stack):
# create a DF to match XGBoost model
df = pd.DataFrame(columns=['ID','RDVI','OSAVI',
'EVI','VARI','MSAVI','GLI',
], dtype='float32')
df['ID'] = range(0, two_dim_vi_stack.shape[1])
df['RDVI'] = two_dim_vi_stack[0,:]
df['OSAVI'] = two_dim_vi_stack[1,:]
df['EVI'] = two_dim_vi_stack[2,:]
df['VARI'] = two_dim_vi_stack[3,:]
df['MSAVI'] = two_dim_vi_stack[4,:]
df['GLI'] = two_dim_vi_stack[5,:]
return df
df = generate_vi_df(vi_array)
# Step 3: Load the XGBoost model
# Able to output in a Pickel a dict with configs (xgb.get_config()). So XGBoost *is* being imported. 22 Nov 2023.
# test = xgb.get_config()
#(The JSON will load independently, so the path is correct. 22 Nov 2023.)
model_fn = 'TreeCAP_Full_US_NoOrgSpecBandsOREco_RDVI_OSAVI_EVI_VARI_MSAVI_GLI.json'
model_filename = os.path.join(xgb_model_directory, model_fn)
## Unclear if model is actually loading. 22 Nov 2023
bst = xgb.Booster(model_file=model_filename)
## This should provide a dict output (this works in and Arcpro notebook) to test if model is loading,
## but Pickle fails to save output. No file is created at all. 22 Nov 2023
output = bst.get_fscore()
########## save a pickle ##########
fname = 'pixelBlocks.p'
pickle_filename = os.path.join(debug_logs_directory, fname)
pickle.dump(output, open(pickle_filename,"wb"))
########## pickle saved ##########
The output cell size is define in the updaterasterInfo method. However, it is for some reason changed inside the updatePixels method. Also, the change is different in the x and y direction.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.