hollance / coreml-survival-guide Goto Github PK

Hi Matthijs,

I downloaded, fine-tuned and froze the model "deeplabv3_mnv2_dm05_pascal_trainval_2018_10_01.tar.gz" from the DeepLab's model zoo.

I am using 'deeplab.py' to convert the model I trained to CoreML format. In your example code in 'deeplab.py', you show how to convert the model "deeplabv3_mnv2_pascal_trainval/frozen_inference_graph.pb". When I inspect the frozen graph of "deeplabv3_mnv2_dm05_pascal_trainval_2018_10_01.tar.gz" with Netron I find that there is no "Bilinear_3:0" layer (it goes up to bilinear_2) and that the mul_1 is called "mul".

May I ask you from what link you downloaded the model you used in your example? I would like to compare that version with the version I downloaded to understand what has changed between them.

Thanks a lot

Kind regards

I tried running your script for ssdlite_mobilenet_v2.. downloaded from tensorflow/models/research/object_detection/ repo.The script runs until after the decoder stage and stops while doing the NMS with the error:

ma_type.shapeRange.sizeRanges.add()
AttributeError: 'ArrayFeatureType' object has no attribute 'shapeRange'

Help would be very much appreciated! I am very new to CoreML.

Hi Matthijis,
I'm trying to use your instructions to convert ssdtflite_v2 to coreml. The problem is; I'm not sure how to create "saved_model.pb" from retrained model on PC properly.
In the "saved_model" you provided on the github, I can successfully create mlmodel, while in the "saved_model" that I'm create myself, I get the following error:

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value BoxPredictor_0/BoxEncodingPredictor_depthwise/depthwise_weights
	 [[Node: BoxPredictor_0/BoxEncodingPredictor_depthwise/depthwise_weights/read = Identity[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](BoxPredictor_0/BoxEncodingPredictor_depthwise/depthwise_weights)]]

while I want to process this line:

ssd_model = tfcoreml.convert(
    tf_model_path         = optimized_frozen_model_file,
    mlmodel_path = coreml_model_path_concat,
    input_name_shape_dict = {input_tensor: [1, input_height, input_width, 3]},
    image_input_names=input_tensor,
    output_feature_names=[bbox_output_tensor, class_output_tensor],
    is_bgr=False,
    red_bias=-1.0,
    green_bias=-1.0,
    blue_bias=-1.0,
    image_scale=2. / 255)

Can you please provide an script or guide for that as well?

Thank you

I have a yolov3.onnx model I've converted to coreml, and pipelined with an nms model and dropped into the example ObjectDetection app. The process seemed smooth enough, but when running on my
iPhoneXS 12.1 I get the following error. I decode natively in yolov3, and I send these boxes and scores of dimension [4, 507, 1] and [80, 507, 1] from my yolov3.mlmodel to my nms.mlmodel and the pipeline compile goes fine.

Your blog post seems to imply that your decoder model is sending these out to the nms in the opposite permutation though, which confuses me:

Now we can permute this from (4, 1917, 1) to (1, 1917, 4) and write the results to the second output of the decoder model, "raw_coordinates":

I have only two models in the pipeline, 0 (yolov3) and 1 (nms), so nms is not getting the shape it wants apparently according to the error. Is there a way to load the pipeline and debug the input/output shape directly? Netron viewer isnt very informative on the pipeline model:
https://storage.googleapis.com/ultralytics/yolov3_tiny_pipelined.mlmodel (15MB)

Failed to perform Vision request: Error Domain=com.apple.vis Code=3 "The VNCoreMLTransform request failed" UserInfo={NSLocalizedDescription=The VNCoreMLTransform request failed, NSUnderlyingError=0x283b1dc20 {Error Domain=com.apple.CoreML Code=0 "Failed to evaluatue model 1 in pipeline" UserInfo={NSLocalizedDescription=Failed to evaluatue model 1 in pipeline, NSUnderlyingError=0x283b1d860 {Error Domain=com.apple.CoreML Code=0 "Dimension 1 of input confidence (-1880735433) is not consistent with the number of classes (80)" UserInfo={NSLocalizedDescription=Dimension 1 of input confidence (-1880735433) is not consistent with the number of classes (80)}}}}}

Hello,

I am manually cropping the pixelbuffer in specific ratios from the center and passing to vision. The following is a descriptive diagram

I tried scaling and transforming but I am not able to figure out the right presets/approach

From detection o/p

var transform = CGAffineTransform()
if(cropRatio == .5){
//50% crop
transform = CGAffineTransform.identity.translatedBy(x:-rect.origin.x * 0.2, y: rect.origin.y * 0.2)
box.boundingBox = rect.applying(transform)
}
else if(cropRatio == .3){
//30% crop
transform = CGAffineTransform.identity.translatedBy(x:-rect.origin.x * 0.3, y: rect.origin.y * 0.3)
box.boundingBox = rect.applying(transform)
	}
else{
box.boundingBox = rect
}

Rendering code is as follows :

          let width = self.bounds.size.width
	 let height = self.bounds.size.height
	 let scale = CGAffineTransform.identity.scaledBy(x: width, y: height)
	 let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -height)
	var rect = CGRect.zero
	let scaleXY = Double(width/height)
	rect = objectBoundingBox.boundingBox.applying(scale).applying(transform)
	info?.bbox = rect

I would need some guidance as to how I can align the boxes. Right now, for 50% and 30% crop the boxes are misaligned.

By any chance I should be using any of these, this
https://developer.apple.com/documentation/vision/2908993-vnimagerectfornormalizedrect

Thanks in advance

Hello 👋 Thanks for your post about converting MobileNetV2_SSDLite to coreml model.

I try to convert this model (hand detection) to coreML, but I'm struggling to find the right bbox output node (even with Netron). Do you have any advices for me ?

Thanks again :)

First of all, always thank you for your blog.

I have a question about performance.

I have two versions of SSDLite (ssd_mobilenet_v1_coco) with same backbone
One version is implemented all components (feature extractor, bbox decoder, and NSM) separately. Feature extractor is implemented by CoreML and both bbox decoder and NSM is by bit of swift code running on CPU. Other version is combined every module in MLModel as you explained

I checked the performance of two versions on iPhoneX and got a result hard to understand.
Processing time of the first version is two times faster than the order one. Although the first one uses more CPU resource to calculate real bbox coordinates and NMS (uses 100 ~ 110% of CPU resource. the other consumes 45 ~ 50%) it's strange in common sense.

Could you give me any opinion or understanding about this status?

Thank you.

Thank you for your posting! I'm also trying to build a mlmodel pipeline with Tiny YOLOV3 and I found your repo really useful and helpful.

I'm following the ssdlite.py. I ran this file on my own laptop without making any changes. Things are clear and I can almost understand all the steps. For the final model generated by making all the 3 models together as a pipeline, I cannot make predictions by running 'final_model.predict({'image': img})' where img is an image with size (300, 300). The errors I got is as following:

final_model.predict({'image': img})
Traceback (most recent call last):
File "", line 1, in
File "/Users/weishunji/PycharmProjects/coreml_pipeline/venv/lib/python3.6/site-packages/coremltools/models/model.py", line 360, in predict
raise Exception('Unable to load CoreML.framework. Cannot make predictions.')
Exception: Unable to load CoreML.framework. Cannot make predictions.

However, I can make predictions successfully using ssd_model and decoder_model. But when they're combined as a pipeline, I'm not able to predict. I'm a little confused about this and wonder whether this is the way it should be.

I'll really appreciate it if you can help me with this!

Error Domain=com.apple.vis Code=3 "Failed to initialize VNCoreMLTransformer" UserInfo={NSLocalizedDescription=Failed to initialize VNCoreMLTransformer, NSUnderlyingError=0x1d0a455b0 {Error Domain=com.apple.vis Code=5 "No valid VNCoreMLModel found in passed in options" UserInfo={NSLocalizedDescription=No valid VNCoreMLModel found in passed in options}}}

@hollance ,hello ,have you come over this error?

Hi there!
Really appreciate your work on explaining CoreML. Super useful!
I cloned your repository to learn about implementing CoreML and to find out how well my SSD_mobilenet_V1 trained on one class via tensorflow object detection API works. I replaced your MobileNetV2_SSDLite.mlmodel with my MobileNetV1_SSD.mlmodel, and changed the coreMLModel variable in the viewcontroller.

Unfortunately it doesn't work. The console output lists two problems:

• Insufficient Permission (to submit GPU work from background)
• Shape (91 x 1 x 1917) was not in enumerated set of allowed shapes
  I find the last error extra strange, since I can't find out where 91 (which is the number of coco classes I assume) comes from, and how I can change it.

The example does work with the originally supplied mobilenet_V2. I'm running on iOS12.
I've verified my model's accuracy on tensorboard, which is quite good, and I've followed your tutorial to convert it to .mlmodel and made sure to change the code for my one-class model.

Failed to perform Vision request: Error Domain=com.apple.vis Code=3 "The VNCoreMLTransform request failed" UserInfo={NSLocalizedDescription=The VNCoreMLTransform request failed, NSUnderlyingError=0x28019f2d0 {Error Domain=com.apple.CoreML Code=0 "Failed to evaluatue model 1 in pipeline" UserInfo={NSLocalizedDescription=Failed to evaluatue model 1 in pipeline, NSUnderlyingError=0x28019e910 {Error Domain=com.apple.CoreML Code=0 "Shape (91 x 1 x 1917) was not in enumerated set of allowed shapes" UserInfo={NSLocalizedDescription=Shape (91 x 1 x 1917) was not in enumerated set of allowed shapes}}}}} 2019-01-22 13:09:23.913001+0100 ObjectDetection[1428:73469] Execution of the command buffer was aborted due to an error during execution. Insufficient Permission (to submit GPU work from background) (IOAF code 6) 2019-01-22 13:09:23.913428+0100 ObjectDetection[1428:73469] Execution of the command buffer was aborted due to an error during execution. Insufficient Permission (to submit GPU work from background) (IOAF code 6) Error: command buffer exited with error status. The Metal Performance Shaders operations encoded on it may not have completed. Error: (null) Insufficient Permission (to submit GPU work from background) (IOAF code 6) <AGXA9FamilyCommandBuffer: 0x12fd4ab00> label = <none> device = <AGXA9Device: 0x105818000> name = Apple A9 GPU commandQueue = <AGXA9FamilyCommandQueue: 0x12fe1a5a0> label = <none> device = <AGXA9Device: 0x105818000> name = Apple A9 GPU retainedReferences = 1

Hi hollance!
Really thank you for your blogs about coreML.

Here is the thing. So I trained a Mobilenetv2+SSDlite model for my own data with 2 classes, and I used ssdlite_mobilenet_v2_coco just like you. When I tried to convert my model to coreML model using your script, I get several errors like below:

1. Traceback (most recent call last): File "ssdlite.py", line 142, in <module> del spec.description.output[1].type.multiArrayType.shape[-1] IndexError: List assignment index out of range
   So I check spec.description, and I get a result like this:

output{
name: "scores"
shortDescription: "Predicted class scores for each bounding box"
type {
multiArrayType {
shape: 3
shape: 1917
dataType: DOUBLE
}
}
}
output{
name: "boxes"
shortDescription: "Predicted coordinates for each bounding box"
type {
multiArrayType {
dataType: DOUBLE
}
}
}

Which means, somehow, the output shape of the "boxes" output is not filled in. And I notices you added the output shape of the "scores" output, so I did the same way and convert coreML model successfully. Then I got another error.

2. validator error: Layer 'Postprocessor/scale_logits:0' consumes an input named 'concat__0' which is not present in this network.
   When I got the final coreML model and add it to xcode project, I found it got an error as shown above. The screen shot of it is added in below.

I used coremltools 2.0, tfcoreml 0.3.0, tensorflow1.10.0(I was trained in this version, so had to convert it in this tensorflow version) and python 3.6.5 . I only changed the number of classes, if you know anything about this, please tell me. Thank you very much!

Please Update for SwiftUI