[Solved] How to interpret the predictions about tfjs-object-detection HOT 8 OPEN

Hi @hugozanini and @saidRaiss

I recently solved this issue and i'd like to share my steps below:

Step 1. Use saved_model_cli to find out what's the number means

Via saved_model_cli we can be able to know what kinda things inside your custom trained model. Here is the example what I got from my saved_model

// saved_model_cli show --dir saved_model --all

...
signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_tensor'] tensor_info:
        dtype: DT_UINT8
        shape: (1, -1, -1, 3)
        name: serving_default_input_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_anchor_indices'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:0
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 4)
        name: StatefulPartitionedCall:1
    outputs['detection_classes'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:2
    outputs['detection_multiclass_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 13)
        name: StatefulPartitionedCall:3
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:4
...
...

Step 2: Use these numbers for interpreting predictions

As you can see, each of the outputs has its own StatefulPartitionedCall such as detection_boxes is 1 and detection_classes is 2. So, we gonna use these numbers just like you already knew :)

// ...
  const boxes = predictions[1].arraySync();
  const scores = predictions[4].arraySync();
  const classes = predictions[2].dataSync();

example

You can find more details from my code sandbox below.
This example has been using the latest version of tfjs (3.3.0) though, it works now (April 1st, 2021)

https://codesandbox.io/s/object-detection-with-tensorflow-js-forked-ezssc?file=/src/index.js:1101-1234

from tfjs-object-detection.

Same problem for me.
Since I don't wanna use the legacy versions, I'm totally wanna know how to interpret tf.tensor.

@saidRaiss
For me, after I downgraded my tfjs version to 1.7.4, it worked.
Please refer to it : https://codesandbox.io/s/object-detection-with-tensorflow-js-forked-6r45k?file=/src/index.js

from tfjs-object-detection.

Hello @dusskapark and @hugozanini

I wanted to follow up on this because I have found something different after some research.

Reading here: https://stackoverflow.com/questions/62270805/get-outputs-in-model-predict-by-name-in-tensorflow-js

I can print the following console.log(model.outputNodes)

I get the resulting output.

[
  'Identity:0',
  'Identity_4:0',
  'detection_multiclass_scores',
  'num_detections',
  'raw_detection_scores',
  'detection_boxes',
  'raw_detection_boxes',
  'Identity_2:0'
]

It appears this is actually the correct order, however I needed to log Identity_4 and Identity_2

At this point I have the following:

  //Getting predictions
  const boxes = predictions[5].arraySync();
  const scores = predictions[1].dataSync();
  const classes = predictions[7].dataSync();

boxes = [
  [
    [
      0.35754913091659546,
      0.24033330380916595,
      0.8018297553062439,
      0.31154873967170715
    ],
    [
      0.28724685311317444,
      0.38544073700904846,
      0.7803705930709839,
      0.47172918915748596
    ],

...

scores = [
    0.5919038653373718,   0.4988971948623657,   0.4108578562736511,
    0.2585921883583069,   0.2472432553768158,   0.2457444965839386,
   0.24459102749824524,  0.23731976747512817,   0.2171754240989685,
...

classes = [
  1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 2,
  2, 1, 2, 2, 1, 2, 1, 1
...

Could this be because I started from a pre-trained model in order to train my custom model? I'm still searching for answers on how to properly choose the correct Tensors from the output for the bounding box. However these seem in line with what I need in order to construct the bounding boxes.

from tfjs-object-detection.

Hi @dusskapark and @saidRaiss, thanks for your question.

I did not find a documentation explaining why the boxes, scores, and classes were in indexes 4, 5 and 6. I figure it out by testing and I recommend to do what @dusskapark said and use version 1.7.4.

If you find smth explaining the reason for this weird sequence, let me know and I will add this info to the project :)

from tfjs-object-detection.

You're a legend @dusskapark! Thanks a lot for your contribution :)

from tfjs-object-detection.

Hello @dusskapark and @hugozanini

Thank you for this example and explanation, however I'm still having a hard time with this. I have generated a saved model .pb using tensorflow and I can successfully use that model to run predictions in python using tensorflow. I used the converter to convert my saved model to a graph model (json).

Using your suggestion above I run the saved_model_cli in the directory of my custom model. Below is the output

saved_model_cli show --dir saved_model --all

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_tensor'] tensor_info:
        dtype: DT_UINT8
        shape: (1, -1, -1, 3)
        name: serving_default_input_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_anchor_indices'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:0
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 4)
        name: StatefulPartitionedCall:1
    outputs['detection_classes'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:2
    outputs['detection_multiclass_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 3)
        name: StatefulPartitionedCall:3
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100)
        name: StatefulPartitionedCall:4
    outputs['num_detections'] tensor_info:
        dtype: DT_FLOAT
        shape: (1)
        name: StatefulPartitionedCall:5
    outputs['raw_detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 51150, 4)
        name: StatefulPartitionedCall:6
    outputs['raw_detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 51150, 3)
        name: StatefulPartitionedCall:7
  Method name is: tensorflow/serving/predict

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          input_tensor: TensorSpec(shape=(1, None, None, 3), dtype=tf.uint8, name='input_tensor')

This all looks ok, so I should interpret the results as

  const boxes = predictions[1].arraySync();
  const scores = predictions[4].dataSync();
  const classes = predictions[2].dataSync();

What I am expecting

• Boxes - is an array of vectors which are size 1 x 4 (for the bounding boxes - xmin,ymin,xmax,ymax)
• Score - array with probability of detection
• Classes - array of integers corresponding to my classes as below (so I should see array with just 1 or 2)

  let classesDir = {
    1: { name: "antenna", id: 1 },
    2: { name: "climbing_peg", id: 2 },
  };

Can you please elaborate further if I am interpreting the outputs correctly? Also can you explain when to use .arraySync() versus .dataSync()

I am working from the code in this repository, as well as the code snippet here: https://codesandbox.io/s/object-detection-with-tensorflow-js-forked-6r45k?file=/src/index.js:1288-1293

Thanks for your help on this.

from tfjs-object-detection.

Hello @dusskapark and @hugozanini

I wanted to follow up on this because I have found something different after some research.

Reading here: https://stackoverflow.com/questions/62270805/get-outputs-in-model-predict-by-name-in-tensorflow-js

I can print the following console.log(model.outputNodes)

I get the resulting output.

[
  'Identity:0',
  'Identity_4:0',
  'detection_multiclass_scores',
  'num_detections',
  'raw_detection_scores',
  'detection_boxes',
  'raw_detection_boxes',
  'Identity_2:0'
]

It appears this is actually the correct order, however I needed to log Identity_4 and Identity_2

At this point I have the following:

  //Getting predictions
  const boxes = predictions[5].arraySync();
  const scores = predictions[1].dataSync();
  const classes = predictions[7].dataSync();

boxes = [
  [
    [
      0.35754913091659546,
      0.24033330380916595,
      0.8018297553062439,
      0.31154873967170715
    ],
    [
      0.28724685311317444,
      0.38544073700904846,
      0.7803705930709839,
      0.47172918915748596
    ],

...

scores = [
    0.5919038653373718,   0.4988971948623657,   0.4108578562736511,
    0.2585921883583069,   0.2472432553768158,   0.2457444965839386,
   0.24459102749824524,  0.23731976747512817,   0.2171754240989685,
...

classes = [
  1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 2,
  2, 1, 2, 2, 1, 2, 1, 1
...

Could this be because I started from a pre-trained model in order to train my custom model? I'm still searching for answers on how to properly choose the correct Tensors from the output for the bounding box. However these seem in line with what I need in order to construct the bounding boxes.

Seems like model.outputNodes is not the right order too？

user my traing model, output is diffrent，(npm package : @tensorflow/tfjs": "^3.15.0"）

console.log(model.outputNodes) output:
 ['num_detections', 'raw_detection_boxes', 'Identity:0', 'Identity_2:0', 'detection_boxes', 'raw_detection_scores', 'Identity_4:0', 'detection_multiclass_scores']

console.log(predictions) output:

(8) [Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]
0: Tensor {kept: false, isDisposedInternal: true, shape: Array(1), dtype: 'float32', size: 1, …}
1: Tensor {kept: false, isDisposedInternal: true, shape: Array(3), dtype: 'float32', size: 7668, …}
2: Tensor {kept: false, isDisposedInternal: true, shape: Array(2), dtype: 'float32', size: 10, …}
3: Tensor {kept: false, isDisposedInternal: true, shape: Array(2), dtype: 'float32', size: 10, …}
4: Tensor {kept: false, isDisposedInternal: true, shape: Array(3), dtype: 'float32', size: 40, …}
5: Tensor {kept: false, isDisposedInternal: true, shape: Array(3), dtype: 'float32', size: 42174, …}
6: Tensor {kept: false, isDisposedInternal: true, shape: Array(2), dtype: 'float32', size: 10, …}
7: Tensor {kept: false, isDisposedInternal: true, shape: Array(3), dtype: 'float32', size: 220, …}

with, 4, 5, 2, I got no predict：

const boxes = predictions[4].arraySync();
const scores = predictions[5].arraySync();
const classes = predictions[2].dataSync();

I should set 4, 6, 3 to get the right result draw on screen。
classes = predictions[2].dataSync(); => got "Cannot read properties of undefined (reading 'name')", maybe class_id overflow。

use "size" of shape, I guess 40 is is the box，but there still three item has size of 10，does there any api description for this ?

from tfjs-object-detection.

It's very confusing. It looks like even though I'm training the same model as in the tutorial, the outputs of my exported model are in a different order.

According to this tensorflow code:

the following output nodes returned by the model.postprocess(..):

• num_detections: Outputs float32 tensors of the form [batch]
  that specifies the number of valid boxes per image in the batch.
• detection_boxes: Outputs float32 tensors of the form
  [batch, num_boxes, 4] containing detected boxes.
• detection_scores: Outputs float32 tensors of the form
  [batch, num_boxes] containing class scores for the detections.
• detection_classes: Outputs float32 tensors of the form
  [batch, num_boxes] containing classes for the detections.

and I can see these names with the saved_model_cli program described above.

But then in JS my model.signature has names like this:

	Identity_1:0:{name: 'Identity_1:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_1:0'
	tensorShape:{dim: Array(3)}
	
	Identity_2:0:{name: 'Identity_2:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_2:0'
	tensorShape:{dim: Array(2)}
	
	Identity_3:0:{name: 'Identity_3:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_3:0'
	tensorShape:{dim: Array(3)}
	
	Identity_4:0:{name: 'Identity_4:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_4:0'
	tensorShape:{dim: Array(2)}
	
	Identity_6:0:{name: 'Identity_6:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_6:0'
	tensorShape:{dim: Array(3)}
	
	Identity_7:0:{name: 'Identity_7:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_7:0'
	tensorShape:{dim: Array(3)}
	
	Identity:0:{name: 'Identity:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity:0'
	tensorShape:{dim: Array(2)}
	
	num_detections:{name: 'Identity_5:0', dtype: 'DT_FLOAT', tensorShape: {…}}
	dtype:'DT_FLOAT'
	name:'Identity_5:0'
	tensorShape:{dim: Array(1)}

And these are out of order from model.predictions but I can pair them up using model.outputs which has the "Identity" names in the same order as predictions.

The confusion remains though as there are multiple outputs that might correspond to the 'detection_boxes'. But only one is a list of integers which must be my class (Identity_4 which for me is at predictions[6]), and then I can find one with a max score which is the maximum of another list which shows the scores of all classes.

I'm left with two possibilities for the bounding boxes though. Unfortunately my model is currently behaving so badly that I can't visually tell which is which.

from tfjs-object-detection.