Difference between Metrics emitted by TF Serving about serving HOT 4 CLOSED

@ndeep27,

Request Latency would account the time taken by Tensorflow Serving to process the request and send predictions. This includes waiting for resources, input/output operations, or other processes.

Runtime Latency would account the time taken by Tensorflow runtime within Tensorflow Serving to process the request and generate predictions.

The difference between metrics of using GPU and CPU might be because of the fact that the overhead of invoking GPU kernels, and copying data to and from GPU, is very high. For operations on models with very little parameters it is not worth of using GPU since frequency of CPU cores is much higher.

Thank you!

from serving.

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

from serving.

This issue was closed due to lack of activity after being marked stale for past 7 days.

from serving.

Are you satisfied with the resolution of your issue?
Yes
No

from serving.