Memory usage of TensorFlow models about serving HOT 7 OPEN

@singhniraj08 Any pointers on above?

from serving.

Hey @singhniraj08 Can you point us to any workarounds in the meantime?

from serving.

@ndeepesh, Can you try using Memory profiling tool if that helps. Since the models are running online on c++ model servers, tracking memory usage becomes difficult in that case.

I am bumping up this issue internally for solution and we will update this thread. Thank you!

from serving.

Thanks @singhniraj08. Without a good estimate this is causing regressions in our hosts where we are going OOM easily with no way to tell which model is the issue

from serving.

@singhniraj08 Will the memory profile tool gets populated for profiling on CPUs? I havent see this getting populated on cpu profiles

from serving.

@ndeepesh, Memory profiling tools monitors the memory usage of your device during the profiling interval. If you are looking for CPU usage while running TF Serving, Profiling inference requests with Tensorboard may help you achieve that. Thanks.

from serving.

@singhniraj08 This only gives us what tensorflow op is taking how much time right? Not the amount of memory it occupies on CPU.

from serving.