I've fine-tuned a few models (inception v3, mobilenet-v2-140-224x224, resnet-v2-50) and attempted to run them using cppflow, but am receiving incorrect values in the output tensor (different than when running python).

I retrained using a retrain.py modification for multi-label classification (https://github.com/isobar-us/multilabel-image-classification-tensorflow/tree/master/tf-multi-label-detection) and this pretrained model (https://tfhub.dev/google/imagenet/inception_v3/feature_vector/3), which I made sure wasn't a TF2 module / model version.

On the c++ side, I'm loading and executing the model (code is inefficient, but it should be producing the float32-normalized, planar image output tensorflow is expecting):

static const int IMAGE_DIM = 299; // for convenience in this code example
Model model("inception-v3/output_graph.pb");
auto inpName = new Tensor(model, "Placeholder");
auto outName = new Tensor(model, "final_result");

cv::Mat img = imread("input.png"); // 3-channel RGB input
cv::Mat inp;

// resize and convert to float
cv::resize(img, inp, cv::Size(IMAGE_DIM, IMAGE_DIM));
inp.convertTo(inp2, CV_32F, 1.0/255.0);
cv::Mat planes[3];
cv::split(inp2, planes);

std::vector<float > img_data;

// write R, G, and B planes into img_data (input planes are BGR)
for (int channel = 2; channel >= 0; channel--) {
    for (int ii=0; ii < IMAGE_DIM*IMAGE_DIM; ii++) {
        img_data.push_back(planes[channel].at<float>(ii));
    }
}
inpName->set_data(img_data, {1, IMAGE_DIM, IMAGE_DIM, 3});
model.run(inpName, outName);

std::vector<float> predictions = outName->get_data<float>();

The output in predictions is wildly different than running the same image through the python label_image, e.g.:

python label_image.py --input_height=299 --input_width=299 --graph=./inception-v3/output_graph.pb --labels=labels.txt --input_layer=Placeholder --output_layer=final_result --image input.png

I'm happy to provide an end-to-end example if that's more beneficial, but before doing that: has anyone experienced a similar issue? Am I missing something in the above code? Any help is much appreciated!

Hello,

I have been trying to test the examples, but every time I try 'make', I face such an error:

cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:79:23: error: expected unqualified-id before ‘[’ token
const auto[op_name, op_idx] = parse_name(std::get<0>(inputs[i]));
^
cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:80:68: error: ‘op_name’ was not declared in this scope
inp_ops[i].oper = TF_GraphOperationByName(this->graph, op_name.c_str());
^~~~~~~
cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:81:32: error: ‘op_idx’ was not declared in this scope
inp_ops[i].index = op_idx;
^~~~~~
cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:96:23: error: expected unqualified-id before ‘[’ token
const auto[op_name, op_idx] = parse_name(outputs[i]);
^
cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:97:68: error: ‘op_name’ was not declared in this scope
out_ops[i].oper = TF_GraphOperationByName(this->graph, op_name.c_str());
^~~~~~~
cppflow-cppflow2/examples/efficientnet/../../include/cppflow/model.h:98:32: error: ‘op_idx’ was not declared in this scope
out_ops[i].index = op_idx;
^~~~~~
CMakeFiles/example.dir/build.make:81: recipe for target 'CMakeFiles/example.dir/main.cpp.o' failed
make[2]: *** [CMakeFiles/example.dir/main.cpp.o] Error 1
CMakeFiles/Makefile2:94: recipe for target 'CMakeFiles/example.dir/all' failed
make[1]: *** [CMakeFiles/example.dir/all] Error 2
Makefile:102: recipe for target 'all' failed
make: *** [all] Error 2

It seems like there is no variable name after auto: const auto[op_name, op_idx] = parse_name(std::get<0>(inputs[i]));, and op_name and op_idx have not been created.

Could any one give me some help on how to resolve this issue?

Thanks in advance.

I have a neural net that has an input of [None, 353] and an ouput of [None, 1]. This however throws this error: You must feed a value for placeholder tensor 'output' with dtype float and shape [?,1] The output tensor in cppflow has shape -1, 1 and the input has -1, 353.
What is a proper way to solve this?

I have tried saving the model.pb again with the shape [1, 353] and [1, 1] but the same error came up.

This code breaks in Model.cpp at : TF_SessionRun(this->session, nullptr, io.data(), iv.data(), inputs.size(), oo.data(), ov, outputs.size(), nullptr, 0, nullptr, this->status);
this->status_check(true);

The interrupted content is：Microsoft C++ exception: std::runtime_error

I don't have any idea of that

#include
#include<stdlib.h>
#include "Model.h"
#include "Tensor.h"

using namespace std;
int main() {

Model model("D:/no_chinese_language/model2.pb");
//model.init();

Tensor input_a{ model, "cond/Merge" };
Tensor input_b{ model, "m_64" };
Tensor input_c{ model, "m_32" };
Tensor output{ model, "output" };
int pi0[16 * 16];
for (int i = 0; i < 16 * 16; i++)
{
	pi0[i] = 106;
}
int width = 16;
float mean = 0;
float std = 0;
float sum = 0;

int stride = 16;
int abe = width*width;
for (int i = 0; i < width; i++)
{
	for (int j = 0; j < width; j++)
	{
		//ofsinfo << pi0[i*stride + j] << endl;
		mean += pi0[i*stride + j];
		sum += pi0[i*stride + j] * pi0[i*stride + j];

	}
}
mean = mean / (abe);
std = sum / (abe)-mean*mean;
std = sqrt(std);
std::vector<float> data(width*width);
if (std == 0)
{
	float  a = 0;
	for (int i = 0; i < width; i++)
	{
		for (int j = 0; j < width; j++)
		{
			a = pi0[i*stride + j] - mean;
			data.push_back(a);
		}
	}
}
else
{
	float a = 0;
	for (int i = 0; i < width; i++)
	{
		for (int j = 0; j < width; j++)
		{
			a = (pi0[i*stride + j] - mean) / std;
			data.push_back(a);
		}
	}
}
vector<int16_t> data64;
data64.push_back(0);
vector<int16_t> data32;
data32.push_back(0);
input_a.set_data(data, { 1, 16, 16, 1 });
input_b.set_data(data64, {1,1});
input_c.set_data(data32,{1,1});
model.run({ &input_a, &input_b, &input_c },  output );
for (float f : output.get_data<float>()) {
	std::cout << (int)f << " ";
}
input_a.clean();
input_b.clean();
input_c.clean();
output.clean();
return 0;

}

meta_graph needs to be deleted. We can also pass in nullptr if it is not used.

The current implementation in raw_ops.h would leak TFE_Op (variable op) if TFE_OpAddInput or TFE_Execute failed and status_check() threw. One possible solution is to wrap TFE_Op with unique_ptr.

Hi!

I was wondering if you can specify which version of C++ you are running, as I am having difficulties running the wrapper (complains about converting "Error C2440 'return': cannot convert from 'T *' to 'std::vector<uint8_t,std::allocator<_Ty>>'"

Hi, serizba :)
a very nice project of yours, finally a nice way to use the Tensorflow API. Using Keras/Tensorflow serving I'm used to do batch interference. I was wondering if it is possible to do multiple inference also with your code.

Is set_data with a shape as a second argument the correct path for me to find the solution?

    int predictions = 2;
    std::vector<int32_t> data_a(predictions * 24);
    std::iota(data_a.begin(), data_a.end(), 1);
   #data_a.resize(24 * predictions);


    std::vector<float> data_b(predictions * 3);
    std::iota(data_b.begin(), data_b.end(), 1);
    #data_b.resize(3* predictions);


    const std::vector<std::int64_t> data_a_shape = { predictions,24};
    const std::vector<std::int64_t> data_b_shape = { predictions, 3 }

	input_a->set_data(data_a, data_a_shape);
	input_b->set_data(data_b, data_b_shape);


	m.run({ input_a, input_b }, output);

Just to add some extra information.
Example network has to input parameters A with 24 parameters and B with 3 parameters.
for prediction=1 it works fine

I ran in some issues following the 'How to run it" from this Github page. The Tensorflow C Api is placed into my home/libtensorflow and I'm running it on Windows 10 64bit.

To get the CMakeLists.txt to work I had to make some adjustments to it:
cmake_minimum_required(VERSION 3.10)
set(CMAKE_C_COMPILER "gcc")
set(CMAKE_CXX_COMPILER "g++")

project(example)

find_library(TENSORFLOW_LIB tensorflow HINT "C:/Users/Menno/libtensorflow/lib")

set(CMAKE_CXX_STANDARD 17)
add_executable(example main.cpp ../../src/Model.cpp ../../src/Tensor.cpp)
target_include_directories(example PRIVATE ../../include "C:/Users/Menno/libtensorflow/include/")
target_link_libraries (example "${TENSORFLOW_LIB}")

Also instead of calling CMake .. I had to use CMake -G "Unix Makefiles" ..

I've made a pastebin with all the output I'm getting: https://pastebin.com/ijn0H9aJ. In short it is a lot of errors with "undefined reference to '_imp__TF_X' where X stands for various TF components.

I'm stuck right now and would really appreciate any help I can get.

Thank you for your great cppLib! I came accross a problem. When I create Tensor like
auto input=new Tensor(model,"input"), a runtime error happened: no operation "input" exist. When I change to auto input=new Tensor(model,"ss") , an error : no operation "ss" happened. Do you know hwo to solve this question?

After following the instructions to load a model from the README, I can successfully compile the load_model example; however, when run, I receive a runtime error: Illegal instruction (core dumped)

I am running Ubuntu 19.10, GCC 9.2.1, the latest TensorFlow C API (libtensorflow-cpu-linux-x86_64-1.15.0).

Any tips on what I could be missing here?

Thank you!

Thank you for creating this awesome wrapper!

Please support model instantiation from already loaded data. In this specific case, I have an unsigned char array with a known length (unsigned int) which holds the data. I have hacked together an appropriate function by basically copying your code for loading the protobuffer for the most part. However, as I am quite new to c++, the code is quite ugly. I think this is not an uncommon case, and the wrapper would be even more useful by supporting it.

Kind regards, and best wishes.

Hello,

I am sorry for bothering you but is there some more help documentation that you would suggest to use your code with following model.
033_Hand_Detection_and_Tracking

and especially palm detection model that can be find here..
saved_model_palm_detection_builtin

I would like to implement TF .pb version and see speed difference in regards to already imeplented tflite.

Thank you very much,
Josip

Hey there!

I converted a Keras model to a .pb file and tested it with python and everything works as expected. The input is a rgb image, every value is converted to a value between 0 and 1. The output is a black and white mask of that image.
When I tried different ways of doing this in Cpp with your coco example as a base I got a wide range of results but none of them make any sense to me. Do you have an example of loading the input of an rgb image and scaling the pixel values to 0-1? I'm not sure if the issue is how I convert Input, Output or both.

Best regards,
Vaan

I'm using windows os , downloaded cppflow windows cpu only version from tensowrflow, unzip the file , and when trying to compile the code :

#include <stdio.h>
#include <tensorflow/c/c_api.h>

int main() {
printf("Hello from TensorFlow C library version %s\n", TF_Version());
return 0;
}

it gave me this error

From what I get .set_data() only takes a flat, 1-dimensional std::vector as input. If my model takes a 4-dimensional tensor as input with the shape (13. 84. 3. 5), how does it transform the flat std::vector into this shape?

Vice versa for output:
.get_data() return a flat vector. But what if my model outputs a multidimensional tensor, how will my data be ordered in this flat vector?

Hi,

what is the suggested model on the fly? lets say I calling model.run (to do the prediction) in production and want to reload a new model every 6 hours and have 50 models.

what tensorflow version do you currently support? 1.x and 2.1.x? does it support both if not why not?

edit: in the Tensorflow page I read: Note: There is no libtensorflow support for TensorFlow 2 yet. It is expected in a future release. any idea why its not out yet? what is improved with tensorflow 2?

the latest you can download seems to be: https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz

also can you say anything about required ram memory to load 1 model? I assume the model will be loaded from disk to memory - after the first model.run call?

    // Read the graph
    Model model("graph.pb");
    model.init();

Thanks

The code below is from Model.cpp, line 148, on the current master branch:

    // Read
    auto data = new char [size];
    file.seekg (0, std::ios::beg);
    file.read (data, size);

    // Error reading the file
    if (!file) {
        std::cerr << "Unable to read the full file: " << filename << std::endl;
        return nullptr;
    }

If the read fails for some reason and we get in the if (!file), the memory in data is not freed. This would be fixed by wrapping data in a std::unique_ptr<char[]> rather than a raw pointer.

Hello,

First of all, I really appreciate your work that enables to use Tensorflow in C++ without bazel.

I am using cppflow 2 to use the pre-trained super resolution model and output the SR images.

Below is the code that I made for it:
`#include
#include
#include <opencv2/opencv.hpp>
#include
#include
#include "../../include/cppflow/cppflow.h"

int main() {

auto input = cppflow::decode_jpeg(cppflow::read_file(std::string("small.jpg")));
input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
input = cppflow::expand_dims(input, 0);

auto shape = input.shape().get_data<int>(); //num_data, height, width, num_channel;
int num_row = shape[1];
int num_col = shape[2];


cppflow::model model("test_model");
auto output = model(input);
//output = cppflow::cast(output, TF_FLOAT, TF_UINT8);

auto data = output.get_data<float>();
std::cout << data.size() << std::endl;

std::vector<uint8_t> sr_img;
sr_img.assign(data.begin(), data.end());

std::cout << sr_img.size()*sizeof(uint8_t) << std::endl;

cv::Mat save = cv::Mat(num_row*4, num_col*4, CV_8UC3);

memcpy(save.data, sr_img.data(), sr_img.size()*sizeof(uint8_t));

//cv::cvtColor(save, save, cv::COLOR_RGB2BGR);
cv::imwrite("result.png", save);
//unsigned char* result = data.data();
//stbi_write_png("sky.png", shape[2]*4, shape[1]*4, shape[3], result, shape[2]*4 * shape[3]);

return 0;

}`

To load the input image, I used 'cppflow:decode_jpeg' function as in 'main.cpp' of efficientnet directory.

After that, I load the pretrained SR model which was generated by this link, and ran the super resolution.

The output from cppflow 2 was saved using opencv image write function (cv2::imwrite).

However, the output from python code and the one from cppflow 2 are different!

cppflow result
python3 result

I checked my SR model several times, but I don't think the model causes such inconsistency since it produces the same results in python.

Could anyone give me some guidance on why such difference happens?

Thanks in advance.

This doesn't work.

Tensor inpName1{ model, "X1" };
Tensor inpName2{ model, "X2" };
Tensor outNames{ model, "Y" };

cv::Mat img1, inp1, img2, inp2;
img1 = cv::imread("img1.png", cv::IMREAD_COLOR);
img2 = cv::imread("img2.png", cv::IMREAD_COLOR);

std::vector<uint8_t > img1_data, img2_data;
img1_data.assign(img1.data, img1.data + img1.total() * img1.channels());
inpName1.set_data(img1_data, { 1, img_size, img_size, 3 });

img2_data.assign(img2.data, img2.data + img2.total() * img2.channels());
inpName2.set_data(img2_data, { 1, img_size, img_size, 3 });

model.run({inpName1, inpName2}, &outNames);

Following the instructions inside README:

You can either install the library system wide by following the tutorial on the Tensorflow page or you can place the contents of the archive in a folder called libtensorflow in the root directory of the repo.

(specifically, using the libtensorflow folder) I got the following error

tensorflow/c/c_api.h: No such file or directory

Which I expected; Nowhere CMAKE is informed of where this libtensorflow folder is, it makes sense it doesn't link to it.

Do you know an easy way to fix this? Also I'd say we need to add the locations of the *.so files from tensorflow, not only headers.

I receive the following two errors while running mnist example using C++11 (cmake 3.10). Can you please help me solve this problem?

error: ‘make_unique’ is not a member of ‘std’
this->actual_shape = std::make_unique<decltype(actual_shape)::element_type>(shape.begin(), shape.end());

error: expected primary-expression before ‘>’ token
this->actual_shape = std::make_unique<decltype(actual_shape)::element_type>(shape.begin(), shape.end());

Hi, may I know where to download the _frozen_inference_graph.pb in the Coco example? Thanks.

Hello

With the commented code it looks like this is already on your radar, but just checking. Is this something you are working on currently to support?

Many thanks!

Hello serizba,

I succeeded with compiling the examples, but when testing on my own model, I ran into an error at the line of model construction, Model model(model_file):
terminate called after throwing an instance of 'std::runtime_error'
what(): Error: No operation named "init" exists
Aborted (core dumped)
Do you know why?

My model was converted from pytorch to tensorflow using onnx and I can load it in python. The following are the codes for testing:
int main() {
Model model("../mymodel.pb");
model.init();

auto input = new Tensor(model, "prefix/input_batch");
auto output_a  = new Tensor(model, "prefix/output_c");
auto output_b  = new Tensor(model, "prefix/output_v");

std::vector<float> img(1*3*361*641);
std::iota(img.begin(), img.end(), 0);
input->set_data(img, {1, 3, 361, 641});

model.run(input, {output_a, output_b});
for (float f : output_a->get_data<float>()) {
    std::cout << f << " ";
}
std::cout << std::endl;

}

Hello,
Interresting and usefull job. Thanks for sharing it !

I've tried to load with CppFlow a Tensorflow pretrained 'ssd_inception_v2_coco' model (for example, but I tried several other models) found here :
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

I wanted to do, in C++, something like what you can find here in Python :
https://github.com/opencv/opencv/blob/master/samples/dnn/mobilenet_ssd_accuracy.py
(or in the attached file)
my_ssd_accuracy.py.txt

So I tried with CppFlow a code like this :
Model model("ssd_inception_v2_coco_2018_01_28/frozen_inference_graph.pb");

That works fine.

Then :

auto outNames1 = new Tensor(model, "num_detections");
auto outNames2 = new Tensor(model, "detection_scores");
auto outNames3 = new Tensor(model, "detection_boxes");
auto outNames4 = new Tensor(model, "detection_classes");

auto inpName = new Tensor(model, "image_tensor");

But I was not able to make it work. Seems that there are errors after such functions as 'TF_GraphGetTensorShape' or 'TF_GraphGetTensorNumDims'.

I'm not a Tensorflow specialist but rather a user. Maybe am I wrong somewhere and it's not possible to load this kind of pretrained model.
Can you confirm it's possible ?

Thanks a lot !

I used cppFlow for TF2 and got following error.
Does it support tensorflow 2?

#include "../../include/Model.h"
#include "../../include/Tensor.h"
#include <opencv2/opencv.hpp>
#include
#include

int main() {

// Create model
Model m("../model.pb");

}

Error LNK2019 unresolved external symbol "public: __cdecl Model::Model(class std::basic_string<char,struct std::char_traits,class std::allocator > const &,class std::vector<unsigned char,class std::allocator > const &)" (??0Model@@qeaa@AEBV?$basic_string@DU?$char_traits@D@std@@v?$allocator@D@2@@std@@aebv?$vector@EV?$allocator@E@std@@@2@@z) referenced in function main ConsoleApplication1 D:\testcode\C++TF\MyCode\ConsoleApplication1\ConsoleApplication1\ConsoleApplication1.obj 1

Hi,

Despite calling TF_DeleteSession & TF_DeleteGraph, GPU memory wasn't released. i'm using TF C api 1.15.
Note:
It only release the memory when it exits the exe.

nice work! I am wondering whether the model will use all available gpus and all memory by default. If it does, how to set it? thanks

Hi. How can i complete batch inference? For example i have a model which input size is (?, 128, 64,3), so in python i can pass for example (8, 128, 64,3) or (16, 128, 64,3) using this code:

   input_var = tf.get_default_graph().get_tensor_by_name("images:0")
    output_var = tf.get_default_graph().get_tensor_by_name("features:0")
    val_x = []
     for filename in glob.glob(*.png"):
        print(filename)
        im = Image.open(filename)
        val_x.append(np.asarray(im))
    val_x_np = np.array(val_x)
    out = np.zeros((len(val_x_np), 128), np.float32)
    out = session.run(output_var, feed_dict={input_var: val_x_np})

but i can't understand how to do this using ccpflow... Could you explain, please?

Why the 'new' s in the basic example?

Also I've some concerns about code like
auto *dims = new int64_t[n_dims];

I see no deletes for this

Of course I suppose you are aware that C++ is not garbage collected.

I've also seen malloc and delete[] mixed, I'm a bit concerned about the whole memory safety of the lib.

Could you please explain me a bit more about this?

Hey, serizba

Firstly, I'd like to thank you for uploading your code that helped me understand how to use the Tensorflow C API :)

I'd also like to ask you about how your Tensor object handles scalars or 0-order tensors.

If I understand correctly, your "Tensor" constructor doesn't modify the "shape" vector in case of a scalar (n_dims == 0) and proceeds constructing the Tensor object yet you throw an exception in the "set_data" method if the vector "shape" is empty.

this->error_check(!this->shape.empty(), "Shape of the input Tensor is not known, please provide a shape");

It'd be great if you could explain the logic behind this. I assume that your code doesn't support 0-rank tensors (scalars), right ?

Thanks!

The current context::get_status() shares the same TF_Status in context, which cause a race condition if executed concurrently.

I am not sure about TFE_Context though. The header mentions some of its internal status are thread local.

One of the solutions is to use thread local context and status, e.g. as in Swift. (thread_local keyword in C++) I think this is the better solution.

The other solution is to allow users to create and manage their own context. We also need to keep the global default context for eager execution.

Edit:
The Context implementation in Swift API for tensorflow is here. It uses a ContextManager (a stack of Context) to manage the default Context for the current thread and stores them in ThreadLocalStorage. I am not sure whether we need ContextManager or not, but it seems thread_local should also work for Context.

Hi recently I use cppflow to load model, witch is already transform into .pb form(used to be .h5 form), , and predict with this model(just simple two classes problem).
The model load is successful done, but when using it predicts resluts, I got different result compare with the original model in python？ Do you have any idea why this may happened?

Model does not have a custom deleter and therefore the Graph and the Session are not deleted properly.

Both Graph and Session should probably be wrapped on a smart pointer specifying their deleting functions.

Hey,

If I want to use this on an ide like QT, what should my lib path and linkers be? Thanks !

I went through the examples. It was well explained.

I wanted to ask if this git repo can be used to Tensorflow lite models too? In this git repo we are loading model graph and then loading weights before the inference. However, in Tensorflow lite we load the model using .tfLite file, like in the following example:

TF-lite:

https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_c

Hello,

In the class Tensor.cpp, the buffer "data" is allocated with "malloc" and then given to "TF_NewTensor" with a custom deleter. This deleter uses "delete[]" to desallocate the buffer.

Should we use "free" instead of "delete []" ?
Am I wrong ?

Thanks

Could you confirm that cppflow should work with TensorFlow 2.2?
And if so, could you provide an example of how to save a model so it works in TensorFlow 2.2?

The reason I ask this is that I get "Invalid GraphDef" when trying to use a model.pb file created by model.save() of TF2.2.

Thanks,

Rani

Hi,

Thanks for this amazing work.

I compiled the cppflow library with visual studio and was able to run the example demo. As a next step, I was trying to load the tensorflow based deepsort model through cppflow. The model files are available in this repo. However, it seems I am getting an error in windows. Looks like cppflow is not able to load the model. Is it possible for you to help me in this regard?

Thanks,
Aditya Vora

Is there any provision for dealing with strings? So, when there tensor is of dtype DT_STRING and shape (-1), will the following work?

Tensor words{model, "sentence"};
std::vector<std::string> words_data{"hello", "world"};
words.set_data(batch_ids_data, {4});

Or should I deal with strings as explained in https://stackoverflow.com/questions/41138822/how-to-create-a-string-type-tensor-in-tensorflow-c-api ?

Hello!
I was trying out this tool and I bumped into an error when trying to run my own model. The example ran flawlessly.
I used Tensorflow 2.3 and Python 3.8 to create a simple sequential Keras model and train it. Then I used method "save" to export it as a Protobuf file (.pb). The C++ file is simple and based on the provided example, and it simply tries to load the model.

Is there any particular way in which Keras models should be saved in order to work with this tool?

Thanks.

This issue serves as a shared note to track the status of SavedModel C API in TF 2.0, and also for discussions on possible implementations in cppflow2.

The original RFC proposal is here. The loading functions are defined here

However, as of this writing, the new SavedModel C API in TF 2.0 is still work-in-progress. It is not enabled in bazel. To enable the API, the following patch is needed. In addition, headers need to be copied manually. Although this patch enables the experimental APIs in the libtensorflow, calling these APIs will throw unimplemented errors. (at least in v2.3) I saw commits were submitted to support the new SavedModel C API in v2.4, so some of them may work now.

diff --git a/tensorflow/BUILD b/tensorflow/BUILD
index d00608ccc9..8bab3a6e3e 100644
--- a/tensorflow/BUILD
+++ b/tensorflow/BUILD
@@ -746,6 +746,7 @@ tf_cc_shared_object(
         "//tensorflow/c:version_script.lds",
         "//tensorflow/c/eager:c_api",
         "//tensorflow/c/eager:c_api_experimental",
+        "//tensorflow/c/experimental/saved_model/public:c_saved_model_api",
         "//tensorflow/core:distributed_tensorflow_dependencies",
         "//tensorflow/core:tensorflow",
     ],

There seems to be two ways to call the model, ConcreteFunction (TF_GetSavedModelConcreteFunction) and SignatureDefFunction (TF_GetSavedModelSignatureDefFunction) defined in this header, but both of them return TFE_Op* (TF_SignatureDefFunctionMakeCallOp and TF_ConcreteFunctionMakeCallOp). They should be compatible with eager execution in cppflow2.

Additional notes

• The ConcreteFunction signature may involve high-level python class, e.g. RaggedTensorSpec for RaggedTensor. They may require significant work to be implemented in C++. The corresponding SignatureDefFunction is rather low-level, where all the Tensors are flattened, e.g. the prevous RaggedTensor is represented by two flat Tensors. From this perspective, starting with SignatureDefFunction seems to be a good idea.

Edit:

• ConcreteFunction (TF_GetSavedModelConcreteFunction) implementation seems to be available on the master branch.
• SignatureDefFunction (TF_GetSavedModelSignatureDefFunction) is not implemented yet.
• The tests provide some examples.

Edit2:
A WIP for the TF v2 SavedModel API in cppflow: https://github.com/ljn917/cppflow/tree/cppflow2-savedmodel

Known limitations:

• Concrete function can only have one input signature. It means it cannot load keras models. You need to annotate the functions with tf.function.
• SignatureDef function is not supported in libtensorflow yet.

Hi,

I am running through the examples for this. I'm just wondering if it works correctly, especially for the mnist example, since the predictions are not accurate at all. I get this:

Real label: 0, predicted: 2, Probability: 1
Real label: 1, predicted: 0, Probability: 0.54904
Real label: 2, predicted: 2, Probability: 1
Real label: 3, predicted: 5, Probability: 0.639081
Real label: 4, predicted: 8, Probability: 0.472427
Real label: 5, predicted: 2, Probability: 1
Real label: 6, predicted: 8, Probability: 0.908674
Real label: 7, predicted: 5, Probability: 0.984732
Real label: 8, predicted: 2, Probability: 0.999436
Real label: 9, predicted: 5, Probability: 0.974823

It could be great to indicate what are the expected outputs for each example.

Thanks,

Carl

Hey there!

I converted a Keras model to a .pb file and tested it with python and everything works as expected. The input is a rgb image, every value is converted to a value between 0 and 1. The output is a black and white mask of that image.
When I tried different ways of doing this in Cpp with your coco example as a base I got a wide range of results but none of them make any sense to me. Do you have an example of loading the input of an rgb image and scaling the pixel values to 0-1? I'm not sure if the issue is how I convert Input, Output or both.

Best regards,
Sally

Hi,

do you have a way to execute model.run on a gpu (NVidia)?

Thanks