Validation and testing questions about deeppath HOT 8 CLOSED

Hi,

1- I haven't seen this error, sorry.

2- Those files just summarize whatever is in the test_####k folders. You should have 1 folder per checkpoint saved. It looks like in your case, you only have the 10000 iteration saved or available. This summary is only done for the "out2*" files (meaning per slide averages), for first 3 labels, macro and micro averages. They show the AUC and CIs. It's just a summary but you can change the code easily if you need to extract other info. Labels "1", "2" and "3" would be whatever order was in your "sorted" folder (before conversion to TFRecord), in alphabetical order.

3- These are the probabilities generated for each tile. You have the tile name and in the brackets, the probabilities generated for the 3 classes (you have an additional class (the first one) called the "background" class which is part of the original inception). The number after the bracket is the corrected probability (ignoring the background class) for the expected label.

HTH
N.

from deeppath.

Thanks.

Bill

from deeppath.

The error that I described above is fixed by changing line 66 in nc_inception_eval.py from:
tf.initialize_all_variables()
to
tf.global_variables_initializer()
ref. here. initialize_all_variables() is deprecated and apparently is broken in newer tensorflow releases.

Bill

from deeppath.

Thanks for the follow up, Bill.

from deeppath.

I am encountering additional errors in attempting to train for mutation classification using ...01_training/xClasses/bazel-bin/inception/imagenet_train (output below). I am presuming it is a similar deprecation issue. I'm running tensorflow-gpu 1.13.1 for compatibility with the latest CUDA. I encountered many issues trying to install earlier CUDA versions.

It appears to not like the labels component, but they look OK to me. Here is an example:

feature {
key: "image/class/label"
value {
int64_list {
value: 0
value: 1
value: 0
value: 0
value: 0
value: 0
value: 0
value: 0
value: 0
value: 1
value: 1
}
}
}

Do I, perhaps, build inception differently for multiclass? If you have any suggestions, would appreciate your help.

Thanks,
Bill

Partial output from failing multiclass training:
WARNING:tensorflow:From /home/bcliffor/DeepPATH/DeepPATH_code/01_training/xClasses/bazel-bin/inception/imagenet_train.runfiles/inception/inception/inception_train.py:339: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use thetf.data` module.
2019-04-29 16:14:14.696263: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-04-29 16:14:24.091919: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: image/class/label. Can't parse serialized Example.
2019-04-29 16:14:24.091920: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: image/class/label. Can't parse serialized Example.
2019-04-29 16:14:24.091920: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: image/class/label. Can't parse serialized Example.
2019-04-29 16:14:24.091935: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: image/class/label. Can't parse serialized Example.
ERROR:tensorflow:Exception in QueueRunner: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample_1/ParseSingleExample}}]]
ERROR:tensorflow:Exception in QueueRunner: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample/ParseSingleExample}}]]
ERROR:tensorflow:Exception in QueueRunner: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample_2/ParseSingleExample}}]]
ERROR:tensorflow:Exception in QueueRunner: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample_3/ParseSingleExample}}]]
Exception in thread QueueRunnerThread-batch_processing/batch_join/fifo_queue-batch_processing/batch_join/fifo_queue_enqueue_3:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample_3/ParseSingleExample}}]]

Exception in thread QueueRunnerThread-batch_processing/batch_join/fifo_queue-batch_processing/batch_join/fifo_queue_enqueue_1:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/bcliffor/dp/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Key: image/class/label. Can't parse serialized Example.
[[{{node batch_processing/ParseSingleExample_1/ParseSingleExample}}]]`

from deeppath.

Hi,

There is an extra "background" class in inception, so an additional one will be automatically added during the processing (you can check the original github page: https://github.com/tensorflow/models/blob/master/research/inception/README.md)

Just be careful also when you convert to TFRecord. You can double check everything is fine with the name of the test TFRecord files. the "true" label appears as a binary at the end of the file name. If you have 10 labels, the label text file must have exactly 10 lines (no empty line at the top or at the bottom)

HTH,
Best,
Nicolas

from deeppath.

from deeppath.

Hi - Glad this is solved.

I indeed used "multiClasses/nc_imagenet_eval.py" originally, but the "xClasses/nc_imagenet_eval.py" should work too (if you use that one, you would need to add the " --mode='1_sigmoid'" option).
At some I will probably delete the "multiClasses/nc_imagenet_eval.py" from the code and just keep the second xClass version.

Best,
Nicolas

from deeppath.