Training Loss about image_captioning HOT 3 CLOSED

Sorry for late response, I found one of my experimental logs on my old laptop, but it's similar to the provided examples, hope it helps.

I0121 11:22:25.310305 6981 image_data_layer.cpp:150] Restarting data prefetching from start.
I0121 11:22:25.736824 6967 solver.cpp:414] Test net output #0: accuracy = 0.00039446
I0121 11:22:25.736872 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 7.73027 (* 20 = 154.605 loss)
I0121 11:22:25.942699 6967 solver.cpp:242] Iteration 0, loss = 151.539
I0121 11:22:25.942756 6967 solver.cpp:258] Train net output #0: accuracy = 0
I0121 11:22:25.942773 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 7.57697 (* 20 = 151.539 loss)
I0121 11:22:25.942802 6967 solver.cpp:571] Iteration 0, lr = 0.01
I0121 11:22:40.078191 6967 solver.cpp:242] Iteration 50, loss = 130.968
I0121 11:22:40.078258 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478
I0121 11:22:40.078274 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.46427 (* 20 = 109.285 loss)
I0121 11:22:40.078290 6967 solver.cpp:571] Iteration 50, lr = 0.01
I0121 11:22:54.318545 6967 solver.cpp:242] Iteration 100, loss = 120.63
I0121 11:22:54.318614 6967 solver.cpp:258] Train net output #0: accuracy = 0.057971
I0121 11:22:54.318639 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.69797 (* 20 = 113.959 loss)
I0121 11:22:54.318652 6967 solver.cpp:571] Iteration 100, lr = 0.01
I0121 11:23:08.571382 6967 solver.cpp:242] Iteration 150, loss = 107.777
I0121 11:23:08.571498 6967 solver.cpp:258] Train net output #0: accuracy = 0.160714
I0121 11:23:08.571521 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.1158 (* 20 = 102.316 loss)
I0121 11:23:08.571535 6967 solver.cpp:571] Iteration 150, lr = 0.01
I0121 11:23:22.804710 6967 solver.cpp:242] Iteration 200, loss = 103.695
I0121 11:23:22.804764 6967 solver.cpp:258] Train net output #0: accuracy = 0.181818
I0121 11:23:22.804780 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.86012 (* 20 = 97.2023 loss)
I0121 11:23:22.804795 6967 solver.cpp:571] Iteration 200, lr = 0.01
I0121 11:23:36.999411 6967 solver.cpp:242] Iteration 250, loss = 102.157
I0121 11:23:36.999465 6967 solver.cpp:258] Train net output #0: accuracy = 0.15625
I0121 11:23:36.999483 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.81321 (* 20 = 96.2642 loss)
I0121 11:23:36.999497 6967 solver.cpp:571] Iteration 250, lr = 0.01
I0121 11:23:51.203235 6967 solver.cpp:242] Iteration 300, loss = 100.736
I0121 11:23:51.203356 6967 solver.cpp:258] Train net output #0: accuracy = 0.132075
I0121 11:23:51.203375 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.8058 (* 20 = 96.116 loss)
I0121 11:23:51.203397 6967 solver.cpp:571] Iteration 300, lr = 0.01
I0121 11:24:05.397879 6967 solver.cpp:242] Iteration 350, loss = 99.7511
I0121 11:24:05.397958 6967 solver.cpp:258] Train net output #0: accuracy = 0.137931
I0121 11:24:05.397976 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.43398 (* 20 = 108.68 loss)
I0121 11:24:05.397989 6967 solver.cpp:571] Iteration 350, lr = 0.01
I0121 11:24:19.592331 6967 solver.cpp:242] Iteration 400, loss = 99.3335
I0121 11:24:19.592398 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478
I0121 11:24:19.592424 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.83965 (* 20 = 96.793 loss)
I0121 11:24:19.592437 6967 solver.cpp:571] Iteration 400, lr = 0.01
I0121 11:24:33.784838 6967 solver.cpp:242] Iteration 450, loss = 98.8016
I0121 11:24:33.784994 6967 solver.cpp:258] Train net output #0: accuracy = 0.15942
I0121 11:24:33.785017 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.7255 (* 20 = 94.51 loss)
I0121 11:24:33.785035 6967 solver.cpp:571] Iteration 450, lr = 0.01
I0121 11:24:47.991668 6967 solver.cpp:242] Iteration 500, loss = 98.3618
I0121 11:24:47.991726 6967 solver.cpp:258] Train net output #0: accuracy = 0.09375
I0121 11:24:47.991745 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.941 (* 20 = 98.8199 loss)
I0121 11:24:47.991763 6967 solver.cpp:571] Iteration 500, lr = 0.01
I0121 11:25:02.184698 6967 solver.cpp:242] Iteration 550, loss = 98.6693
I0121 11:25:02.184752 6967 solver.cpp:258] Train net output #0: accuracy = 0.111111
I0121 11:25:02.184772 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.78535 (* 20 = 95.7069 loss)
I0121 11:25:02.184784 6967 solver.cpp:571] Iteration 550, lr = 0.01
I0121 11:25:16.382648 6967 solver.cpp:242] Iteration 600, loss = 98.394
I0121 11:25:16.382751 6967 solver.cpp:258] Train net output #0: accuracy = 0.20339
I0121 11:25:16.382772 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.87894 (* 20 = 97.5788 loss)
I0121 11:25:16.382788 6967 solver.cpp:571] Iteration 600, lr = 0.01
I0121 11:25:30.572062 6967 solver.cpp:242] Iteration 650, loss = 97.412
I0121 11:25:30.572118 6967 solver.cpp:258] Train net output #0: accuracy = 0.0983607
I0121 11:25:30.572135 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.96279 (* 20 = 99.2559 loss)
I0121 11:25:30.572150 6967 solver.cpp:571] Iteration 650, lr = 0.01
I0121 11:25:44.765990 6967 solver.cpp:242] Iteration 700, loss = 97.6146
I0121 11:25:44.766044 6967 solver.cpp:258] Train net output #0: accuracy = 0.226415
I0121 11:25:44.766060 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.87698 (* 20 = 97.5396 loss)
I0121 11:25:44.766072 6967 solver.cpp:571] Iteration 700, lr = 0.01
I0121 11:25:58.954264 6967 solver.cpp:242] Iteration 750, loss = 96.491

.......
I0121 23:45:21.838233 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 2.1057 (* 20 = 42.1139 loss)
I0121 23:45:21.838251 6967 solver.cpp:571] Iteration 109850, lr = 0.0025
I0121 23:45:35.991255 6967 solver.cpp:242] Iteration 109900, loss = 44.926
I0121 23:45:35.991304 6967 solver.cpp:258] Train net output #0: accuracy = 0.396552
I0121 23:45:35.991319 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 2.51174 (* 20 = 50.2349 loss)
I0121 23:45:35.991331 6967 solver.cpp:571] Iteration 109900, lr = 0.0025
I0121 23:45:50.162571 6967 solver.cpp:242] Iteration 109950, loss = 45.8994
I0121 23:45:50.162673 6967 solver.cpp:258] Train net output #0: accuracy = 0.377049
I0121 23:45:50.162689 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 3.05881 (* 20 = 61.1762 loss)
I0121 23:45:50.162700 6967 solver.cpp:571] Iteration 109950, lr = 0.0025
I0121 23:46:04.124423 6967 solver.cpp:449] Snapshotting to binary proto file ./examples/flickr8K/trained_models/lrcn20160121_iter_110000.caffemodel
I0121 23:46:04.124464 6967 net.cpp:948] Serializing 34 layers
I0121 23:47:45.034051 6967 solver.cpp:734] Snapshotting solver state to binary proto file./examples/flickr8K/trained_models/lrcn20160121_iter_110000.solverstate
I0121 23:47:47.449722 6967 solver.cpp:326] Iteration 110000, loss = 36.1393
I0121 23:47:47.449769 6967 solver.cpp:346] Iteration 110000, Testing net (#0)
I0121 23:47:47.449782 6967 net.cpp:781] Copying source layer data
I0121 23:47:47.449792 6967 net.cpp:781] Copying source layer data
I0121 23:47:47.449798 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split
I0121 23:47:47.449803 6967 net.cpp:781] Copying source layer target_sentence_data_2_split
I0121 23:47:47.449811 6967 net.cpp:781] Copying source layer silence
I0121 23:47:47.449816 6967 net.cpp:781] Copying source layer conv1
I0121 23:47:47.449825 6967 net.cpp:781] Copying source layer relu1
I0121 23:47:47.449829 6967 net.cpp:781] Copying source layer pool1
I0121 23:47:47.449834 6967 net.cpp:781] Copying source layer norm1
I0121 23:47:47.449839 6967 net.cpp:781] Copying source layer conv2
I0121 23:47:47.449847 6967 net.cpp:781] Copying source layer relu2
I0121 23:47:47.449852 6967 net.cpp:781] Copying source layer pool2
I0121 23:47:47.449857 6967 net.cpp:781] Copying source layer norm2
I0121 23:47:47.449862 6967 net.cpp:781] Copying source layer conv3
I0121 23:47:47.449868 6967 net.cpp:781] Copying source layer relu3
I0121 23:47:47.449873 6967 net.cpp:781] Copying source layer conv4
I0121 23:47:47.449880 6967 net.cpp:781] Copying source layer relu4
I0121 23:47:47.449887 6967 net.cpp:781] Copying source layer conv5
I0121 23:47:47.449892 6967 net.cpp:781] Copying source layer relu5
I0121 23:47:47.449898 6967 net.cpp:781] Copying source layer pool5
I0121 23:47:47.449903 6967 net.cpp:781] Copying source layer fc6
I0121 23:47:47.449908 6967 net.cpp:781] Copying source layer relu6
I0121 23:47:47.449915 6967 net.cpp:781] Copying source layer drop6
I0121 23:47:47.449920 6967 net.cpp:781] Copying source layer fc7
I0121 23:47:47.449926 6967 net.cpp:781] Copying source layer relu7
I0121 23:47:47.449933 6967 net.cpp:781] Copying source layer drop7
I0121 23:47:47.449937 6967 net.cpp:781] Copying source layer fc8
I0121 23:47:47.449944 6967 net.cpp:781] Copying source layer embedding
I0121 23:47:47.449949 6967 net.cpp:781] Copying source layer lstm1
I0121 23:47:47.449957 6967 net.cpp:781] Copying source layer lstm2
I0121 23:47:47.449964 6967 net.cpp:781] Copying source layer predict
I0121 23:47:47.449970 6967 net.cpp:781] Copying source layer predict_predict_0_split
I0121 23:47:47.449975 6967 net.cpp:781] Copying source layer cross_entropy_loss
I0121 23:47:47.449990 6967 net.cpp:781] Copying source layer accuracy
I0121 23:49:40.741700 6967 solver.cpp:414] Test net output #0: accuracy = 0.524921
I0121 23:49:40.741817 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 1.99833 (* 20 = 39.9666 loss)
I0121 23:49:40.741830 6967 solver.cpp:346] Iteration 110000, Testing net (#1)
I0121 23:49:40.741847 6967 net.cpp:781] Copying source layer data
I0121 23:49:40.741857 6967 net.cpp:781] Copying source layer data
I0121 23:49:40.741864 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split
I0121 23:49:40.741871 6967 net.cpp:781] Copying source layer target_sentence_data_2_split
I0121 23:49:40.741881 6967 net.cpp:781] Copying source layer silence
I0121 23:49:40.741890 6967 net.cpp:781] Copying source layer conv1
I0121 23:49:40.741905 6967 net.cpp:781] Copying source layer relu1
I0121 23:49:40.741914 6967 net.cpp:781] Copying source layer pool1
I0121 23:49:40.741925 6967 net.cpp:781] Copying source layer norm1
I0121 23:49:40.741932 6967 net.cpp:781] Copying source layer conv2
I0121 23:49:40.741945 6967 net.cpp:781] Copying source layer relu2
I0121 23:49:40.741955 6967 net.cpp:781] Copying source layer pool2
I0121 23:49:40.741966 6967 net.cpp:781] Copying source layer norm2
I0121 23:49:40.741976 6967 net.cpp:781] Copying source layer conv3
I0121 23:49:40.741986 6967 net.cpp:781] Copying source layer relu3
I0121 23:49:40.741997 6967 net.cpp:781] Copying source layer conv4
I0121 23:49:40.742007 6967 net.cpp:781] Copying source layer relu4
I0121 23:49:40.742018 6967 net.cpp:781] Copying source layer conv5
I0121 23:49:40.742029 6967 net.cpp:781] Copying source layer relu5
I0121 23:49:40.742040 6967 net.cpp:781] Copying source layer pool5
I0121 23:49:40.742049 6967 net.cpp:781] Copying source layer fc6
I0121 23:49:40.742061 6967 net.cpp:781] Copying source layer relu6
I0121 23:49:40.742074 6967 net.cpp:781] Copying source layer drop6
I0121 23:49:40.742089 6967 net.cpp:781] Copying source layer fc7
I0121 23:49:40.742105 6967 net.cpp:781] Copying source layer relu7
I0121 23:49:40.742116 6967 net.cpp:781] Copying source layer drop7
I0121 23:49:40.742127 6967 net.cpp:781] Copying source layer fc8
I0121 23:49:40.742139 6967 net.cpp:781] Copying source layer embedding
I0121 23:49:40.742152 6967 net.cpp:781] Copying source layer lstm1
I0121 23:49:40.742166 6967 net.cpp:781] Copying source layer lstm2
I0121 23:49:40.742179 6967 net.cpp:781] Copying source layer predict
I0121 23:49:40.742192 6967 net.cpp:781] Copying source layer predict_predict_0_split
I0121 23:49:40.742203 6967 net.cpp:781] Copying source layer cross_entropy_loss
I0121 23:49:40.742214 6967 net.cpp:781] Copying source layer accuracy
I0121 23:51:33.523701 6981 image_data_layer.cpp:150] Restarting data prefetching from start.
I0121 23:51:33.947804 6967 solver.cpp:414] Test net output #0: accuracy = 0.369726
I0121 23:51:33.947850 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 3.00547 (* 20 = 60.1093 loss)
I0121 23:51:33.947860 6967 solver.cpp:331] Optimization Done.
I0121 23:51:33.947875 6967 caffe.cpp:214] Optimization Done.

from image_captioning.

from image_captioning.

@deepsemantic can you please help me out in training the model. I have trained the model with 100 batch size as my GPU gives out of memory exception if batch size is 150.

The model gets trained and after 25000 iteration accuracy is 0.35 and loss is 3.7 but when i test the model using "bi_generation_retrieval.py", the output is as follows :

...................
755 ./data/flickr8K/images/1056249424_ef2a2e041c.jpg
Two dog play in the water.
756 ./data/flickr8K/images/3265864834_e0229020dd.jpg
Two dog play in the water.
757 ./data/flickr8K/images/802594049_289e3c8420.jpg
Two dog play in the water.
758 ./data/flickr8K/images/3430779304_43a2146f4b.jpg
A man in a red shirt be stand on a sidewalk.
759 ./data/flickr8K/images/2455286250_fb6a66175a.jpg
Two dog play in the water.
760 ./data/flickr8K/images/1425919702_ddb761aeec.jpg
Two dog play in the water.
761 ./data/flickr8K/images/1088767354_2acee738cf.jpg
A man in a red shirt be stand on a bench.
762 ./data/flickr8K/images/540338917_57069687be.jpg
Two dog play in the water.
763 ./data/flickr8K/images/3459871361_92d1ecda36.jpg
A man in a red shirt be stand on a bench.
764 ./data/flickr8K/images/576920249_df1bdc2068.jpg
A man in a red shirt be stand on a sidewalk.
765 ./data/flickr8K/images/2872197070_4e97c3ccfa.jpg
Two dog play in the water.
766 ./data/flickr8K/images/3495349745_1b29a63571.jpg
Two dog play in the water.
767 ./data/flickr8K/images/2943334864_6bab479a3e.jpg
Two dog play in the water.
....................................

Its just these two sentences repeated for all 1000 val images. Any clue what I can do to trained it right? and will it be possible for you to share a trained model?
My email is [email protected].

Thank you

from image_captioning.