Use 'evaluate' without the `HO_superpopulation` file about genocae HOT 6 CLOSED

Now this fails with error (from this GHA log):

Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1777, in genfromtxt
    fhd = iter(fid)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_gcae.py", line 1616, in <module>
    main()
  File "run_gcae.py", line 1561, in main
    write_f1_scores_to_csv(results_directory, "epoch_{0}".format(epoch), superpopulations_file, f1_scores_by_pop, coords_by_pop)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/visualization.py", line 603, in write_f1_scores_to_csv
    superpop_pop_dict = get_superpop_pop_dict(superpopulations_file)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/data_handler.py", line 869, in get_superpop_pop_dict
    pop_superpop_list = get_pop_superpop_list(pop_superpop_file)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/data_handler.py", line 626, in get_pop_superpop_list
    pop_superpop_list = np.genfromtxt(file, usecols=(0,1), dtype=str, delimiter=",")
  File "/home/runner/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1779, in genfromtxt
    raise TypeError(
TypeError: fname must be a string, filehandle, list of strings, or generator. Got <class 'NoneType'> instead.

I can imagine the problem is that I trained the CNN too short, all shapes were skipped and hence a division by zero (successfull shapes) occurred:

Imputing originally missing genotypes to most common value.
Evaluating epochs [1, 2, 3]
########################### epoch 1 ###########################
Too few for hull: 1
--  shape: (1, 2): skipping
Too few for hull: 10
--  shape: (1, 2): skipping
Too few for hull: 100
--  shape: (1, 2): skipping
Too few for hull: 101
# [more]
--  shape: (1, 2): skipping
Too few for hull: 97
--  shape: (1, 2): skipping
Too few for hull: 98
--  shape: (1, 2): skipping
Too few for hull: 99
--  shape: (1, 2): skipping
------ hull error : 0.0
------ f1 score with 3NN :0.1807228915662651

from genocae.

Also 100 epochs is not sufficient. Trying 1000.

python3 run_gcae.py train --datadir example_tiny --data issue_6_bin --model_id M1  --epochs 1000 --save_interval 1  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py project --datadir example_tiny --data issue_6_bin --model_id M1 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py evaluate --metrics "hull_error,f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1

from genocae.

Hmmm, if I run for 1000 epochs with the HO_superpopulations file ...

python3 run_gcae.py train --datadir example_tiny --data issue_6_bin --model_id M1  --epochs 1000 --save_interval 1  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py project --datadir example_tiny --data issue_6_bin --model_id M1 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py evaluate --metrics "hull_error,f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1

... I get a hint:

Too few for hull: Uzbek
--  shape: (1, 2): skipping
Too few for hull: Xibo
--  shape: (1, 2): skipping
Too few for hull: Yakut
--  shape: (1, 2): skipping
Too few for hull: Yemeni
--  shape: (1, 2): skipping
Too few for hull: Yi
--  shape: (1, 2): skipping
Too few for hull: Yoruba
--  shape: (1, 2): skipping
Too few for hull: Yukagir
--  shape: (1, 2): skipping
Too few for hull: Zapotec
--  shape: (1, 2): skipping
------ hull error : 0.0
------ f1 score with 3NN :0.1927710843373494

It means that there are too few individuals per ?family. Aha, it is in the README.md:

hull_error: for every population p: define the convex hull created by the points of samples of p. calculate the fraction that other population's samples make up of all the points inside the hull. the hull error is the average of this over populations.

from genocae.

Nope, this has nothing to do with the hull error, as when I run evaluate with ...

python3 run_gcae.py evaluate --metrics "f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1

I still get:

------ f1 score with 3NN :0.1927710843373494
Traceback (most recent call last):
  File "/home/richel/miniconda3/lib/python3.9/site-packages/numpy/lib/npyio.py", line 1796, in genfromtxt
    fhd = iter(fid)
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1616, in <module>
    main()
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1561, in main
    write_f1_scores_to_csv(results_directory, "epoch_{0}".format(epoch), superpopulations_file, f1_scores_by_pop, coords_by_pop)
  File "/home/richel/GitHubs/GenoCAE/utils/visualization.py", line 603, in write_f1_scores_to_csv
    superpop_pop_dict = get_superpop_pop_dict(superpopulations_file)
  File "/home/richel/GitHubs/GenoCAE/utils/data_handler.py", line 869, in get_superpop_pop_dict
    pop_superpop_list = get_pop_superpop_list(pop_superpop_file)
  File "/home/richel/GitHubs/GenoCAE/utils/data_handler.py", line 626, in get_pop_superpop_list
    pop_superpop_list = np.genfromtxt(file, usecols=(0,1), dtype=str, delimiter=",")
  File "/home/richel/miniconda3/lib/python3.9/site-packages/numpy/lib/npyio.py", line 1798, in genfromtxt
    raise TypeError(
TypeError: fname must be a string, filehandle, list of strings, or generator. Got <class 'NoneType'> instead.

from genocae.

Ah, with the HO_superpop file it does work:

python3 run_gcae.py evaluate --metrics "f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1       --superpops example_tiny/HO_superpopulations

Gives:

...
2021-12-09 17:51:55.681304: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-12-09 17:51:55.681334: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-12-09 17:51:58.451444: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2021-12-09 17:51:58.451471: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (N141CU): /proc/driver/nvidia/version does not exist
2021-12-09 17:51:58.451766: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
tensorflow version 2.7.0

______________________________ arguments ______________________________
train : False
datadir : example_tiny/
data : None
model_id : None
train_opts_id : None
data_opts_id : None
save_interval : None
epochs : None
resume_from : None
trainedmodeldir : None
pheno_model_id : None
project : False
superpops : example_tiny/HO_superpopulations
epoch : None
pdata : None
trainedmodelname : ae.M1.ex3.b_0_4.issue_6_bin.p1
plot : False
animate : False
evaluate : True
metrics : f1_score_3

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}

Imputing originally missing genotypes to most common value.
Evaluating epochs [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000]
########################### epoch 1 ###########################
------ f1 score with 3NN :0.1927710843373494
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_1.csv
########################### epoch 2 ###########################
------ f1 score with 3NN :0.1686746987951807
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_2.csv
########################### epoch 3 ###########################
------ f1 score with 3NN :0.1927710843373494
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_3.csv
[...]
########################### epoch 998 ###########################
------ f1 score with 3NN :0.1807228915662651
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_998.csv
########################### epoch 999 ###########################
------ f1 score with 3NN :0.1927710843373494
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_999.csv
########################### epoch 1000 ###########################
------ f1 score with 3NN :0.18674698795180722
writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_1000.csv

from genocae.

Cannot do this, you really need that superpop file :-)

from genocae.