grahamgower / genomatnn Goto Github PK
View Code? Open in Web Editor NEWPredicts adaptive introgression using a CNN trained on genotype matrices.
License: Other
Predicts adaptive introgression using a CNN trained on genotype matrices.
License: Other
I tried to update the .toml config to use my own simulated .trees files. I changed it to:
[sim.tranche]
# The labels and modelspec(s) for each tranche. The network will be trained to
# classify data as coming from one of these tranches. Each tranche consists of
# a list of simulation modelspecs.
# Only two tranches are supported.
"constant_2pop" = [
"genomatnn_data/trees/constant_2pop",
# Skip this for now, as it's too computationally intensive
# to do many replicates for training. :-(
#"HomSap/HomininComposite_4G20/DFE",
]
single_pulse_uni_AB = [
"genomatnn_data/trees/single_pulse_uni_AB",
]
But, I get the error:
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/bin/genomatnn", line 33, in <module>
sys.exit(load_entry_point('genomatnn==0.1.dev115+g4e4a918', 'console_scripts', 'genomatnn')())
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/cli.py", line 587, in main
args = parse_args(args_list)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/cli.py", line 540, in parse_args
args.conf = config.Config(args.conf)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/config.py", line 79, in __init__
self._getcfg_sim()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/config.py", line 142, in _getcfg_sim
self._getcfg_tranche(self.sim["tranche"])
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/config.py", line 150, in _getcfg_tranche
model = sim.get_demog_model(modelspec)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/genomatnn-0.1.dev115+g4e4a918-py3.6-linux-x86_64.egg/genomatnn/sim.py", line 841, in get_demog_model
raise ValueError(f"{modelspec} not found")
ValueError: genomatnn_data/trees/constant_2pop not found
So, am I right in interpreting this that genomatnn is not currently setup to use any arbitrary simulations defined in the toml? Or am I missing something about how to define them?
E.g. something like:
[calibrate.weights]
"HomSap/HomininComposite_4G20/Neutral/slim" = 1
"HomSap/HomininComposite_4G20/Sweep/CEU" = 1
"HomSap/HomininComposite_4G20/AI/Nea_to_CEU" = 0.01
Or maybe per-tranche?
[calibrate.weights]
"AI" = 1
"not AI" = 0.01
Hi @grahamgower
It looks like the population size of YRI should be 27600 (Gower et al. 2021 Appedix 3 Table 1), while the population size is 27000 in the demographic models.
https://github.com/grahamgower/genomatnn/blob/main/demographic_models/HomininComposite_4G20.yaml#L30
https://github.com/grahamgower/genomatnn/blob/main/demographic_models/HomininComposite2_4G20.yaml#L45
Move code under a subdirectory, add setup.py
, etc.
Update requirements.txt to lockdown numpy and tensorflow version. Also, add nose.
Clarify that all bits not using GPL imports can be freely used under a permissive licence.
I tried following the instructions in the README. And I got some errors when I ran the tests.
Here is what I did:
module load anaconda5 ## necessary for the HPC
conda create -n genomatnn gsl cudnn "numpy<1.19" "blas=*=mkl"
source activate genomatnn
pip install git+https://github.com/grahamgower/stdpopsim.git@selection --user
git clone https://github.com/grahamgower/genomatnn.git
cd genomatnn
conda update numpy ## fixes the numpy-tensorflow conflict
python setup.py install
pip install nose
python setup.py build_ext -i
nosetests -v tests
Here is the output from the tests
2020-11-24 14:08:34.157097: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/itac/2020.3.036/intel64/slib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.4.304/linux/tbb/lib/intel64/gcc4.8:/opt/intel/debugger_2020/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/daal/lib/intel64_lin
2020-11-24 14:08:34.157205: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
test_upsample_indexes (tests.test_calibrate.TestUpsample) ... ok
test_upsample_indexes_weighted (tests.test_calibrate.TestUpsample) ... ok
2020-11-24 14:08:40.758042: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-11-24 14:08:40.758587: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/itac/2020.3.036/intel64/slib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.4.304/linux/tbb/lib/intel64/gcc4.8:/opt/intel/debugger_2020/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/daal/lib/intel64_lin
2020-11-24 14:08:40.758626: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2020-11-24 14:08:40.758665: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (br006.pvt.bridges.psc.edu): /proc/driver/nvidia/version does not exist
2020-11-24 14:08:40.761023: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-11-24 14:08:41.002596: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_410"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:08:41.011092: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2020-11-24 14:08:41.011665: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2297105000 Hz
2020-11-24 14:08:45.722282: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_2355"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ERROR
2020-11-24 14:08:50.673550: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_3263"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:08:54.761367: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_5208"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ERROR
2020-11-24 14:08:59.817739: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_6116"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:09:03.661341: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_8061"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ERROR
2020-11-24 14:09:08.947115: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_8969"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:09:12.812368: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_10914"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ERROR
test_list_modelspecs (tests.test_cli.TestSim) ... ok
test_missing_config_file (tests.test_cli.TestSim) ... ok
test_sim (tests.test_cli.TestSim) ... ok
test_missing_config_file (tests.test_cli.TestTrain) ... ok
test_train (tests.test_cli.TestTrain) ... 2020-11-24 14:09:19.118835: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_11822"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:09:23.264315: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_13767"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ok
test_train_from_cache (tests.test_cli.TestTrain) ... 2020-11-24 14:09:25.810186: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_14675"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:09:28.671301: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_16620"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ok
2020-11-24 14:09:32.904075: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_17528"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2020-11-24 14:09:36.717616: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:654] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
key: "Targuments"
value {
list {
}
}
}
attr {
key: "f"
value {
func {
name: "__inference_Dataset_flat_map_slice_batch_indices_19473"
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
dim {
size: -1
}
}
}
}
}
attr {
key: "output_types"
value {
list {
type: DT_INT64
}
}
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
ERROR
test_min_coalescent_time_ancient_samples (tests.test_contact.TestContact) ... ok
test_split_time (tests.test_contact.TestContact) ... ok
test_tmcra (tests.test_contact.TestContact) ... ok
test_exclusion_of_drawn_mutation (tests.test_convert.TestDrawnMutation) ... ERROR
test_compare_ts_vcf_genotype_matrixes (tests.test_convert.TestGenotypeMatrixes) ... ERROR
test_ts_genotype_matrix (tests.test_convert.TestGenotypeMatrixes) ... ok
test_reorder (tests.test_convert.TestSorting) ... ok
test_sort_similarity (tests.test_convert.TestSorting) ... ok
test_ts_pop_counts_indices (tests.test_convert.TestSorting) ... ok
test_verify_partition (tests.test_convert.TestSorting) ... ok
test_gt_bytes2vec (tests.test_misc.TestMisc) ... ok
test_basic_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_per_population_permutation_invariant_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_permutation_invariant_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_accumulate_matrices (tests.test_vcf.TestVCF) ... ERROR
test_bcftools_query (tests.test_vcf.TestVCF) ... ERROR
test_genotypes (tests.test_vcf.TestVCF) ... ERROR
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestApply'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 588, in main
args.func(args.conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/save.py", line 207, in load_model
compile)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestApply_phased'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 588, in main
args.func(args.conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/save.py", line 207, in load_model
compile)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestEval'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 588, in main
args.func(args.conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/save.py", line 207, in load_model
compile)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestEval_phased'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 588, in main
args.func(args.conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/save.py", line 207, in load_model
compile)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestVcfplot'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 588, in main
args.func(args.conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/save.py", line 207, in load_model
compile)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/site-packages/tensorflow-2.4.0rc2-py3.6-linux-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test_exclusion_of_drawn_mutation (tests.test_convert.TestDrawnMutation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_convert.py", line 260, in test_exclusion_of_drawn_mutation
seed=1,
File "/home/aglad/.local/lib/python3.6/site-packages/stdpopsim/slim_engine.py", line 1022, in simulate
dry_run=dry_run)
File "/home/aglad/.local/lib/python3.6/site-packages/stdpopsim/slim_engine.py", line 1058, in _run_slim
stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc:
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'slim': 'slim'
======================================================================
ERROR: test_compare_ts_vcf_genotype_matrixes (tests.test_convert.TestGenotypeMatrixes)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_convert.py", line 167, in test_compare_ts_vcf_genotype_matrixes
subprocess.run(["bgzip", vcf_file])
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bgzip': 'bgzip'
======================================================================
ERROR: test_accumulate_matrices (tests.test_vcf.TestVCF)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_vcf.py", line 71, in test_accumulate_matrices
max_missing_thres=max_missing_thres,
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 178, in accumulate_matrices
rng=rng,
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 110, in genotypes
regions=regions,
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 29, in bcftools_query
stderr=subprocess.PIPE,
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bcftools': 'bcftools'
======================================================================
ERROR: test_bcftools_query (tests.test_vcf.TestVCF)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_vcf.py", line 16, in test_bcftools_query
for line in vcf.bcftools_query("%CHROM\t%POS\n", self.vcf_file):
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 29, in bcftools_query
stderr=subprocess.PIPE,
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bcftools': 'bcftools'
======================================================================
ERROR: test_genotypes (tests.test_vcf.TestVCF)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/pylon5/mcz3c2p/aglad/genomatnn/tests/test_vcf.py", line 33, in test_genotypes
self.vcf_file, maf_thres=maf_thres, max_missing_thres=max_missing_thres,
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 110, in genotypes
regions=regions,
File "/pylon5/mcz3c2p/aglad/genomatnn/genomatnn/vcf.py", line 29, in bcftools_query
stderr=subprocess.PIPE,
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/home/aglad/.conda/envs/genomatnn/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bcftools': 'bcftools'
----------------------------------------------------------------------
Ran 25 tests in 61.832s
FAILED (errors=10)
E.g. it should be easy to share Sweep/CHB
between a Nea->CHB analysis and a Den->CHB analysis.
It's inconvenient when testing hyperparameters to have to manually run eval
on each trained model.
Some of the *.py files have __main__
blocks I used for testing, etc. Useful functionality should be moved into genomatnn.py
subcommands, or converted into unit tests, and the rest deleted.
E.g. DFE simulations.
Hi! I was following the installing instructions and got 6 errors. I'm wondering whether I could get some help with this? The errors are "AttributeError: 'str' object has no attribute 'decode'" and "FileNotFoundError: [Errno 2] No such file or directory: 'bgzip'" (I tried to fix the bgzip error by uninstalling and reinstalling the bgzip package, but still got the same error when running the tests again.) I've attached the full error message below.
Thank you in advance for your help!
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestApply'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/util.py", line 471, in try_run
return func()
File "/Users/yiningliu/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 608, in main
args.func(args.conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/save.py", line 206, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestApply_phased'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/util.py", line 471, in try_run
return func()
File "/Users/yiningliu/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 608, in main
args.func(args.conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/save.py", line 206, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestEval'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/util.py", line 471, in try_run
return func()
File "/Users/yiningliu/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 608, in main
args.func(args.conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/save.py", line 206, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestEval_phased'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/util.py", line 471, in try_run
return func()
File "/Users/yiningliu/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 608, in main
args.func(args.conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/save.py", line 206, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test suite for <class 'tests.test_cli.TestVcfplot'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 210, in run
self.setUp()
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 293, in setUp
self.setupContext(ancestor)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/suite.py", line 316, in setupContext
try_run(context, names)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/nose/util.py", line 471, in try_run
return func()
File "/Users/yiningliu/genomatnn/tests/test_cli.py", line 52, in setUpClass
cli.main(f"train --seed 1 {cls.config_file}".split())
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 608, in main
args.func(args.conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 120, in do_train
do_eval(conf)
File "/Users/yiningliu/genomatnn/genomatnn/cli.py", line 171, in do_eval
model = models.load_model(conf.nn_hdf5_file)
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/save.py", line 206, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/site-packages/tensorflow-2.4.1-py3.8-macosx-10.9-x86_64.egg/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
======================================================================
ERROR: test_compare_ts_vcf_genotype_matrixes (tests.test_convert.TestGenotypeMatrixes)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/yiningliu/genomatnn/tests/test_convert.py", line 167, in test_compare_ts_vcf_genotype_matrixes
subprocess.run(["bgzip", vcf_file])
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/yiningliu/opt/anaconda3/envs/genomatnn/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bgzip'
----------------------------------------------------------------------
Ran 25 tests in 1053.544s
FAILED (errors=6)
Hi @grahamgower
I have another issue encountered while trying to run the tutorial on human data. I got an error after running this line: $ genomatnn sim -n 1000 Nea_to_CEU.toml
. Here is the error message:
Thanks for help !
Hi Graham! Is there an easy way to convert the simulated tree sequences to vcf-like files (preferably with annotation of which mutation being the AI beneficial allele)? I'm asking for the purpose of cross-comparison between methods. I see that there's genotype matrices conversion option (train -c), which I thought could work too, but I don't see the matrices being written out as actual files in the zarrcache directory. Maybe I'm missing something here? Thanks!
A small proportion of simulation jobs run forever, maybe due to AF conditioning where the condition is very unlikely. Should figure out what's going on and if there's a simple fix. E.g. for Nea -> CEU
:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
205154 srx907 20 0 87012 69876 3252 R 100.0 0.0 1254:50 slim -s 795194818 /tmp/tmpyu7uzdva.slim
215055 srx907 20 0 87472 68476 3224 R 100.0 0.0 1150:49 slim -s 1675221105 /tmp/tmpptuoy1cw.slim
218061 srx907 20 0 96640 80084 3252 R 100.0 0.0 1118:18 slim -s 1390539401 /tmp/tmpg7ao8pck.slim
237002 srx907 20 0 89860 64548 3252 R 100.0 0.0 910:50.10 slim -s 558990568 /tmp/tmpf1ywxpsb.slim
253361 srx907 20 0 87268 56436 3252 R 100.0 0.0 728:48.05 slim -s 2920547319 /tmp/tmpy5911jdq.slim
212630 srx907 20 0 86828 64724 3224 R 100.0 0.0 1176:05 slim -s 1042124329 /tmp/tmpamxyj_g8.slim
222823 srx907 20 0 103012 77716 3252 R 100.0 0.0 1066:16 slim -s 3986576135 /tmp/tmpq5i7e47m.slim
238159 srx907 20 0 80780 53624 3224 R 100.0 0.0 898:03.37 slim -s 442496812 /tmp/tmpb1w1ss13.slim
242698 srx907 20 0 85988 58664 3224 R 100.0 0.0 847:44.36 slim -s 3650201765 /tmp/tmpkoh9pv5q.slim
tmpamxyj_g8.slim.txt
tmpb1w1ss13.slim.txt
tmpf1ywxpsb.slim.txt
tmpg7ao8pck.slim.txt
tmpkoh9pv5q.slim.txt
tmpptuoy1cw.slim.txt
tmpq5i7e47m.slim.txt
tmpy5911jdq.slim.txt
tmpyu7uzdva.slim.txt
Somehow detect overlap and adjust vertical coordinates?
Github recently disabled the add-path
command from their actions scripts, so now the 'github actions' continuous integration tests don't run. E.g.
https://github.com/grahamgower/genomatnn/pull/34/checks?check_run_id=1454407206
Should probably reject these and sample again.
The current structure is awkward and inefficient. There's too many wrapper functions in cli.py, and the vcf is queried multiple times for overlapping queries. It's also not really sensible to cache the matrices after processing (#19) with this structure.
vcf2mat
should be changed to yield genotype vectors for a single site, which are then accumulated into matrices for potentially overlapping windows. The matrices can then be resized/sorted after accumulation. This will avoid querying the vcf multiple times, which should give a nice speedup and hopefuly avoid some of the wrappers at the higher level.
Hi,
Having successfully installed the package under the conda environment on CentOS Linux, I encountered this error while doing the test. It seems to me that this is a problem with python, but I don't really know how to debug it.
Thanks in advance
Sincerely,
Apdulai
$ nosetests -v tests
test_upsample_indexes (tests.test_calibrate.TestUpsample) ... ok
test_upsample_indexes_weighted (tests.test_calibrate.TestUpsample) ... ok
test_apply_creates_predictions (tests.test_cli.TestApply) ... ok
test_missing_config_file (tests.test_cli.TestApply) ... ok
test_missing_nn_hdf5_file (tests.test_cli.TestApply) ... ok
test_apply_creates_predictions (tests.test_cli.TestApply_phased) ... ok
test_missing_config_file (tests.test_cli.TestApply_phased) ... ok
test_missing_nn_hdf5_file (tests.test_cli.TestApply_phased) ... ok
test_eval_creates_plots (tests.test_cli.TestEval) ... ok
test_missing_config_file (tests.test_cli.TestEval) ... ok
test_missing_nn_hdf5_file (tests.test_cli.TestEval) ... ok
test_eval_creates_plots (tests.test_cli.TestEval_phased) ... ok
test_missing_config_file (tests.test_cli.TestEval_phased) ... ok
test_missing_nn_hdf5_file (tests.test_cli.TestEval_phased) ... ok
test_list_modelspecs (tests.test_cli.TestSim) ... ok
test_missing_config_file (tests.test_cli.TestSim) ... ok
test_sim (tests.test_cli.TestSim) ... ok
test_missing_config_file (tests.test_cli.TestTrain) ... ok
test_train (tests.test_cli.TestTrain) ... ok
test_train_from_cache (tests.test_cli.TestTrain) ... ok
test_missing_config_file (tests.test_cli.TestVcfplot) ... ok
test_vcfplot_creates_plot (tests.test_cli.TestVcfplot) ... ok
test_min_coalescent_time_ancient_samples (tests.test_contact.TestContact) ... ok
test_split_time (tests.test_contact.TestContact) ... ok
test_tmcra (tests.test_contact.TestContact) ... ok
test_exclusion_of_drawn_mutation (tests.test_convert.TestDrawnMutation) ... ERROR
test_compare_ts_vcf_genotype_matrixes (tests.test_convert.TestGenotypeMatrixes) ... ok
test_ts_genotype_matrix (tests.test_convert.TestGenotypeMatrixes) ... ok
test_reorder (tests.test_convert.TestSorting) ... ok
test_sort_similarity (tests.test_convert.TestSorting) ... ok
test_ts_pop_counts_indices (tests.test_convert.TestSorting) ... ok
test_verify_partition (tests.test_convert.TestSorting) ... ok
test_gt_bytes2vec (tests.test_misc.TestMisc) ... ok
test_basic_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_per_population_permutation_invariant_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_permutation_invariant_cnn (tests.test_tfstuff.TestModelConstruction) ... ok
test_accumulate_matrices (tests.test_vcf.TestVCF) ... ok
test_bcftools_query (tests.test_vcf.TestVCF) ... ok
test_genotypes (tests.test_vcf.TestVCF) ... ok
Traceback (most recent call last):
File "/shared/ifbstor1/home/$USER/genomatnn/tests/test_convert.py", line 253, in test_exclusion_of_drawn_mutation
ts = slim.simulate(
File "/shared/home/$USER/.local/lib/python3.8/site-packages/stdpopsim/slim_engine.py", line 1027, in simulate
ts = pyslim.load(ts_file.name)
File "/shared/ifbstor1/home/$USER/genomatnn/.conda/envs/lib/python3.8/site-packages/pyslim/slim_tree_sequence.py", line 38, in load
ts = SlimTreeSequence.load(path, legacy_metadata=legacy_metadata)
File "/shared/ifbstor1/home/$USER/genomatnn/.conda/envs/lib/python3.8/site-packages/pyslim/slim_tree_sequence.py", line 225, in load
return cls(ts, reference_sequence=reference_sequence, legacy_metadata=legacy_metadata)
File "/shared/ifbstor1/home/$USER/genomatnn/.conda/envs/lib/python3.8/site-packages/pyslim/slim_tree_sequence.py", line 170, in init
self.reference_sequence = reference_sequence
AttributeError: can't set attribute
Is conda activate genomatnn
correct?
For me, only source activate genomatnn
works.
Hi Graham, I was trying to apply the pre-trained model to empirical data included in the /example/ directory using command
genomatnn apply Nea_to_CEU.toml Nea_to_CEU_af-0.05_2250018620.hdf5
and I encountered the below error message saying that cache data is missing. Is it possible to directly use one of the pre-trained models (for AI in modern humans), or do I have to train my own CNN with new simulations for it to work properly? Thanks so much!!
WARNING: There are non-GPU devices in tf.distribute.Strategy
, not using nccl allreduce.
Traceback (most recent call last):
File "/opt/anaconda3/bin/genomatnn", line 33, in
sys.exit(load_entry_point('genomatnn==0.1.dev138+g7a51abd', 'console_scripts', 'genomatnn')())
File "/opt/anaconda3/lib/python3.8/site-packages/genomatnn-0.1.dev138+g7a51abd-py3.8-macosx-10.9-x86_64.egg/genomatnn/cli.py", line 625, in main
args.func(args.conf)
File "/opt/anaconda3/lib/python3.8/site-packages/genomatnn-0.1.dev138+g7a51abd-py3.8-macosx-10.9-x86_64.egg/genomatnn/cli.py", line 360, in do_apply
get_predictions(conf, pred_file, samples_file)
File "/opt/anaconda3/lib/python3.8/site-packages/genomatnn-0.1.dev138+g7a51abd-py3.8-macosx-10.9-x86_64.egg/genomatnn/cli.py", line 305, in get_predictions
data = convert.load_data_cache(cache)
File "/opt/anaconda3/lib/python3.8/site-packages/genomatnn-0.1.dev138+g7a51abd-py3.8-macosx-10.9-x86_64.egg/genomatnn/convert.py", line 383, in load_data_cache
raise RuntimeError(f"{cache} doesn't exist")
RuntimeError: zarrcache_256-rows doesn't exist
Its too easy to mix up parameter order otherwise.
.
predict_on_batch
outputs the warning below. Possibly need to change the generator to produce a fixed size per batch.
5 out of the last 27 calls to <function Model.make_predict_function..predict_function at 0x7f012446a050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Hello,
I'm trying to install the conda environment which is in the yml file on the gith but it doesn't seem to work "Solving environment" doesn't resolve.
I guess some package versions don't seem to be available anymore in 2023, for example what version of stdpopsim are you using? Or do you have an updated version of the environment?
I tried with stdpopsim version 0.1 but after installing genomatnn the tool doesn't seem to recognize msprime even though it exists in the environment (the package loads with python on the command terminal).
genomatnn.txt
the txt file contains the conda environment that I tried to build to run genomatnn (with stdpopsim=0.1) but which doesn't work either.
Thanks in advance!
Ps: I don't understand, I managed to install genomatnn in 2022 just by replacing the numpy version with 1.20.0.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.