janclemenslab / das Goto Github PK
View Code? Open in Web Editor NEWDeep Audio Segmenter
Home Page: http://janclemenslab.org/das/
das's People
Contributors
Stargazers
Watchers
das's Issues
Improvement: Print time point of deleted or removed annotations in terminal
when annotating, it is printed in the terminal at which time point an event type is added or to which time point the event type is moved but when deleting, the time point is not printed. it would be nice, if also this deletion or removal time point is printed.
this is what it look like at the moment:
INFO:root: Moved vibration_manual from t=1977.4847 to 1978.1313 seconds.
INFO:root: Added pulse_manual at t=2039.4525 seconds.
INFO:root: Removed 1 duplicates in pulse_manual.
INFO:root: Deleted all 5 of pulse_manual in view.
issue installing das
Hello! I am trying to install DAS on my local computer but unfortunately, I have not been able to do so because the installation has gotten stuck on the solving environment step for a long amount of time (>16 hours). I have previously downloaded DAS and have not had this issue before, are there any steps I can take to eliminate this problem?
Unable to execute "das gui"
Hello! I'm using a Mac. I installed using the terminal and I am able to execute the commands "conda activate das" and "das train --help" with no problem. However, when I type in "das gui" and hit enter, I get the following:
My computer is definitely attempting to run something because it gets loud and hot until I Force Quit the Terminal. Any assistance in getting the program to run would be greatly appreciated.
Best,
Jen
das.train_tune : Error
Hello,
I am trying to use the das.train_tune function on Colab.
das.train_tune.train(model_name='tcn_stft', data_dir=path_data, save_dir=path_res, save_name=save_name, batch_size = 8, #32 verbose=2 )
But I get the following error :
ERROR:das.train_tune:Something went wrong. Will try to continue. Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/das/train_tune.py", line 109, in run_trial verbose=verbose) File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/lib/python3.7/dist-packages/das/data.py", line 249, in __getitem__ replace=False) File "mtrand.pyx", line 915, in numpy.random.mtrand.RandomState.choice ValueError: 'a' cannot be empty unless no samples are taken --------------------------------------------------------------------------- ValueError Traceback (most recent call last) [<ipython-input-12-ba0751aa2c8c>](https://localhost:8080/#) in <module> 4 save_name=save_name, 5 batch_size=8, # 32 ----> 6 verbose=2 7 )
`
6 frames
/usr/local/lib/python3.7/dist-packages/das/train_tune.py in train(data_dir, x_suffix, y_suffix, save_dir, save_prefix, save_name, model_name, nb_filters, kernel_size, nb_conv, use_separable, nb_hist, ignore_boundaries, batch_norm, nb_pre_conv, pre_nb_dft, pre_kernel_size, pre_nb_filters, pre_nb_conv, upsample, dilations, nb_lstm_units, verbose, batch_size, nb_epoch, learning_rate, reduce_lr, reduce_lr_patience, fraction_data, seed, batch_level_subsampling, augmentations, tensorboard, wandb_api_token, wandb_project, wandb_entity, log_messages, nb_stacks, with_y_hist, balance, version_data, tune_config, nb_tune_trials, _qt_progress)
445 verbose=verbose,
446 callbacks=callbacks,
--> 447 class_weight=params['class_weights'],
448 )
449 logger.info(tuner.results_summary())
/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/base_tuner.py in search(self, *fit_args, **fit_kwargs)
207 step=tuner_utils.get_best_step(results, self.oracle.objective),
208 )
--> 209 self.on_trial_end(trial)
210 self.on_search_end()
211
/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/base_tuner.py in on_trial_end(self, trial)
262 self.logger.report_trial_state(trial.trial_id, trial.get_state())
263
--> 264 self.oracle.end_trial(trial.trial_id, trial_module.TrialStatus.COMPLETED)
265 self.oracle.update_space(trial.hyperparameters)
266 # Display needs the updated trial scored by the Oracle.
[/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/oracle.py](https://localhost:8080/#) in end_trial(self, trial_id, status)
259 trial.status = status
260 if status == trial_lib.TrialStatus.COMPLETED:
--> 261 self.score_trial(trial)
262 self.end_order.append(trial_id)
263 self._save_trial(trial)
[/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/oracle.py](https://localhost:8080/#) in score_trial(self, trial)
162 trial: A completed `Trial` object.
163 """
--> 164 trial.score = trial.metrics.get_best_value(self.objective.name)
165 trial.best_step = trial.metrics.get_best_step(self.objective.name)
166
[/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/metrics_tracking.py](https://localhost:8080/#) in get_best_value(self, name)
229
230 def get_best_value(self, name):
--> 231 self._assert_exists(name)
232 return self.metrics[name].get_best_value()
233
[/usr/local/lib/python3.7/dist-packages/keras_tuner/engine/metrics_tracking.py](https://localhost:8080/#) in _assert_exists(self, name)
284 def _assert_exists(self, name):
285 if name not in self.metrics:
--> 286 raise ValueError("Unknown metric: %s" % (name,))
287
288
ValueError: Unknown metric: val_loss
`
Any help is much appreciated!
Applicability of pretrained model
Hi!
I am using your pretrained model for mouse vocalizations on the parallel recordings of two sets of microphones.
Surprisingly, the model works great for one set of microphones, but not for the other.
For example, this is the spectrogram from mic set 1 (4 mics, 200kHz) with the output of the model in red:
The vocalization probability clearly goes up at the start of the vocalization in the spectrogram.
For mic set 2 (64 mics, 250kHz), the model's output is almost always around 0.99, except for some edge effects and unreasonable band pass filters:
(Ignore the difference in shape of the images, both show roughly the same time window of 32ms and maximum frequency range)
The results are the same when I use a single mic from the set and when I use an average in frequency domain.
Furthermore, I am using the exact same function for resampling and reshaping the model input for the two mic sets.
Admittedly, the noise levels are slightly higher for mic set 2. Do you think this is what causes the difference?
Is there anything else I can try except for training a new model, or is that really the way to go?
Thanks!
Ruben
with latest das environment installation I cannot open the gui on ubuntu
after conda update conda, I ran: conda create python=3.8 das -c conda-forge -c ncb -n das. When trying to execute das gui I get:
Traceback (most recent call last):
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/das/bin/das", line 7, in <module>
from das.cli import main
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/das/lib/python3.8/site-packages/das/cli.py", line 3, in <module>
from . import train, predict, train_tune
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/das/lib/python3.8/site-packages/das/train_tune.py", line 17, in <module>
import keras_tuner as kt
ModuleNotFoundError: No module named 'keras_tuner'
with conda list, i see that keras-tuner is installed (version 1.0.1). I ran conda install -c conda-forge keras-tuner (and also tried it with a pip install another time), for das gui I get now:
E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (xxx): /proc/driver/nvidia/version does not exist
The X11 connection broke: Maximum allowed requested length exceeded (code 4)
XIO: fatal IO error 0 (Success) on X server ":0"
after 368 requests (368 known processed) with 0 events remaining.
Optimizing model parameters and training outputs
Dear Jan and team,
I am using das through Python and would like to fit the network with different configurations to compare performance and find the best model.
I see there is a shell script for the command-line on the terminal (janclemenslab: TRAIN) but is there a way to do this with the train function in the das.train module in python?
Additionally, how can I get python to print the training outputs?
If I run my training in the GUI, I get a printout like below that includes the f1-scores.
INFO:das.train:{'noise': {'precision': 0.9175990751370318, 'recall': 0.979656218576971, 'f1-score': 0.9476127362477954, 'support': 231717}, 'pulse': {'precision': 0.05627009646302251, 'recall': 0.015928398058252427, 'f1-score': 0.024828564672499408, 'support': 6592}, 'sine': {'precision': 0.8592689767483941, 'recall': 0.5747178602765423, 'f1-score': 0.6887611726526097, 'support': 33051}, 'accuracy': 0.9069243808962264, 'macro avg': {'precision': 0.6110460494494828, 'recall': 0.5234341589705885, 'f1-score': 0.5537341578576348, 'support': 271360}, 'weighted avg': {'precision': 0.8895708148582068, 'recall': 0.9069243808962264, 'f1-score': 0.8936685429716721, 'support': 271360}}
But I don't get this python. I am using the code below with the same files (including test set) and parameters as the GUI. Am I missing an argument to print the results?
model, params = das.train.train(model_name='tcn', data_dir='Quickstart3.npy', save_dir='Quickstart3.res', nb_hist= 256, kernel_size=16, nb_filters=16, batch_size=16, ignore_boundaries=True, verbose=1, nb_epoch=4, log_messages= True)
Any advice is much appreciated!
Error when training with nb_hist = 2048 in gui, but not via local py script or colab script
Dear Jan & team,
Thanks for being so responsive with my many questions thus far!
I think I've found another potential bug (?) or at least an error I'm consistently getting when trying to train a model with nb_hist = 2048. I tried training in 3 possible ways (GUI, py script in colab, py script run via command-line) and found that the error only occurs when I run the training in the GUI.
While training from the GUI with nb_hist = 1024, everything works as expected. But setting this value to 2048 yields the following error:
INFO:das.train:Loading data from F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.npy.
INFO:das.train:Versioning the data:
INFO:das.train: MD5 hash of F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.npy is
INFO:das.train: c7dea09bbf41af8e387916c18dfff10a
INFO:das.train:Parameters:
INFO:das.train:{'data_dir': 'F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.npy', 'x_suffix': '', 'y_suffix': '', 'save_dir': 'F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.res', 'save_prefix': '', 'save_name': None, 'model_name': 'tcn_stft', 'nb_filters': 32, 'kernel_size': 16, 'nb_conv': 4, 'use_separable': [False, False, False], 'nb_hist': 2048, 'ignore_boundaries': True, 'batch_norm': True, 'nb_pre_conv': 4, 'pre_nb_dft': 64, 'pre_kernel_size': 3, 'pre_nb_filters': 16, 'pre_nb_conv': '16', 'upsample': True, 'dilations': [1, 2, 4, 8, 16], 'nb_lstm_units': 0, 'verbose': 2, 'batch_size': 32, 'nb_epoch': 100, 'learning_rate': 0.0001, 'reduce_lr': False, 'reduce_lr_patience': 5, 'fraction_data': None, 'first_sample_train': 0, 'last_sample_train': None, 'first_sample_val': 0, 'last_sample_val': None, 'seed': None, 'batch_level_subsampling': False, 'augmentations': None, 'tensorboard': False, 'wandb_api_token': None, 'wandb_project': None, 'wandb_entity': None, 'log_messages': True, 'nb_stacks': 2, 'with_y_hist': True, 'balance': False, 'version_data': True, 'post_opt': False, 'fill_gaps_min': None, 'fill_gaps_max': None, 'fill_gaps_steps': None, 'min_len_min': None, 'min_len_max': None, 'min_len_steps': None, 'resnet_compute': False, 'resnet_train': False, 'sample_weight_mode': 'temporal', 'data_padding': 64, 'return_sequences': True, 'stride': 1920, 'y_offset': 0, 'output_stride': 1, 'class_names': ['noise', 'comm', 'echo'], 'class_types': ['segment', 'segment', 'segment'], 'data_folder': 'F:/Ava/Experiment_2/Exp2_DAS_selections/training_set', 'data_splits': {'test': 0.2, 'train': 0.6, 'val': 0.2}, 'delete_intermediate_store': True, 'event_std_seconds': 0.0016, 'file_splits': {}, 'filename_endsample_test': [], 'filename_endsample_train': [], 'filename_endsample_val': [], 'filename_startsample_test': [], 'filename_startsample_train': [], 'filename_startsample_val': [], 'filename_test': [], 'filename_train': [], 'filename_val': [], 'gap_seconds': 0.0, 'make_single_class_datasets': False, 'samplerate_x_Hz': 250000.0, 'samplerate_y_Hz': 250000.0, 'split_in_two': False, 'store_folder': 'F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.npy', 'file_split_dict': {}, 'data_hash': 'c7dea09bbf41af8e387916c18dfff10a', 'nb_freq': 1, 'nb_channels': 1, 'nb_classes': 3}
INFO:das.train:Preparing data
INFO:das.train:Training data:
INFO:das.train: AudioSequence with 411 batches each with 32 items.
Total of 25260018 samples with
each x=(1,) and
each y=(3,)
INFO:das.train:Validation data:
INFO:das.train: AudioSequence with 137 batches each with 32 items.
Total of 8420006 samples with
each x=(1,) and
each y=(3,)
INFO:das.train:building network
Traceback (most recent call last):
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\utils.py", line 222, in run
self.fn(*self.args, **self.kwargs)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\das\train.py", line 361, in train
model = models.model_dict[model_name](**params)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\das\models.py", line 109, in tcn_stft
x = tcn_layer.TCN(nb_filters=nb_filters,
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\das\tcn\tcn.py", line 152, in __init__
use_separable.extend(use_separable[-1] * nb_stacks)
TypeError: 'int' object is not iterable
I initially though the problem may have been due to memory constraints (and that the int not iterable error may be just due to it breaking at an uninformative point).
I don't think this is the case however, as running an almost identical model with nb_hist = 2048 via the terminal works fine (at least, starts/is still running).
In addition, trying a model with nb_hist = 4096 from the GUI gets past the above error message and begins training, then crashes (this time, I believe, due to memory overload).
I'm a bit puzzled by why this parameter should derail the GUI-originated training. I'll report back if I discover that there's anything else I've been doing that might be causing the issue.
Thanks!
For reference, my system info is:
INFO:das.cli: Windows-10-10.0.19041-SP0
INFO:das.cli: DAS v0.26.6
INFO:das.cli: GUI is available.
INFO:das.cli: xarray-behave v0.33.0
INFO:das.cli: pyqtgraph v0.12.4
INFO:das.cli: PySide2 v5.13.2
INFO:das.cli: Qt v5.12.9
INFO:das.cli:
INFO:das.cli: tensorflow v2.6.0
INFO:das.cli: keras v2.6.0
INFO:das.cli: GPU is available.
INFO:das.cli:
INFO:das.cli: python v3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:50:36) [MSC v.1929 64 bit (AMD64)]
INFO:das.cli: pandas v1.4.2
INFO:das.cli: numpy v1.22.4
INFO:das.cli: h5py v3.6.0
INFO:das.cli: scipy v1.8.1
INFO:das.cli: scikit-learn v1.1.1
INFO:das.cli: xarray v2022.3.0
On Windows, TensorFlow/CUDA lib not found
Hi Jan & Team,
I'm trying to train my model through the gui and am running into a problem with some of das's dependencies. Apologies if this post is inappropriate here, but I thought it might be helpful if any other users encouter a similar issue.
I'll emphasize that I think this is not a das issue, but rather an issue to do with the tensorflow and/or CUDA (NVIDI's GPU driver) packages not looking in the right places for a needed library.
Here is the problem:
The training initializes fine, but then I get the following error:
INFO:das.train:Will save to F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.res/20220705_144815.
INFO:das.train:start training
2022-07-05 14:48:15.133820: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/400
2022-07-05 14:48:21.231684: I tensorflow/compiler/xla/service/service.cc:171] XLA service 0x21d111ef360 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-07-05 14:48:21.231839: I tensorflow/compiler/xla/service/service.cc:179] StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2022-07-05 14:48:21.243508: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:75] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
2022-07-05 14:48:21.243765: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:76] Searched for CUDA in the following directories:
2022-07-05 14:48:21.244053: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79] ./cuda_sdk_lib
2022-07-05 14:48:21.244239: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79] C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.3
2022-07-05 14:48:21.244466: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79] /usr/local/cuda
2022-07-05 14:48:21.244651: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79] .
2022-07-05 14:48:21.244850: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:81] You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions. For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2022-07-05 14:48:21.245787: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2022-07-05 14:48:21.245978: I tensorflow/compiler/jit/xla_compilation_cache.cc:363] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2022-07-05 14:48:21.246128: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at xla_ops.cc:432 : Internal: libdevice not found at ./libdevice.10.bc
2022-07-05 14:48:21.327881: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:210] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2022-07-05 14:48:24.722848: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8201
2022-07-05 14:48:26.940443: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2022-07-05 14:48:26.978341: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at xla_ops.cc:432 : Internal: libdevice not found at ./libdevice.10.bc
Traceback (most recent call last):
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\utils.py", line 222, in run
self.fn(*self.args, **self.kwargs)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\das\train.py", line 400, in train
fit_hist = model.fit(
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1193, in fit
tmp_logs = self.train_function(iterator)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
result = self._call(*args, **kwds)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 950, in _call
return self._stateless_fn(*args, **kwds)
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3039, in __call__
return graph_function._call_flat(
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 1963, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call
outputs = execute.execute(
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: libdevice not found at ./libdevice.10.bc
[[{{node cluster_1_1/xla_compile}}]] [Op:__inference_train_function_13833]
The critical line appears to be:
2022-07-05 14:48:21.243765: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:76] Searched for CUDA in the following directories:
This appears to be a known issue when using tensor/cuda in anaconda, see here and here. However, I haven't been able to solve it. (There are some suggestions for how to set XLA's search path, but I have not been able to change the search path whether through the terminal or a py script.)
Before I proceed with debugging this, I wanted to ask what the latest version of TensorFlow and CUDA is that might still be compatible with das? Or, if you happen to have run into this problem already, what you might recommend.
For reference, my conda das environment has the following (again, recently downloaded as per install instructions and updated xarray via pip):
(das) C:\Users\AC-Lab-AKK>conda list -n das
# packages in environment at C:\Users\AC-Lab-AKK\anaconda3\envs\das:
#
# Name Version Build Channel
_tflow_select 2.1.0 gpu anaconda
abseil-cpp 20210324.2 h0e60522_0 conda-forge
absl-py 1.1.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.1 py39hb82d6ee_1 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
aom 3.3.0 h0e60522_1 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
asciitree 0.3.3 py_2 conda-forge
astor 0.8.1 pyh9f0ad1d_0 conda-forge
asttokens 2.0.5 pyhd8ed1ab_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
audioread 2.1.9 py39hcbf5309_3 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.1 pyhd3eb1b0_0 anaconda
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
blinker 1.4 py_1 conda-forge
blosc 1.21.1 h74325e0_3 conda-forge
bokeh 2.4.3 py39hcbf5309_0 conda-forge
brotli 1.0.9 h8ffe710_7 conda-forge
brotli-bin 1.0.9 h8ffe710_7 conda-forge
brotlipy 0.7.0 py39hb82d6ee_1004 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
c-blosc2 2.1.1 hdf67494_2 conda-forge
ca-certificates 2022.6.15 h5b45459_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 4.2.4 pyhd8ed1ab_0 conda-forge
certifi 2022.6.15 py39hcbf5309_0 conda-forge
cffi 1.15.0 py39h0878f49_0 conda-forge
cfitsio 4.1.0 h5a969a9_0 conda-forge
charls 2.3.4 h39d44d4_0 conda-forge
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
click 8.1.3 py39hcbf5309_0 conda-forge
cloudpickle 2.1.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
colorcet 3.0.0 pyhd8ed1ab_0 conda-forge
cryptography 37.0.2 py39h7bc7c5c_0 conda-forge
cudatoolkit 11.3.1 h280eb24_10 conda-forge
cudnn 8.2.1.32 h754d62a_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.11.2 py39hb82d6ee_2 conda-forge
das 0.26.4 py39_0 ncb
dask 2022.6.0 pyhd8ed1ab_0 conda-forge
dask-core 2022.6.0 pyhd8ed1ab_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defopt 6.3.0 pyhd8ed1ab_0 conda-forge
distributed 2022.6.0 pyhd8ed1ab_0 conda-forge
docutils 0.18.1 py39hcbf5309_1 conda-forge
executing 0.8.3 pyhd8ed1ab_0 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
flammkuchen 0.9.2 py_0 conda-forge
flatbuffers 2.0.6 h0e60522_0 conda-forge
fonttools 4.33.3 py39hb82d6ee_0 conda-forge
freeglut 3.2.2 h0e60522_1 conda-forge
freetype 2.10.4 h546665d_1 conda-forge
frozenlist 1.3.0 py39hb82d6ee_1 conda-forge
fsspec 2022.5.0 pyhd8ed1ab_0 conda-forge
future 0.18.2 py39hcbf5309_5 conda-forge
gast 0.4.0 pyh9f0ad1d_0 conda-forge
giflib 5.2.1 h8d14728_2 conda-forge
google-auth 1.35.0 pyh6c4a22f_0 conda-forge
google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpcio 1.46.3 py39hb76b349_0 conda-forge
h5py 3.6.0 nompi_py39hd4deaf1_100 conda-forge
hdf5 1.12.1 nompi_h2a0e4a3_104 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 68.2 h0e60522_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
imagecodecs 2022.2.22 py39h43fe4e9_6 conda-forge
imageio 2.19.3 pyhcf75d05_0 conda-forge
importlib-metadata 4.11.4 py39hcbf5309_0 conda-forge
importlib_metadata 4.11.4 hd8ed1ab_0 conda-forge
intel-openmp 2022.1.0 h57928b3_3787 conda-forge
ipython 8.4.0 py39hcbf5309_0 conda-forge
jasper 2.0.33 h77af90b_0 conda-forge
jedi 0.18.1 py39hcbf5309_1 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h8ffe710_1 conda-forge
jxrlib 1.1 h8ffe710_2 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
keras-tuner 1.1.0 pyhd8ed1ab_0 conda-forge
kiwisolver 1.4.3 py39h2e07f2f_0 conda-forge
krb5 1.19.3 h1176d77_0 conda-forge
kt-legacy 1.0.4 pypi_0 pypi
lcms2 2.12 h2a16943_0 conda-forge
lerc 3.0 h0e60522_0 conda-forge
libaec 1.0.6 h39d44d4_0 conda-forge
libavif 0.10.1 h8ffe710_0 conda-forge
libblas 3.9.0 15_win64_mkl conda-forge
libbrotlicommon 1.0.9 h8ffe710_7 conda-forge
libbrotlidec 1.0.9 h8ffe710_7 conda-forge
libbrotlienc 1.0.9 h8ffe710_7 conda-forge
libcblas 3.9.0 15_win64_mkl conda-forge
libclang 11.1.0 default_h5c34c98_1 conda-forge
libcurl 7.83.1 h789b8ee_0 conda-forge
libdeflate 1.12 h8ffe710_0 conda-forge
libffi 3.4.2 h8ffe710_5 conda-forge
libflac 1.3.4 h0e60522_0 conda-forge
libiconv 1.16 he774522_0 conda-forge
liblapack 3.9.0 15_win64_mkl conda-forge
liblapacke 3.9.0 15_win64_mkl conda-forge
libogg 1.3.5 h2bbff1b_1 anaconda
libopencv 4.5.1 py39h27d8466_0 conda-forge
libopus 1.3.1 h8ffe710_1 conda-forge
libpng 1.6.37 h1d00b33_2 conda-forge
libprotobuf 3.17.2 h7755175_1 conda-forge
librosa 0.9.1 pyhd8ed1ab_0 conda-forge
libsndfile 1.0.31 h0e60522_1 conda-forge
libssh2 1.10.0 h680486a_2 conda-forge
libtiff 4.4.0 h2ed3b44_1 conda-forge
libvorbis 1.3.7 h0e60522_0 conda-forge
libwebp 1.2.2 h57928b3_0 conda-forge
libwebp-base 1.2.2 h8ffe710_1 conda-forge
libxcb 1.13 hcd874cb_1004 conda-forge
libxml2 2.9.14 hf5bbc77_0 conda-forge
libxslt 1.1.35 h34f844d_0 conda-forge
libzlib 1.2.12 h8ffe710_1 conda-forge
libzopfli 1.0.3 h0e60522_0 conda-forge
llvmlite 0.38.1 py39ha0cd8c8_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.0.0 py39h0878066_2 conda-forge
lz4-c 1.9.3 h8ffe710_1 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.3.7 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py39hb82d6ee_1 conda-forge
matplotlib 3.5.2 py39hcbf5309_0 conda-forge
matplotlib-base 3.5.2 py39h581301d_0 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
matplotlib-scalebar 0.8.1 pyhd8ed1ab_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
msgpack-python 1.0.4 py39h2e07f2f_0 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multidict 6.0.2 py39hb82d6ee_1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mypy_extensions 0.4.3 py39hcbf5309_5 conda-forge
networkx 2.8.4 pyhd8ed1ab_0 conda-forge
numba 0.55.2 py39hb8cd55e_0 conda-forge
numcodecs 0.9.1 py39h415ef7b_3 conda-forge
numexpr 2.7.3 py39h2e25243_2 conda-forge
numpy 1.22.4 py39h0948cea_0 conda-forge
oauthlib 3.2.0 pyhd8ed1ab_0 conda-forge
opencv 4.5.1 py39hcbf5309_0 conda-forge
openjpeg 2.4.0 hb211442_1 conda-forge
openssl 1.1.1o h8ffe710_0 conda-forge
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.2 py39h2e25243_2 conda-forge
param 1.12.1 pyh6c4a22f_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
peakutils 1.3.3 py_0 conda-forge
pickleshare 0.7.5 py39hde42818_1002 conda-forge
pillow 9.1.1 py39ha53f419_1 conda-forge
pip 22.1.2 pyhd8ed1ab_0 conda-forge
pockets 0.9.1 py_0 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
portaudio 19.6.0 h0e60522_5 conda-forge
prompt-toolkit 3.0.29 pyha770c72_0 conda-forge
protobuf 3.17.2 py39h415ef7b_0 conda-forge
psutil 5.9.1 py39hb82d6ee_0 conda-forge
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
py-opencv 4.5.1 py39h832f523_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.8 py_0 anaconda
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pygments 2.12.0 pyhd8ed1ab_0 conda-forge
pyjwt 2.4.0 pyhd8ed1ab_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyqt 5.12.3 py39hcbf5309_8 conda-forge
pyqt-impl 5.12.3 py39h415ef7b_8 conda-forge
pyqt5-sip 4.19.18 py39h415ef7b_8 conda-forge
pyqtchart 5.12 py39h415ef7b_8 conda-forge
pyqtgraph 0.12.4 pyhd8ed1ab_0 conda-forge
pyqtwebengine 5.12.1 py39h415ef7b_8 conda-forge
pyside2 5.13.2 py39hc084194_4 conda-forge
pysocks 1.7.1 py39hcbf5309_5 conda-forge
pysoundfile 0.10.3.post1 pyhd3deb0d_0 conda-forge
pytables 3.7.0 py39hbcfe41f_0 conda-forge
python 3.9.13 h9a09f29_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge
python-sounddevice 0.4.1 pyh9f0ad1d_0 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2022.1 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyvideoreader 0.5.6 pyhfe9d6e3_0 conda-forge
pywavelets 1.3.0 py39h5d4886f_1 conda-forge
pyyaml 6.0 py39hb82d6ee_4 conda-forge
qt 5.12.9 h5909a2a_4 conda-forge
qtpy 2.1.0 pyhd8ed1ab_0 conda-forge
requests 2.28.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
resampy 0.2.2 py_0 conda-forge
rsa 4.8 pyhd8ed1ab_0 conda-forge
samplestamps 0.4.4 py39_0 ncb
scikit-image 0.19.3 py39h2e25243_0 conda-forge
scikit-learn 1.1.1 py39he931e04_0 conda-forge
scipy 1.8.1 py39hc0c34ad_0 conda-forge
setuptools 62.6.0 py39hcbf5309_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.9 h82413e6_1 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sphinxcontrib-napoleon 0.7 py_0 conda-forge
sqlite 3.38.5 h8ffe710_0 conda-forge
stack_data 0.3.0 pyhd8ed1ab_0 conda-forge
tabulate 0.8.9 pyhd8ed1ab_0 conda-forge
tbb 2021.5.0 h2d74725_1 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tensorboard 2.6.0 pyhd8ed1ab_1 conda-forge
tensorboard-data-server 0.6.0 py39hcbf5309_2 conda-forge
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
tensorflow 2.6.0 gpu_py39he88c5ba_0 anaconda
tensorflow-base 2.6.0 gpu_py39hb3da07e_0 anaconda
tensorflow-estimator 2.6.0 pyh7b7c402_0 anaconda
tensorflow-gpu 2.6.0 h17022bd_0 anaconda
termcolor 1.1.0 py_2 conda-forge
terminaltables 3.1.10 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2022.5.4 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h8ffe710_0 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py39hb82d6ee_3 conda-forge
tqdm 4.64.0 pyhd8ed1ab_0 conda-forge
traitlets 5.3.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.2.0 hd8ed1ab_1 conda-forge
typing_extensions 4.2.0 pyha770c72_1 conda-forge
typing_inspect 0.7.1 pyh6c4a22f_0 conda-forge
tzdata 2022a h191b570_0 conda-forge
ucrt 10.0.20348.0 h57928b3_0 conda-forge
unicodedata2 14.0.0 py39hb82d6ee_1 conda-forge
urllib3 1.26.9 pyhd8ed1ab_0 conda-forge
vc 14.2 hb210afc_6 conda-forge
vs2015_runtime 14.29.30037 h902a5da_6 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
werkzeug 2.1.2 pyhd8ed1ab_1 conda-forge
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
win_inet_pton 1.1.0 py39hcbf5309_4 conda-forge
wrapt 1.14.1 py39hb82d6ee_0 conda-forge
xarray 2022.3.0 pyhd8ed1ab_0 conda-forge
xarray-behave 0.33.0 pypi_0 pypi
xorg-libxau 1.0.9 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xz 5.2.5 h62dcd97_1 conda-forge
yaml 0.2.5 h8ffe710_2 conda-forge
yarl 1.7.2 py39hb82d6ee_2 conda-forge
zarr 2.11.3 pyhd8ed1ab_0 conda-forge
zfp 0.5.5 h0e60522_8 conda-forge
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zipp 3.8.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.12 h8ffe710_1 conda-forge
zlib-ng 2.0.6 h8ffe710_0 conda-forge
zstd 1.5.2 h6255e5f_1 conda-forge
Thanks a lot!
Segment Thresholds + h5 export in GUI not working?
Hi,
I have noticed that the segment thresholds in the GUI do not seem to be working? My predictions seem to be the same regardless of the threshold settings.
I have also noticed that export as .h5 does not seem to work, it exports as a .csv even when .h5 is selected.
I have tested this on windows 10 and 11. Are these known issues?
Working on Das v0.26.8
Otherwise all working great, thanks very much!
No predictions appearing
Hello! I have tried annotating a file and training a model with these annotations but when I run predictions on the same file, no predictions show up. There is also no error code for it.
I'd appreciate any help you can offer!
Error upon starting to train: weights = self.weights() TypeError: 'numpy.ndarray' object is not callable
__________________________________________________________________________________________________
activation_15 (Activation) (None, 62, 32) 0 add_15[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 62, 2) 66 activation_15[0][0]
__________________________________________________________________________________________________
up_sampling1d (UpSampling1D) (None, 992, 2) 0 dense[0][0]
==================================================================================================
Total params: 270,626
Trainable params: 270,626
Non-trainable params: 0
__________________________________________________________________________________________________
INFO:das.train:None
INFO:das.train:Will save to //DAS.res/20220611_151440.
INFO:das.train:start training
Traceback (most recent call last):
File "//anaconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/utils.py", line 222, in run
self.fn(*self.args, **self.kwargs)
File "//anaconda3/envs/das/lib/python3.9/site-packages/das/train.py", line 400, in train
fit_hist = model.fit(
File "//anaconda3/envs/das/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 1050, in fit
data_handler = data_adapter.DataHandler(
File "//anaconda3/envs/das/lib/python3.9/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1100, in __init__
self._adapter = adapter_cls(
File "//anaconda3/envs/das/lib/python3.9/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 902, in __init__
super(KerasSequenceAdapter, self).__init__(
File "///anaconda3/envs/das/lib/python3.9/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 779, in __init__
peek, x = self._peek_and_restore(x)
File "//anaconda3/envs/das/lib/python3.9/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 913, in _peek_and_restore
return x[0], x
File "///anaconda3/envs/das/lib/python3.9/site-packages/das/data.py", line 282, in __getitem__
weights = self.weights()
TypeError: 'numpy.ndarray' object is not callable
After annotations and starting train, I'll get the following error. Is this solvable trivially by initializing differently?
Error when running predict
During the predict phase I am getting an error and no segments are showing up in the gui. Below you can see the output. Looks like the main problem is "ufunc 'add' did not contain a loop with signature matching types (dtype('<U11'), dtype('<U11')) -> dtype('<U11')"
INFO:root:Predicting song using DeepSS:
INFO:root: Running inference on audio.
INFO:root: Model from C:/Users/Andrew/Desktop/quickstart.res/20210421_161522.
c:\users\andrew\anaconda3\envs\dss\lib\site-packages\tensorflow\python\keras\engine\training.py:2325: UserWarning: Model.state_updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
warnings.warn('Model.state_updates will be removed in a future version. '
INFO:root: found 824 instances of segments '[0 1 2 3 4]'.
Traceback (most recent call last):
File "c:\users\andrew\anaconda3\envs\dss\lib\site-packages\xarray_behave\gui\app.py", line 1315, in
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "c:\users\andrew\anaconda3\envs\dss\lib\site-packages\xarray_behave\gui\app.py", line 1859, in deepss_predict
self.event_times.add_time(name + suffix, onset_seconds, offset_seconds, category='segment')
numpy.core._exceptions.UFuncTypeError: ufunc 'add' did not contain a loop with signature matching types (dtype('<U11'), dtype('<U11')) -> dtype('<U11')
[feature request] undo/redo & annotation duration indicator
Dear Jan & team,
I wanted to mention two features I thought would be very useful for the DAS gui. (My sincere apologies if these are somehow already implemented and I may have missed them!):
-
undo/redo feature: The existing keyboard shortcuts are extremely useful and well thought-out. I did however find myself often wishing there was a way to undo or redo a change I made to an annotation, especially when trying to remove annotations (Y), re-threshold and re-detect events (I) in a section of the spectrogram.
-
annotation duration display: In my particular case (and perhaps for many others), the duration of audio events is very diagnostic as to whether an event belongs in one class or another. However, as the events I work with (bat echolocation calls vs. communication calls) are very short in duration, it can be somewhat challenging to see how long the current annotation is. A small display in one corner that may indicate live (i.e. as you are creating/editing an annotation, or for a selected annotation) the duration in seconds would be extremely useful!
If I get some time in the next few weeks, I may try to play around with the code to see if I can implement a rough version myself and open a pull request.
Thanks!
TIPS to improve the model training?
Hi, we could installed the DAS on my Ubuntu server and trained our data to detect the marmoset calls which were recorded in the sound chamber room. We got a great results but I wonder if I improve the prediction much more by adding more data or tuning some parameters.
Now we used the 1680 min including 5240 call annotations, as the training data set. The trained model predicted the call region correctly for the new data set (good performance for hit), but the model failed to recognize the “pulse-like” noises as a call regions (frequent false alarms).
Our data were obtained in the sound chamber, meaning the S/N ratios were relatively higher.
Do you have any suggestions or TPIS to refine the model?
- Change parameter settings?
- Adding more data?
- Additional annotation (now we input the "call region annotation", but we should include noise region as "noise" annotation?)
- Data augumantaion techniques?
K
Make dataset for training error
Hello ! I am following the quickstart tutorial and whenever I try to "Make dataset for training" from the GUI, the log shows the following and nothing is created.
INFO:xarray_behave.gui.das:Collecting info on song types from annotation and definition files: INFO:xarray_behave.gui.das: /home/emilierp/Birds/quickstart/dmel_annotations.csv INFO:xarray_behave.gui.das:Done. INFO:xarray_behave.gui.das: Identifying song types: INFO:xarray_behave.gui.das: Found pulse of type event INFO:xarray_behave.gui.das: Found sine of type segment INFO:xarray_behave.gui.das:Done. INFO:xarray_behave.gui.das:Collecting info on sample rates and channels from audio files: Traceback (most recent call last): File "/home/emilierp/miniconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/app.py", line 1892, in <lambda> menuitem.triggered.connect(lambda: callback(qt_keycode)) File "/home/emilierp/miniconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/app.py", line 417, in das_make das.make(data_folder=form_data['data_folder'], File "/home/emilierp/miniconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/das.py", line 169, in make fss.add(fs) TypeError: unhashable type: 'numpy.ndarray'
Any help ? Thank you in advance !
Could not load the Qt platform plugin "windows" in "" even though it was found
When I ran "dss gui" in a freshly installed dss environment I got:
QObject::moveToThread: Current thread (0x2048d3fc860) is not the object's thread (0x2048d3fc400).
Cannot move to target thread (0x2048d3fc860)
qt.qpa.plugin: Could not load the Qt platform plugin "windows" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: direct2d, minimal, offscreen, webgl, windows.
Reinstalling Miniconda3 completely new fixed the problem.
Latency tips
Hi!
I'm trying to use your mouse model for a real-time application. Unfortunately, the prediction call is taking about 20ms for me, while in the paper you report that it can be done under 10ms. I am starting and stopping the timing right around the call, like:
start = time.time()
output = self.das_model.predict_on_batch(input)
print(" DAS single forward pass: ", time.time() - start)
Do you have any tips on how I could increase the inference speed?
Thanks!
Ruben
Details:
input shape: ((1,8092,1),)
12th Gen Intel(R) Core(TM) i9-12900K
128GB RAM
GeForce RTX 3090
Confidence scores of predictions missing
Dear Jan and team,
I am using python to evaluate the networks and predictions, but I am having trouble with some of my graphs. I am following the code provided by janclemenslab, but the confidence scores are not showing up. What am I missing?
Graph 1
Graph2
I am making my predictions using the following code (the folder contains a test set).
%%time
samplerate, x = scipy.io.wavfile.read('Dmel_male.wav')
print(f"DAS requires [T, channels], but single-channel wave files are loaded with shape [T,] (data shape is {x.shape}).")
x = np.atleast_2d(x).T
events, segments, class_probabilities, class_names = das.predict.predict(x,
model_save_name='Quickstart3.res/20220830_141833',
verbose=2,
segment_minlen=0.02,
segment_fillgap=0.02)
Also, is it possible to make my predictions in the GUI, save them with their confidence score, and then feed the file into python for evaluation?
Many thanks!
Input data formats and data across multiple files when using pre-existing annotations
Two questions:
-
The information in thee data formats page seems to assume that the data will be contained within a single file. What if data is spread across multiple input wav files each with a separate set of annotations? One possibility might be to concatenate the wav files into a single numpy array, but this doesn't seem satisfying as it could lead to edge artifacts where the signals are appended. Are there other ways to manage inputs that are distributed across multiple recordings?
-
I'm a bit confused by the information describing audio/annotations/song definitions (at the top of the page) and the data structure used for training (at the bottom). How are the data and annotation files related to the data structures for training? It appears that the data structures for training are assuming that the data is already stored in memory (e.g., the wav files are converted to numpy arrays and the labels are converted to a series of labels for each time point). Is that correct? It's not clear how one is supposed to go from the files described at the top to the data structures described at the bottom.
If the data structure assumes all of the data is a dictionary in memory, it seems that the top part about the annotation files can just be skipped if we are converting our labels into time series ourselves, right? Just create our own pre-processing and load the data into memory?
If so, we'd still need to figure out a way to deal with multiple input files, since I'm presuming that
data['train']['x']is a single array with the entire series of audio values for all of the training data, right?
Restart training in COLAB
I currently am training a model in COLAB. However, I have reached my user limits.
Can I restart training from the last saved epoch?
Carmen
get_pulseshapes window not centered
The function to extract the waveform around pulse center is creating the window 2*win_hw samples before the given center, rather than centering +- win_hw.
At src/das/pulse_utils.py lines 104-105.
Suggested change would be:
t0 = int(p - win_hw)
t1 = int(p + win_hw)
But I haven't tested this further than normalizing.
Error messages popping up during training and prediction after reinstalling DAS
Hello Jan and team,
I recently ran into some issues with the program that resulted in my removing and reinstalling anaconda on my computer. I reinstalled DAS and the GUI appears to be working normally except now I receive the below error message every time I train a network. The error message pops up at the very end of running the function.
INFO:das.train: Saving to C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Dataset_ChimpNetwork/Norm_PH_noise_FR200_2000hz.res/test_20221010_152304_results.h5.
Traceback (most recent call last):
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\xarray_behave\gui\utils.py", line 222, in run
self.fn(*self.args, **self.kwargs)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\train.py", line 486, in train
fl.save(save_filename, ddd)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py", line 638, in save
_save_level(h5file, group, value, name=key,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py", line 211, in _save_level
_save_level(handler, new_group, v, name=k, filters=filters,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py", line 211, in _save_level
_save_level(handler, new_group, v, name=k, filters=filters,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py", line 207, in _save_level
new_group = handler.create_group(group, name,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tables\file.py", line 908, in create_group
return Group(parentnode, name,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tables\group.py", line 221, in init
super().init(parentnode, name, _log)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tables\node.py", line 240, in init
parentnode._g_refnode(self, name, validate)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tables\group.py", line 492, in _g_refnode
check_name_validity(childname)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tables\path.py", line 144, in check_name_validity
raise ValueError("the/character is not allowed "
ValueError: the/character is not allowed in object names: 'C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Dataset_ChimpNetwork/Norm_PH_noise_FR200_2000hz\PH_noise10'
I'm not really sure how and if this error is causing a problem though because the training does complete, giving me the training results and producing the usual 3 output files (results.h5, model.h5, and params.yaml). However, I am also seeing other issues that perhaps all stem from the same problem.
For instance, I also get an error while trying to predict on a new audio file:
ERROR:root:Error processing file C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Dataset_ChimpNetwork/test\chimp_22.wav.
Traceback (most recent call last):
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 618, in cli_predict
events, segments, class_probabilities, class_names = predict(x,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 490, in predict
x = np.pad(x, ((0, pad_len), (0, 0)), mode='edge')
File "<array_function internals>", line 180, in pad
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\numpy\lib\arraypad.py", line 793, in pad
padded, original_area_slice = _pad_simple(array, pad_width)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\numpy\lib\arraypad.py", line 114, in _pad_simple
padded = np.empty(new_shape, dtype=array.dtype, order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.24 TiB for an array with shape (270336, 1261714) and data type float32
And now when I try to upload the model results into python, this code from the tutorial that used to work, gives me an error:
import flammkuchen as fl
import logging
logging.basicConfig(level=logging.INFO)
res_name = 'Norm_PH_noise_FR200_2000hz.res/Final_8192_bat16_lr002reduce_20221007_134838'
res = fl.load(f'{res_name}_results.h5')
KeyError Traceback (most recent call last)
File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:479, in _load_level(handler, level, pathtable)
478 try:
--> 479 return pathtable[pathname]
480 except KeyError:KeyError: '/'
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:479, in _load_level(handler, level, pathtable)
478 try:
--> 479 return pathtable[pathname]
480 except KeyError:KeyError: '/y_pred'
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
File ~\miniconda3\envs\das\lib\site-packages\zarr\core.py:241, in Array._load_metadata_nosync(self)
240 mkey = _prefix_to_array_key(self._store, self._key_prefix)
--> 241 meta_bytes = self._store[mkey]
242 except KeyError:File ~\miniconda3\envs\das\lib\site-packages\zarr\storage.py:1074, in DirectoryStore.getitem(self, key)
1073 else:
-> 1074 raise KeyError(key)KeyError: '.zarray'
During handling of the above exception, another exception occurred:
ArrayNotFoundError Traceback (most recent call last)
Cell In [5], line 6
4 logging.basicConfig(level=logging.INFO)
5 res_name = 'Norm_PH_noise_FR200_2000hz.res/Final_8192_bat16_lr002reduce_20221007_134838'
----> 6 res = fl.load(f'{res_name}_results.h5')
7 resFile ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:718, in load(path, group, sel, unpack)
715 raise ValueError("Must specify group withselunless it "
716 "automatically unpacks")
717 else:
--> 718 data = _load_level(h5file, grp, pathtable)
720 if DEEPDISH_IO_VERSION_STR in grp._v_attrs:
721 v = grp._v_attrs[DEEPDISH_IO_VERSION_STR]File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:481, in _load_level(handler, level, pathtable)
479 return pathtable[pathname]
480 except KeyError:
--> 481 pathtable[pathname] = _load_nonlink_level(handler, node, pathtable,
482 pathname)
483 return pathtable[pathname]File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:370, in _load_nonlink_level(handler, level, pathtable, pathname)
368 # Load sub-groups
369 for grp in level:
--> 370 lev = _load_level(handler, grp, pathtable)
371 n = grp._v_name
372 # Check if it's a complicated pair or a string-value pairFile ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:481, in _load_level(handler, level, pathtable)
479 return pathtable[pathname]
480 except KeyError:
--> 481 pathtable[pathname] = _load_nonlink_level(handler, node, pathtable,
482 pathname)
483 return pathtable[pathname]File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:437, in _load_nonlink_level(handler, level, pathtable, pathname)
435 elif isinstance(level, tables.VLArray):
436 if level.shape == (1,):
--> 437 return _load_pickled(level)
438 else:
439 return level[:]File ~\miniconda3\envs\das\lib\site-packages\flammkuchen\hdf5io.py:343, in _load_pickled(level)
342 def _load_pickled(level):
--> 343 if isinstance(level[0], ForcePickle):
344 return level[0].obj
345 else:File ~\miniconda3\envs\das\lib\site-packages\tables\vlarray.py:662, in VLArray.getitem(self, key)
660 key += self.nrows
661 (start, stop, step) = self._process_range(key, key + 1, 1)
--> 662 return self.read(start, stop, step)[0]
663 elif isinstance(key, slice):
664 start, stop, step = self._process_range(
665 key.start, key.stop, key.step)File ~\miniconda3\envs\das\lib\site-packages\tables\vlarray.py:806, in VLArray.read(self, start, stop, step)
804 atom = self.atom
805 if not hasattr(atom, 'size'): # it is a pseudo-atom
--> 806 outlistarr = [atom.fromarray(arr) for arr in listarr]
807 else:
808 # Convert the list to the right flavor
809 flavor = self.flavorFile ~\miniconda3\envs\das\lib\site-packages\tables\vlarray.py:806, in (.0)
804 atom = self.atom
805 if not hasattr(atom, 'size'): # it is a pseudo-atom
--> 806 outlistarr = [atom.fromarray(arr) for arr in listarr]
807 else:
808 # Convert the list to the right flavor
809 flavor = self.flavorFile ~\miniconda3\envs\das\lib\site-packages\tables\atom.py:1175, in ObjectAtom.fromarray(self, array)
1173 if array.size == 0:
1174 return None
-> 1175 return pickle.loads(array.tobytes())File ~\miniconda3\envs\das\lib\site-packages\zarr\core.py:2405, in Array.setstate(self, state)
2404 def setstate(self, state):
-> 2405 self.init(**state)File ~\miniconda3\envs\das\lib\site-packages\zarr\core.py:215, in Array.init(self, store, path, read_only, chunk_store, synchronizer, cache_metadata, cache_attrs, partial_decompress, write_empty_chunks, zarr_version, meta_array)
212 self._metadata_key_suffix = self._hierarchy_metadata['metadata_key_suffix']
214 # initialize metadata
--> 215 self._load_metadata()
217 # initialize attributes
218 akey = _prefix_to_attrs_key(self._store, self._key_prefix)File ~\miniconda3\envs\das\lib\site-packages\zarr\core.py:232, in Array._load_metadata(self)
230 """(Re)load metadata from store."""
231 if self._synchronizer is None:
--> 232 self._load_metadata_nosync()
233 else:
234 mkey = _prefix_to_array_key(self._store, self._key_prefix)File ~\miniconda3\envs\das\lib\site-packages\zarr\core.py:243, in Array._load_metadata_nosync(self)
241 meta_bytes = self._store[mkey]
242 except KeyError:
--> 243 raise ArrayNotFoundError(self._path)
244 else:
245
246 # decode and store metadata as instance members
247 meta = self._store._metadata_class.decode_array_metadata(meta_bytes)ArrayNotFoundError: array not found at path %r' ''
I'm totally lost on what the issue is. Any thoughts on what could be causing all these errors? I'm not sure if this is related to my needing to delete and reinstall python. I followed the tutorial same as before, but perhaps DAS didn't install correctly?
Any advice is appreciated.
Many thanks!
Error during training when optimizing postprocessing
Dear Jan,
I am trying to optimize postprocessing during training but I keep getting this error.
TypeError: train() got an unexpected keyword argument 'fill_gaps_min'
I've tried in python and in the GUI with no luck. I know it can be a computationally intensive addition, but even with very small networks it doesn't work.
Any idea what's going on?
Thanks!
pyqtgraph error - levels argument required
I downloaded the new version from the GitHub page and unfortunately, there seems to be a new error - now whenever I load any audio file, this pops up
Originally posted by @emorysaba in #10 (comment)
DAS gui freezes if threshold smoothing factor is set to 0
Hi there!
I'm using DAS to detect some bat calls in my audio files. Installation and initial use works well, but as I'm trying to annotate my files using the thresholding mode, I'm running into an issue that I'm not quite sure how to solve.
My workflow is to (1) add a new type of song, in this case "echo"; (2) manually set the threshold; then (3) generate detections (shortcut "i"). This works well but I find that if I then adjust the threshold manually and try to generate detections again (for example because calls were not detected in a given section), DAS starts throwing errors and stops properly responding to commands.
More specifically:
- any attempt to change the view of the spectogram/oscillogram via the keyboard commands A/D/W/S or the scrollbar results in the following error in the terminal, while the spectrogram doesn't systematically change:
Traceback (most recent call last):
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1533, in <lambda>
self.scrollbar.valueChanged.connect(lambda value: setattr(self, 't0', value))
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1602, in t0
self.update_xy()
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1939, in update_xy
self.envelope = self.get_envelope()
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1754, in get_envelope
env = np.sqrt(np.convolve(self.y.astype(np.float)**2, win, mode='same'))
File "<__array_function__ internals>", line 180, in convolve
File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\numpy\core\numeric.py", line 843, in convolve
raise ValueError('v cannot be empty')
ValueError: v cannot be empty
-
Another oddity is that the scrollbar develops erratic behavior. Clicking on a region on the scrollbar usually does nothing, but sometimes does. Pulling it quickly in one direction or another does get the spectrogram to shift, but the display and the scrollbar are uncalibrated, e.g. when the scrollbar should be at timepoint zero, the spectro shows some other time value at point 0. You can see this in this screencast.
-
Meanwhile, mouse-clicking on the spectrogram to add an annotation results in a normal-looking terminal message, like so:
INFO:root: Added echo at t=[0.9677:0.9677] seconds.
but no event appears on the spectrogram.
Here too is a screencast showing the workflow leading up to an including the error. It occurs at about 2:20 minutes.
For reference, my machine is running Windows 10, and my conda version is 4.12.
I initially ran into this problem annotating long files (7.5 minutes long), so I cut a 30-second segment and used that but the problem still occurred. (Errors above reported from 30-second .wav file.)
Not sure if this may play a role, but when initializing das gui, I get the following warning on my terminal (even after installing Visual C++):
warning:root:no tensorflow-probability
Finally, I should mention that I'm actually running all of this not on my local machine but on a lab computer controlled through Google's Remote Desktop via Chrome. However, I don't suppose this should have an influence, as I have full control of the computer from remote.
I'd be grateful for any advice! Thanks!
EDIT: In fact, without doing anything the gui became responsive again. Now I'm finding that as I go through my file to annotate every few seconds, I sometimes end up in that non-responsive loop again, and some clicking around eventually "releases" it and I can continue. Not sure how to systematically describe now under what conditions it freezes/unfreezes...
EDIT 2 - 3: I discovered that if the smoothing factor under Adjust thresholding mode is set to zero, the un-responsiveness is triggered until it is changed back to a value greater than zero! I don't think this accounts for [pretty much] all the occasions on which some freezing occurred , but I think a significant amount is explained by this !
Annotations do not load unless file ends in "annotations.csv"
Hi Jan,
When loading .wav files into the das gui for annotating/updating annotations, it seems that the gui does not load them unless the annotations file ends in "annotations.csv" (the default). This can cause issues if one needs to create different sets of annotations for different purposes for the same .wav file.
| Edit: Perhaps this behavior is intended, in which case this is more of a feature request that annotation files be read regardless of filename.
Note that, while it's possible to select the correct .csv file from the load data pop-up, no annotations appear unless the file has the default filename ending.
For reference, my system/install is:
INFO:das.cli: Windows-10-10.0.22000-SP0
INFO:das.cli: DAS v0.26.6
INFO:das.cli: GUI is available.
INFO:das.cli: xarray-behave v0.33.0
INFO:das.cli: pyqtgraph v0.12.4
INFO:das.cli: PySide2 v5.13.2
INFO:das.cli: Qt v5.12.9
INFO:das.cli:
INFO:das.cli: tensorflow v2.6.0
INFO:das.cli: keras v2.6.0
INFO:das.cli: GPU is available.
INFO:das.cli:
INFO:das.cli: python v3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:50:36) [MSC v.1929 64 bit (AMD64)]
INFO:das.cli: pandas v1.4.3
INFO:das.cli: numpy v1.23.1
INFO:das.cli: h5py v3.7.0
INFO:das.cli: scipy v1.8.1
INFO:das.cli: scikit-learn v1.1.1
INFO:das.cli: xarray v2022.3.0
- ava
Predict in GUI not working
Hello again! I apologize for posting so much here, but in trying to use the predict function in the GUI today I was presented with this error when I selected a model to use:
Traceback (most recent call last):
File "/Applications/anaconda3/envs/base2/envs/das/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 1677, in <lambda>
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "/Applications/anaconda3/envs/base2/envs/das/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 611, in das_predict
dialog.form['segment_fillgap'] = params['post_opt']['gap_dur']
TypeError: 'bool' object is not subscriptable
Any help with how to fix this would be much appreciated!
Memory+ errors when predicting - tips for optimization?
Hi Jan,
I'm encountering a (possibly memory) issue when trying to get to das to predict call events via the command line (with a custom script), which returns some cryptic error messages.
It looks like it enters into the process as expected, then may be encountering some issues with the GPU driver or with tensorflow. I wonder if it's just a memory issue breaking at a strange point, since the first segment it throws up is the duration of the whole file (0 - 449.94 s).
(das) C:\Users\Ava Kiai\OneDrive - Johann Wolfgang Goethe Universität\#Bat Lab\_Projects\colony_jam\Experiment_2\2-call_detection_and_classifier_model>python predict_v1.py "E:\Ava\Experiment_2\Exp2_DAS_selections\test_set\outtest_set" "E:\Ava\Experiment_2\Exp2_DAS_selections\training_set_v2.res\20220804_110323_model.h5" -0
WARNING:root:No tensorflow-probability.
data from: E:\Ava\Experiment_2\Exp2_DAS_selections\test_set\outtest_set
model from: E:\Ava\Experiment_2\Exp2_DAS_selections\training_set_v2.res\20220804_110323_model.h5
loading model 20220804_110323
rec2021-08-23_17-37-00_02_04.wav
reading audio file...
predict...
segment 0: [ 0. 449.94] s...
2022-08-10 16:34:44.131524: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-10 16:34:45.102154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1249 MB memory: -> device: 0, name: NVIDIA GeForce MX450, pci bus id: 0000:01:00.0, compute capability: 7.5
2022-08-10 16:34:45.829214: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-08-10 16:34:47.445444: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8201
2022-08-10 16:34:49.159561: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2022-08-10 16:34:49.159757: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at conv_ops.cc:1272 : Not found: No algorithm worked!
Traceback (most recent call last):
File "C:\Users\Ava Kiai\OneDrive - Johann Wolfgang Goethe Universität\#Bat Lab\_Projects\colony_jam\Experiment_2\2-call_detection_and_classifier_model\predict_v1.py", line 136, in <module>
events, segments, class_probabilities, class_names = das.predict.predict(
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\das\predict.py", line 343, in predict
class_probabilities = predict_probabililties(x, model, params, verbose, prepend_data_padding)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\das\predict.py", line 28, in predict_probabililties
y_pred = model.predict(pred_gen, verbose=verbose) # run the network
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1749, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
result = self._call(*args, **kwds)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 956, in _call
return self._concrete_stateful_fn._call_flat(
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 1963, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call
outputs = execute.execute(
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: No algorithm worked!
[[node TCN/conv1d/conv1d (defined at \.conda\envs\das\lib\site-packages\das\predict.py:28) ]] [Op:__inference_predict_function_2194]
Function call stack:
predict_function
2022-08-10 16:34:49.410589: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]
(das) C:\Users\Ava Kiai\OneDrive - Johann Wolfgang Goethe Universität\#Bat Lab\_Projects\colony_jam\Experiment_2\2-call_detection_and_classifier_model>
(das) C:\Users\Ava Kiai\OneDrive - Johann Wolfgang Goethe Universität\#Bat Lab\_Projects\colony_jam\Experiment_2\2-call_detection_and_classifier_model>
Note that I get a different, plain memory error if I run the prediction command from the GUI:
INFO:root:Predicting song using DAS:
INFO:root:Loading model from E:/Ava/Experiment_2/Exp2_DAS_selections/training_set_v2.res/20220804_110323.
2022-08-10 17:03:25.063690: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-10 17:03:25.491788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1249 MB memory: -> device: 0, name: NVIDIA GeForce MX450, pci bus id: 0000:01:00.0, compute capability: 7.5
INFO:root: Loading data from E:/Ava/Experiment_2/Exp2_DAS_selections/test_set/outtest_set\rec2021-08-23_17-37-00_02_04.wav.
INFO:root: Annotating using model at E:/Ava/Experiment_2/Exp2_DAS_selections/training_set_v2.res/20220804_110323.
2022-08-10 17:03:26.023220: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-08-10 17:03:27.022580: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8201
2022-08-10 17:03:27.595744: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2
2022-08-10 17:03:27.596390: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2
2022-08-10 17:03:27.596901: W tensorflow/stream_executor/gpu/asm_compiler.cc:77] Couldn't get ptxas version string: Internal: Couldn't invoke ptxas.exe --version
2022-08-10 17:03:27.602882: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2
2022-08-10 17:03:27.603240: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
1191/1772 [===================>..........] - ETA: 39s2022-08-10 17:04:50.969756: I tensorflow/stream_executor/cuda/cuda_driver.cc:732] failed to allocate 512.00M (536870912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-08-10 17:04:50.995580: I tensorflow/stream_executor/cuda/cuda_driver.cc:732] failed to allocate 460.80M (483183872 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2022-08-10 17:04:51.018966: I tensorflow/stream_executor/cuda/cuda_driver.cc:732] failed to allocate 414.72M (434865664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
1772/1772 [==============================] - 129s 71ms/step
2022-08-10 17:05:45.346386: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 886.00MiB (rounded to 929038336)requested by op ConcatV2
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2022-08-10 17:05:45.346816: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc
2022-08-10 17:05:45.350113: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 94, Chunks in use: 94. 23.5KiB allocated for chunks. 23.5KiB in use in bin. 9.1KiB client-requested in use in bin.
...
2022-08-10 17:05:45.987255: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at concat_op.cc:158 : Resource exhausted: OOM when allocating tensor with shape[56704,2048,2] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
ERROR:root:Error processing file E:/Ava/Experiment_2/Exp2_DAS_selections/test_set/outtest_set\rec2021-08-23_17-37-00_02_04.wav.
Traceback (most recent call last):
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\das\predict.py", line 446, in cli_predict
events, segments, class_probabilities, class_names = predict(x, None, verbose, batch_size,
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\das\predict.py", line 343, in predict
class_probabilities = predict_probabililties(x, model, params, verbose, prepend_data_padding)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\das\predict.py", line 28, in predict_probabililties
y_pred = model.predict(pred_gen, verbose=verbose) # run the network
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1766, in predict
all_outputs = nest.map_structure_up_to(batch_outputs, concat, outputs)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\util\nest.py", line 1376, in map_structure_up_to
return map_structure_with_tuple_paths_up_to(
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\util\nest.py", line 1474, in map_structure_with_tuple_paths_up_to
results = [
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\util\nest.py", line 1475, in <listcomp>
func(*args, **kwargs) for args in zip(flat_path_gen, *flat_value_gen)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\util\nest.py", line 1378, in <lambda>
lambda _, *values: func(*values), # Discards the path arg.
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 2853, in concat
return array_ops.concat(tensors, axis=axis)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1769, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1212, in concat_v2
_ops.raise_from_not_ok_status(e, name)
File "C:\Users\Ava Kiai\.conda\envs\das\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[56704,2048,2] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:ConcatV2] name: concat
I'm wondering:
- Are these two error messages referencing the same problem?
- Might there be a way I can optimize my graphics card or memory allocation to make this run on my local machine?
- Could it be that DAS believes the whole 450 s file is one segment (as per some of the indications above) and that's why it's crashing, but would work fine if some parameters were better tuned?
I've sent you the model I'm using, train/predict scripts, and some data for testing via email. I hope this might help parse this.
Thanks again!
DAS version info:
INFO:das.cli: Windows-10-10.0.22000-SP0
INFO:das.cli: DAS v0.26.6
INFO:das.cli: GUI is available.
INFO:das.cli: xarray-behave v0.33.0
INFO:das.cli: pyqtgraph v0.12.4
INFO:das.cli: PySide2 v5.13.2
INFO:das.cli: Qt v5.12.9
INFO:das.cli:
INFO:das.cli: tensorflow v2.6.0
INFO:das.cli: keras v2.6.0
INFO:das.cli: GPU is available.
INFO:das.cli:
INFO:das.cli: python v3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:50:36) [MSC v.1929 64 bit (AMD64)]
INFO:das.cli: pandas v1.4.3
INFO:das.cli: numpy v1.23.1
INFO:das.cli: h5py v3.7.0
INFO:das.cli: scipy v1.8.1
INFO:das.cli: scikit-learn v1.1.1
INFO:das.cli: xarray v2022.3.0
System info: I'm running on Windows 11 64-bit, with an NVIDIA GeForce MX450.
Predict error
Hello! I'm running into an issue with the predict function (screenshot attached below) - I've tried re-assembling datasets and re-training models every which way and still run into this issue about 2/3rds of the time when using it. Any idea as to what could be causing this? Thank you in advance!
*_results.h5 not created after training
Hi, I am trying training,prediction in das.
I would like to ask you a question. I was unable to get the result.h5 file when I performed training with the das.train function in python. Test data has been prepared in step of making dataset.
Could you please tell me how I could do this?
Originally posted by @yHamazaki84 in #16 (comment)
Create a folder in /res when saving annotations
When I click File > Save annotations and there is no localhost folder in /res, it throws the following error (put no pop up or so). It would be cool if either a localhost folder is created in res/ or if a pop up tells that the annotations.csv could not be saved.
Saving annotations to /home/local/UG-STUDENT/elsa.steinfath/ukme04/#Data/chainingmic/dat/localhost-20220217_090732/home/local/UG-STUDENT/elsa.steinfath/ukme04/#Data/chainingmic/res/localhost-20220217_090732/localhost-20220217_090732_annotations.csv.
Traceback (most recent call last):
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 1651, in
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 214, in save_annotations
self.export_to_csv(savefilename)
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 259, in export_to_csv
df.to_csv(savefilename, index=False)
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/pandas/core/generic.py", line 3563, in to_csv
return DataFrameRenderer(formatter).to_csv(
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/pandas/io/formats/format.py", line 1180, in to_csv
csv_formatter.save()
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/pandas/io/formats/csvs.py", line 241, in save
with get_handle(
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/pandas/io/common.py", line 697, in get_handle
check_parent_directory(str(handle))
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/xarray_behave/lib/python3.8/site-packages/pandas/io/common.py", line 571, in check_parent_directory
raise OSError(fr"Cannot save file into a non-existent directory: '{parent}'")
OSError: Cannot save file into a non-existent directory: '/home/local/UG-STUDENT/elsa.steinfath/ukme04/#Data/chainingmic/dat/localhost-20220217_090732/home/local/UG-STUDENT/elsa.steinfath/ukme04/#Data/chainingmic/res/localhost-20220217_090732'
Error opening wav file
Hello again! Unfortunately I've run into another issue - when I try to open some wav files, I get the following error. Any help debugging what could be going wrong here (including there being potential issues with my wav file) would be much appreciated!
Preciting error when using Colab model
Dear Jan,
I am training in Colab using the below code
from numpy import False_
path_to_data = '/content/panthoot.npy'
das.train.train(model_name='tcn_stft',
data_dir=path_to_data,
save_dir='/content/drive/MyDrive/panthoot.res',
save_prefix= '4096_bat32_lr002_PP',
pre_nb_conv= 16,
pre_nb_dft= 64,
nb_hist=4096,
kernel_size=16,
nb_filters=32,
ignore_boundaries=False,
verbose=2,
nb_conv=3,
batch_size= 16,
learning_rate=0.001,
reduce_lr=True,
reduce_lr_patience= 4,
nb_epoch= 10)
Everything seems to work fine, but then when I try to predict I get the below errors and the function fails. This only happens with colab-trained models, it does not happen with models I have trained in the GUI. Why is this?
Also, the WARNING -"input with incompatible shape (32, 8192, 2)" - always pops up when I am predicting on any model. What is this and is it a problem? Could it be because I trained with mono channel audio but prediction on stereo?
INFO:root: Loading data from C:/Users/chita/OneDrive/MMNP_Back_UP_AP/Acoustic Sensors_B-J/Site B - Kasiha/test\SITEB-CORE3_20180405_200000.wav.
INFO:root: Resampling. Audio rate is 24000Hz but model was trained on data with 16000Hz.
INFO:root: Annotating using model at C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Dataset_ChimpNetwork/Colab/8192_bat32_lr002_PP_20221014_183516.
0%| | 0/41 [00:00<?, ?it/s]WARNING:tensorflow:Model was constructed with shape (None, 8192, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, 8192, 1), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (32, 8192, 2).
WARNING:tensorflow:Model was constructed with shape (None, 8192, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, 8192, 1), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (32, 8192, 2).
0%| | 0/41 [00:00<?, ?it/s]
ERROR:root:Error processing file C:/Users/chita/OneDrive/MMNP_Back_UP_AP/Acoustic Sensors_B-J/Site B - Kasiha/test\SITEB-CORE3_20180405_200000.wav.
Traceback (most recent call last):
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 618, in cli_predict
events, segments, class_probabilities, class_names = predict(x,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 492, in predict
class_probabilities = predict_probabilities(x, model, params, verbose, prepend_data_padding)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 52, in predict_probabilities
y_pred_batch = model.predict_on_batch(batch_data) # run the network
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1945, in predict_on_batch
outputs = self.predict_function(iterator)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
result = self._call(*args, **kwds)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 924, in _call
results = self._stateful_fn(*args, **kwds)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3038, in __call__
filtered_flat_args) = self._maybe_define_function(args, kwargs)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3459, in _maybe_define_function
return self._define_function_with_shape_relaxation(
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3381, in _define_function_with_shape_relaxation
graph_function = self._create_graph_function(
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3298, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\framework\func_graph.py", line 1007, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 668, in wrapped_fn
out = weak_wrapped_fn().__wrapped__(*args, **kwds)
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\framework\func_graph.py", line 994, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py:1584 predict_function *
return step_function(self, iterator)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py:1574 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1286 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2849 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3632 _call_for_each_replica
return fn(*args, **kwargs)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py:1567 run_step **
outputs = model.predict_step(data)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py:1540 predict_step
return self(x, training=False)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1057 __call__
outputs = call_fn(inputs, *args, **kwargs)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\functional.py:420 call
return self._run_internal_graph(
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\functional.py:556 _run_internal_graph
outputs = node.layer(*args, **kwargs)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1040 __call__
input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
C:\Users\chita\miniconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\input_spec.py:251 assert_input_compatibility
raise ValueError(
ValueError: Input 0 of layer conv1d is incompatible with the layer: expected axis -1 of input shape to have value 1 but received input with shape (32, 8192, 2)
Play waveform through speakers does not work.
I am using deepss on Ubuntu and did sudo apt-get install libsndfile1, still when pressing 'E' to listen to the audio, I get the following error:
Expression 'paInvalidSampleRate' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2043
Expression 'PaAlsaStreamComponent_InitialConfigure( &self->playback, outParams, self->primeBuffers, hwParamsPlayback, &realSr )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2717
Expression 'PaAlsaStream_Configure( stream, inputParameters, outputParameters, sampleRate, framesPerBuffer, &inputLatency, &outputLatency, &hostBufferSizeMode )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2838
Traceback (most recent call last):
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 1333, in <lambda>
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 1787, in play_audio
sd.play(y, self.fs_song)
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/sounddevice.py", line 174, in play
ctx.start_stream(OutputStream, samplerate, ctx.output_channels,
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/sounddevice.py", line 2573, in start_stream
self.stream = StreamClass(samplerate=samplerate,
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/sounddevice.py", line 1488, in __init__
_StreamBase.__init__(self, kind='output', wrap_callback='array',
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/sounddevice.py", line 892, in __init__
_check(_lib.Pa_OpenStream(self._ptr, iparameters, oparameters,
File "/home/local/UG-STUDENT/elsa.steinfath/miniconda3/envs/dss/lib/python3.8/site-packages/sounddevice.py", line 2738, in _check
raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Error opening OutputStream: Invalid sample rate [PaErrorCode -9997]
Delete annotations with right click again
This latest version does not have the feature anymore to delete single annotations by right click. I found it very useful and suggest that we add this feature again.
DAS predict clean up fails if only used for events
File "anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1669, in <lambda>
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 729, in das_predict
detected_segment_names = [segments['names'][ii] for ii in detected_segment_names]
File "anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 729, in <listcomp>
detected_segment_names = [segments['names'][ii] for ii in detected_segment_names]
TypeError: list indices must be integers or slices, not NoneType
at least it looks like it is searching for segments, but I only defined events.
Installation did not complete.
Hi, I am trying to install DAS (Ideally, no-gui mode in ubuntu-gpu sever), but I could not complete it.
Thinking that the Linux server was the cause, I tried to install it on conda environment of my Mac as well, but the installation did not complete in the same way. Here is the status report.
- issues on Linux server
OS: Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-91-generic x86_64)
GPU: NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4
First, I made a docker container including tensor flow with gpu, by using Nvidia's official container (nvcr.io/nvidia/tensorflow 21.10-tf2-py3). Then, I ran the containter and built the anaconda environment at the /workspace location (docker host location) , by
wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.shbash Anaconda3-2021.11-Linux-x86_64.sh
anaconda3directory was placed under/workspace
After installation, I activate base environment by runningsource /.bashrc-- it seems that I made conda environment at container.
Then, I ranconda env create ncb/das-nogui-env -n das, but it was not competely proceeded like
% conda env create ncb/das-env -n das
Collecting package metadata (repodata.json): done
Solving environment: \
After solving environment: \, it does not proceed at all.
- issues on Mac
OS: macOS 0.15.7（19H1519）
CPU: Intel Core i9
conda info
conda info
active environment : base
active env location : /Users/XXX/.pyenv/versions/anaconda3-2021.05
shell level : 1
user config file : /Users/XXXXX/.condarc
populated config files :
conda version : 4.10.3
conda-build version : 3.21.4
python version : 3.8.8.final.0
virtual packages : __osx=10.15.7=0
__unix=0=0
__archspec=1=x86_64
base environment : /Users/XXXXXX/.pyenv/versions/anaconda3-2021.05 (writable)
conda av data dir : /Users/XXXXXX/.pyenv/versions/anaconda3-2021.05/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/XXXXXX/.pyenv/versions/anaconda3-2021.05/pkgs
/Users/XXXXXXXX/.conda/pkgs
envs directories : /Users/XXXXXXX/.pyenv/versions/anaconda3-2021.05/envs
/Users/XXXXXXXX/.conda/envs
platform : osx-64
user-agent : conda/4.10.3 requests/2.26.0 CPython/3.8.8 Darwin/19.6.0 OSX/10.15.7
UID:GID : 501:20
netrc file : None
offline mode : False
I thought the error might be in the ubuntu server or the docker configuration, so I tried to confirm the installation in my conda environment (my anaconda was installed via pyenv). I ran conda env create ncb/das-env -n das, but it was completely same like the ubuntu server:
conda env create ncb/das-env -n das
Collecting package metadata (repodata.json): done
Solving environment: \
I really needs your help and advice!
K
After fresh installation: "Could not import the GUI."
I ran conda update conda and conda env create -f https://raw.githubusercontent.com/janclemenslab/deepss/master/env/deepss_gui.yml -n dss
after activating the environment, when running dss gui, I get:
2021-06-11 13:14:33.126988: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
Could not import the GUI.
For instructions on how to install the GUI,
check the docs janclemenslab.org/deepss/install.html.
in the dss environment pyside2 is installed - version 5.15.2
Index error when predicting: index 0 is out of bounds for axis 0 with size 0
Dear Jan and team,
I am using the DAS GUI and working through the tutorial but I keep running into the same error when I try to predict.
INFO:root:Predicting song using DAS:
INFO:root: Running inference on audio.
INFO:root: Model from C:/Users/chita/OneDrive/Desktop/Chimp_vocalizations/Quickstart.res/20220819_125409.
40/40 [==============================] - 2s 34ms/step
Traceback (most recent call last):
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1809, in
menuitem.triggered.connect(lambda: callback(qt_keycode))
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 709, in das_predict
events, segments, _, _ = das.predict.predict(
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 353, in predict
events, segments = predict_song(class_probabilities=class_probabilities, params=params,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 242, in predict_song
events = predict_events(class_probabilities, samplerate,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\predict.py", line 212, in predict_events
good_event_indices = event_utils.event_interval_filter(events_seconds,
File "C:\Users\chita\miniconda3\envs\das\lib\site-packages\das\event_utils.py", line 109, in event_interval_filter
ipi_too_short[0] = False # otherwise first event will always be removed
IndexError: index 0 is out of bounds for axis 0 with size 0
I have tried this many times with the Drosophila dataset, as well as my own, but this doesn't make a difference. A couple of times (but not every time) when I checked the box for 'Training targets for individual song times' when doing 'DAS/Make dataset for training', I was able to run the model to make predictions without getting the above error. However, I don't think it worked as it is supposed to because there were very few proposals made. Even the pulse song came back with only 2-3 proposals.
Any insight is greatly appreciated! Thank you.
GUI fails to start on linux machines w/o GPU
part of #18:
with conda list, i see that keras-tuner is installed (version 1.0.1). I ran conda install -c conda-forge keras-tuner (and also tried it with a pip install another time), for das gui I get now:
E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (xxx): /proc/driver/nvidia/version does not exist
The X11 connection broke: Maximum allowed requested length exceeded (code 4)
XIO: fatal IO error 0 (Success) on X server ":0"
after 368 requests (368 known processed) with 0 events remaining.
A question about installing DeepSS
thanks to your replay. the problem is "FileNotFoundError: Could not find module 'd:\anaconda\envs\dss\lib\site-packages\scipy.libs\libbanded5x.EHDKC2XVYTQQ5MALRS6XN2CUSS6SRL6P.gfortran-win_amd64.dll"
sorry, I'm new to python, I followed the steps to install it, and Anaconda is the latest version.
is it because of the version of python?Which version should I install?
I don't know what that means "output of conda list"
thank you.
speeding up das.predict.predict
Hello! I was wondering if there were any way to speed up the predict function when using the python module - I am currently annotating several hundred wav files each 10 minutes long, and it takes my local computer ~3 minutes/file, which is slow. Are there any steps I can take to speed up this process in the notebook? Thank you for all the help!
Spectrogram does not show in DAS
I am working through the tutorial of DAS and loaded the Dmel audio file but it only shows the waveform in the display and not the spectrogram. I have checked the box for the view spectrogram. I also tried another wav file and the same issue. I hope you can look into this.
current das cannot load newly generated idswaps.txt
das gui saves fly idswaps in time format (float) but can only load frames (integers). upgrading xarray-behave in the das environment fixes the error.
das comes with xarray-behave version 0.28.2. the current xarray-behave version (0.29.6) has no problems loading idswaps in time format.
DAS GUI Linux trouble
After a successful (labored due to mamba, but successful) Windows install, I'm encountering more resistance on Ubuntu 22.04 LTS (freshly installed ubuntu-22.04.1-desktop-amd64, single boot, running on Wayland) on the XPS-13-Plus with the following mamba/conda versions:
mamba 0.7.6
conda 4.12.0
DAS installs (no warnings raised), but getting the following error when trying to run the GUI:
2022-09-29 15:45:00.510866: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set Traceback (most recent call last): File "/home/bartulem/anaconda3/envs/das/bin/das", line 11, in <module> sys.exit(main()) File "/home/bartulem/anaconda3/envs/das/lib/python3.9/site-packages/das/cli.py", line 67, in main import xarray_behave.gui.app File "/home/bartulem/anaconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/app.py", line 47, in <module> from . import (colormaps, File "/home/bartulem/anaconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/views.py", line 19, in <module> from . import formbuilder File "/home/bartulem/anaconda3/envs/das/lib/python3.9/site-packages/xarray_behave/gui/formbuilder.py", line 43, in <module> class YamlDialog(QtGui.QDialog): AttributeError: module 'pyqtgraph.Qt.QtGui' has no attribute 'QDialog'
Please advise if you have any idea what's going on (I suspect it's not something superficial), I have no time to troubleshoot. :)
GUI cannot File/Export for DAS
When attempting to export manual annotations for training via File/Export for DAS, the consequent dialog box does not pop up, and instead the following error is displayed on the console:
Traceback (most recent call last): File "c:\users\jeffr\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 1367, in <lambda> menuitem.triggered.connect(lambda: callback(qt_keycode)) File "c:\users\jeffr\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\app.py", line 284, in export_for_das with open(yaml_file, "r") as form_yaml: FileNotFoundError: [Errno 2] No such file or directory: 'c:\\users\\jeffr\\anaconda3\\envs\\das\\lib\\site-packages\\xarray_behave/gui/forms/export_for_das.yaml'
This is probably also due to the rename and you are probably working on it, but again, just in case.
possible to use 'das train' with non GUI version?
is it possible to use 'das train' and 'das predict' in command line to use non GUI version?
or should i run with python script?
below is error info:
ERROR:root:Failed to import GUI. You can still use the non GUI part of DAS. Traceback (most recent call last): File "/opt/conda/envs/das/lib/python3.8/site-packages/das/cli.py", line 62, in main import xarray_behave.gui.app File "/opt/conda/envs/das/lib/python3.8/site-packages/xarray_behave/gui/app.py", line 27, in <module> from qtpy import QtGui, QtCore, QtWidgets File "/opt/conda/envs/das/lib/python3.8/site-packages/qtpy/QtGui.py", line 30, in <module> from PyQt5.QtGui import * ImportError: libGL.so.1: cannot open shared object file: No such file or directory
Could not import the GUI after fresh install
Hello,
I recently installed das following the instructions here: https://janclemenslab.org/das/installation.html (ie, pasting conda env create ncb/das-env -n das into the terminal app and running it from my base environment).
The installation proceeds without errors, but when I then activate the environment with conda activate das and run the command das gui to open the GUI I get the following:
/Applications/anaconda3/envs/das/lib/python3.8/site-packages/pyqtgraph/debug.py:21:
UserWarning: Due to PYSIDE-1140, ThreadChase and ThreadColor won't work.
Could not import the GUI.
For instructions on how to install the GUI,
check the docs janclemenslab.org/das/install.html.
Running other commands gives the same warning, although they appear to work (see example below for the output of
das version)
Things I have tried that did not fix the problem:
- completely removing the das environment I just made with
conda env remove --name dasand reinstalling - updating pyside2 (following the solution to a previous issue):
conda install -c conda-forge pyside2 - reverting to an earlier version of pyqtgraph (following the solution to a previous issue):
conda install pyqtgraph==0.12.1
Running das version gives:
/Applications/anaconda3/envs/das/lib/python3.8/site-packages/pyqtgraph/debug.py:21: UserWarning: Due to PYSIDE-1140, ThreadChase and ThreadColor won't work
warnings.warn("Due to PYSIDE-1140, ThreadChase and ThreadColor won't work")
The following versions are available to DAS:
DAS v0.22.4
GUI is NOT AVAILABLE
tensorflow v2.4.3
keras v2.4.0
GPU is NOT AVAILABLE
python v3.8.12 (default, Oct 12 2021, 06:23:56)
[Clang 10.0.0 ]
pandas v1.3.3
numpy v1.21.2
h5py v2.10.0
scipy v1.7.1
scikit-learn v1.0.1
Note that under the current version of DAS it says "GUI is not available". Any help fixing this would be greatly appreciated - apologies if I'm missing an obvious solution!
installation error with updated das name
I just tried to install das with conda env create -f https://raw.githubusercontent.com/janclemenslab/das/master/env/das_gui.yml -n das, and a Pip subprocess error pops out which prevents das from installing properly. The following is the exact message:
`Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement das-unsupervised (from versions: none)
ERROR: No matching distribution found for das-unsupervised
failed
CondaEnvException: Pip failed`
I tried doing pip install das-unsupervised by itself, but it was also not working and gives the same error.
Seeing that the name change for the package just happened few hours ago, you are probably already aware of this issue, but just in case.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.