pyannote / pyannote-core Goto Github PK

Hi, I was recently working on DER and found that Annotation shows inconsistency between plot and legend when tensors are used.
For example,

from pyannote.core import Annotation, Segment
import torch
import numpy as np

annotation = Annotation()
for i in range(10):
    annotation[Segment(i, i + 1)] = torch.tensor(0)
annotation

Executing the above code would produce the following output

However assigning int 0 rather tensor produces the accurate plot,

If only numpy data is being handled properly, there is no exception generated when tensors are used.

The default implementation seems to separate overlapping segments into different lines (vertically), by parsing the start and end times of all segments, linearly from left to right. [source]

While this is pretty useful visually, there is another way to handle overlaps: plot differently-labelled segments on separate lines. In particular, this is useful if the different labels represent different audio environments and you want to visualize differences in your model's performance across environments/labels.

The actual implementation of this is pretty simple, and adds only three new lines of code:

  def plot_annotation(self, annotation: Annotation, ax=None, time=True, legend=True, separate_by="optimal"):

      if not self.crop:
          self.crop = annotation.get_timeline(copy=False).extent()

      cropped = annotation.crop(self.crop, mode='intersection')
      labels = cropped.labels()
      labels_dict = {label:i for i, label in enumerate(labels)}
      segments = [s for s, _ in cropped.itertracks()]

      ax = self.setup(ax=ax, time=time)

      for (segment, track, label), y in zip(
              cropped.itertracks(yield_label=True),
              self.get_y(segments)):
          if separate_by == "labels":
              y = 1. - 1. / (len(labels) + 1) * (1 + labels_dict.get(label))
          self.draw_segment(ax, segment, y, label=label)

Let me know if this works and i can make a PR!

An example of this implementation at work. The model clearly underperforms in the SYS label, by comparison:

I went ahead and wrote a piece of code that does option (1.)

from matplotlib import pyplot as plt
from pyannote.core import notebook
from pyannote.core import SlidingWindowFeature, SlidingWindow

num_chunks = len(segmentations)

notebook.crop = Segment(start=segmentations.sliding_window.start, end=segmentations.sliding_window[num_chunks].end)

chunks = segmentations.sliding_window
num_overlap = round(chunks.duration // chunks.step + 1)

fig, axes = plt.subplots(nrows=num_overlap, ncols=1, figsize=(10, 10))
mini, maxi = np.min(segmentations.data), np.max(segmentations.data)
ylim = (mini - 0.2 * (maxi - mini), maxi + 0.2 * (maxi - mini))
for c, (chunk, data) in enumerate(segmentations):
    ax = axes[c % num_overlap]
    step = duration = chunk.duration / len(data) 
    frames = SlidingWindow(start=chunk.start, step=step, duration=step)
    chunk_segmentation = SlidingWindowFeature(data, frames)
    notebook.plot_feature(chunk_segmentation, ax=ax, time=c % num_overlap == (num_overlap - 1), ylim=ylim)
    ax.set_prop_cycle(None)

Originally posted by @hbredin in pyannote/pyannote-audio#773 (comment)

Hi,
I ran into an error when using pyannote.core.utils.distance.pdist. I would appreciate any thoughts about whether I'm up against a bug, or this is unsupported usage.

Reproducing code example:

from pyannote.core.utils import distance
import numpy as np
a  = np.array([[1,2,3], [4,5,6], [7,8,9]])
dis = distance.pdist(arr, metric='minkowski', w=None)

Error message:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\scipy\lib\site-packages\pyannote\core\utils\distance.py", line 133, in pdist
    return scipy.spatial.distance.pdist(fX, metric=metric, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\scipy\lib\site-packages\scipy\spatial\distance.py", line 1734, in pdist
    dm = pdist(X, _TEST_METRICS[mstr], **kwargs)
  File "C:\ProgramData\Anaconda3\envs\scipy\lib\site-packages\scipy\spatial\distance.py", line 1690, in pdist
    dm[k] = metric(X[i], X[j], **kwargs)
  File "C:\ProgramData\Anaconda3\envs\scipy\lib\site-packages\scipy\spatial\distance.py", line 481, in minkowski
    dist = norm(u_v, ord=p)
  File "C:\ProgramData\Anaconda3\envs\scipy\lib\site-packages\scipy\linalg\misc.py", line 129, in norm
    a = np.asarray_chkfinite(a)

version information:
Python 3.6.8
NumPy 1.15.0
SciPy 1.0.0

How can we get label from part of segment in Annotation object.

Consider the snippet below:

from pyannote.core import Annotation, Segment
annotation = Annotation()
annotation[Segment(0, 2)]='A'
annotation[Segment(1, 3)]='B'

I am looking to get label for Segment(0,1). As per above assignment I was expecting get_labels would return A for that section.

segment = Segment(0, 1)
labels = annotation.get_labels(segment, unique=False)
print(labels)
assert 'A' in labels 

But ironically it returns empty. How can I get labels for a segment part of the whole segment in annotation.

Description

Annotation().repr_png() does not reset the legend so when next time a new Annotation object is created, the range of the legend will not change

Steps/Code to Reproduce

from pyannote.core import Annotation, Segment
hypothesis = Annotation(uri='hypothesis')
for i in range(10):
st=i
ed=i+1
hypothesis[Segment(st, ed)] = i

hypothesis.repr_png()

hypothesis = Annotation(uri='hypothesis')
for i in range(10,20,1):
st=i
ed=i+1
hypothesis[Segment(st, ed)] = i

hypothesis.repr_png()

Expected Results

every new Annotation(uri='hypothesis') are called should start with empty legend and the code provided above should not cause error

Actual Results

The above code causes error below:

File "/homes/ssd1/jackchen/jackchen/ICASSP2010/workspace/Pyannote_issue.py", line 23, in
hypothesis.repr_png()

File "/homes/ssd1/jackchen/anaconda3/lib/python3.7/site-packages/pyannote/core/annotation.py", line 1314, in repr_png
return repr_annotation(self)

File "/homes/ssd1/jackchen/anaconda3/lib/python3.7/site-packages/pyannote/core/notebook.py", line 328, in repr_annotation
notebook.plot_annotation(annotation, ax=ax)

File "/homes/ssd1/jackchen/anaconda3/lib/python3.7/site-packages/pyannote/core/notebook.py", line 234, in plot_annotation
H, L = zip(*list((next(h_l)[0], l) for l, h_l in HL))

Versions

pyannote.core==2.2.2
pyannote.database==2.1
pyannote.metrics==2.0.2

Suggested solution

The error is caused by not resetting the notebook object so if we modify the repr_annotation() function to

def repr_annotation(annotation):
"""Get png data for annotation"""
import matplotlib.pyplot as plt
figsize = plt.rcParams['figure.figsize']
plt.rcParams['figure.figsize'] = (notebook.width, 2)
fig, ax = plt.subplots()
notebook.plot_annotation(annotation, ax=ax)
data = print_figure(fig, 'png')
plt.close(fig)
plt.rcParams['figure.figsize'] = figsize
notebook.reset()
return data

and the original code is :

def repr_annotation(annotation):
"""Get png data for annotation"""
import matplotlib.pyplot as plt
figsize = plt.rcParams['figure.figsize']
plt.rcParams['figure.figsize'] = (notebook.width, 2)
fig, ax = plt.subplots()
notebook.plot_annotation(annotation, ax=ax)
data = print_figure(fig, 'png')
plt.close(fig)
plt.rcParams['figure.figsize'] = figsize
return data

When doing speech activity detection on multiple files, I noticed that the x axis limit is set in the first instance and is not updated in other files even though they have different duration. Should I do anything else for restarting the axes? Looking at the source code, it appears to me that it should already reset the axis at line 349 of the Notebook class in pyannote.core.

Below I am calling different filepaths from a dataframe to illustrate my problem.

import torch
import matplotlib.pylab as plt
from scipy.io import wavfile

def plot_audio_signal(filepath: str):
    sr, signal = wavfile.read(filepath)
    time = np.linspace(0, len(signal) / sr, num=len(signal))

    _ = plt.figure(figsize=(16, 2))
    _ = plt.title("Audio Signal")
    _ = plt.xlabel("Time (s)")
    _ = plt.ylabel("Amplitude")
    _ = plt.plot(time, signal)

i = 0
sad = torch.hub.load('pyannote/pyannote-audio', 'sad', pipeline=True)
plot_audio_signal(X["file"].iloc[i])
sad({"audio": X.iloc[i]["file"]})

First audio file with ~3 seconds duration.

i = 3
plot_audio_signal(X["file"].iloc[i])
sad({"audio": X.iloc[i]["file"]})

Second audio file with ~4 seconds duration but the x limit in the annotation object is the same as previous. You can't even see where the blue line speaker stops talking.

Version

pyannote.audio                1.1.1
pyannote.core                 4.1
pyannote.database             4.1.1
pyannote.metrics              3.0.1
pyannote.pipeline             1.5.2

Description

I tried to use matplotlib to plot some figures from pyannote-metrics, and I faced some problems

Steps/Code to Reproduce

This is the code I tried, but it didn't work.

import numpy
import matplotlib.pyplot as plt
from pyannote.core import Segment, Timeline, Annotation

plt.width = 10
plt.rcParams['figure.figsize'] = (plt.width, 3)
plt.crop = Segment(0, 40)
plt.subplot(211)

reference = Annotation()
reference[Segment(0, 10)] = 'A'
reference[Segment(12, 20)] = 'B'
reference[Segment(24, 27)] = 'A'
reference[Segment(30, 40)] = 'C'
pdb.set_trace()
# plt.plot cannot take the reference.
plt.gca().text(0.6, 0.15, 'reference', fontsize=16)
plt.show()

Expected Results

I hoped I can get the same figure in
https://github.com/pyannote/pyannote-metrics/blob/master/notebooks/pyannote.metrics.diarization.ipynb

Versions

python 3.5

from pyannote.core import Scores, Segment
scores = Scores()
scores[Segment(0, 1), 'foo', 'bar'] = 42

results in the following error

Traceback (most recent call last):
  File "pyannote/core/scores.py", line 412, in __setitem__
    self._df = self._df.set_value((segment, track), label, value)
  File "pandas/core/frame.py", line 1675, in set_value
    self.loc[index, col] = value
  File "pandas/core/indexing.py", line 117, in __setitem__
    indexer = self._convert_tuple(key, is_setter=True)
  File "pandas/core/indexing.py", line 161, in _convert_tuple
    idx = self._convert_to_indexer(k, axis=i, is_setter=is_setter)
  File "pandas/core/indexing.py", line 1096, in _convert_to_indexer
    return labels.get_locs(obj)
  File "pandas/core/index.py", line 4151, in get_locs
    ranges.append(reduce(np.logical_or,indexers))
TypeError: reduce() of empty sequence with no initial value

in some rare cases, we might loose existing edges in the process of merging

Annotation.itertracks iteration order is not deterministic when it comes to tracks sharing the very same segment. Thanks @gw17 for finding this bug.

Checking equality between two Annotation instances does not work, because SortedDict (Annotation._tracks) does not provide the __eq__ or __ne__ methods.

Thanks @gw17 for detecting this bug.

If pyannote.audio's Binarize method got prediction scores that all elements are below threshold in the condition when the Segment.set_precision() is defined. Float Infinity Error occurs.

Here's Example Case Code

case = 

import numpy as np
from pyannote.audio.utils.signal import Binarize  # pyannote.audio version 1.1.2
from pyannote.core import Segment, SlidingWindow, SlidingWindowFeature  # pyannote.core version 4.2

threshold = 0.2
binarize = Binarize(offset=threshold, onset=threshold)
input_segment_window = SlidingWindow(duration=1.0, step=0.5)
output_frame_window = SlidingWindow(duration=0.1, step=0.05)

if case==int(1):  # No Segment.set_precision(). All prediction scores are smaller than threshold
    pseudo_prediction = np.zeros((100, 1), dtype=float)
elif case==int(2):  #  No Segment.set_precision(). Some prediction scores are larger than threshold
    pseudo_prediction = np.zeros((100, 1), dtype=float)
    pseudo_prediction[50][0] = 0.5 # any number larger than threshold
elif case==int(3):  # Segment.set_precision() defined. Some prediction scores are larger than threshold
    Segment.set_precision(3)
    pseudo_prediction = np.zeros((100, 1), dtype=float)
    pseudo_prediction[50][0] = 0.5 # any number larger than threshold
# Error Case
elif case==int(4):  # Segment.set_precision() defined. All prediction scores are smaller than threshold
    Segment.set_precision(3)
    pseudo_prediction = np.zeros((100, 1), dtype=float)
else:
    raise NotImplementedError


pred_window = SlidingWindowFeature(data=pseudo_prediction, sliding_window=output_frame_window)
binarized_timeline = binarize.apply(pred_window, input_segment_window)  # Problem Line. Error occurs when all prediction scores are below threshold under the condition that 'Segment.set_precision(N)' has been defined

http://www.grantjenks.com/docs/sortedcollections/#

Apart from a direct translation from banyan SortedDict/SortedSet to SortedCollections', this will require to implement the following missing methods:

• co_iter
• overlapping
• extent

I am trying to use the get_overlap() function on some output from the speaker diarization pipeline, but I get a key error on certain elements...for example:

KeyError: <Segment(463.447, 464.358)>

The actual Segment object has the following start and end times:

start end
463.4465625 464.3578125

It looks like those values are being rounded up somewhere, therefore causing the KeyError. The actual Segment object has the value rounded down:

[ 00:07:43.446 --> 00:07:44.357]

Hello,
Apologies in advance if there is something super obvious I'm missing.

when running this line of code from the documentation tutorial:
partition = peak.apply(scd_scores, dimension=1)

i get the following error:
TypeError: bool should return bool, returned numpy.bool_

I followed the install instructions to the letter with the exception that I used conda activate vs. source activate when activating the conda environment. I have also tried running the code python 3.6 and 3.7, and tried my best to trace what was happening in the background, but I wasn't able to find what i need to change.

Thanks in advance for any advice

After going through the diarization tutorial, I use the following script

fn = "example.wav"
out_rttm = "output.rttm"

torch.hub.load('pyannote/pyannote-audio', 'dia',device="cuda",batch_size=64)

diarization = pipeline({'audio': fn})

with open(out_rttm,"w") as f:
diarization.write_rttm(f)

When I write to the out_rttm file, one expectation is for the 1st label to be associated with the 1st speaker (i.e. speaker 1 = A, speaker 2 = B, etc.). Currently the 1st speaker can be any letter.

Is there a function I can call or parameter I can set to achieve the 1st label being associated with 1st speaker and so on in the output labels? Thank you.

Hi,

Isn't there a mistake in Annotation.crop ?
You say that "If support is a Timeline, its support is used." but isn't it the other way around ?
Because you turn support into Timeline if it's a Segment

Bests,

Hi Hervé,

On file feature.py, line 199, I don't understand why you don't define the end of the SlidingWindow when you crop the SlidingWindowFeature ?
I have got on error based on this issue.
Thanks.

sliding_window = SlidingWindow(
start=self.sliding_window[ranges[0][0]].start,
duration=self.sliding_window.duration,
step=self.sliding_window.step)

There are breaking changes:
http://www.grantjenks.com/docs/sortedcontainers/history.html#v2

Would be nice to be able to (elementwise) add (or multiply) two SlidingWindowFeature with different sliding window.

Behind the scene, it would either shift or resample the right-hand side term.

This would make a lot of code much more readable.

It's all in the title : tutorial notebooks still use python2 print command.

The fixed argument in the crop method addresses the issue of rounding/numerical precision for the duration/end of the segment. However, the same issue exists for the beginning of the segment in the strict mode, potentially giving your the wrong segment in pyannote-audio.

One way to handle this would be to add a tolerance of, say, 0.0001 when rounding. E.g.
i = int(np.ceil(i_))
changes to
i = int(np.ceil(i_ + 0.0001))

I don't think such a tolerance value would have a detrimental practical impact.

From matplotlib 3.9.0 matplotlib.cm.get_cmap is no longer available due to refactoring of the colormaps:

  File "/opt/conda/lib/python3.10/site-packages/diart/utils.py", line 7, in <module>
    from pyannote.core import Annotation, Segment, SlidingWindowFeature, notebook
  File "/opt/conda/lib/python3.10/site-packages/pyannote/core/notebook.py", line 366, in <module>
    notebook = Notebook()
  File "/opt/conda/lib/python3.10/site-packages/pyannote/core/notebook.py", line 126, in __init__
    self.reset()
  File "/opt/conda/lib/python3.10/site-packages/pyannote/core/notebook.py", line 129, in reset
    from matplotlib.cm import get_cmap
ImportError: cannot import name 'get_cmap' from 'matplotlib.cm' (/opt/conda/lib/python3.10/site-packages/matplotlib/cm.py)

Documentation as well as fixes can be found here:
https://matplotlib.org/stable/api/prev_api_changes/api_changes_3.9.0.html

Scores is buggy and used nowhere in pyannote ecosystem.
Let's remove this completely.

Hi,

I would like to take an original rttm file and merge the tracks across contiguous labels and write the annotation back to another rttm file. I've attached the original rttm (1_original.rttm) and merge contiguous labels file (1_contLabel.rttm) a reference to where I have observed the two issues below:

Here is my code:
file = '1_orginal.rttm'
out_file = '1_contLabel.rttm'
groundtruth = load_rttm(file)['1']
groundtruth = groundtruth.support(collar=1)
with open(out_file, 'w') as f:
groundtruth.write_rttm(f)

Issues 1: the duration changes for some non-contiguous labels

In 1_orignal.rttm the annotation is:
SPEAKER 1 1 48.954 1.8230000000000004 sp2

In 1_contLabel.rttm the annotation changes to:
SPEAKER 1 1 48.954 3.882 sp2

Is there another parameter I need to set in support() to prevent this from happening?

Issues 2: Overlaps removed

In the original rttm if the chunks are overlapping, they are just taken out when support is used. Is there a way I can maintain these labels?

In 1_orignal.rttm the annotation is:
SPEAKER 1 1 50.641 0.3760000000000048 sp1
SPEAKER 1 1 50.785 2.051000000000002 sp2

In 1_contLabel.rttm the annotation changes to:

Thank you in advance for your help

pyannote_metrics_nsp.zip

Hi, thanks for the awesome tool!

I am trying to make some cool visualizations of my model on tensorboard. I need to crop my Annotations because the audio files that I am using are too long to be visualized correctly. I decided to use the .crop() function you provide to do that. This is the code:

def build_annotation(sequence, meta):
    annotation = Annotation(uri=meta[0][0])
    for segment, m in zip(sequence, meta):
        start, end = float(m[-2]), float(m[-2]) + float(m[-1])
        annotation[Segment(start, end)] = segment
    return annotation


def draw_annotations(hp, ref, collar=0.0):
    fig, (ax1, ax2) = plt.subplots(2, 1)
    notebook.plot_annotation(hp.support(collar=collar), ax=ax1, legend=False)
    notebook.plot_annotation(ref.support(collar=collar), ax=ax2, legend=False)
    ax1.autoscale()
    ax2.autoscale()
    fig.tight_layout()
    fig.set_dpi(180)
    fig.canvas.draw()
    img = np.array(fig.canvas.renderer.buffer_rgba()).astype(np.float32)
    img = np.transpose(img,(2,0,1))/np.amax(img)
    return img


def visual_annotations(pred, gt, pred_meta, gt_meta, max_len=30):
    plots = {}
    hp_annotation = [build_annotation(s, m) for s, m in zip(pred, pred_meta)]
    ref_annotation = [build_annotation(s, m) for s, m in zip(gt, gt_meta)]
    for hp, ref, gt_m in zip(hp_annotation, ref_annotation, gt_meta):
        end = math.ceil(max(hp.get_timeline().duration(),
                            ref.get_timeline().duration()))
        for x in range(0, end, max_len):
            print('/'.join([gt_m[0][0], str(x//max_len)]))
            sub_hp = hp.copy().crop(Segment(x, x + max_len))
            sub_ref = ref.copy().crop(Segment(x, x + max_len))
            img = draw_annotations(sub_hp, sub_ref)
            plots['/'.join([gt_m[0][0], str(x//max_len)])] = img
    return plots

Note that in visual_annotations I use .copy() in order to preserve the original annotation for the next iteration.

This is the result:

As you can see just the first subannotation gets rendered correctly, the others are completely blank. It is also weird because it looks like the time does not go forward (the support is always the same).
I checked and the segments show up when I print the subannotations after cropping, so the data seems to be there. Is it just a problem with the support? I tried passing a cropped Timeline instead of a Segment but the result is the same.

I tried with different solutions but I cannot get it to work. Is there something very obvious I cannot see?

I found when import a module that contains commands "from pyannote.core import Segment, Annotation" in a python module, it reports error: ‘ImportError: No module named 'pyannote.core'

My solution is change the import order. But Pycharm offers automatic import reorder, if using this, it reports the error.

Could u plz solve it ?

SlidingWindowFeature is a very powerful class but its documentation is nearly inexistant...

One could update the behavior of crop with center mode to always return at least one entry.

See this issue for reference

Hi,
annotation.get_labels returns dict_values if unique=False.
Shouldn't it return a list since dict_values are not subscriptable ?

Hi, I needed a smart way to fill the gaps of an annotation. I wrote it, maybe it could be useful to someone else if you want. I don't know if it's the best way to do it, but may be a starting point.
Regards,
Francesco

crop = annotation.crop(Segment(60, 120))

The above returns an annotation with nothing in time [0, 60], then the cropped segments in the interval [60, 120] (which is obviously intended behavior).

Is there a way to "shift" the crop so that the segments begin at time 0?

Code

Here, we only set maximum, which should waste some time on no-intersections?

temp = Segment(start=segment.end, end=segment.end)
for other_segment in other.segments_list_.irange(maximum=temp): # <= HERE 
# it should return all starting from the other.segment_list_'s minimum segment. 
# But we should only need the list starting from the target-searching's start?

How about set the minimum as following:

min_seg_flag = Segment(start=segment.start, end=segment.start)
max_seg_flag = Segment(start=segment.end, end=segment.end)
for other_segment in other.segments_list_.irange(minimum=min_seg_flag, maximum=max_seg_flag):

Should this correct? or I misunderstand somethings?

Getting this error on running overlapping module on my custom wav file:

ovl_scores = ovl({'uri': 'sample', 'audio': 'overlap.wav'})

from pyannote.audio.utils.signal import Binarize
binarize = Binarize(offset=0.55, onset=0.55, log_scale=True, 
                    min_duration_off=0.1, min_duration_on=0.1)

overlap = binarize.apply(ovl_scores, dimension=1)

On printing "ovl_scores ", I get this

`/usr/local/lib/python3.6/dist-packages/numpy/lib/nanfunctions.py in nanmin(a, axis, out, keepdims)
    319         # Fast, but not safe for subclasses of ndarray, or object arrays,
    320         # which do not implement isnan (gh-9009), or fmin correctly (gh-8975)
--> 321         res = np.fmin.reduce(a, axis=axis, out=out, **kwargs)
    322         if np.isnan(res).any():
    323             warnings.warn("All-NaN slice encountered", RuntimeWarning,

ValueError: zero-size array to reduction operation fmin which has no identity`

Why does it work only on AMI files and not custom files? Both audios are 16K

The current implementation relies on lots (LOTS) of calls to Annotation.setitem, which is far from optimal.

hi, I have a problem 'ERROR: No matching distribution found for pyannote.core>=5.0.0', could you help me figure out the possible causes?

A method to draw a legend in a dedicated (sub)plot.
This would contain all labels that have been encountered so far

If we type
reference
this generates a plot of reference in notebook.

However, the x range seems to be fixed in [0, 40]

Is there a way to change this x range?

will raise

ValueError: need at least one array to concatenate` 

when clipped_ranges is empty.

This issue was first encountered in pyannote/pyannote-audio#117 but is actually a pyannote.core bug.

Hi,

I am amused by your work. I would like to apply your "voice-activity-detection" class in order to extract speech from wav files. I am stucked because I don't know how to convert "pyannote.core.annotation.Annotation" type to array or something where I can exclude the non-speech related parts. May you give me some hint?