yikai-liao / symusic Goto Github PK

Following #6, we found an issue with the order of the velocity and NoteOff messages when parsing / writing notes having the same onset time and pitch values.

This is likely to be a FIFO / LIFO issue, either one of these principle should be applied for both parsing and writing in order to keep the data integrity.

Each MIDI (and in turn ScoreTick) has a time division express in ticks per beat, which often ranges between 120 and 480 (multiples of 2, 3 and 4).

It could be useful for some users to provide a feature allowing to resample a ScoreTick by changing the time signature. This would imply resample the times of all the events of a MIDI, and durations of events concerned.
For the durations especially, a min_duration expressed in tick could be very useful for cases where the resampling reduces the time division and turns some durations to 0.
You can take a look at how it is done in miditok (quantize_* methods), though there are probably better ways to do it, I was thinking of using numpy to batch the computations.

That's not an important feature, this can wait or be refused without major justification.

There must be some wrong ...

Pedals are actually stored twice, in controls and pedals of a track

If someone change controls or pedals, there might be some inconsistence between them.

Now, in symusic, we just write all the controls back and ingoring the pedal events in pedals.

Well, a solution is to remove all the corresponding control events in controls, but I'm not sure if it is a good design.

What's your opinion @Natooz ?

Following #6, there is a float conversion somewhere that can slightly alter the Tempo.tempo values, causing the tests (__eq__) to fail.

For higher information density, I'm considering changing the repr of list of event ( like NoteTickList ), like this

When showing a list, I tend to hide they arguments' names, while the names are reserved for showing the single one

Note(time=0, duration=118, pitch=69, velocity=117)
[Note(0, 118, 69, 117), Note(0, 118, 77, 62), Note(240, 238, 76, 62), Note(480, 238, 72, 62), Note(720, 238, 67, 62)]

And for track and score, I will turn to show the summary, following miditoolkit, like this:

Score(ttype=Tick, tpq=480, begin=780, end=1431755, tracks=51, notes=60411, time_sig=97, key_sig=97, markers=97, lyrics=0)
Track(ttype=Tick, program=72, is_drum=false, name=PICCOLO, notes=1053)

And, when showing TrackList, the arguments' names won't be removed

@Natooz Do you have any suggestions?

We are designing interface for pianoroll representation. There are different existing interface designs:

• miditoolkit :
  • Only accept List[Note] as input. (I think that' s weird because miditoolkit itself has containers as Instrument.
  • resample_factor, resample_method, time_portion, keep_note_with_zero_duration are provided as arguments, because division of it is tpq. (I think we accept Score<Tick>, Track<Tick>, Note<Tick> should be enough)
  • velocity_threshold is provided as argument. (I think we can ignore it and implement filter_notes)
• pypianoroll:
  • Accept encode_velocity as argument to decide whether to binarize the pianoroll.
  • No much arguments.
• muspy:
  • Basically the same as pypinaoroll

I think the interface must satisfy:

• Provide encode_velocity as argument to decide whether to binarize the pianoroll.
• Let user use the global resample interface instead of resolution -related argument.
• Provide pitch_range, pitch_offset to clip pitch range.
• modes of pianoroll (onset, offset, frame), accept as a List[str]?

Here are my considerations, do you have ideas? @Natooz @Yikai-Liao

Hi! Many thanks for developing Symusic, a really great addition to the community.

I've been using miditoolkit for MIDI related processing but plan to switch to Symusic as it's really much faster. The only thing I miss now is the conversion between ticks and seconds. I need this for music transcription, alignment and synchronization tasks.

What are your plans for implementing the conversion of event times between ticks/quarters and seconds? Natooz/MidiTok#112 (comment)

In miditoolkit there are get_tick_to_time_mapping and _get_tick_to_second_mapping methods that create an array with indices providing a map between tick positions in MIDI to their time positions in seconds. It's very memory inefficient, but allows you to convert any tick to seconds.

I propose to have in Symusic a function that accepts a time in ticks/quarters and returns the time in seconds. And the reverse function (second -> tick/quarter). Converting the whole score to/from seconds is nice, but being able to map any point in time between time domains is also a needed feature.

Possible implementation: having precomputed times in ticks and seconds for all Score tempos, for any arbitrary tick/second we can find the closest tempo and compute the delta shift using the tempo.

SoA support would be an important feature for symusic, since it is more suitable for AI applications than the current AoS (Array of Struct) interface.

SoA interface could enable lots of flexible conversion, resampling, quantization and other operations. #10
These functions take full advantage of numpy's eco (like numba), and could be very fast.

It's also important that we don't need to introduce more time_unit types (like beat) in symusic, which would make the general purpose code in c++ part more and more complex.

However, the interface for SoA is still to be determined. I'd like to hear your advice @Natooz @ilya16 . And of course, other design options are welcome!

Here, I will list some possible options I could think of.

Option 1: Dict of Numpy Array

In this case, we won't introduce new classes in symusic, but only use dict and numpy.ndarray.

from symusic import Score, Note, ControlChange
s = Score(...)

# get the soa of controls
# because "controls" is not a python list, but a c++ vector
# we could bind a .numpy() method for it
controls_arr: Dict[str, np.ndarray] = s.controls.numpy()
notes_arr: Dict[str, np.ndarray] = s.tracks[0].notes.numpy()

# create traditional AoS from numpy array
# Here, we could utilize the existing factory class for events like notes 
s.controls = ControlChange.from_numpy(controls_arr['time'], controls_arr['number'], controls_arr['value'])
s.tracks[0].notes = Note.from_numpy(notes_arr['time'], notes_arr['duration']， notes_arr['pitch']， notes_arr['velocity'])
# or we could use ** to shorten this
s.controls = ControlChange.from_numpy(**controls_arr)
s.tracks[0].notes = Note.from_numpy(**notes_arr)

Option 2: New SoA Classes in C++

In this case, we will define new classes for SoA, and use them in symusic. It seems more object-oriented.

The problem is, they will be defined 3 times, because of time unit. (The same reason for NoteTick, NoteQuarter and NoteSecond)
We will define a Union for them in symusic.types

from symusic import Score
import symusic.types as smt

# get the soa of controls and notes
controls_arr: smt.ControlChangeArr = s.controls.numpy()
notes_arr: smt.NoteArr = s.tracks[0].notes.numpy()

# convert them back to AoS
s.controls = controls_arr.list()
s.tracks[0].notes = notes_arr.list()

Also, although we have switched to nanobind, which get a much smaller overhead on accessing class attributes, the overhead is still there. Note that those overhead are almost constant, so it's not a problem for those "ms level" functions.

So if not necessary, I would not recommend create new class in c++. (Well, this overhead should be considered more in AoS part, not the SoA part)

Here is a benchmark for those tiny operations.

Option 3: New SoA Classes in Python

In this case, we define the new class in python. It is more flexible, python native (no overhead).

But, these class can't be called in c++ (At least I don't know how to achieve this now. Maybe it's possible).
So, we won't get the .numpy() function here.

from symusic import Score, NoteArr, ControlChangeArr

controls_arr = ControlChangeArr(s.controls)
notes_arr = NoteArr(s.tracks[0].notes)

# convert them back to AoS
s.controls = controls_arr.list()
s.tracks[0].notes = notes_arr.list()