For an HTML version check out the Manual under Software Projects.

[TOC]

MidGen is a Perl based standard midi file reader/processor/generator. It includes Perl modules and packages to create/read/write and process standard midi (smf) files based on Perl input scripts. Essentially this tool provides a framework, which is specifically designed for experiments with musical pattern, arpeggios, accompaniment styles, arrangements or entire scores. However MidGen is not only limited to musical data processing since the framework provides furthermore access to all smf event types such as Channel-/SysEx-/Escape- and Meta-events. In general there is no limitation since everything representable by smf can be generated, processed and provided by MidGen.

For musical data, a build-in "micro sequencer" function translates text-based micro-sequences into smf midi events. Since everything runs in perl environment, you can take advantage from programming languages like using variables for micro-sequences, sequence substitutions, storing arrangement parts in sub-procedures, repeat sequence iterations with loops, etc. together with convenient smf input/output functionality. This approach allows writing music in traditional sequenced format in combination with algorithmic or procedural programming functionalities. In addition, there are many functions for SMF data manipulation implemented such as copy/paste/insert/transpose which are working either on track level or across the entire arrangement. Also you can include and merge additional external smf midi data into your arrangement so that for instance live played parts get merged with generated accompaniment pattern. Essentially there is no limit for creativity since its an open framework were you can easily add features, functions, procedures or other extensions with your own ideas.

Example smf output arrangement in score representation:

The score was entirely generated from the example source code below using micro sequences:

To compile the midi output file, just run:

The output is typically a type 1 smf midi file saved in your current working directory from where the script gets executed. In addition, the console output provides an overview and displays general song- and track-information of your arrangement. For more detailed result debug, individual event lists per track are written as regular text files along with the smf output.

Once you have installed perl, just run:

perl MidGen.pl <project.pl>

If there is no project-file specified, the default project (Projects\Current.pl) will get processed.

To get started quickly and to see how the output looks like, you can also just process a smf midi file thru the program or try one of the provided project examples. MidGen was tested with ActiveState Perl, Strawberry Perl portable and various Linux/Ubuntu Perl versions.

Example: read/write a smf midi file:

The easiest way to explore MidGen in order to see how it works and what it can do, is to process some of the example files. Since some of them are made specifically for playsmf containing loops and arpeggios, it would be helpful to play the output smf using this tool.

• MidGenPrj\BalladePourAdeline.pl - simple score showing how the micro-sequencer works
• MidGen\Projects\ReadSty.pl <filename.sty> - converts Yamaha Style files into looped SMF (usable with playsmf)
• MidGen\Projects\ReadCmf.pl <filename.cmf>- converts CMF (Creative Music File) to SMF
• MidGen\Projects\DancingSliders.pl - shows how continous controller data can be generated
• MidGen\Projects\PortamentoExample.pl - demonstrates the portamento function
• MidGenPrj\STYLES\Laeufer_Humanized.pl - piano style arpeggio with timing and velocity randomization
• MidGenPrj\STYLES\Current51_GuitarStrumEFX.pl - guitar strumming style example w/ Roland EFX effects attached
• MidGenPrj\STYLES\Current30_SoloMatrix.pl - style example with drum, bass and guitar strumming
• MidGenPrj\STYLES\Current38_GuitarLive.pl - guitar strumm style example (keys down and up will trigger guitar chords down/up)
• MidGen\Projects\RolandHp503Test0.pl - demontrates how to use smf phrases (live played parts) and generated styles
• MidGenPrj\STYLES\Current53_MC.pl - demonstrates midi clock
• MidGenPrj\STYLES\Current54_MTC.pl - demonstrates midi time code
• MidGenPrj\STYLES\Current55_ChordCheck.pl - simple style pattern with few guitar chords to check all implemented chord types and inversions

• MidGen.pl - top level framework entry point
• MIDI.pm - basic smf input/output module (contains general smf read/write and event-insert functions)
• MidiDebug.pm - not necessarily required, but contains additional track and event list output functions
• Edit.pm - contains general smf editor functions such as
  • micro-sequencer
  • step-sequencer
  • copy/paste tracks, regions, etc.
  • quantize tracks (note timestamps, durations)
  • fade-in/out single tracks or entire songs
  • portamento
  • randomize (humanize) timestamps, durations, note on/off velocities

and much more.

The Devices sub-folder contains mainly device specific packages and modules.

• GM.pm - GM controller definitions including RPN/NRPN and GM reset SysEx function
• GM2.pm - contains mainly GM2 universal SysEx functions such as MasterVolume, FineTune, Reverb, Chorus and MMC functionality
• GS.pm - Roland GS functions (mostly SysEx functionality)
• XG(lite).pm - very few Yamaha XG(lite) definitions
• MMC.pm - Multimedia control (mostly SysEx transport control functions - moved later into GM2 standard)
• MG.pm - few MidGen specific definitions
• SMF.pm - few SMF specific definitions
• Sync.pm - midiclock and midi time code functions
  • InsertMC() - insert midi clock event series using escape sequence events
  • InsertMTC() - insert midi timing clock event series using escape sequence events
• playsmf.pm - definitions and functions specifically for playsmf midi player/looper/arpeggiator
  • OverrideCmdLineArgs() - override command line arguments using sequencer specific meta events
  • PreProc() - event pre-processor especially for jump on jump and zero shot sequence preparation (dummy event insertion)

This sub-folder was originally meant to store a sequence library for musical pattern like bass lines, drum pattern, styles, etc., however it evolved over time to a library of tools, helper functions and other miscellaneous perl packages.

Internally the smf is nothing else than a multidimensional hash structure with integer key values across all hierarchy levels. The top level key represents the track number (except "track" -1 which is used for smf specific information like filename, smf type, PPQ, etc.), the 2nd level key represents the eventtime in ticks based on the smf PPQ setting and the 3rd level key is the event ID representing the event order within a given tick. This allows simple access and iterations across all smf events for easy data manipulation.

Reading and writing a midi file is as simple as:

my %smf = MIDI::Read("filename.mid"); MIDI::Write(\%smf, "filename_new.mid");

In addition, you can also display general track overview information on the console output screen plus getting individual event lists per track by calling MidiDebug::PrintInfo() such as:

my %smf = MIDI::Read("filename.mid"); MidiDebug::PrintInfo(\%smf); MIDI::Write(\%smf, "filename_new.mid");

Since MidGen has already defined a default smf datastructure handle called %main::out runnig thru MidiDebug::PrintInfo() and MIDI::Write() functions at the very end of the program flow, you dont need necessarily taking care about those functions unless you work with multiple smf structures in parallel. Therefore its also possible to read and write a smf by just running the following instruction:

%main::out = MIDI::Read("filename"); $main::out{-1}{0} = "filename_new.mid";

Make sure that you provide a new filename (smf "track" -1, argument 0), else the original smf will be overwritten.

track event list view output

As mentioned, each individual track provides additional event lists within separate text (.txt) files which are written in parallel to the smf output. Those lists might be helpful for debugging or just to get an event overview by track. The following information is organized in individual columns:

• event timestamp in PPQ ticks (1st level smf hash key)
• event timestamp in bar:beat:tick
• ordered event ID (2nd level smf hash key)
• event data in raw hex representation
• event data interpretation (type, channel, value, duration, etc.)
• graphical value representation (ascii bar)

Example track lists:

special "track" -1

As already mentioned, there is one specific "track" (-1) holding generic smf information by 2nd level key values such as:

• -3: smf position pointer - only valid after reading a smf. This field contains the file position after reading the smf. Typically it points to EOF (end of file = file size), however is some cases the file contains more than just smf data which is indicated by smf position < smf size. In this case the pointer can be used to continue reading data from there. For instance the Yamaha style reader makes use of this parameter since the the style CASM sections starts after the smf data block.
• -2: smf size - only valid after reading a smf. This field can be used in combination with the smf position pointer to determine if there is more than just smf data stored with the file.
• -1: smf timestamp - only valid after reading a smf
• 0: smf name
• 1: smf type (typically type 1 since type 0 gets automatically converted to type 1 if not otherwise specified)
• 2: number of tracks - only valid after reading a smf for information, else unused
• 3: PPQ (pulses per quarter note / smf timing resolution)
• 4: initial tempo
• 5/6: initial time signature nominator/denominator
• 7: initial key signature
• 8: copyright text

timestamps and durations

Timestamps and note- or continuous controller-durations for functions such as (insert, micro-sequencer, copy, etc.) are typically provided in whole notes as floating point values to decouple them from timesignatures. Internally all times are stored in ticks based on the smf PPQ setting. Therefore its important to define the required resolution in the beginning of a project.

supported event types

MidGen supports all types of midi events including channel-, SysEx-, Meta- and Escape-messages. NoteOn-Off pairing is done internally while reading smf files. That means all internal operations refer to Note- rather than Note-On/Off events unless they are inserted individually on purpose. Therefore each note is represented as a single event with a given duration and a optional note-off velocity.

The micro-sequencer is a central and important function especially written for convenient note- , controller- and other data insertions. Essentially it inserts a sequence of notes and/or controller/meta/sysex events into a specified track, based on a given start-time, scale (e.g. chromatic, major, minor, etc.) and base note (e.g. 60 - middle C). The sequence is provided as a consecutive, white space separated list of <duration>:<data> pairs representing duration timestamps with associated event data in text format separated by the colon : sympol. The micro-sequencer keeps internally track of timestamps, note-pitches and other data and typically advances in time with each provided event-duration value unless otherwise specified (e.g. for overlap notes or chords). In addition, the sequencer returns the total duration of the provided micro-sequence in whole note units. This allows the caller thread keeping track about the total timestamp if you simply sum up all micro-sequence durations in case they belong logically together. The micro sequencer function is called from Edit package as follow:

<duration> = Edit::Seq(<smf-hash-pointer>, <track>, <start-time>, <base-note>, <scale>, <sequence>);

Actually there are much more arguments available, but those are the most important ones to start with.

• <smf-hash-pointer> is simply a pointer into your smf midi data structure where the sequence get inserted. Typically you'll use the standard smf output structure %main::out, but in case you work on multiple smf structures in parallel, you need to specify which one to use.
• <Track> is almost self-explanatory. It simply specifies the tracknumber where the sequence goes into. Track numbers start from zero.
• <start-time> is basically the timestamp in whole note units where the sequence gets inserted.
• <base-note> determines the sequence reference note 0 as the sequencer works relative from there. In conjunction with the scale it provides finally the sequence key.
• <scale> 0:chromatic (typically used in combination with base-note zero to access regular midi note numbers - e.g. for percussion sequences); 2:major; 3:minor
• <sequence> is a text string argument representing the actual sequence to insert

Example - simple micro sequence:

The sequence above is mostly self explanatory.

Timestamps and event-durations are typically provided in whole note units either as integer or floating point numbers or as equations such as 1/4 or 1/4+1/8, 1/4+1/8+1/16 etc. To shortcut dotted note equations, you can just append tailing + signs to the duration value. For instance a dotted quarter can be written either as 1/4+1/8 or alternatively as 1/4+. Double or triple dotted notes are written respectively with tailing ++ or +++ signs. Similarly you can shorten a note by its half length with tailing - signs. The example below shows few different variants of duration timestamps.

Note: If there is no timestamp provided, the sequencer automatically applies the latest valid timestamp value.

To avoid complex arithmetic equations and to keep sequences more readable, it is possible to align event durations to given timestamp grid boundaries by using the alignment operator | in front of the duration value. This advices the sequencer to proceed in time until the next grid boundary timestamp is reached. So in the example above, the |1/1:% event will simply insert a rest in alignment with the next whole note timestamp value. In this case it proceeds to the next bar since the time signature is 4/4 (whole note grid boundary).

In order to keep the sequence more structured and readable, it is possible to insert additional | character symbols to visualize bar, beat or other timestamp separations. Semantically they dont have any meaning and are just ignored like comments or remarks by the sequencer as long as they stay in clear separation from events.

The microsequencer supports different smf event types such as

• note events
• controller events including after touch, sysex, program change and tempo events either as single event or as continous controller series
• marker and lyric text events
• time- and key-signature changes
• pause/rest/NOP "events"

Events are always paired with duration timestamps even though some event types do not really require a duration such as marker or lyric events. In this case a zero duration can be used if the sequencer should not advance in time. Events can also get paired to insert multiple event types at the same time with the same duration in one sequencer event. For instance you can insert a note and in parallel a continous controller using the same timestamp and duration.

Note events are typically provided as numerical values rather than traditional musical symbols. This decouples them from scales and tonal systems and keeps the flexibility for additional arithmetic operations. They can get specified either in absolut- or relative (interval) values by preceding ^ (up) or v (down) symbols. Absolute values can be either positive or negative in case they refer to notes below the given basenote. Notes will follow the given scale and key provided by micro sequencer arguments <scale> and <basenote>. In result, note zero is basically the basenote.

Example - Two consecutive micro sequences with additional marker M<text> events and bar separators:

The example above just demonstrates how multiple micro sequences can get concatenated using a global timestamp variable.

Since the micro sequencer keeps internally track about note values, it is possible to use relative note intervals in addition to absolute ones. Relative notes are determined by preceding up ^ or down v symbols in front of the note values. If there is no note number specified, the sequencer assumes just one relative note step.

Note values can be increased or decreased in semitones by putting either flat and/or sharp symbols after the note value. Traditional music puts them in front of the note, but here it is required to put them after the note value as additional attributes. The sequencer allows having multiple consecutive and/or mixed symbols by simply summing up all flats and sharps to get the final value.

Example - C major with few additional flats and sharps:

rests and pauses are simply represented by the % character symbol.

To repeat single events or rests with the same duration and/or note values, you can just use the . (dot) symbol for either duration and/or event data values. The following scenarios are possible:

• <duration>:<event> - regular duration/event data type pair w/o repeat
• <duration>:. - repeat the latest event note value with a new applied duration value
• .:<event> or <event> - repeat the latest duration value with a new applied event note value. in this case, the duration repeat operator and the colon separator .: is not necessarily required and you can shortcut by just writing the new event value as already done in many examples before.
• .:. or shortcut . - repeat the whole event using the latest applied duration and event values (true event repeat)

The continue event operator > is similar to regular repeats, but ignores rest events and simply 'continues' playing with the latest note value. This event type is mainly used in combination with pattern sequence templates.

In addition, the micro-sequencer comes with several build-in loop and sub-sequence functions for sequence repetitions. Essentially you can enclose sequences or parts of sequences within parentheses or braces n{<sub-sequence>} and put a repetition number n in front of them. If no repetition number is given, the sequencer assumes one insertion w/o repetition, otherwise the sequencer will insert the enclosed sub-sequence(s) n times. Since the sequencer allows the insertion of recursively nested sub-sequences, repetition pattern can quickly get large and complex by just a few instructions. The example below shows a small input sequence with two nested sub-sequences.

Finally, different types of parentheses have different meanings:

• curly n{<sub-sequence>} - normal repeat
• regular n(<sub-sequence>) - preserve/restore duration and note values when a sub-sequence gets entered, but advance in time
• square n[<sub-sequence>] - preserve/restore duration/note and timestamp values when a sub-sequence gets entered (doesnt advance in time)

Since the sequencer keeps track of timestamps, durations and notes, you can preserve the previous duration/note and timestamp values when a sub-sequence get entered. In result, the sequencer can restore those values when returning to the main-sequence. This is especially important when building pattern templates using relative note values.

Usually the sequencer takes care about timestamp values and advances in time with each provided event duration. This function is typically usefull for monotonic lines like lead melodies, bass lines, etc. with non-overlapping notes. However in many cases you need note or event overlaps such as in chords or many other situations. Therefore the sequencer provides additional timestamp directives by having additional < symbols in front and/or after the given timestamp value.

The examples above show 4 different scenarios:

The 1st one is a regular insert of a quarter note where the sequencer advances in time with the event duration. Therefore the total sequence length is effectively one quarter note.

The 2nd scenario shows that the return marker merges with the entry marker because the sequencer doesnt advance in time when the quarter note gets inserted. In result, the total sequence length is effectively zero although a note was inserted.

The 3rd scenario demonstrates how the sequencer goes back in time before the note gets inserted. Therefore the note appears earlier in time as the sequence entry point. Since the sequencer still advances in time with the inserted event, in result the return marker merges again with the entry marker and the total effective sequence length is zero.

The 4th scenario demonstrates the combination of both previous scenarios. The sequencer goes back in time before the event gets inserted, advances in time with the event and finally goes back in time again after the event insertion. In result, the note and the return marker appear earlier in time than the entry marker. Effectively the sequencer is running backward and returning a negative sequence length in this case.

Finally timestamp directives can be used to write overlap notes, chords or sub-sequence overhangs at the beginning or ending of an sequence. The example below shows a simple chord triad with 3 stacked notes. In this case only the last note get advanced in time while the 1st and 2nd notes doesnt advance. If there is no timestamp value provided, the given timestamp directive is still valid and the sequencer will not advance after the event insertion.

micro-sequence overlaps (overhang) Another example below shows how to setup sequence overlaps or overhangs using timestamp directives. In some cases it is required having extra events or overlap events either before the sequence actually starts or after the sequence has been finished. This can easily get achived by timestamp directives going back in time before the sequence starts or after the sequence has been finished. The example below shows two concatenated micro-sequences with additional overhang events on each sequence side.

Pattern sequence templates are typically sub- or partial sequences refering exclusively to relative notes based on the sequencers internal states. This decouples them from absolute note values and keeps the templates more generic. Usually they are assigned to perl variables for further usage in higher level sequence strings.

This allows for instance building different chord-type templates or small arpeggio pattern templates without attaching them to absolute note values for later instanciation within sequences.

The example above shows how a simple chord triad is setup as a template and beeing used in a small sequence.

Remark: Some strange looking events like 1/1<:0_% are so called NOP (no operation) events since they do not advance in time neither inserting a note since a rest is applied in combination with a note value, however they instruct the sequencer to set the current timestamp and the current note value for further sequence events.

The following example is very similar and shows how to insert a small arpeggio sequence using templates.

Now one template example demonstrating how to use subsequences with and without sequencer value restoration. Actually when we simply concatenate a sub-sequence template such as " > ^2 ^2 . " multiple times, the sequencer will increase continously all note values with each insertion. In contrast, if we put the sub-sequence into parentheses, the sequencer will take care about preserving and restoring note values when the template gets entered or left. Therefore the resulting sequence is totally different.

The next example shows the effect of additional square brackets around the template sequence and their effect. In this case not only note values, but also timestamp values get preserved when the subsequence get entered. In result, the sequencer jumps back in time once the sub-sequence template has been processed and starts over again from the beginning. Therefore the sequence appears now stacked rather than consecutive in time.

Each note- or pause-event of the micro sequencer supports additional (optional) attributes such as on/off velocity values and/or attached controller data events or event series. This way you can easily attach additional articulation attributes to each individual note in alignment with note-on and duration timestamps. Note attributes are typically appended by undersscore _ separations. For example " 1/4:0_.75_r.25 " inserts a quarter note with note value 0 (relative to scale and base note), NoteOn velocity 0.75 and release velocity 0.25.

Note-On velocities are simply written as normalized (0.0 ... 1.0) floating point values in adition to note events. If a note event is not specified, the sequencer will repeat the previous note with the provided velocity values. This can be useful to shortcut for instance percussion sequences repeating the same note with different velocities.

Note-Off velocities are specified by r<velocity> floating point values. They are rarely used and most sequencers doesnt really display them, but here an Cubase list edit example showing both Note-On and -Off velocities together.

Controller-type attribute data are either additional single events inserted together with NoteOn or rest events at the current timestamp position in sequence or continous event series inserted along the given note duration. They are appended to note or rest events with additional arguments by one of the following syntax formats.

Single event controller (e.g. foot, switch, pedals, etc) are provided by either two or three additional arguments like:

<duration>:<event>[_C<Controller#>_<Value>]*

<duration>:<event>[_C<Controller#>_<Centered>_<Value>]*

For continous controller data series, the following format is used:

<duration>:<event>[_C<Controller#>_<Function>_<Centered>_<Value>_<Value1>[_<align>]]*

Controller type data can be one of the following event types specified by the Controller# argument:

• 0x0..0x7f: regular (7bit) MIDI controller
• 0x80..0xff: poly after touch by key
• 0x100: channel after touch
• 0x101: NoteOn velocity - no real extra events, but modify existing note-on velocities
• 0x102: NoteOff velocity - no real extra events, but modify existing note-off velocities
• 0x103: tempo adjustment in % (1.0=100%) based on initial song tempo setting - makes only sense in track zero
• 0x104: pitchbend
• 0x105: program change - makes only sense as single event, but can attach dedicated program changes to individual notes
• 0x200..0x21f: extended (14 bit) MIDI controller
• 0x300..0x303: extended (14 bit) MIDI controller GP #5-#8
• 0x304..0x3fff: SysEx callback functions - only available if sysex callback functions are assigned by device specific packages
• 0x4000..0x7fff: RPN
• 0x8000..0xbfff: NRPN

The Function argument specifies one of the following data sweep shapes:

• 0: unused
• 1: linear transition sweep from Value to Value1 or single event if Value1=Value or if Value1 is not provided
• 2: full swing sinusoid [0 , 2pi] sweep from Value to Value with elongation = Value1 (sign determines up/down swing direction)
• 3: half swing sinusoid [0 , pi] sweep from Value to Value with elongation = Value1 (sign determines up/down swing direction)
• 4: half sinuoid [-pi/2 , pi/2] transition sweep from Value to Value1
• 5: sigmoid transition sweep from Value to Value1

Remark: If the function argument is provided as a negative number, the data sweep will be inverted (up<--->down).

The Centered argument determines if the Value arguments are provided in centered (signed) or uncentered (unsigned) format:

• 0: uncentered - normalized values provided as [0.0 ... 1.0]
• 1: centered - normalized values provided as [-1.0 ... 0.0 ... 1.0]

The centered format makes especially sense for centered controller types such as pan, balance, pitch-bend, etc. while other controllers such as volume, expression, etc. make more sense in uncentered format representation - although there is no restriction for one or the other format type.

The optional aligned argument determines if controller data series get inserted continously (default) or in alignement with Note-On events. This feature can be used to reduce the amount of data in case controller changes are only required when notes get triggered.

Example - switch controller: attach additional sustain pedal press/release events to 1st and last note in sequence

Example - continous controller: sweep MIDI controller 7 (volume) using full swing sinusoid function along the given note

Below a few more examples inserting whole note rests along with marker text and midi controller 11 (0xb hex) demonstrating different swing and transition types.

Example - alignment: Insert 16 x 1/16th notes, go back in time and insert controller 11 (0xb) in alignment with notes, but 1/64th ahead of note events.

Example - tempo adjustment: Insert initial tempo and sweep +/- 25%

In general the Micro-Sequencer contains an embedded Step-Sequencer which kicks in if a sequence text string is enclosed within | bar character symbols. In this case the enclosed step sequences get internally converted to micro-sequences before the micro sequencer compiles the smf similar to a pre-processor. Step-sequences are basically text strings were each single character or symbol refers to an individual micro-sequence. Those individual sequences can be either quite simple just containing single notes or more complex containing chords, guitar strums or other pattern types.

A typical use case for step sequences are for instance drum patterns since they usually run on a regular fixed time base with static key->voice assignments. In this case a time base (or step granularity resolution) and a note definition is required before the step sequence is applied. This is simply done with a dummy rest event which does not advance in time (see example below).

Example: insert few micro-step-sequences with 1/16 resolution across multiple drum voices

Lets take a closer look at the example above: The initial dummy event 1/16<:35_% is just a rest event without advancing in time, but defines the resolution and the absolute key value based on chromatic scale with basenote=0. The remaining characters are all enclosed within bars very left and very right belonging to the step-sequence.

Remark: Additional bar | characters within a step sequence are just visual separator symbols for better readability without any semantic meaning. Internally they get removed before parsing the sequence.

By default, the step sequence characters have the following interpretation:

• . - rest
• 0,1,2,3,4,5,6,7,8,9,X,x - notes with different velocity values 0%, 10%, 20% .... 100% (X,x)
• > - repeat or stretch the previous note
• # - one semitone up
• b - one semitone down

Example: demonstrating different velocities and stretching notes.

The Edit::Portamento() function emulates portamento functionality by collapsing monotonic melody tracks into single note pitch tracks with fixed keys while the pitches are provided by pitch bend events.

<Pitch-Bend-Sensitivity> = Edit::Portamento(<smf-hash-pointer>, <track>, <portamento-time>);

• <smf-hash-pointer> - smf hash pointer handle
• <track> - selected track
• <portamento-time> - gliding time

The function returns the required pitch bend sensitivity which must be inserted in the very beginning of the track, else the sequence will sound off.

eof