The Architecture of a Synthesizer Sound Editor

Summary

With extensive use of parsing and runtime code generation together with a relational data model, it is possible to reach a 20-fold reduction in code size. Patched, a Nord Modular patch editor, uses less than 5,000 lines of Python, PyMeta2, and various DSLs to do the same job as Nomad, another Nord Modular patch editor, which is using more than 100,000 lines of Java.

Introduction

Patched (patch-ed) is the most recent result of my 10+ year long project to reverse engineer and reimplement the Clavia Nord Modular patch editor. But to complete the cross platform replacement is not my only goal. I also want to investigate interesting programming techniques.


A common way to program is to use a generic programming language to interpret input and generate output, updating internal data structures in between. This approach often results in lots of code, like Nomad, another editor for the same synth, with more than 100,000 lines of Java. For Patched, I was inspired by the research of VPRI and its somewhat different approach which results in a drastic reduction of code size. As I don't have very much spare time in front of a keyboard to type lots of code, this sounded attractive. Short code is also easier to overview and share. At the center of the VPRI approach is the concise representation of meaning using domain specific languages and runtime code transformations (compilation) to the target platform.

Another common coding practice I wanted to avoid was the object oriented data model. It requires lots of API code and structures the data in ways that make multiple simultaneous access patterns cumbersome. The relational model is a better way to organize data. A major drawback is the weak integration of query languages with the generic programming languages in common use. Using the VPRI approach, a DSL can be tailored to have good support for embedded queries when necessary.

Patched can be downloaded as a tarball from CVS. Check the README file for installation instructions.

Problem Domain

A Nord Modular patch editor uses MIDI to communicate with the synth to get the current patch and make changes to it. The editor also needs to present the patch in a human comprehensible form and let the user update it. Finally, it has to be able to load and save patch files to share with others and for backup purposes. The MIDI protocol is given by the synth. The patch file format is given by the legacy patch editor. What is left is the user interface, and I have chosen to implement a text based console UI.

Pattern Matching and Parsing

Patched is written in Python and PyMeta2, an implementation of OMeta for Python. OMeta is a language for pattern matching and parsing which is quite easy to work with. I use it extensively throughout the codebase...

• to parse the three DSLs for Protocol definition, View composition and Model construction.
• as a code generation target for the Protocol definition DSL.
• to parse the MIDI communication.
• to parse user commands and generated queries.
• to parse patch file format.

All parsing results in a translation from the source language to a target language, Python or OMeta, which can be interpreted by the platform.

• Model construction -> Python
• View composition -> Python
• Protocol definition (message parser) -> OMeta
• Protocol definition (message generator) -> Python
• MIDI message -> Mixcode
• Patch file format -> Mixcode
• Mixcode -> Python

Model Construction Language

The DSL used for data model construction makes it possible to create simple tables and fill them with data. Variables can be bound to items in Python lists to generate a span of rows where there is a correlation between columns.

map (name string, value number, caption string)
map 'knob' x:range(0,18) 'k{}'.format(x+1)
map 'knob' 18 'pedal'
map 'knob' 19 'after touch'
map 'knob' 20 'on/off switch'

The code above creates the table map with three columns and then fill it with data. The parser will translate it to the following code before it is evaluated by the Python interpreter in the model.py context.

query("create table map (name string, value number, caption string)")
for x in range(0,18):
 query("insert into map values ('{}',{},'{}')".format('knob',x,'k{}'.format(x+1)))
query("insert into map values ('{}',{},'{}')".format('knob',18,'pedal'))
query("insert into map values ('{}',{},'{}')".format('knob',19,'after touch'))
query("insert into map values ('{}',{},'{}')".format('knob',20,'on/off switch'))
 
It is not a huge reduction in number of lines for this DSL, but the number of bytes is somewhat reduced. An in memory instance of sqlite3 is used as database engine. As much of the program state as possible is stored in the relational model. Certain data is also added to the relational model to make it possible to solve problems directly in SQL and avoid the use of Python for data processing.

Patched data model: modular.model 
Model compiler and API: model.py

View Composition Language 

View composition is done using a language for 2D text layout with tight SQL integration. Blocks of text can be arranged in columns or rows to form new blocks of text. A block will expand to align with the blocks it is composed with, to make the resulting composition rectangular. Each block is configured individually how it should grow to fit.

A literal block of text is surrounded by parenthesis.

block1 =
  ('A typical block of text '
   'with two -$lines'
   ' '                       )
 
The second line has an extension point marked with '$'. This means that if the line needs to be expanded, it will insert copies of the character right before, in this case '-'. When the line is long enough both characters ('-$') are removed. The first and third line has an implicit '$' at the end of the line, which means they will expand using ' ', and the last ' ' will be removed when the line is long enough. The third line is not normally displayed, as it is the line to use for vertical expansion. If the block is put next to another block with more lines, this block will grow with space filled lines until aligned. The last line is removed when the block is long enough.

block2 =
  ('1 ' '2 ' '3 ' '4 ' ' ')

block3 =
  [ block1 block2 ]

Above, block2 is introduced with four numbered lines. Then the two blocks are composed as a row (brackets) in block3 with block1 to the left of block2. If block3 is rendered, the result would look like this.

A typical block of text1
with two ---------lines2
                       3
                       4

Column composition is the default behaviour for blocks, but can be selected explicitly using curly braces. This can be used to create columns of blocks inside a row composition.

block4 =
{ block1
  block2 }

is the same as 

block4 =
  block1
  block2

and will render as

A typical block of text
with two lines
1
2
3
4
 
Here it is also apparent that lines are not aligned until a composition so requires. To get a right aligned rendering of block4, it must be composed with the minimal block (' ') placed on the right side.

block5 =
  [ block4 (' ') ]

will render as

A typical block of text
with two ---------lines
1
2
3
4

A block can be inserted for each row in a query result set, to dynamically add zero or more blocks to the composition. Variables are bound to the columns in the result set, which can be inserted into blocks of text and other queries.

block6 =
  ('datum: {datum} ' ' ')
  * "select [datum] from x where time={t}"

For each row that match in table x, [datum] will bind to the value and one instance of the literal block will be created and added to the column. {t} is substituted with the value of t, which must have been bound earlier in the dynamic scope. When rendered it would look something like this.

datum: 12
datum: 54
datum: 6
datum: 112

A view composition is translated to Python code and evaluated in the context of a relational data model and predefined variable bindings.

Patch rendering: patch.view
Memory bank rendering: bank.view
Help rendering: help.view
View compiler and API: view.py

Protocol Definition Language

Binary protocols may not respect the character quanta that OMeta works with. Therefore it is necessary to make each singe bit accessible by OMeta by expanding the character vector into a bit vector. If [0x15, 0x7f] is a 7 bit per byte MIDI Sysex message, it will be expanded to the 14 bits [0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1] before it is processed by OMeta.

The protocol definition syntax is then further simplified to be even shorter than what can be achieved with OMeta directly. A protocol definition is translated by OMeta to OMeta before messages are parsed. The same definition can also be translated by OMeta to Python, to make it possible to generate messages.

IAmPC
  0x33:7 0x00:5 slot:2 0x06:7
  0x00:7 versionHigh:7 versionLow:7;

IAmPC is the first message Patched sends to the synth to establish contact. '0x33:7' is a constant value represented by seven bits. 'slot:2' is a variable represented by two bits. To generate the binary representation of the message, one value per variable must be supplied to the generator.

generate('IAmPC', [0, 3, 3])
=> [0x33, 0x00, 0x06, 0x00, 0x03, 0x03]

The response to this message will be parsed with the following message definition.

IAmNM
  0x33:7 0x00:5 slot:2 0x06:7
  0x01:7 versionHigh:7 versionLow:7
  serial:21 deviceId:7
  @[
  "print 'Modular {} model {} v{}.{} connected.'".format(serial, deviceId, versionHigh, versionLow),
  "insert into synth (serial) values ({})".format(serial),
  "ack({})".format(serial)
  ];

After @ follows a Python list with Mixcode statements. The message parser will return a dictionary, where the key 'actions' is bound to this list. It will later be evaluated in the modular.py context.

The pad operator (%) will make sure the length of a message is evenly divisible with the given number, and pad with 0 to make it so.

It is possible to nest message definitions and also pass parameters when doing so.

The minus operator (-) will exclude the definition when searching for a match with a complete message. The definition can only be used as part of another message.

The multiplication operator (*) makes it possible to say how many times a definition should be repeated in sequence. The slash operator (/) can make the repetition terminate when the value specified after it is found.

Patch
  0x33:7 0x07:3 last:1 first:1 slot:2 0x06:7
  command:1 pid:6 paddedpart:Paddedpart(slot pid) checksum:7;
 
-Paddedpart(:slot :pid)%7
  partid:8 part:Patchpart(slot pid partid);

-Patchpart(:slot :pid 90)%8
  section:1 nmodules:7 names:Name*nmodules
  @[
  'delete from name where slot={} and section={}'.format(slot, section),
  'update name set slot={}, section={} where slot=-1'.format(slot, section),
  'delete from name where section=2',
  "insert into name (slot, section, indx, name) values ({}, 2, 1, 'MORPH')".format(slot),
  "tx('GetPatchPart2', {})".format([slot, pid] + [[0x4f, 0x01], [0x4e, 0x00]][section] + [0])
  ];

-Name
  indx:8 name:8*16/0
  @[
  "insert into name (slot, section, indx, name) values (-1,-1,{},'{}')".format(indx, ''.join([chrn(y) for y in name]))
  ];


Common protocol definitions: patch.protocol 
Host protocol definition: host.protocol
Modular protocol definition: modular.protocol, patch2.protocol
Protocol parser/generator and API: protocol.py

Mixcode

Mixcode is a mix of Python, SQL and user commands that is translated to pure Python before it is evaluated in the modular.py context.

Similar to the View Composition Language, it is possible to iterate over result sets and bind columns to Python variables. The following Mixcode

["select [datum] from x where time={t}",
 " print datum"                        ]

will compile to

for (datum,) in model.query("select datum from x where time={}".format(t)):

 print datum

User commands start with '#'. The command frontend just appends a '#' in front of user input to conform with Mixcode syntax. Everything else in Mixcode is assumed to be pure Python.

MIDI messages generated from user commands that are sent to the synth to update the patch state are usually injected back into the MIDI message parser, to also have them update the local state. This means user commands don't have to duplicate the database updates necessary to keep the local state in sync with the synth.

User commands can call other user commands by invoking the Mixcode interpreter call eval_mixcode() in the code they emit.

The patchfile compiler is used by the Mixcode compiler to generate code for the import command.

Mixcode compiler: command.py

Patchfile compiler

An OMeta parser is used to read pch files. The file is translated to MIDI messages which are transmitted to the synth and then read back from the synth to update the user view.

Patchfile compiler: patchfile.py

Patchfile writer

A View Composition specification is used to render a patch in the relational model to the pch file format.

Patchfile writer: patchfile.view

modular.py - the integration point

All components are tied together in modular.py, which interprets instructions written in Mixcode generated by the components in the system. The relational model, the MIDI protocol compiler/generator, the patchfile writer and the Mixcode compiler are all created by modular.py and can be used by Mixcode.

Distilled, there are a few fundamental operations available for Mixcode.

• Transmit message to synth.
• Wait for ack message.
• Loopback generated MIDI message to parser for local state updates.
• Update the relational model.
• Render patch and bank view to terminal.
• Render patchfile view to file.
• Interpret more Mixcode.

Below is a flowchart that illustrates the flow of data/code in Patched. Code is data and data is code.

And here is a simplified component diagram.

Mixcode interpreter: modular.py

Simplifications Made Possible by the Nord Modular Deisgn

In the protocol and in the pch file format, a morph group is referenced as a parameter 0-3 in a module with index 1 located in module section 2. Instead of handling this as a special case, the morph module is added to the relational model as an ordinary module. This makes it possible to apply rendering code for modules in general to the morph groups.

The command 'new' is used to fill a slot with an empty patch. This is just syntactic sugar for 'import new.pch'. To treat 'new' as an ordinary patch import means we don't have to write any special code to initialize the empty patch. It is also possible for the user to 'export new.pch' to make changes to the initial state.

Observations

The API disappears (almost). Traditionally, I have spent a lot of time writing and calling APIs. Not so when using runtime code generation. The code generator can skip over the API, and generate highly expressive code directly in the target language. A little API is required to integrate the various components like sqlite3 and Port Midi, to reach through to the metal in an efficient manner. Each parser also needs a compile or interpret call, and that's it.

You don't need to go all the way to a completely modifiable metacircular programming system to make good use of runtime code generation. Patched is a traditional write-compile-run application.

The code is compact, but not incomprehensible. Everything is there right in front of you at the right level of abstraction when it needs  to be modified. Code written in a generic programming language tends to have much more accidental complexity, imposed by the generality of the language.

Code is data - Code can be constructed and manipulated just like data.
Data is code - All data is encoded in some language that can be compiled and/or interpreted to manipulate the system state.