Finally, after the stages of acquisition, filtering, conversion, and transfer, the data was available to be mapped to musical structures in C++. A range of algorithms were planned and implemented, with which to explore a variety of musical and gestural possibilities with the data. I initially proposed to build a general-purpose toolkit of Etudes, simple test pieces that applied mapping structures one-by-one. Like traditional instrumental Etudes, the idea was that these could be practiced and performed as studies of technique. I intended for the primitives to be written on a low-enough level of abstraction that they could be useful for creating a broad range of musical styles; the idea was not to create a system for recreating orchestra conducting, but rather a construction kit for almost any gestures. One of my aims was to make the system unpretentious -- not motivated by any overriding aesthetic considerations, but rather by an attempt to build straightforward relationships between gesture and sound.
The Etudes were also to be organized into three categories in increasing complexity and design: technical, gestural, and dramatic. The Technical Etudes were to be a systematic set of direct mappings between sensor values and musical structures, while the Gestural Etudes would use more sophisticated algorithms to recognize complete gestures. The Dramatic Etudes were to be more akin to finished compositions for the stage with dramatic and aesthetic content. In these dramatic forms I planned to explore ideas about what makes a compelling stage performance. I thought that the Romantic era’s "caprice" form would be a good model for these; caprices were extremely challenging and expressive etudes written specifically for master performers, to show off their dexterity and skill while standing on their own as "show pieces." Over a few weeks I developed over twenty such etudes, increasing in complexity, but soon abandoned them. The mappings were too simple and spare when performed one-on-one, and a series of one-to-one mappings quickly became repetitive and dull to perform. Instead it became clear that I should attempt to build bigger, multi-layered systems instead that would manipulate many aspects of the music simultaneously. I decided that the system should be general-purpose enough to incorporate either existent pieces of music from the classical repertoire or new works composed specifically for the Conductor’s Jacket. The remainder of this chapter describes the structure and functionality of the resultant system.
6.4.1 Musical Algorithms of the Gesture Construction
Using C++ algorithms in combination with the Rogus MIDI library, there are a number of different musical quantities that are controllable in real-time by the Gesture Construction. These include such things as beats, tempo, cutoffs, holds, pauses, note volumes, channel volumes, articulations, panning, octave doublings, triggers, pitch choices, accents, timbre morphing, balances, meter, and number of voices. Each one of these is described in detail in the following section.
Perhaps the most important musical quantities for conductors are the individual beats that they generate with their gestures. These beats are indicated by an accelerational downward motion, called an ictus, which changes direction and appears to ‘bounce’ back upward. The beat itself is indicated at the moment of the change in direction. There are several ideas about whether or not conductors are in direct control of the individual beats of an orchestra, and so I started out by exploring a few different methods. I ended up using a ‘direct-drive’ model, meaning that each beat was directly given by a specific gesture, and the music stops and starts instantaneously. If no beats occur, then the variable emg0.beatData remains at 0 and no notes play. If a beat occurs, then emg0.beatData changes to 1 and one beat’s worth of notes play. The current beat number is kept track of by a counter (i) that increments at every beat and pushes the current point in the score forward:
In most implementations, I encoded the notes in a fixed order and rhythm, reflecting the Western classical convention of the written score. Once started, the score could only progress in one direction until complete, although no beats would play unless first given by the performer. Beats were also sometimes used as discrete triggers; in addition to performing notes from a score, I also sometimes used them to initiate a sequence, a stream of notes, or a process.
Secondly, particularly for conducting mappings, tempo
detection was essential for determining the distances between consecutive
notes, both within and across beats. Labview was set up to detect
beats and to send corresponding timer messages when they occurred. When
a beat was received in C++, the previous beat’s timer data was subtracted
the latest beat’s timer data; this provided the inter-onset interval
in milliseconds. Tempo was then defined in terms of this inter-onset interval;
a variable waittime was defined to be the distance between successive
sixteenth notes. (Since sixteenth notes were the smallest duration in all
the pieces I worked on, I could simply define the distance between longer
notes – eighths, quarters, and larger -- to be n*waittime.) An elaborate
smoothing function was developed so that the rapid tempo adjustments (particularly
in the presence of noisy EMG signals and double-triggered beats) would
not sound too abrupt and so that the tempo couldn’t get too fast too quickly:
There was also a threshold for waittime so that it could not be greater than 220 milliseconds; this was necessary since there would occasionally be a glitch in the Labview timer when it would flip over and start from 0, and this would cause waittime to become impossibly large. Also, the tempo was fixed for the very first beat of the piece, since it would be impossible to guess the tempo from only one EMG peak. My tempo algorithms reflected a practical approach; others have used the slightly more elaborate absolute root mean square error calculation, which determines tempo deviation or rate of change:
Thirdly, cutoffs are another critical aspect of the conductor’s gestural arsenal. Similar to beats, they are strong gestures that indicate the stop time of a held note or a pause. I set up a structure such that if a cutoff was needed, it could be fulfilled with a beat. That is, if a particular score had a musical fermata indicated (i.e., a note of arbitrary length to be decided by the conductor), then the end of the fermata would be triggered by a beat. Thus, beats and cutoffs are indicated by the same gestures, but the context provided by the score determines how the gestures are interpreted and used.
The volumes of individual notes were also controlled by the Gesture Construction. Although methods for doing this varied significantly depending upon the musical context, one way was to use a weighted combination of three values: emg0.bAmpData (the force with which the current beat was made), resp.instValue (the current respiration value), and the previous volume (to average or smooth the contour). Typical code looked like this; the weighting algorithm is given in the first line, followed by a threshold cutoff, and completed by the line that sends a MIDI note-on command with an associated channel, pitch, volume (called ‘velocity’ in MIDI), and the output node (synthesizer):
For conducting-style examples, I typically would only run an elaborate algorithm for the first two notes within a beat. For subsequent notes within a beat, I would compare the current value of the right bicep signal to its value at the beginning of the beat; if the current value was greater, then I would add 8 to the previous volume value, otherwise I would subtract 8. The exact volume contour of notes within one beat is not well understood; it is possible that work such as Manfred Clynes’ SuperConductor could help to make decisions about things such as the volume contour for notes within one beat.
In some cases I found that it was useful to combine the
control of individual note volumes with channel volumes; this allowed the
performer to have a lot more dynamic range of control between a subtle
whisper and a loud sweep. So in addition to algorithms to control the relative
volumes of notes relative to each other, I also used some techniques to
raise and lower the overall volume of voices. In one case I called this
function morph, and identified each morph function with a channel #. The
channel volume code for channel 0 is given below:
One issue with channel volumes was the extreme control that they allowed over the dynamic range; if the sensors were noisy, they were difficult to control. In situations where the sensors were well under the control of the performer, the effect could sometimes be dramatic and compelling.
Another musical structure to be controlled in real-time is articulation, which is defined as "the characteristics of attack and decay of single tones or groups of tones." Given the finding presented in section 4.1.9 of this thesis, namely that the forearm EMG signal seems to indicate the quality of articulation such as legato and staccato, I decided to tie the lengths of individual notes to the usage of the forearm. Since MIDI does not allow for the shaping of the attack and decay of single notes in real-time, I approximated articulation to be the lengths of individual notes. The code below shows how the EMG1 sensor directly determines the length of each sixteenth-note (weighted equally with the previous value for length, which smoothes the result):
Panning, a MIDI construction that allows the performer
to control the left-right spatialization of sounds, is a musical quality
that is not in control of the traditional conductor. However, in order
to give the sounds a little more life and energy to the ear, I added a
small panning function that moved them around a bit in the stereo field:
Panright, the strength of the right speaker, was controlled by the three EMG sensors on the right arm, whereas panleft, the strength of the left speaker, was defined by the equivalent sensors on the left arm. This way, aggregated values for the overall activity of both arms could be compared and used to move the sounds a little bit.
Octave doublings are another quantity that is not normally
under a conductor’s control; however, I used it as a technique to fill
out the perceived volume of the bass voices. If the left biceps EMG signal
was higher than usual, it would send out extra notes that were 12 semitones,
or one octave lower than notes on other channels. This generated the effect
that if the left arm was used, more sounds would join in to make the volume
and bass frequencies stronger:
Finally, other musical algorithms were constructed with which to control pitch, accents, timbre morphing, balances, and voice numbers. While there is not the time or space in which to describe them in detail, they were used in several compositions that will be described in the next section.