Inside the Conductor’s Jacket:
Analysis, Interpretation and Musical Synthesis of Expressive Gesture
Teresa Marrin Nakra
Submitted to the Department of Media Arts and Sciences,
School of Architecture and Planning,
in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
at the 
February 2000
copyright Massachusetts Institute of Technology 2000. All rights reserved.
Thesis Supervisors: Tod Machover (MIT) and Rosalind W. Picard (MIT)
Thesis Readers:  John Harbison (MIT) and David Wessel (U. C. Berkeley)

Table of Contents
Title, Abstract, and Committee Members




1.1 Vision
1.2 Overview 1.2.1 The slow asphyxiation of classical music
1.2.2 Why our new instruments are not expressive enough
1.2.3 What needs to improve for interactive music to become an art form
1.2.4 Instruments for Free Gesture
1.3 Motivation 1.3.1 Hyperinstruments, the Brain Opera, and the Digital Baton
1.3.2 Why continue with conducting as a model?
1.3.3 Conducting Technique
1.3.4 Why conducting might not be a good model for interactive music systems
1.3.5 Interpretive variation as the key to emotion in music
1.3.6 The Significance of Music for Us
1.4 Approach 1.4.1 Framing the Problem
1.4.2 Results
Chapter 2: BACKGROUND AND RELATED WORK 2.1 Conducting and Interpretation Pedagogy
2.2 Previous Conductor Study
2.3 Theories of Expression and Emotion in Music 2.3.1 Leonard Bernstein
2.3.2 Manfred Clynes
2.3.3 Expression "Rules" Research
2.4 Theoretical frameworks for mappings between gestures and music 2.4.1 David Efron
2.4.2 Joel Ryan
2.4.3 Teresa Marrin
2.5 Interactive systems for conductors and conductor-like gestures 2.5.1 Hyperinstruments
2.5.2 Radio Baton
2.5.3 The Virtual Orchestra
2.5.4 A MultiModal Conducting Simulator
2.5.5 The Conductor Follower of the MIT Electronic Music Studio
2.5.6 Gesture Recognition and Computer Vision
2.6 Wearable interfaces for real-time interactive music 2.6.1 BodySynth
2.6.2 BioMuse
2.6.3 Lady’s Glove
2.6.4 DancingShoes
2.6.5 Miburi
2.6.6 Benoit Maubrey’s Electro-Acoustic Clothing
2.6.7 The Musical Jacket
2.6.8 Chris Janney’s HeartBeat
2.6.9 Others
Chapter 3: THE CONDUCTOR’S JACKET SYSTEM 3.1 Background 3.1.1 Preliminary Investigations 3.2 System Design
3.3 Implementation Details and Issues 3.3.1 Design Criteria
3.3.2 Measures of expression that were not used
3.3.3 Design/Implementation Problems
3.4 Data Collection Experiments
3.5 Formatting, Timing, Graphing and Filtering the Data 3.5.1 Non-real-time filters in Labview
3.5.2 Non-real-time filters in Matlab
3.5.3 General Issues with this Data Set
Chapter 4: VISUAL ANALYSIS OF CONDUCTOR DATA 4.1 Interpretive Feature Identification 4.1.1 Use of the left hand for expressive variation
4.1.2 The flatlining effect
4.1.3 The direct, one-to-one correlation between muscle tension and dynamic intensity
4.1.4 Predictive indications
4.1.5 Repetitive signals minimized until new information appears
4.1.6 Treatment of information-bearing vs. non-information bearing gestures
4.1.7 Frequency of unnecessary actions decreases with experience
4.1.8 Clarity of signal during slow, legato passages correlates with experience
4.1.9 Division of labor between biceps, triceps, and forearm
4.1.10 Rate encoding
4.1.11 The link between respiration and phrasing
4.1.12 Large GSR peaks at the beginning of every piece
4.1.13 GSR baseline variance as a strong indicator of experience
4.1.14 Temperature baselines
4.2 Other features for future treatment
Chapter 5: HYPOTHESES OF EXPRESSION 5.1 Interpretation of results from analysis
5.2 Hypotheses of Expression 5.2.1 Efficiency
5.2.2 Intentionality
5.2.3 Polyphony
5.2.4 Signal-to-Noise Ratio of Expertise
5.2.5 Tri-Phasic Structure of Communicative Gestures
5.2.6 Bi-Phasic Pulse Structure
5.2.7 Evolution of Conducting Gestures
5.2.8 Unidirectional Rate Sensitivity
5.2.9 Musical Flow State
5.3 What is Expression?
Chapter 6: THE GESTURE CONSTRUCTION 6.1 System Architecture 6.1.1 Jacket Design
6.1.2 System Design
6.2 Real-time Signal Processing
6.3 Code Interfacing between Filters and Mapping Structures
6.4 C++ Mapping Algorithms 6.4.1 Musical Algorithms of the Gesture Construction 6.5 Resulting pieces 6.5.1 Etude 1: Tuning
6.5.2 Etude 2: One-to-One Relationship
6.5.3 Etude 3: Beats, Cutoffs, and Crescendo/Diminuendo on Sustain
6.5.4 Bach’s Toccata and Fugue in D minor
6.5.5 Song for the End
6.6 Public Performances and Demonstrations
Chapter 7: EVALUATION AND FUTURE WORK 7.1 Evaluation
7.2 Biggest Lessons
7.3 Design Wins
7.4 Future Work 7.4.1 Analytical Improvements
7.4.2 Hardware Improvements 
7.4.3 Software Improvements
7.4.4 Artistic and Theoretical Improvements
Chapter 8: CONCLUSIONS 8.1 Design Issues for Sensor Instruments 8.1.1 The Disembodiment Problem
8.1.2 Mimesis
8.1.3 Traditional vs. Digital Instruments
8.1.4 Distinctions between Musical and Physical Gesture
8.2 A Framework for Future Research 8.2.1 Extensions to the Conductor’s Jacket Project
8.2.2 Implications from the Conductor’s Jacket for Other Work
8.3 The Future of Musical Performances 8.3.1 The Need for New Live Concerts
8.3.2 Possibilities for Great Art with Sensors
8.4 Coda

Appendix: Information on the Conductor’s Jacket datafiles