Inside the
Conductor’s Jacket:
Analysis, Interpretation
and Musical Synthesis of Expressive Gesture
by
Teresa Marrin
Nakra
Submitted to the Department
of Media Arts and Sciences,
School of Architecture and
Planning,
in partial fulfillment of
the requirements for the degree of
Doctor of Philosophy
at the
MASSACHUSETTS INSTITUTE OF
TECHNOLOGY
February 2000
copyright Massachusetts Institute
of Technology 2000. All rights reserved.
Thesis Supervisors: Tod Machover
(MIT) and Rosalind W. Picard (MIT)
Thesis Readers: John
Harbison (MIT) and David Wessel (U. C. Berkeley)
Table of Contents
Title,
Abstract, and Committee Members
Dedication
Acknowledgements
Chapter 1: INTRODUCTION
1.1 Vision
1.2 Overview
1.2.1 The slow
asphyxiation of classical music
1.2.2 Why our new
instruments are not expressive enough
1.2.3 What needs
to improve for interactive music to become an art form
1.2.4 Instruments
for Free Gesture
1.3 Motivation
1.3.1 Hyperinstruments,
the Brain Opera, and the Digital Baton
1.3.2 Why continue
with conducting as a model?
1.3.3 Conducting
Technique
1.3.4 Why conducting
might not be a good model for interactive music systems
1.3.5 Interpretive
variation as the key to emotion in music
1.3.6 The Significance
of Music for Us
1.4 Approach
1.4.1 Framing
the Problem
1.4.2 Results
Chapter 2: BACKGROUND
AND RELATED WORK
2.1 Conducting
and Interpretation Pedagogy
2.2 Previous Conductor
Study
2.3 Theories of
Expression and Emotion in Music
2.3.1 Leonard
Bernstein
2.3.2 Manfred Clynes
2.3.3 Expression
"Rules" Research
2.4 Theoretical frameworks
for mappings between gestures and music
2.4.1 David Efron
2.4.2 Joel Ryan
2.4.3 Teresa Marrin
2.5 Interactive systems
for conductors and conductor-like gestures
2.5.1 Hyperinstruments
2.5.2 Radio Baton
2.5.3 The Virtual
Orchestra
2.5.4 A MultiModal
Conducting Simulator
2.5.5 The Conductor
Follower of the MIT Electronic Music Studio
2.5.6 Gesture Recognition
and Computer Vision
2.6 Wearable interfaces
for real-time interactive music
2.6.1 BodySynth
2.6.2 BioMuse
2.6.3 Lady’s Glove
2.6.4 DancingShoes
2.6.5 Miburi
2.6.6 Benoit Maubrey’s
Electro-Acoustic Clothing
2.6.7 The Musical
Jacket
2.6.8 Chris Janney’s
HeartBeat
2.6.9 Others
Chapter 3: THE
CONDUCTOR’S JACKET SYSTEM
3.1 Background
3.1.1 Preliminary
Investigations
3.2 System Design
3.3 Implementation
Details and Issues
3.3.1 Design Criteria
3.3.2 Measures
of expression that were not used
3.3.3 Design/Implementation
Problems
3.4 Data Collection
Experiments
3.5 Formatting,
Timing, Graphing and Filtering the Data
3.5.1 Non-real-time
filters in Labview
3.5.2 Non-real-time
filters in Matlab
3.5.3 General Issues
with this Data Set
Chapter 4: VISUAL
ANALYSIS OF CONDUCTOR DATA
4.1 Interpretive
Feature Identification
4.1.1
Use of the left hand for expressive variation
4.1.2
The flatlining effect
4.1.3
The direct, one-to-one correlation between muscle tension and dynamic intensity
4.1.4
Predictive indications
4.1.5
Repetitive signals minimized until new information appears
4.1.6
Treatment of information-bearing vs. non-information bearing gestures
4.1.7
Frequency of unnecessary actions decreases with experience
4.1.8
Clarity of signal during slow, legato passages correlates with experience
4.1.9
Division of labor between biceps, triceps, and forearm
4.1.10
Rate encoding
4.1.11
The link between respiration and phrasing
4.1.12
Large GSR peaks at the beginning of every piece
4.1.13
GSR baseline variance as a strong indicator of experience
4.1.14
Temperature baselines
4.2 Other features
for future treatment
Chapter 5: HYPOTHESES
OF EXPRESSION
5.1 Interpretation
of results from analysis
5.2 Hypotheses
of Expression
5.2.1 Efficiency
5.2.2 Intentionality
5.2.3 Polyphony
5.2.4 Signal-to-Noise
Ratio of Expertise
5.2.5 Tri-Phasic
Structure of Communicative Gestures
5.2.6 Bi-Phasic
Pulse Structure
5.2.7 Evolution
of Conducting Gestures
5.2.8 Unidirectional
Rate Sensitivity
5.2.9 Musical Flow
State
5.3 What is Expression?
Chapter 6: THE
GESTURE CONSTRUCTION
6.1 System Architecture
6.1.1 Jacket Design
6.1.2 System Design
6.2 Real-time Signal
Processing
6.3 Code Interfacing
between Filters and Mapping Structures
6.4 C++ Mapping
Algorithms
6.4.1 Musical
Algorithms of the Gesture Construction
6.5 Resulting pieces
6.5.1 Etude 1:
Tuning
6.5.2 Etude 2:
One-to-One Relationship
6.5.3 Etude 3:
Beats, Cutoffs, and Crescendo/Diminuendo on Sustain
6.5.4 Bach’s Toccata
and Fugue in D minor
6.5.5 Song for
the End
6.6 Public Performances
and Demonstrations
Chapter 7: EVALUATION
AND FUTURE WORK
7.1 Evaluation
7.2 Biggest Lessons
7.3 Design Wins
7.4 Future Work
7.4.1 Analytical
Improvements
7.4.2 Hardware
Improvements
7.4.3 Software
Improvements
7.4.4 Artistic
and Theoretical Improvements
Chapter 8: CONCLUSIONS
8.1 Design Issues
for Sensor Instruments
8.1.1 The Disembodiment
Problem
8.1.2 Mimesis
8.1.3 Traditional
vs. Digital Instruments
8.1.4 Distinctions
between Musical and Physical Gesture
8.2 A Framework for
Future Research
8.2.1 Extensions
to the Conductor’s Jacket Project
8.2.2 Implications
from the Conductor’s Jacket for Other Work
8.3 The Future of
Musical Performances
8.3.1 The Need
for New Live Concerts
8.3.2 Possibilities
for Great Art with Sensors
8.4 Coda
REFERENCES
Appendix:
Information on the Conductor’s Jacket datafiles