For example, results from a study on Affective Wearables by Healey and Picard yielded promising physiological data containing salient features of stress. They were able to find five physiological correlates to stressful states, including increasing slope in skin conductivity, average heart rate, average respiration rate, blood pressure, and constriction of the peripheral blood vessels. While these measures were aversely affected by motion artifact, they were still significant, because they nonetheless led to 90-100% accuracy rates in distinguishing the high arousal state of anger from a class of low arousal states, including love and reverence.43 Earlier, Ward Winton, Lois Putnam and Robert Krauss found in studies where subjects viewed emotion-eliciting images that an increase in heart rate indicated the valence of a reaction, and that the skin conductance divided by the heart rate gave a good measure of arousal.44 Arousal and valence form the two axes that many researchers use to define the state-space of emotion. Internal affective states can be plotted on a two-dimensional graph using just these two coordinates.
Physiological correlations with arousal, valence, and affect seemed extremely promising for my interests, since music has often been described as a medium for emotional communication. The scope for possible research seemed very broad. One area that suggested further investigation was the ‘contagion effect,’ which was suggested to me by Professor Picard. This psychological phenomenon, which has been shown to exist for stress, is the transmission of internal states from one human to another. To the extent that people claim to be ‘moved’ by a musical performance, it might be said that they have been contagiously affected by it.
In the case of an emotionally moving performance by a symphony orchestra, it might be said that the primary contagious agent is the composition, whereas the second agent is the conductor.45 In the transmission of the contagion, the conductor’s signals are transduced through the orchestra. She communicates to the players in the orchestra by generating visible and perceivable signals, including gesture, speech, and facial expression. While this contagious relationship between conductor, musicians, and audience has not been empirically shown to be true, I have heard numerous anecdotal stories to support it. For example, an anonymous person associated with the American Symphony Orchestra League once described to me what he saw as a clear example of affective contagion in an orchestra. He had investigated one orchestra where, during the course of a few years, nearly every member of the first violin section had contracted a debilitating case of tendonitis. After observing several rehearsals and performances, he realized that the conductor also had painful tendonitis, to such an extent that he needed to ice down his arm after conducting. This person suggested to me that the conductor’s internal stress was silently and effectively being communicated to the musicians through his tense body language and physical movements. The ASOL representative told this story in the context of encouraging future conductors to keep in mind that they have a great responsibility not only to convey musical ideas, but to refrain from conveying any unhealthy conditions directly to the members of orchestra.
My approach to the issues raised by the Digital Baton and the general questions of expression and emotion has been to develop my own unique synthesis-by-analysis method. That is, I decided to go into the ‘field’ to collect data on real musicians and then feed what I learned back into a new real-time music system. I believed that a quantitative study would yield information that could not be acquired by inspection, and would ultimately enable the building of a better, more musical system. The approach that I have taken has been specifically designed to achieve meaningful answers about one of the most mysterious of human behaviors. In the process I have attempted to remain respectful of the complexity of the subject, while also choosing practical and achievable goals. I describe my methods below in detail.
Given that questions of musical meaning and expression tend to be difficult to define and constrain, I posed quantitative instead of artistic questions. I decided to continue to focus on the performance parameters of a trained musician, and chose to stay with conducting as the primary musical activity to study. Instead of forming any concrete initial hypotheses, I first gathered data to see what it would yield. After several initial pilot tests and research46, I set about building a wearable sensor system with which to measure expert conductors47 and followed that with a series of data collection sessions for six conductors in real-life situations. It soon became obvious that the physiological and motion data that I was collecting contained clearly repeatable patterns and trends. After these preliminary observations, I made some revisions to the data collection system, finished the data collection events, and then launched a much deeper investigation of the data.
This thesis project was designed and carried out in five interwoven stages. The first task was to model the body as a signal generator and design a system to optimally sense the most important signals. This involved extended investigations into physiological sensors and practical data-gathering methods, as well as constructing several versions of the interface and sensor hardware and collecting data from numerous subjects. Secondly, six local orchestra conductors with a wide a range of expertises and styles agreed to wear a personalized jacket and let us collect data during their rehearsals and performances. Thirdly, I designed and performed a visual analysis to extract the most promising features from the data and explored useful filtering, segmentation, and recognition algorithms for exposing the underlying structural detail in those features. Fourthly, a more in-depth interpretation project was done to explain the stronger underlying phenomena in the data; this consisted of interpreting the results of the analysis phase and making decisions about which features are most salient and meaningful. Finally, for the last stage, I built a instrument to recognize musical features in real-time and synthesize music that reflected their structure and character; this system has two complete pieces as well as a set of technical ‘etudes,’ and has been successfully demonstrated and performed publicly.
The four phases of this work have consistently overlapped each other – in the best cases, analyses have been directly followed by syntheses in etude mappings. The focus from the beginning has been to discover the significant and meaningful features in different gestures and find ways to make the music reflect that meaning; the overriding goal of the entire project has been to build a much more expressive and responsive gestural interface.
The tangible, final results of the Conductor’s Jacket project include:
This thesis presents novel methods for sensing, analyzing,
interpreting, and accompanying expressive gestures in real-time with flexible
and responsive music. In the following chapter I present numerous theoretical
and practical precedents for my work. Chapter 3 describes the system design
and implementation of the Conductor’s Jacket sensing and data collection
hardware. Chapter 4 presents the visual analysis of the conductor data,
including a full account of fourteen expressive features and descriptions
of twenty-one others. In Chapter 5 I discuss the implications of my analytical
results and present my theories of expression and meaning. Chapter 6 details
the system architecture and details of the Gesture Construction,
the interactive music software system I built. In Chapter 7 I evaluate
my results and discuss the implications, and in Chapter 8 I conclude with
some thoughts and ideas for future work.