1.2 Overview

While the vision I’m presenting is a fanciful, idealistic one, it suggests one future evolution for the new forms that are currently developing in the space between classical performing arts and interactive technologies. It’s a hybrid form, a carnival atmosphere that results from the gathering together of many disparate elements. No longer the rarefied, quiet atmosphere of concert hall; this is a festive embrace between art and the general public. And there are already several precedents for it in today’s popular culture, including productions like Riverdance, Blue Man Group, the Brain Opera, the TOUCH Festival, and Monsters of Grace.

The new technologies that today attract the public’s imagination may be incorporated into the performing arts in a variety of ways. The most promising methods will encourage people to gather together in shared spaces and cause them to interact meaningfully with each other. My vision proposes that we should not try to carefully preserve any ‘pure’ forms; this kind of classical sentimentalism and reverence for a perfect high art will only continue to make museums out of our concert halls. However, that doesn’t mean that we should throw away all our forms and start designing art forms from scratch; this has been tried in the computer music community and it has not succeeded on a large scale. Rather, I suggest that we open up and transform our performance models with a fun sense of abandon and experimentation. Without meaning to be unrealistic and hyperbolic, I have a deep hope for the possibility that new technologies might resuscitate the musical culture that I cherish so deeply and which, I fear, is in the process of being lost.

1.2.1 The slow asphyxiation of classical music

As the twentieth century wanes, American performing arts organizations are encountering tremendous uncertainty. Everyone knows that the traditional forms, particularly symphony orchestra, chamber music and ballet, are not connecting with the majority of society. On the other hand, by their very definitions they cannot change enough to meet the interests of the general public. Their repertoires and styles, which were defined in previous centuries, lack the strong rhythms and amplified sounds that modern audiences prefer. Certain special events have been designed to reverse the trends, such as crossover Pops concerts and month-long Nutcracker series, but these only provide temporary budgetary relief.

"In the United States, a typical big-city orchestra gave one hundred concerts in 1946 and broke even. Twenty years later, it played a hundred and fifty dates and lost forty thousand dollars. In 1991, it put on two hundred concerts and shed seven hundred and thirty-five thousand dollars. At this rate, the orchestra would not exist by the end of the century. ‘The five biggest orchestras are protected by large endowments, but many of our municipal orchestras are simply not going to survive,’ warned Deborah Borda, executive director of the New York Philharmonic." Given the decreasing support for classical music forms, something should be created to bolster and extend (not replace) their function in society. Ideally, this new form would retain the flavor and essence (and many of the instruments) of classical music while modernizing it enough to make it enticing to current audiences. This new form would bridge the growing divide between traditional performing arts and popular culture and sustain a place in society for enriching entertainment. It might also ensure the continuance of live performances, which are inherently valuable.

 

As the mass production of the piano transformed the American musical experience during the nineteenth century by introducing recreational performance into the homes of middle-class families, it is possible that a new means for creating music might attract a large segment of the technologically savvy populace of the twenty-first century. One way in which this might happen would be through a new class of instruments. Not only could novel instruments captivate the imaginations of vast masses of amateurs, but they might also change and update our perceptions of art music by inspiring performing artists to develop new forms for the stage that preserve the role of music for its own sake in our culture. This collection of new instruments could leverage available technologies to do more than just play notes; they could generate and vary complex patterns, perform higher-level functions like conducting, and even generate graphics, lighting, and special effects. But in order for the performing arts community to consider and embrace these possibilities, it must have instruments that are at the very least as expressive as the traditional, mechanical ones have been.

1.2.2 Why our new instruments are not expressive enough

"One of the most telling, and annoying, characteristics of performance by computers is precisely its ‘mechanical’ nature – the nuance and expression performed instinctively by human players after many years of intensive practice is completely squeezed out of a quantized sequencer track. If more of the musical sensibilities informing human expressive performance can be imparted to computer programs, the range of contexts into which they can usefully be inserted will grow markedly." During the past twenty years there have been tremendous innovations in the development of interfaces and methods for performing live music with computers. However, most of these systems, with a few notable exceptions, have not been widely adopted by performing musicians. There are good reasons why most musicians have not yet traded in their guitars, violins, or conducting batons for new technologies. The most important is that these technologies do not yet convey the most deeply meaningful aspects of human expression. That is, they do not capture and communicate the significant and emotional aspects of the gestures that are used to control them. This is a very deep and complex problem, and it’s not clear that it can ever be solved; some people say that computer technology can never achieve the fine sound and responsiveness of a great violin.

However, before considering the most difficult cases, such as replacing the violin (which is not an attractive idea anyway), there are some much more basic technical issues that must be addressed in order for us to improve upon what we already have. First of all, many current interfaces do not sample their input data fast enough or with enough degrees of freedom to match the speed and complexity of human movement. Secondly, the nature of the sensing environment often inappropriately constrains the range and style of movement that the performer can make. Thirdly, the software that maps the inputs to musical outputs is not powerful enough to respond appropriately to the structure, quality, and character in the gestures. There tend to be simplistic assumptions about what level of complexity and analysis of the data is sufficient. For example, one computer musician has written that "only the current value of the continuous control is of interest – there is no reason to try to characterize the shape of change over time." This is short sighted -- ultimately, all musical systems could benefit from prior knowledge about how their parameters vary dynamically and what the variations mean. Without this knowledge, interactive systems cannot anticipate and respond appropriately to their control inputs.

And it is not just the modeling of the input data that needs improvement; musical outputs tend to be either too simple or too confusing for the audience. The hardest thing to get right with electronic instruments, according to Joel Ryan, is the ‘shape of the response’:

"Often, the controller feels right, but the shape of the response does not fit your musical idea. This is a huge area..." The musical response from an instrument should, of course, always fit the musical idea that generates it, but designing a system where that is always true could take an infinitely long time. So one of the first challenges in designing new musical systems is to carefully choose which issues to tackle. The tools of our medium have not yet been perfected or standardized, but we have to begin somewhere.

1.2.3 What needs to improve for interactive music to become an art form

"[if an instrument] makes the player happy by providing some direct opportunity for expression, then it has a chance as a musical tool." In order for new computer technologies to replace, enhance, or transform the capabilities of traditional instruments, they need to convey affect, interpretation, and skill. In order for skilled performers to express themselves artfully, they require musical instruments that are not only sensitive to subtle variations in input but can be used to control multiple modulation streams in real-time; these instruments must also be repeatable and deterministic in their output. What often happens with the replacement of traditional mechanical instruments with sensor-based interfaces is that the many dimensions in the input stream are reduced in the transduction process, and thereby effectively projected down to minimal axes of control. While the reduction of dimensionality in an input stream is often good for automatic recognition and other engineering tasks, it is not always ideal for music, where subtlety in the ‘microstructural variation’ provides much of the interesting content for the performer and the listener. So the first thing that has to improve is that the full dimensionality of the input gestures must be acquired by sensors, and the second thing is that this data must be acquired at high rates, so as to preserve the information without aliasing and other subsampling pathologies.

A second big problem with many sensor-based systems is the large disconnect between the way the gesture looks and the way the music sounds. This is because of brittle, unnatural, or overly constrained mappings between gesture and sound, and these make it difficult for an audience to understand or be able to respond to performance. These instruments can quickly become frustrating for people on both sides of the stage. Bert Bongers once heard Pierre Boulez say that computers will never be able to solve this problem:

"Pierre Boulez…mentioned that there always will be a difference between computer music and traditional instrumental music, because musicians can perform (or interpret) gestures, and computers cannot. We think that this is the previous generation’s view on computers. One of the most important goals for us is to make gestural music performances with computers." I think that we will have to find ways of making stirring gestural music performances in order for interactive music to become an art form. The inherent problems in sensing and gesture recognition must be solved before sensor-based instruments will be widely adopted. Until then, interactive music will remain a nascent form for the stage.

1.2.4 Instruments for Free Gesture

I’m especially interested in a subset of performance interfaces: systems for free gesture. These are instruments that sense the motion of the body without changing or constraining it physically. The Theremin was an early example, although its initial versions, due to the nature of their analog internals, had fixed mappings between hand position, pitch, and volume. From what I have seen, computer music created with free gestures began to be possible around 1989, with the integration of real-time MIDI performance systems and novel sensors. The Radio Drum was an early example, developed at CCRMA in 1989, followed by the BioMuse system in 1992. Soon afterwards, the MIT Media Lab developed the noncontact, field-sensing Sensor Chair in 1994.

These free gesture instruments are generally most appropriate for gestures of the larger limbs, such as the torso and arms, as opposed to most traditional instruments, which make use of the dexterity and nimbleness of the fingers. Fingers are ideal for quick and accurate triggering of individual events, whereas the limbs are useful for larger-scale gestures of shaping and coordinating.
 

 Chapter 1.3