A Project in Music Perception:

Linking Compositional Techniques with Perceived Emotion Categories

Motivation

What is it about classical music that, when it is well performed, makes the piece or the phrase sound musical. Suppose that an evoked emotional response is necessary for this to occur. What in the music evokes this response? Are there typical patterns that composers use to create these effects? If so, perhaps a computer can recognize these patterns and use them to help classify the passage into a specific emotion category.

Problem

This project explores the relationship between specific compositional techniques and perceived emotion categories.

Approach

To investigate this relationship, one must first choose specific emotion categories. Based off of Eckman's emotion categories, I used the following three categories in this experiment:

Fear
Joy
Sorrow

Next, I composed and arranged three melodies for each category and recorded them in a MIDI format. Each melody was about ten seconds long and written using the following techniques:

Melodic contours
Dynamics
Tempo
Key signature

For the "Joy" category, I chose to use the famous melody from Beethoven's "Ode to Joy" as one example, mainly to test if people have preconceived ideas when they hear familiar music. The other two examples were written with ascending and descending lines, moderately loud dynamics, quick tempos, and major keys. "Sorrow's" examples had descending lines, varying dynamics with an emphasis on decays, slow tempos, and both major and minor key signatures. Fear's examples had both ascending and descending lines, undulating dynamics with both swells and decays, mainly fast tempos (one example had a slow tempo), and both major and minor key signatures.

Now, these nine short melodies were placed on a laptop computer and played for the subjects. Before the experiment began, each subject was given a form ( .ps, .pdf) and instructed not to speak or ask questions during the experiment. They were also told that the example would be announced before it was played ("This is example number one") so that there would be no ambiguity between the end of one melody and the beginning of another, and that they were to circle which category they thought the example best fit.

Once the subjects understood the instructions, the experiment began. Each example was played in a predetermined order, as follows (the following files are in MIDI format):

The subjects' forms were collected once the experiment concluded. After the forms were collected, each subject was asked to provide some feedback regarding the experiment.

Results

Fifteen subjects were tested after a church service one Sunday afternoon. All subjects were volunteers and all subjects were informed that the test was about music perception. The table below depicts their responses. Note, in trials two, three, and eight, one subject did not circle a response, which is why there are only fourteen total responses in these trials.

Fear

Joy

Sorrow

None of the Above

Bar graphs

% correct match

Joy 1

0

12

0

3

.ps, .pdf

80

Joy 2

0

14

0

1

.ps, .pdf

93

Joy 3

0

10

0

5

.ps , .pdf

67

Sorrow 1

0

0

11

4

.ps , .pdf

73

Sorrow 2

1

1

6

6

.ps , .pdf

43

Sorrow 3

0

0

11

4

.ps , .pdf

73

Fear 1

8

4

2

1

.ps , .pdf

53

Fear 2

8

2

3

1

.ps , .pdf

57

Fear 3

4

4

0

6

.ps , .pdf

28.5

The table below shows the percentage of subjects that classified the music example into the given emotion category.

Fear

Joy

Sorrow

None of the Above

Joy 1

0%

80%

0%

20%

Joy 2

0%

93%

0%

7%

Joy 3

0%

67%

0%

33%

Sorrow 1

0%

0%

73%

27%

Sorrow 2

7%

7%

43%

43%

Sorrow 3

0%

0%

73%

27%

Fear 1

53%

27%

13%

7%

Fear 2

57%

14%

21%

7%

Fear 3

28.5%

28.5%

0%

43%

Conclusion

A correlation appears to exist between compositional techniques and perceived emotion categories. Joy seemed to be the category subjects had the easiest time identifying correctly but had a non-negligible false positive classification with the Fear examples. This means people associated this particular mixture of compositional techniques with Joy when they were intended to but confused these techniques with Fear's examples.

Now there are two possible explanations for this. One, the subjects really did associate the compositional techniques with the Joy category. Or two, the subjects had an easier time categorizing the Joy examples because the other two categories were significantly different from Joy. To further test this result, one should create and test examples that belong to emotion categories similar to Joy.

But just as people could easily categorize Joy, people had difficulty categorizing Fear and Sorrow. The subjects did not classify fear as sorrow, or vice versa, which is rather encouraging. Rather when subjects misclassified examples, they perceived the Sorrow examples as not belonging to any of the categories and the Fear examples as being Joy examples.

This is not encouraging. Possibly this disproves my theory, showing that there isn't a direct, easy correlation between compositional techniques and perceived emotion categories. Another explanation is that the written melodies were not sufficient to portray these categories and better examples are needed. I believe that better examples should be created and tested before concluding the aforementioned correlation does not exist.

Finally, there is evidence that people have preconceived perceptions about familiar music. This was shown in the selection "Joy 2," which was really the melody from Beethoven's "Ode to Joy." Nearly all the subjects categorized this example as Joy, even though the specific example they heard had no expressive content. Of the techniques listed above, it used a moderate tempo, constant dynamics, major key, and varying melodic line, similar to the techniques used in all the Joy examples. To further test this, one can explore other familiar passages as well as re-arrange these familiar passages and attempt to convey a completely different emotion.

Overall, the results of this project do not allow one to conclusively claim there is a correlation between compositional techniques and perceived emotion categories. It appears that recognizing Joy patterns is promising, yet the results are not strong enough to be conclusive. In this test, humans had trouble correctly classifying Fear and Sorrow examples, so with these results one cannot decisively conclude much.

To further improve this project, one can create new examples using additional techniques, such as instrumentation, harmonic support, and underlying chord progressions. One can also test examples that belong to an emotion category similar to Joy. And finally, one can test other examples and see if people have preconceived perceptions associated with familiar music.

email comments to:
mtdeplon@mit.edu