Changing Musical Emotion through Score and Performance with a Computational Rule System - Ph.D.

« August 2008
Su	Mo	Tu	We	Th	Fr	Sa
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31

Steven @ Wed, 2006-01-25 19:47

Abstract

This page discusses CMERS - a Computational Music Emotion Rule System for the control of perceived musical emotions, that modifies a musical work at the levels of score and performance in real-time. I researched, designed, programmed, and tested CMERS, which handles all modifications to the musical work. CMERS achieves a change in perceived musical emotion through the application of music-emotion rules; these rules quantify the empirically observed relations between musical features and specific emotions (for example, major mode ≈ happy, minor mode ≈ sad).

Employing a 2-dimensional representation of emotion (seen below), CMERS was shown in testing to be successful in changing the perceived emotion of all selected music works to each of the four emotion space quadrants, referred to loosely as happy, angry, sad, and tender, with a mean accuracy of 78% and a multinomial logistic regression analysis of Χ²(9) = 11183.0, p < 0.0005 (N = 20).

Download Full Abstract

Music Samples

The above music samples are produced from Beethoven’s Piano Sonata No. 20, Op. 49 No. 2 in G Major. In its unmodified state, this work is generally described by listeners as a happy upbeat work. Four emotionally modified versions have been produced by CMERS: sad, angry, tender, and happy (happier than original). To maximise the impact, and to best highlight the differences between tracks, try listening to them in order. Question: can you tell which musical features were modified?

All five samples have had expressive performance rules applied; this process attempts to make the expressionless computer MIDI file sound like a human performance, mimicking the subtle rubato and dynamic modulation through sophisticated score analysis. Question: could you tell that the unmodified version was generated by a computer? In user testing, very few participants were able to.

Browse all music samples

Computational Music-Emotion Rule System

At the highest level, CMERS can be viewed as a filter-based system for real-time MIDI modification (see Figure 1). At start up the MIDI file of the music to be modified is input, along with a small amount of additional mark-up (expressive performance markings from the score which MIDI does not support). Prior to runtime, a series of pre-processing operations are carried out. These include the appliaction expressive performance rules (making an expressionless computer MIDI file sound like a human performance), and the calculation of some music-emotion rule components.

When instructed by the user, CMERS begins to play the audio. The music data passes through a series of real-time filters, where each filter modifies a particular aspect of the music, for example the loudness. Individual filters are aggregated into Filter Control Sets, which correspond to the four quadrants of the 2DES representation. The user can then modify the emotion of the music in real-time, with negligble latency, through the graphical user interface; (similar to Figure 2 below).

A Computational Music-Emotion Rule System operating at the levels of score and performance, with automated expressive performance features, provides researchers with a powerful tool for exploring emotional relationships within music. CMERS ability to isolate individual structural, performance, and expressive performance elements, provides researchers with a precise and extensible platform from which to examine the contributions and conflating behaviour of specific features to musical emotion. CMERS real-time emotion modification capability, automated expressive performance, and simple emotion message format, allows for its use as a computer gaming music-emotion engine, in addition to a multitude of other real-time application environments.

Figure 1. High-level CMERS Architecture

Music-Emotion Rules

For over a century psychologists have studied the relationships between specific musical features and their corresponding emotions. One well known example in the Western tradition is the mode’s strong association with valence, where major mode ≈ happy and upbeat, and minor mode ≈ sad or agitated. Typically, these studies reported the musical features present (e.g., staccato articulation, major mode, key of G, moderate loudness) and emotion of the music, as reported by listeners.

With over 150 empirical studies having been conducted, strong statistical trends can extracted from this data. Consequently, empirical studies of emotion in music constitute the most practical resource in the development of a rule-based computational system for controlling musical emotions.

In CMERS, rule types are grouped by emotion. This grouping allows multiple rules to be applied simultaneously to bring about a change in emotion. For example, when the user selects "sad", the following rule types are invoked: mode minor, articulation legato, loudness soft, and many others.

Emotion Concepts

The dissertation makes an important distinction between two types of emotion: perceived, and induced. Perceived emotion is the emotion you think a person or stimuli is expressing/feeling, while induced emotion is that felt by you after viewing the stimulus. For example, when viewing a photograph of an angry person, or someone expressing anger, you perceive the emotion of the person to be anger, but what emotion is induced in you as a result of viewing the photograph could be a variety of emotions (e.g., fear, anger, humour). Importantly, this dissertation claims only to control perceived emotions of music, not what that music makes you feel.

In the dissertation, emotion is represented visually using a 2-dimensional form, displayed below. The vertical axis represents the energy level, or activity, of the emotion. For example, emotions low in energy are located at the bottom of the figure, and might be ‘bored’, or ‘sleepy’, while high energy emotions at the top might ‘excited’ or ‘tense’. The horizontal axis indicates whether it is a good emotion or a bad emotion. On the right side of the figure are good (or positive) emotions, while those on the left are bad (or negative) emotions. The two axes combine to cover many emotion types. For example, happy has above average energy and is considered a good emotion, so happy will be found in the top right quarter. Sad is considered to be a low energy emotion, and a negative one; so sad will be found in the lower left corner.

The four emotionally modified music samples (see above) correspond to falling within the each of the quadrants: 3 (sad), 2 (angry), 4 (tender), and 1 (happy).

Figure 2. 2-dimensional representation of emotion used by CMERS

All Audio Downloads

Beethoven’s Piano Sonata No. 20, Op. 49 No. 2 in G Major

This Beethoven sonata is typically described by listeners as being happy and upbeat. Five samples are provided here: unmodified, sad, angry, tender, and happy (happier than original). To maximise the impact, and to best highlight the differences between tracks, try listening to them in order.

Mendelssohn’s Songs without Words, Op. 19 No. 2 in A Minor

This Mendelssohn song is typically described by listeners as being sad or melancholic. Five samples are provided here: unmodified, happy, tender, angry, and sad (sadder than original). Note: the order is different for the Mendelssohn work. This was done as the unmodified work is described as sad, so the listed order has been changed to maximise the impact.

Mozart’s Piano Sonata No. 12, KV 332 in F Major

This Mozart sonata is typically described by listeners as being happy and upbeat. Five samples are provided here: unmodified, sad, angry, tender, and happy (happier than original). To maximise the impact, and to best highlight the differences between tracks, try listening to them in order.

add new comment |