Final Project

Kwan Hong Lee
kwan@media.mit.edu
May 2001
MAS.630 Affective Computing Seminar
Final Project

People are using more and more mobile devices to access information and mobile phones have become a common place in our daily communication. In this project, I am trying to find out if there is any correlation between frustration and hand pressure on the mobile devices while people are communicating using voice through a mobile phone like device. The communication channel is IP (Internet Protocol) based audio channel which introduces delays and packet losses.

Compaq iPaq was used as the handheld platform for mobile voice communication. It runs Linux and connects to the network through 802.11 wireless LAN card. People can have a full duplex audio communication with the iPaq. The Audio Proxy Server takes care of audio forwarding and dropping packets in order to simulate a lossy voice channel. A parameter can be adjusted in the Audio Proxy Server in order to increase or decrease packet loss. It also saves audio data to the file during data collection. The audio from one iPaq goes through the Audio Proxy Server and depending on the Audio Proxy Server voice channel settings, it may drop some audio or directly forward it.

The Sensor Server records the data received from the IRX board that is collected from the force resistive sensor around the iPaq.

The Audio Proxy Server was implemented in C. The Sensor Server was implemented in Java. The clients on the iPaqs were also implemented in C. The following figure illustrates how the system is setup to collect data.

Figure 1. System Architecture

The following figures illustrate how the force resistive sensor has been placed around the iPaq in order to measure pressure from the hands while user communicates.

Figure 2. The strip around the iPaq is the force resistive sensor.

Figure 3. A side view of the iPaq and how sensor is attached around the iPaq.

Figure 4. How a user would use to talk with other people.

Figure 5. Shows how the sensor is connected to the IRX board.

Pairs of users play tic-tac-toe against each other while they are placed in two separate rooms with their iPaqs. They are made to communicate the coordinates of where they are placing their marks (circle or x) to the other party through the voice channel established by the iPaq. The users have to play the game several times and the channel characteristics are modified every 5 minutes. Currently one channel is a good quality channel without any packet loss with less than 500 ms of delay. The other channel is a lossy channel which drops about 33% of the packets that are transmitted. Pressure data is collected as they are playing the game, communicating through these two types of voice channels.

Data collected could be graphed using Matlab as following where the x-axis is the time in seconds. Different parts of the audio can be played by Matlab during data analysis.

Mobile devices would always be attached to user's hands when they are used. If we can conclude any kind of affective state of the user through the pressure detected when the user is speaking or using the device, this information can be used to design and build new affective interfaces that can be used as a secondary channel to communicate with other users. For example, when user is very frustrated, the user's audio that is heard on othe other end could be amplified. The device itself could adjust to itself also in order to reduce tension or frustration on the user. More data needs to be collected and analyzed and different experiments need to be carried out in order affirm whether there is any useful information in the hand pressure of the user. Also, current experiments have been all carried out while stationary due to the sensor data being collected through a serial port connected to a PC. However, more interesting data can be collected if the sensor data are collected on the mobile device as the user is moving around.