A wearable computer with bio-metric sensors, in constant contact with its user, offers an opportunity to obtain unprecedented amounts of physical and physiological data about a single user. This advantage also presents many challenges. To use the data effectively in learning algorithms, and to interpret the results, the data must be accurately labeled. We are in the early stages of developing ways to record the physical and psychological events of the wearer's life without interrupting their normal activities. Currently, an automatically time stamped notes file allows the user to annotate data from the sensor screen. Adding an audio recording device or video camera would improve the users ability to annotate data after the fact, but would make the annotation cumbersome. We propose to add two sensors to the system to improve annotation: a foot pressure sensor to indicate ambulatory movement and an audio detector that would detect if the wearer was making any vocal sounds (speech, sneezing, etc.) The latter could record and analyze the audio if privacy was not an issue, or if privacy was an issue, it would not record voice, but only use features of the sound signal to prompt the user for annotations, e.g. computer detects the presence of speech and prompts the user ``Did you just talk to someone?'' to which the wearer could respond ``Yes, my boss.''
It is important to annotate the physical activities of the user because such physical activities can overwhelm the physiological effect of psychological events. Even under ``ideal'' laboratory conditions, with a resting subject responding to a directed stimulus, scientists have yet to find reliably discernible features of physiological signals for affect detection. In a natural environment where weather, diet, and physical activity are present, the patterns corresponding to particular affective states are at a much greater risk of being obscured. We must identify and account for these confounding variables.
In psychological studies by Lang [LGea93] and Winton, Putnam and Krauss [WPK84], subjects were monitored while looking at a series of photographs which were supposed to elicit an emotional response. These studies showed that heart rate variability was an indication of valence (whether or not the person found the photograph pleasant) and that the ratio of skin conductance to heart rate variability was an indicator of arousal. However, the greatest changes in skin conductance and heart rate in their study were 0.6 micro-Siemens, and 8 beats per minute respectively, in the ten second period after viewing the slides. These results are significant for a resting subject under controlled laboratory conditions, but we have found that such results can be overwhelmed by physical activity in an ambulatory subject.
We conducted experiments in the laboratory to show that the physical activity of ambulatory subjects is an important confounding variable to detecting emotional responses from bio-sensors. Five different subjects were run through the ambulatory bio-sensing task, the results of which are reported in Table 1. The experimental protocol involved subjects wearing the bio-sensors from the FlexComp system while performing a series of ambulatory tasks. The subjects were first shown how to attach the bio-sensors. All subjects wore a respiration sensor, a BVP sensor, and a GSR sensor on the hand and another one on the arch of the foot. They were then asked to perform the following tasks: sit in a chair for one minute, resting quietly, stand up and sit down twice, walk around the room for one minute, sit in a chair normally for two minutes, stand up and walk around the room for one minute, sit normally for another minute, jog in place for one minute, then sit for a minute and finally the subjects were asked to cough as if they had a cold twice. An experimenter was in the room with the subjects at all times during the experiment to instruct the subjects to perform the activities. Subjects sometimes had questions during the experiment, and talking was noted in the experimental record. The rest times between the activities sometimes varied, due to questions, so the beginning of the tasks were marked by the experimenter with a specially designed mouse equipped with a sensor which would make a spike in the recording data when depressed.
The data for this experiment was saved at 16 samples per second using the FlexComp with a resolution of 0.01 micro-Siemens. Changes in the skin conductivity (GSR) were measured from a resting state to an active state. The changes were calculated as the difference between the baseline measurement, taken at the time each task was begun, and the first significant local maximum. The heuristic used for determining the significant local maxima is that the difference between the local maximum and the preceding local minimum is greater than 0.5 micro-Siemens. If there was no significant maximum, then the difference in the signal between the beginning and end of the activity was recorded. The heart rate was calculated from the peak-to-peak intervals and stored at 16 samples per second. To calculate the change in heart rate the average of the heart rate for 10 seconds before the task was subtracted from the average for the heart rate ten seconds following the task. An example of data from the experiments is shown in Figure 4.
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
It is useful to have an accurate recording of the physical activity of the wearer and to understand how this affects the user's physiology and expression of affect. From this information we can attempt to decouple the psychological effects from the physical. For example, although the heart rate of the wearer changes drastically with inhalation and exhalation, the heart rate and respiration signals are highly correlated as shown in Figure 5. By recording respiration, we can state which heart rate changes are effected by this source and which changes must be attributed to other sources.
 |
 |
The selection of which signals to measure is another challenge for an affective wearable. There is evidence to believe that physiological signals including accelerated heart rate [Lev92], raised blood pressure, increased skin conductivity, and constriction of the peripheral blood vessels [Hel78] are associated with stress in humans and heart rate variability [WPK84], pupillary dilation and right and left hemispheric activation in the brain [Dav93] are associated with like/dislike. These states are particularly useful ones for interactions involving humans and computers, especially as computers try to understand and adapt better to their wearers.
We are experimenting to determine which physiological signals can be best monitored under ambulatory conditions to produce the most salient affective features, given the constraint that the sensors should be comfortable to wear. Such a system needs to be unobtrusive, lightweight, and easy to use. It should not require the user to make exceptions to her daily routine. Our goal is to create an affective wearable system that is comfortable and robust, able to extract and analyze complex features in both the time and frequency domain, and able to record annotations and context information, all for providing pleasantly useful services to its wearer.
The GSR sensor is one of the most robust wearable sensors because it is a bulk measurement and not very sensitive to exact placement of the sensor. Traditionally, this measurement is taken across the palm of the hand. However, the hand is one of the least ideal places for sensor placement. Ordinary activities such as washing your hands changes the baseline reading and repetitive motion of the fingers in activities like typing, creates a noise signal that confounds the signal. To alleviate this problem, we have explored the options of placing the electrodes on the toes, and measuring skin conductivity across the sole of the foot.
In this experiment, another five subjects were seated and startled with five white noise bursts while wearing skin conductivity sensors on three locations. The sensors were placed on the two middle sections of the first and second finger on the dominant hand, on the two middle sections of the first and second toes on the same foot, and on either end of the arch of the same foot. An electret microphone was used to record the startling tone burst as a reference. All skin conductivity data was sampled and saved at 16 samples per second with 0.01 micro-Siemens resolution. An example of the data recorded during these experiments is shown in Figure 6.
 |
 |
| ||||||||||||||||||||||||||||||||||||
The startle experiment was designed to test the correlation of the hand and foot responses under extreme audio stimuli. The correlation coefficient was used to measure similarity between the signals taken from the three locations. The results of these correlations are reported in Table 2. In the natural sciences, the correlation coefficient is used in the Pearson correlation to test significance of traits, based on the number of data points. If we consider all samples of the signal data points for this test, all the results are extremely significant with p>0.01.
Strong correlations were found between the hand and foot responses in the ambulatory study with correlation coefficients of .90, .85, .83 and .88 for the first four subjects respectively. The fifth subject inadvertently detached the hand GSR during the experiment. From these results we concluded that the skin conductivity responses of the hand and the foot to a startle signal are highly correlated. Therefore, either placement of the sensor could be used to measure the GSR response for these subjects. Only one subject, subject four, showed an unusually smooth response from the toe sensor, as if the signal had been low pass filtered, resulting in poor correlations with the other signals. This could be due to sensor placement error or to a naturally sluggish sweat response along this pathway, indicating that different individuals may have different optimal sensor sites. The blood volume pressure sensor uses photoplethysmography, a technique where an LED is used to ``look'' at the amount of blood flowing in the vessel. From this reading both the heart beat and constriction of the blood vessel can be determined. This sensor is sensitive to correct placement and motion artifacts; however, in a single-subject experiment we conducted driving through Boston at rush hour, where the wearer moved the hand with the sensors routinely to shift gears, adjust the radio, and turn the steering wheel, we found that although the signal was disturbed during motion, the signal re-stabilized after movements. Hence, we think that motion artifacts can be compensated for. Nonetheless, the driver reported that the wires felt like they were in the way.
We are interested in moving sensors off the hands to sense what people are feeling during a variety of consumer tasks, such as diapering an infant. This is a particularly challenging task as it is important not to drag the wires through the dirty diaper, or let the infant grab them and pull them, or put them in his mouth. In other words, we are investigating options for moving the sensors off the hands.
Respiration is the easiest sensor to wear and tends to be the most immune from motion artifacts of the sensors investigated here. It uses a constrained Hall effect sensor to measure the expansion and contraction of the chest cavity. The confounding variables of this measurement mostly audio (e.g. talking, sneezing, coughing and sighing) all of which we hope to compensate for with the audio detector.
We have been investigating the optimal placement for the EMG sensor to detect stress. The greatest challenge of EMG placement is to find a muscle to which the EMG electrode pad will stick well and which is not involved in muscle motion. To solve the problem of affixing the sensors we are currently considering the use of new electrodes and glue designed by the Boston University Neuro-muscular Research Center which have been demonstrated to stay affixed through rigorous physical activity. To solve the problem of motion we may need to create sensors which detect motion and use this to correct the straight EMG reading. We are also developing alternative sensors for muscular movement using AMP piezo tape. These have the advantage of not requiring glues or gels.
 |
 |
 |
We are in the process of creating a wireless interface for all of these sensors. There are two possibilities currently being investigated for wireless transmission of data from the sensors: infrared transmission, and constrained short wave FM transmission using the personal area network (PAN). We have currently designed working prototypes of analog sensors such as the skin conductance sensor shown in Figure 9 and the blood volume pressure sensor shown in Figure 10. These sensors are sampled using an A/D converter chip. The digital bio-metric signal can then be transmitted either by the iRx PIC configuration or the PAN-CBPS system.
The PIC iRx infrared system allows line-of-sight transmission of data over short distances and does not interfere with electrical signals. It is ideally used in situations where there is little distance or obstruction between the sensing system and the target. Examples of situations in which this might be useful include transmission of data between two people who are in close physical proximity as shown in Figure 8 or between a person and a desktop computer. For example, a shoe sensor with an embedded iRx chip could transmit data to a receiver beneath the workstation.
The PAN system which uses short wave FM to transmit data beneath the surface of the skin [Zim96] proves to be more useful in situations where the receiver and the sensor are not in direct line of site as would be the case for the bio-sensors and a wearable computer. In the system shown in Figure 9, a transceiver which attaches to the serial port of a computer interrogates a remote skin conductance sensor. The system attaches to the person's body using conductive rubber electrodes which require no gel. The wearer's body acts like a bus for the digital data [PRG +97]. By using a transceiver controlled interrogation scheme, this bus can be shared by up to fifteen sensors, with a shared rate of up to 2400 baud.