Before the data could be analyzed, there was an enormous amount of processing that was required. For some of the largest files (the largest being 371 Megabytes, representing one continuous, high-data-rate scan for 45 minutes), the first requirement was to split the files up into practical chunks. For this I wrote a utility in C to split each file into a series of 1 Megabyte-sized smaller files, with all the channels properly aligned. After the data files became more manageable the next task was to graph and filter them for the visualization analysis.

3.5.1 Non-real-time filters in Labview

One of the first things I wanted to do was to graphically format each data file so that I could set it underneath its corresponding video segment and watch both the data and the video at the same time. To do this, I wrote a routine in Labview to take a previously-acquired data file, graph it in color, and have it scroll past the field of view as if it were being acquired in real-time. I got it to work by reading the ASCII-based data file into a while loop, sending it to a shift register, and adding timers into the loop to emulate the sampling rate. This worked reasonably well, but took a very long time to precompute. Filesizes larger than 1 Megabyte did not work well for this method and would frequently crash the machine. Since files under 1 Megabyte corresponded to relatively short video segments (less than one minute), I eventually abandoned this approach in favor of printing the data in long paper rolls.

I also wrote a utility in Labview to do frequency-domain analyses such as FFTs and spectrograms; these yielded interesting results but basically showed a lack of interesting structure in the frequency domain of the data I had collected.

3.5.2 Non-real-time filters in Matlab

In order to solve the glitching problem in one subject’s concert data mentioned in section 3.3.3, I wrote a filter in Matlab to reassign the misplaced samples. This solved most of the problem, although a few outliers remained (particularly in channels EMG3 and Heart rate). I wrote a second filter that removed these outliers and replaced them with the previous value in that channel. This noise had occurred in channels 3-7, corresponding to his EMG3, Respiration, Temp, GSR, and Heart Rate signals. The basic problem was that there was noise at pseudoregular intervals in each channel, and that the noise in channel x+1 was actually data from channel x. It initially appeared as if this wrapped around at the ends; that the data in channel 7 should have been in channel 3 (this ended up not to work). My filter first looked for large outliers (generally any values greater than 2 standard deviations) in all the channels, and compare their mean with the mean of the following channel to determine that they matched. I then moved the samples to their corresponding indices in the correct channel. Then, if a few outliers remain which do not have matches in the next channel, it replaces them with the previous sample in the same file. Here’s a segment of the Matlab code that accomplished this:

  Chapter 4.1.1

3.5.3 General Issues with this Data Set

In addition to processing, graphing, and noise removal, there were several issues that were particular to this data set that needed to be addressed. For example, sampling rates were extremely important, since the time-scale resolution for conducting is extremely small. Events had to be timed precisely to the millisecond in order to line them up with the video and extract meaning from them; glitches and inconsistencies in the data could quickly render it useless. There were some unresolved inconsistencies in timing mechanisms that caused difficulties in resolving a sampling rate. For example, the final subject’s data was sampled at 4 kHz according to the Labview software. However, from inspecting the data, it seemed to be around 3 kHz. I found that in many of the cases the data lined up much better with one or two extra seconds per page, which I attribute to some delay in the use of Labview in ‘continuous run’ mode. Since continuous run causes a buffer to be continuously filled and then flushed, I assume that the time taken to flush and then refill the buffer accounts for this timing inconsistency. For his data, which was sampled at 4kHz, for every 44,000 samples, instead of it lining up with 11 seconds, it seems to line up with approximately 12-13 seconds. For the first 12’16" I found that the data lined up well with 12 seconds per 44,000 samples; for the remainder I found that it lined up better with 13 seconds per 44,000 samples.

Also, differing environmental conditions contribute to differences between subjects’ baselines and noise levels. For example, the noise level for one student subject’s data was much higher than it had been during the previous week in the same room for the two other students. Also, during one professional subject’s rehearsal, the respiration band was extremely tight and therefore amplified tiny fluctuations in movement. The tightness of the respiration band seems to influence its overall range; the band was extremely tight on him during the rehearsal, but less tight at the concert. A tight band will mean a high baseline, which will also influence the resultant signal. That is, if the band starts tight, it might not expand to its fullest extent.

Another issue had to do with the printing of the data sets; depending on how the data had been sampled, the printing of it made a difference in how much structure could be obtained visually. With one professional subject’s data, which had been initially sampled at 300 Hz, when I initially printed it at a density of 5,000 samples per page, it was unintelligible. When I printed it a second time at a density of 500 samples per page, the structure in it suddenly became much clearer. The opposite was true for another subject’s data, which was not intelligible when I initially printed at 14,000 samples per page. Upon reprinting it at 44,000 samples per page, the features became instantly recognizable. Also, low scaling of data relative to the height of the signal reduces the visibility of the features. This may sound obvious, but it is tricky to do correctly without having to rescale each successive page of data to the local value of the sensor stream. Finally, it was crucial to maintain consistent vertical scaling across all-printouts within one file; a change in scale deeply affects perceptual cues about the prominence of features.

Then there are examples where noise conditions yield interesting results. For example, when a sensor is not attached or attached without a signal, the noise is approximately Gaussian; when generating a signal or actively generating noise, the signal radically changes. This could be used to detect whether or not the sensor is sitting correctly on the skin. Also, in the EMG signals, the signal-to-noise ratio is very high; usually, if there is no noticeable motion, then the signal is almost completely flat. The signal-to-noise ratio for each sensor is different; some (particularly the GSR) seem particularly sensitive to motion artifacts.

Finally, there were several instances when noise conditions yielded problems. For example, I found a great deal of apparent motion artifact in the respiration and GSR signals; this seemed to mostly arise from large movements of the arms and shoulders. I also encountered difficulty in collecting the heart rate signal from the subjects. It was in general unusable because it was intermittent at best across most of the subjects.