We just had a paper accepted to the ICMI 2011 workshop on “Inferring Cognitive and Emotional States from Multimodal Measures (MMCogEmS)“! The paper is called “Gesture Dynamics: Features Sensitive to Task Difficulty and Correlated with Physiological Sensors” and reports partial results from our recent study in the Multimodal Stress Detection project. The study included many modalities of input, but this paper focuses on the gesture modality. Here is the abstract:
This paper presents preliminary results regarding which features of pen-based gesture input are sensitive to cognitive stress when manipulated via changes in task difficulty. We conducted a laboratory study in which participants performed a vigilance-oriented continuous attention and visual search task. Responses to the search stimuli were entered via pen gestures (e.g., drawing a letter corresponding to the stimulus). Task difficulty was increased during predefined intervals. Participants’ input behaviors were logged, allowing for analysis of gesture input patterns for features sensitive to changes in task difficulty. We also collected physiological sensor readings (e.g., skin temperature, pulse rate, and respiration rate). Input behavior features such as gesture size and pen pressure were not affected by task difficulty, but gesture duration and length were affected. Task difficulty also affected physiological sensors, notably pulse rate. Results indicate that both gesture dynamics and physiological sensors can be used to detect changes in difficulty-induced stress.
Here’s the camera-ready version of the paper.
Lisa Anthony 