Recent investigations into infants' intersensory perception suggest a specific developmental pattern for infants' attention to visible and auditory attributes of dynamic human faces. This work has proposed that infants' perception seems to progress along a sensory continuum: beginning with multimodal sensory cues (e.g., auditory and visual), then visual-cues alone, and finally auditory-cues alone. Additionally, research has proposed that amodal or invariant sensory information directs infants' attention to specific redundant aspects in the surrounding environment (e.g., temporal synchronicity). The current research attempted to clarify the potential methodological confounds contained in previous investigations into infant intersensory development by contrasting infant behavior within fixed trial and infant-controlled habituation procedures. Moreover, the current research examined infants' attention to auditory manipulations within multimodal displays when redundant sensory information (synchronicity) was or was not available. In Experiment 1, 4-month-old infants were habituated to complex audiovisual displays of a male or female face within an infant controlled habituation procedure, and then tested for response recovery to a change in voice. For half the infants, the change in voice maintained synchronicity with the face, and for the other half, it did not. The results showed significant response recovery (i.e., dishabituation) to the change in voice regardless of the synchronicity condition. In Experiment 2, 4-month-old infants received the same face+voice test recordings used in Experiment 1, but now within a fixed trial habituation procedure. Again, synchronicity was manipulated across groups of infants. In contrast to Experiment 1, the infants in the fixed-trial experiment failed to show evidence of voice discrimination. These results suggest that infant controlled procedures may be more sensitive to infant attention, especially in terms of complex social displays. In addition, synchronicity appeared to be unnecessary in terms of infants' ability to detect vocal differences across multimodal displays. In sum, these results highlight the importance of research methodology (e.g., infant control) and overall stimulus complexity (e.g., discrete vs. complex) involving studies of infants' intersensory development. / Ph. D.

It has been well documented that dual-task performance is more accurate when each task is based on a different sensory modality. It is also well documented that the memory for each sense has unequal durations, particularly visual (iconic) and auditory (echoic) sensory memory. In this dissertation I address whether differences in sensory memory (e.g. iconic vs. echoic) duration have implications for the design of a multimodal display. Since echoic memory persists for seconds in contrast to iconic memory which persists only for milliseconds, one of my hypotheses was that in a visual-auditory dual task condition, performance will be better if the visual task is completed before the auditory task than vice versa. In Experiment 1 I investigated whether the ability to recall multi-modal stimuli is affected by recall order, with each mode being responded to separately. In Experiment 2, I investigated the effects of stimulus order and recall order on the ability to recall information from a multi-modal presentation. In Experiment 3 I investigated the effect of presentation order using a more realistic task. In Experiment 4 I investigated whether manipulating the presentation order of stimuli of different modalities improves humans' ability to combine the information from the two modalities in order to make decision based on pre-learned rules. As hypothesized, accuracy was greater when visual stimuli were responded to first and auditory stimuli second. Also as hypothesized, performance was improved by not presenting both sequences at the same time, limiting the perceptual load. Contrary to my expectations, overall performance was better when a visual sequence was presented before the audio sequence. Though presenting a visual sequence prior to an auditory sequence lengthens the visual retention interval, it also provides time for visual information to be recoded to a more robust form without disruption. Experiment 4 demonstrated that decision making requiring the integration of visual and auditory information is enhanced by reducing workload and promoting a strategic use of echoic memory. A framework for predicting Experiment 1-4 results is proposed and evaluated.

<p>Adults aged 65 years and older have become the fastest-growing age group worldwide and are known to face perceptual, cognitive, and physical challenges in later stages of life. Automation may help to support these various age-related declines. However, many current automated systems often suffer from design limitations and occasionally require human intervention. To date, there is little guidance on how to design human-machine interfaces (HMIs) to help a wide range of users, especially older adults, transition to manual control. Multimodal interfaces, which present information in the visual, auditory, and/or tactile sensory channels, may be one viable option to communicate roles in human-automation systems, but insufficient empirical evidence is available for this approach. Also, the aging process is not homogenous across individuals, and physical and cognitive factors may better indicate one’s aging trajectory. Yet, the benefits that such individual differences have on task performance in human-automation systems are not well understood. Thus, the purpose of this dissertation work was to examine the effects of 1) multimodal interfaces and 2) one particular non-chronological age factor, engagement in physical exercise, on transitioning from automated to manual control dynamic automated environments. Automated driving was used as the testbed. The work was completed in three phases. </p><p><br></p> <p>The vehicle takeover process involves 1) the perception of takeover requests (TORs), 2) action selection from possible maneuvers that can be performed in response to the TOR, and 3) the execution of selected actions. The first phase focused on differences in the detection of multimodal TORs between younger and older drivers during the initial phase of the vehicle takeover process. Participants were asked to notice and respond to uni-, bi- and trimodal combinations of visual, auditory, and tactile TORs. Dependent measures were brake response time and maximum brake force. Overall, bi- and trimodal warnings were associated with faster responses for both age groups across driving conditions, but was more pronounced for older adults. Also, engaging in physical exercise was found to be correlated with smaller maximum brake force. </p><p><br></p> <p>The second phase aimed to quantify the effects of age and physical exercise on takeover task performance as a function of modality type and lead time (i.e., the amount of time given to make decisions about which action to employ). However, due to COVID-19 restrictions, the study could not be completed, thus only pilot data was collected. Dependent measures included decision making time and maximum resulting jerk. Preliminary results indicated that older adults had a higher maximum resulting jerk compared to younger adults. However, the differences in decision-making time and maximum resulting jerk were narrower for the exercise group (compared to the non-exercise group) between the two age groups. </p><p><br></p> <p>Given COVID-19 restrictions, the objective of phase two shifted to focus on other (non-age-related) gaps in the multimodal literature. Specifically, the new phase examined the effects of signal direction, lead time, and modality on takeover performance. Dependent measures included pre-takeover metrics, e.g., takeover and information processing time, as well as a host of post-takeover variables, i.e., maximum resulting acceleration. Takeover requests with a tactile component were associated with the faster takeover and information processing times. The shorter lead time was correlated with poorer takeover quality.</p><p><br></p> <p>The third, and final, phase used knowledge from phases one and two to investigate the effectiveness of meaningful tactile signal patterns to improve takeover performance. Structured and graded tactile signal patterns were embedded into the vehicle’s seat pan and back. Dependent measures were response and information processing times, and maximum resulting acceleration. Overall, in only instructional signal group, meaningful tactile patterns (either in the seat back or seat pan) had worse takeover performance in terms of response time and maximum resulting acceleration compared to signals without patterns. Additionally, tactile information presented in the seat back was perceived as most useful and satisfying.</p><p><br></p> <p>Findings from this research can inform the development of next-generation HMIs that account for differences in various demographic factors, as well as advance our knowledge of the aging process. In addition, this work may contribute to improved safety across many complex domains that contain different types and forms of automation, such as aviation, manufacturing, and healthcare.</p>