THE USE OF A MULTI-MODAL INTERFACE TO INTEGRATE

IN-VEHICLE INFORMATION PRESENTATION

David J. Wheatley & Joshua B. Hurwitz

User Centered Research, Motorola Labs

1301, E. Algonquin Road

Schaumburg, Illinois, USA

E-mail: David.j.Wheatley@motorola.com

E-mail: Joshua.Hurwitz@motorola.com

Summary: The car of the future will have many new information sources—including telematics systems, navigation systems and Advanced Driver Assistance Systems (ADAS)—that will compete for a driver’s limited cognitive attention. If they are implemented as completely separate systems then cognitive overload and driver distraction are inevitable outcomes. However, if they are implemented as an integrated intelligent system with a multi-modal interface, then the benefits of such functionality will be achieved with much less impact on driving safety.  Such a system will support the task of safe driving by filtering and mediating information in response to real-world driving demands.  This paper outlines the Human Factors research program being undertaken by Motorola Labs to evaluate key elements of such a multi-modal interface as well as the key human factors issues involved in a multi-modal interface.

INTRODUCTION

With the advent of new technologies in the vehicle—such as cellular communications and navigation, telematics and driver-assistance systems—comes understandable concern about the potential safety hazards associated with multi-tasking in the car. The current focus is on the potential safety hazards of using cellphones whilst driving.  Results from the research literature indicate that, while drivers using a cellphone can adequately perform everyday driving tasks, they show delayed responses to unexpected hazards.  For example, when following other vehicles, drivers display increased reaction times to avoid collisions with the lead vehicles (Nilsson, 1993; Alm & Nilsson, 1995).  Furthermore, epidemiological studies show an association between accidents and cellphone usage (Redelmeier & Tibshirani, 1997).

While such studies provide evidence of the dangers of driving while using cellphones, there are many benefits of this now ubiquitous technology (over 110 million cellphone subscribers in the US).  It was estimated that 2.8 million emergency calls were made from cellphones in 1995 (ComCare Alliance, 2001).   Industry estimates put this now at 18 million calls nationwide.  Consequently, road-crash notification times have fallen by 30% in the past ten years (Potts J. 2000). This has resulted in reduced trauma and fatalities due to faster medical attention being more rapidly available in the “golden hour” after an accident.

However, there remain concerns about the distraction effects of cellphone use by drivers, and these are not without precedence. Concerns were also raised in the 1930’s when Motorola first introduced the car radio. There were complaints about driver distraction and disturbance of the peace, and there were even suggestions that the radio be turned off whilst driving (NHTSA, 1997). This is remarkably similar to the current concerns about cellphone use in the vehicle, but without the associated safety benefits.

Perhaps it is in the nature of such “new” technologies these issues are raised and subsequently addressed, and are thereby solved. There is, however, a qualitative difference in the current situation: since several in-vehicle systems at various stages of maturity are or will be introduced into the vehicle. These include

Over time, such systems will be introduced separately by different suppliers, cascading down from high-end to mass-market vehicles.  This is the general pattern of technology introduction, particularly in the automotive industry. This process is almost certain to ensure that, among the various systems, there will be little consistency in the user interface and little coordination to prioritise or filter the information presented to the driver.

 

SYSTEM INTEGRATION

Given this situation, there are good reasons why the safety concerns may be valid. In order to avoid potential hazards arising from inconsistency and cognitive overload, an in-vehicle system consisting of any number of the subsystems described above needs to be developed as an integrated whole and not as a loose assembly of separate parts. In other words, the focus needs to be on on the totality of the information exchange that takes place between a driver on the one hand, and the vehicle, environment and other systems on the other. Integration of telematics systems (including navigation, infotainment etc.) with Advanced Driver Assistance Systems (ADAS) and all wireless communications is the only way in which this information exchange can be controlled, managed and prioritised in the context of the variable demands of the driving task.

However, this is a complex issue, and the quantity and nature of information available to the driver is such that simply using visual display will not suffice. Furthermore, since driving is a predominantly visual task and the visual channel is frequently overloaded, other sensory channels need to be used to present information to the driver.  To be fully effective, an integrated in-vehicle system—such as that being researched and developed at Motorola Labs—will require a multi-modal interface with the driver.

The choice of modality to present certain information will depend on the nature of this information.  In general, the auditory modality is better than the visual modality for providing an initial hazard alert to a driver and for rapidly conveying the magnitude of the potential hazard.  The visual modality, on the other hand, is more appropriate for supporting a driver’s continuous awareness of the surrounding traffic.  One example would be a visual display that depicts the road environment immediately around the driver’s vehicle.  In such a display, an appropriate set of visual icons can inform the driver of objects located in blind spots, and can make road signs more salient and remind drivers of road signs they recently passed.  Another reason for using a multi-modal interface is information redundancy. For example, being temporally transient, an auditory signal may be missed, whilst the visual display may be seen. Finally, haptic feedback could be conveyed in the form of a “virtual rumble strip” when the driver is in danger of unintentionally going off the road.  Such feedback takes advantage of drivers’ experiences with infrastructure support devices, such as rumble strips, that are likely to be available only in certain locations and extends the safety benefits to all locations.

DRIVER ADVOCATETM APPLICATION

The integrated system that is currently being addressed combines the functionality of both telematics and driver assistance systems. By integrating these together, this system is able to manage and prioritise the presentation of information to the driver. The Driver AdvocateTM system enables cellular wireless communications to be available in a moving vehicle without compromising driving safety. It does this by monitoring the status of the vehicle (steering wheel movements, brake, throttle actuation, speed and other data), the immediate traffic environment (proximity, trajectory and speed of nearby vehicles and other sensor data) and the driver state (alertness, drowsiness).  This data is combined and is used by an embedded intelligent system to estimate the driver’s cognitive (mental) workload at that point in time. The system will, for example, suspend a voice call, or direct it to voicemail if it determines that the driver is inattentive or if driving performance is degraded and will present visual, auditory and haptic alerts and alarms to the driver to enhance his situational awareness.

The system concept is that the data derived from the on-board sensors is used for two purposes.  The first is to derive an estimate of current driver workload and relate this to the driver state. If there is cognitive overload, then non-urgent and non-time critical information is filtered or perhaps even not presented. The second application of the sensor data is to generate alerts and alarms relating to events in the driving environment to which the driver has not responded—in other words, to increase situational awareness.

Developing a multi-modal interface for a new system concept such as this requires considerable secondary research on these issues.  However, it also requires a number of primary experimental studies.  Such studies will be designed to evaluate whether the performance effects hypothesised with certain presentation modes are found in the simulated driving task.

RESEARCH PROGRAM

A series of experimental human factors studies is underway to quantify the performance effects of in-vehicle warnings on driver performance. These studies will focus on warnings presented in a particular modality, or compare performance for warnings presented in different modalities. 

The first of these was a driving simulator study carried out in association with the University of Iowa (a detailed report on this study will be published elsewhere). The driving task was to follow a lead vehicle, which periodically decelerated at two different rates.  Subjects had to brake to avoid hitting the lead vehicle from the rear. In one condition of the study, they performed the driving task alone.  In another, they performed this task along with a distractor task (a cognitive task, presented via the auditory modality), and also each of these conditions both with and without an auditory headway warning. The warning consisted of an auditory alarm that was generated if appeared that the two vehicles might get too close to each other. The aim of the study was to quantify the degradation effect of the distractor task on brake reaction time and headway maintenance, and to ascertain whether the auditory headway warning mitigated any degradation and restored the safety margin in terms of headway.

The results of this study suggest that an auditory headway warning helps drivers maintain safer distances with lead vehicles.  Temporal headways increased significantly on trips in which warnings were presented.  Furthermore, this effect was greater on trips in which drivers were engaged in a secondary task. We can conclude therefore that a headway warning does lead to significant improvements in headway maintenance or safety margin.

However, our initial studies have also shown that the subjective reactions to the auditory alerting system have not generally been positive. In fact, 53% of subjects felt that the auditory warning tone actually interfered with their ability to drive safely and 59% indicated that it was more difficult to concentrate on driving. This occurred despite the fact that the warning actually improved safety according to the objective performance measures. Seventy-eight percent of the subjects mostly or definitely agreed that the warning tone was annoying. Though performance and safety may improve using such an auditory headway warning alert, in some cases it may not enhance the pleasure of driving. 

Our second simulator research study builds upon the conclusions drawn from the first and will explore the use of a visual situational awareness display for headway warning. The intent is to determine whether comparable levels of performance and safety improvement can be achieved without the negative subjective reactions found with the warning tone used in the first study.  If drivers are more aware of the surrounding traffic, then hazardous situations can be avoided without intense and potentially annoying alarms.

      

CONCLUSIONS

The safe and effective implementation of future in-vehicle information and safety systems will require a multimodal interface using visual, auditory and haptic channels. The aim of this research program is to hypothesise elements of this interface based upon secondary research and to evaluate the resulting performance effects in empirical research carried out in a driving simulator environment. Further research studies are planned to evaluate elements of a voice dialogue interface with this type of in-vehicle system.

 

 

REFERENCES

Alm, H., & Nilsson, L. 1995. The effects of a mobile telephone task on driver behavior in a car following situation. Accident Analysis and Prevention, Vol 27 No. 5 pp 707-715.

ComCare Alliance 2001, 911 – America’s Universal Emergency Phone Number, Comcare Alliance, 888 17th St.NW. 12th Floor,  Washington DC.

Hurwitz J.B & Wheatley D.J. 2000  Alerting Drivers with Headway Warning Signals, Internal Motorola Report, Dec 2000

Liu Yung-Ching 2001, Comparative Study of the effects of auditory, visual and multi-modality displays on drivers’ performance in advanced traveller information systems. Ergonomics Vol 44 No. 4 pp 425 – 442.

National Highway Transportation Safety Administration 1997 ; An Investigation of the Safety Implications of Wireless Communications in Vehicles, Report No. DOT HS 808-635. Nov 1997

Nilsson, L. 1993 Behavioural research in an advanced driving simulator: Experiences of the VTI system. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, pp. 612-616.

Potts J. 2000 Wireless Phone Calls to 911; Steps Toward a more Effective System, Currents, summer 2000.

Redelmeier, D.A. & Tibshirani, R.J. 1997 . Association between Cellular Telephone calls and Motor Vehicle Collisions. The New England Journal of Medicine 336, 2,  pp. 453 – 458.