MEASURES OF DRIVER BEHAVIOR AND COGNITIVE WORKLOAD
IN A DRIVING SIMULATOR AND IN REAL TRAFFIC
ENVIRONMENT
– experiences
from two experimental studies in Sweden
*Swedish National Road Administration, 781 87 Borlänge, SWEDEN
**Department of Psychology, Stockholm University, S-10691
Stockholm,
SWEDEN
E-mail: ruggero.ceci@vv.se
Summary: The use of ITS (Intelligent Transportation System) in general is increasing in road traffic with external demands on driver attention and cognitive functioning. Also in-vehicle systems such as navigation and onboard PCs with Internet and e-mail connections are on the market in many parts of the world. Two different studies are presented in this paper. These have focused upon mental performance as a result of driving in a tunnel simulation with a route choice task and in a real traffic environment with the effect of various in-vehicle navigation tasks. Results indicate future orientation and road choice problems, as much as 50% of test-drivers missed important road sign information and made critical road choice errors at specific points, i.e. entering the tunnel system from main roads. In the second study significant effects of visual and visual/verbal but no significant effects of verbal instructions on mental performance were obtained. These results are discussed with respect to requirements regarding suitable standard methods for assessment of cognitive workload caused by external information (i.e. road/tunnel environment) and from in-vehicle systems.
BACKGROUND
New in-vehicle technologies such as navigation systems and onboard Internet connections etc. are going to increase in availability and use. It has been estimated that in Japan about 15% of the total vehicle fleet is already equipped with navigation and other in-vehicle information systems. Also, the use of ITS (Intelligent Transportation System) in general is increasing in road traffic with external demands on driver attention and cognitive functioning. The present studies have focused upon the development of new methods to study the effects of 1) ambient information demands in a driving simulator situation and 2) in-vehicle information systems (IVIS) in a real-life situation, by measures of driver behavior, cognitive workload and subjective as well as physiological responses.
METHODS
Study 1
The first study comprises a series of simulator trials with 21 subjects driving five different predefined routes in a visualization of a future Stockholm road tunnel system, Södra Länken. The driving simulator has an advanced construction with a motion system, a wide angle (120) visual system, a vibratory generating system, a sound system and a temperature regulating system (Nilsson, 1989, 1993; Nordmark, 1994). All subjects were unfamiliar with the presented routes.
All trials were videotaped and driver performance parameters were registered on-line by the simulator computer system, in addition electro dermal activity (EDA) and mental workload ratings according to a modified NASA-TLX scale was used (Hart and Staveland, 1988). The electrodes were attached to the left-hand long- and ring fingers. The following five different subjective ratings were used: driver demands, time pressure, feeling of uncertainty, performance, and overall difficulty (which is an index based on the first three ratings). Five different tunnel routes were administered to each subject after one trial route and instructions regarding the subjective ratings and driving procedure.
After each route session subjects were interviewed
regarding their experiences from the driving including specific problems of
orientation, road sign placement, speed choice and other perceived
irregularities. Each route was only performed once in order to avoid learning
effects, all routes were rotated between subjects in a balanced order to avoid
serial effects. NASA-TX ratings were made after each route and EDA was recorded
continuously during the experimental session in addition to speed,
acceleration, lateral position and brake activity.
Study 2
The second study was performed as a field study in Linköping City with twenty-four professional drivers testing a peripheral detection task (PDT) device as a standard method for measuring cognitive load as an evaluation of IVIS. The PDT-method was first used in a simulator study of an advanced driver assistance system (ADAS) in Holland (Martens and Van Winsum, 1999). The PDT stimuli consisted of a (3x20 cm) base with 23 light emitting diodes (LED). Subjective mental workload was measured according to the simple version of the NASA-TLX questionnaire.
The present study compared memory-based driving in a built-up area with driving based on instructions from a GPS-based navigation system in the same area. An instrumented car (Volvo, Model 850S, 2.5, 1996 with manual gearbox) equipped with an in-vehicle navigation system (VDO Dayton, MS 5000) was used for the experiment. The test vehicle was equipped with an advanced data collection system that included full video coverage of the trials, headway, speed variation, brake activity, lateral position.
All drivers were highly skilled and familiar with both
IT-components in vehicles and the area in which they were required to drive.
Two different driving routes were used in the experiment. All drivers had to
drive one route from the memory and one based on instructions from the navigation system. Based
upon subgroup inclusion subjects were either exposed to visual, verbal or
visual and verbal navigation instructions. During the different driving tasks
the subjects were instructed to respond as fast as possible to as many
PDT-stimuli as possible without reducing the attention from the road scene.
RESULTS and DISCUSSION
Study 1
Table 1 shows the results from the NASA-TLX measures for all five routes (1-5) as mean values and standard deviations. It can be seen that the index variable overall difficulty identified route 5 as the most difficult (36.0) followed by routes 4, 1,3 and 2. (see route map, appendix 1).
As expected the mean results from the NASA-TLX are relatively low, around 30 as compared to the maximum rating of 140. The simulated drive in the Södra Länken tunnel is a fairly easy task as compared to a real road or tunnel experience with heavy traffic and complex decision making. An earlier study (Wilson and O´Donnel, 1988) has demonstrated that variability in physiological responses can be a useful measure of a driver’s mental strain and perceived uncertainty. A higher variability indicates an increase in cognitive workload.
Table 1. Means and standard deviations (Sd) from
NASA-TLX ratings for five different routes.
________________________________________________________________________________NASA-TLX Route n:r
________________________________________________________________________________
1 2 3 4 5
________________________________________________________
Driver demands 27.5+/-18 30.4+/-19 26.2+/-14 31.0+/-21 32.9+/-21
Time pressure 23.4+/-22 19.9+/-10 29.1+/-23 39.1+/-23 30.8+/-23
Feeling of uncertainty 30.5+/-25 22.2+/-15 18.9+/-12 25.9+/-14 44.0+/-21
Performance 36.6+/-19 26.6+/-19 29.5+/-21 22.2+/-15 29.6+/-17
Overall difficulty* 27.1 24.3 24.7 32.0 36.0
_________________________________________________________________________________
* overall difficulty is an index based on the mean from the ratings driver
demand, time pressure and feeling of uncertainty (Sd cannot be calculated for
this index).
Table 2 shows the results from measures of EDA (electro dermal activity), driving speed, acceleration and braking activity as group means of coefficients of variation (CV). An interesting finding from the
Table 2. Means of coefficients of variation of EDA,
driving speed, acceleration and brake activity for the five different routes.
_______________________________________________________________________________
Variability of
driver responses Route n:r
_______________________________________________________________________________
1 2 3 4 5
_______________________________________________________
EDA 0.171 0.161 0.155 0.181 0.179
Driving speed 0.091 0.073 0.079 0.075 0.084
Acceleration 0.313 0.307 0.323 0.298 0.285
Braking activity 7.166 20.27 18.22 14.25 12.51
_______________________________________________________________________________
EDA measures is the correspondence with the NASA-TLX results regarding routes 4 and 5. This is also confirmed by the high correlations between the psycho-physiological reaction (i.e. EDA) and subjective ratings of overall difficulty (rxy = 0.90, p<0.05) and feeling of uncertainty (rxy = 0.85).
Results from the driving performance and error analysis, based on video recording, indicates future orientation and road choice problems. A total of 50% of the drivers missed important road sign information and made critical road choice errors at specific points, i.e. entering the tunnel system system from main roads. Also within the tunnel system many subjects (30%-50%) made lane choice errors resulting in loss of orientation and missed their target exits.
Finally, the main purpose of this study is to demonstrate a new methodology to be used in the planning process of a road or a tunnel construction. The idea of having subjects unfamiliar with the road/tunnel environment performing a simulated driving task with the use of EDA and subjective ratings is fairly new in Sweden (and presumably in the rest of the world). Also, to use the results from this type of experimental studies in the ongoing construction process of a large tunnel like the 6 km Södra Länken (the largest ever built in Sweden) is definitely a challenge. A follow-up study is planned in the real tunnel environment when it opens for traffic in late 2003.
Study 2
In this study subjects judged the items according to the NASA-TLX scale very differently, however, pairwise T-tests for each variable within each group (18 comparisons) showed a significant difference (p<0.05) only for frustration between trials with full navigation. All in all, 15 out of 18 possible comparisons represent a more positive assessment of memory based driving as compared with navigation based driving.
Table 3. Average subjective scores of workload after
the simplified NASA-TLX workload questionnaire for combinations of instructions
and navigation conditions (N = 24)
________________________________________________________________________________
Verbal
navigation Visual
Navigation Full
Navigation
(n=8) (n=8) (n=8)
Navigation Memory Navigation Memory Navigation
Memory
Mental demand 47.1
30.1 53.2
41.0 28.7
21.8
Physical demand 25.0 20.3 44.5 33.1 18.2 16.3
Time pressure 27.2 15.1 34.7 29.6 18.3 13.8
Performance 73.1 80.0 68.5 68.0 73.3 82.2
Effort 37.8 36.1 39.0 42.7 29.5 21.0
Frustration 25.6 26.5 50.6 33.1 20.6 14.2
________________________________________________________________________________
The results showed little effect on driving performance of navigation mode but PDT performance was influenced by navigation mode. Reaction time was longer and hit rates lower in trials with the navigation system than in trials based on memory (See Table 4). Significant effects of visual and visual/verbal but no significant effects of verbal instructions on PDT performance were obtained as a result of a more detailed analysis according to a 2x3 (instructionsxlevels) analysis of variance (ANOVA).
In line with results from previous studies PDT-performance showed a remarkable sensitivity to distraction caused by the navigation instructions also in the present experiment. By presenting subjects with the navigation messages with the same content and on identical locations during driving, only the mode of presentation was varied in the present experiment.
Table 4. PDT - Reaction Time (RT) in milliseconds
(ms) and Hit Rate (HR) for individual subjects (n=24).
Group means (M) and standard deviation (Sd) are
presented for the different variables.
_______________________________________________________________________
VERBAL NAVIGATION VISUAL NAVIGATION FULL
NAVIGATION
Navigation Memory Navigation Memory Navigation Memory
S RT
HR RT HR
S RT HR RT HR
S RT HR RT HR
3 770
.59 698 .72
1 582 .82 523
.93 5 508 .94 387 .94
4 625
.98 520 .98
2 548 .89 410
.97 6 636 .76 489 .98
9
761 .83 734 .93
7 786 .80 737 .92 11 628 .92 571 .92
10 590 .96 686 .95
8 772 .80 686
.82 12 695 .91 633 .91
15 812 .83 955 .86
13 632 .96 624 .95
17 785 .96 829 .90
16 628 .89 616 .91
14 698 .91 725
.91 18 831 .75 657 .97
21 666 .88 593 .94
19 780 .67 832
.68 23 704
.89 746 .90
22 511 .95 416 1.00
20 623 .89 507
.93 24 694 .90 601 .94
________________________________________________________________________________
M:
670.4 .86 652.3 .91 677.6 .84 630.5 .89 685.1
.88 614.1
.93
Sd:
102.8 .12 160.2 .09 94.5 .09 141.1 .09
99.2 .08 138.9 .03
______________________________________________________________________________________
Thus, it could be demonstrated that PDT-performance was related to the mode of presentation. The driving task requires continuous visual information processing and visual distraction is a very important component in safety evaluation of IVIS. Therefore, the method should have a predominant status in a test battery for traffic safety evaluations of IVIS and ADAS (advanced driver assistance systems).
The authors are grateful to Dr. Lisbeth Harms and Dr. Lena Nilsson for their collaboration in study 2.
REFERENCES
Hart, S.G. & Staveland, L.E. (1988). Development of NASA-TLX. In P. Hancock & N. Meshkati. (Eds). Human Mental Workload. Amsterdam: North-Holland.
Martens,
M. & Van Winsum, W. (1999).
Activation criteria and warning strategies: Driving Simulator Results. IN-ARTE
Project (TR4014), ID 7.1 Unpublished technical report. TNO Soesterberg.
Nilsson, L. (1989). The VTI driving
simulator. Description of a research tool. Drive project V1017 (Bertie). VTI särtryck 150.
Nilsson, L. (1993). Behavioral
Research in an Advanced Driving
Simulator – Experiences of the VTI System. Proceedings of the Human Factors and
Ergonomic Society, 37th Annual Meeting, Seattle, Washington. VTI
särtryck 197.
Nordmark, S. (1994). Driving simulators, trends and experiences. RTS’ 94 driving Simulator Conference, Paris. VTI särtryck 204.
Wilson, G.F. & O´Donnel, R.D. (1988). Measurement of operator workload with the neuropsycho-logical workload test battery. In: P. Hancock & N. Meshkati. (Eds). Human Mental Workload. Amsterdam: North-Holland.