Autonomous transportation is a field of innovation being rapidly advanced in the past decade. Most major car manufacturers are racing towards mass production of partly or fully autonomous vehicles. Moreover, mobility providers like Uber or Lyft have started their own programs, and tech companies like Google or Nvidia have also entered the field. Despite all the effort that goes into the engineering and the design of self-navigation, there are concerns about the general acceptance of autonomous vehicles. Those concerns include uncertainty whether the general public would be willing to purchase semi or fully autonomous cars at a rate that justifies mass production, and ethical uncertainty about safety and responsibility, among others. Since the advent of testing self-driving cars in real traffic situations there have been numerous efforts to assess acceptance, anxiety, arousal and other emotional aspects of this particular new technology [1]. Besides asking people to report their feelings and experiences, a reliable source of information is tracking biological measures like heart rate, muscle activity, eye movements, or electroencephalography (EEG) signals. Up until very recently, the non-portability of certain advanced neurological and behavioral techniques such as motion tracking, EEG or electromyography (EMG) often confined these investigations to the laboratory. Instead of real-world passenger experience, researchers often used video recordings, virtual reality (VR) or computer simulations [2-5]. However, recent improvements in measurement technology have made it viable to record these biological signals in the actual vehicles in focus. Our study is aimed 1) establishing a viable research method for the psychological and physiological measurement of passenger experience in self-navigating vehicles and 2) finding useful dimensions of covariability in the recorded data. The overall aim of our research is to establish a set of measurements that are sensitive in gauging affective engagement in experiencing autonomously driven passenger cars.
Self-driving vehicles are radical innovations which according to our current knowledge will overturn the daily lives and decades-old habits of all people in developed countries – whether they are involved in transport as drivers, cyclists, pedestrians, passengers, etc. [6]. Technological development related to self-driving vehicles has greatly accelerated recently: at present, self-driving vehicle tests are being conducted in nearly 200 cities - a figure which has doubled in one year. Hence, it seems that more developers are in the last phase of technical development, and are ready to enter production [7].
However, with a very few exceptions, developers focus solely on technological development, thus the spread of self-driving vehicles depends not only on the technological development shown, but also on legislative framework, infrastructure, and social acceptance [8]. One of the most important factors regarding innovation diffusion are decisions on innovation: the acceptance or the rejection of innovations and their expansion is based on people’s judgments and decisions [9]. Therefore, development work shall be extended in the direction of mapping social acceptance as accurately as possible, thereby accelerating social adaptation in the interest of the society and its members’ ability to process the projected drastic change.
In the last decades of the 20th century Information Systems (IS) were adopted in both organizational and domestic environments. Behavioral researchers wanted to know which factors influenced the acceptance and use of these new technologies. As a result, several technology acceptance research models were developed. Initially, there were the Theory of Reasoned Action (TRA) [10], and the Theory of Planned Behavior (TPB) [11]. Davis et al. [12] merged these theories into the Technology Acceptance Model (TAM). All the above-mentioned models assume that the actual use of a new technology depends on one’s behavioral intention (BI) to apply the technology itself. BI is directly moderated by one’s attitude towards use (A). In the TAM model, Perceived Ease of Use (PEU) and Perceived Usefulness (PU) have a direct aggregated moderating effect on A. In the TAM 2 model, PEU is described by Davis [12]; later [13] described PU. Venkatesh et al. [14] proposed a unification of the existing technology acceptance models and presented the Unified theory of Acceptance and Use of Technology (UTAUT) adding Social Influence (SI), Facilitating Conditions (FC) and moderating variables (Age, Gender, Experience and Voluntariness) to the model. The UTAUT 2 research model [15] is designed to allow researchers to investigate consumer acceptance and the use of new technologies, adding new moderating factors (Hedonic Motivation, Price Value, Habit) and removing one (Voluntariness of Use) from the model.
The UTAUT 2 model is widely used to predict technology acceptance and thus future behaviour. However, it is an interesting question how well the commonly used questionnaire survey is capable of collecting real and valid information. Consider for example the reporting of the subject's own emotional state (called Hedonic motivation in the UTAUT 2 model). Respondents often find it difficult to identify and report their current emotional states. This is especially true when they were asked about their future (expected) emotional states when traveling in a self-driving vehicle. Concurrently, Nordhof et al [16] asked 9,118 car drivers from eight European countries using the UTAUT2 model to explain public acceptance of conditionally automated (L3) cars and found that hedonic motivation was identified as the strongest predictor from all UTAUT 2 variables of individuals’ behavioural intention. Their results indicated that individuals who found conditionally automated cars to be fun and enjoyable were more likely to intend to use them. However, the method for measuring hedonic motivation was still the traditional online survey.
On this basis, it is worth considering what alternative data collection methods could be used to explore these emotional motivations. For this purpose, the UTAUT 2 model can be taken as a framework, where variables can be measured with a more holistic approach – for example, to examine emotional responses by neuroscientific methods rather than direct interviewing. Most studies on attitudes and trust towards autonomous technology utilize self-reporting questionnaires (e.g. [17, 18]). This approach has several limitations: first, the vast majority of participants respond without any prior experience as a passenger or driver of self-driving vehicles. Second, social desirability factors also might bias the results, therefore the objectivity of such data might be questioned. Recording physiological responses like EEG, galvanic skin response (GSR) or eye movements might be a useful method for eliminating such biases, and results of these measurements can be easily compared with self-reported data. Although a growing amount of research attempted to measure passengers' biological reactions in autonomous vehicles either utilizing EEG (e.g. [3, 4]) or eye-tracking methods [19, 20], we are not aware of any studies in which EEG and eye-tracking data were recorded simultaneously.
Eye movements indicating emotional and cognitive transitions
Muscle movements in perceptual systems including the head, hands, and feet have been established to indicate cognitive and affective states and transitions. Spectral analyses of head movements and eye movements were repeatedly shown to indicate intentional, goal related cognitive transitions such as problem solving, recognition, or comprehension. The general reasoning behind using complexity measures and spectrum analysis on continuous physiological data focuses on the source of cognitive and emotional change. Such changes are likely to be distributed within the complex network of physical, electro-chemical, nervous, muscular, and behavioral interactions. These are fundamental to complex systemic behavior, such as weather patterns, ecological dynamics, stock market fluctuations, or functions of the nervous system. Natural systems often exhibit power-law distributions and the variation of their components is correlated across both spatial and temporal scales [21, 22]. These correlations allow inquiry into the number and strength of interactions between the active components. In recent years, the concept of multifractality has become more popular in modeling, exploration, and prediction of complex dynamical system behavior [23, 24]. In psychology, multifractal concepts and tools became available right after the introductory reports (i.e. [25]) and they were followed by tutorials and practical guidance [26, 27].
Differences in the distribution of fractal dimension and the width of the multifractal spectrum have been reported as significant and reliable markers in various cognitive tasks including problem solving [28], magnitude perception [29, 30], perceptual intent [31], visual recognition [32], comprehension [33], and memory [34]. Inspired by these relatively recent findings, our experimental research methods on the eye-tracking data included fractal and multifractal analysis.
EEG signatures of emotional and cognitive transitions
The human brain is constantly active, resulting in electrical signals that can be measured from the scalp by EEG. The EEG oscillations are mainly classified according to their frequency bands: for example, the delta band is referred to as activation below 4 Hz, the theta band reflects activity between 4-8 Hz, the alpha band between 8-12 Hz, beta between 13-30 Hz, and activity higher than 20 Hz is referred to as the gamma band [35].
Numerous studies target EEG correlates of emotional and motivational states by differences in alpha band power between the right and left hemispheres [36, 37]. Frontal alpha asymmetry is a result of the subtraction of left frontal alpha power from right frontal alpha power after log-transforming the values to normalize distributions (log F4 minus log F3). The result is referred to as relative left or relative right frontal activity: when the result is more positive, relative left activation is present and when the result is more negative, relative right activation is present. High scores therefore indicate more positive or approaching attitudes while lower scores indicate more negative or withdrawal attitudes [37, 38].
When classifying mental states or cognitive processes from relaxed to alerted or stressed states, usually the ratio of higher frequency (beta, gamma) and lower frequency (alpha, theta) powers are compared [39, 3]. Lower frequencies are dominating in more relaxed states while higher frequencies are present in more aroused or stressed states. For example, increased beta/alpha ratio [40, 30], decreased alpha/beta and theta/beta ratio [41] or increased relative gamma ratio [42] was found to correlate with stress level. Higher relative gamma was also detected during enhanced attention and concentration [3].
Several studies record EEG while participants are exposed to self-driving technology, mostly by sitting in a simulator [2-5]. For example, Park and colleagues [4] found an increasing beta-alpha power ratio when a participant was passively watching positive scenarios (the car performing smoothly on a highway) and negative scenarios (the same car driving erratically and violating common rules of the road) of self-driving cars, revealing elevated stress levels when the participants were exposed to simulated dangerous situations. In a similar simulator study, the most effective takeover warning signals were accompanied by enhanced ratios of higher frequencies, suggesting enhanced stress, attention, and alertness [3].
Most recently, Seet and colleagues [5] investigated the impact of autonomous vehicle malfunction on human trust by combining EEG and self-reported questionnaires in a simulation environment. During the simulation, participants actively drove the vehicle in an urban environment and from time to time, scenarios with malfunctions (e.g. the vehicle drove at high speed through junctions or even crashed) occurred. In the Conditional Automation Driving phase, participants were able to take over control when needed, while in the Full Automation Driving condition no takeover function was present. Results showed that participants’ preference was significantly higher toward the situation when they were able to take over control. Besides, the frontal alpha power reduction in the right but not in the left hemisphere during malfunctions in the fully automated condition can also be interpreted as an enhanced motivation of the driver towards controlling the vehicle.
Since we have limited experience with autonomous navigation systems and real-world physiological measurements our hypotheses were outlined carefully. First, we expected that our complexity measure — the multifractal spectrum width — would be sensitive to possible differences in the human driver versus autonomous driving conditions. Looking at the MF width literature, we could not predict the exact direction of the difference. Second, we expected to find that the novelty and excitement of the experience would manifest itself in higher levels of valence and arousal in the EEG data. Third, we expected that some personality or anxiety measures in psychological surveys would correlate with physiological measures.