Digital and Traditional Executive Function Assessment Tools (EFATs) for Healthy Adults: A Systematic Review of Psychometric Properties

doi:10.21203/rs.3.rs-5161193/v1

Objective: To examine the available data on executive function assessment tools (EFATs) for healthy adults with respect to the following: assessed constructs; reliability; validity; and other psychometric properties. This review also seeks to pinpoint the most employed EFATs in research, those adapted for the Brazilian population, and those suitable for online administration.

Methods: This review adheres to the Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Searches were conducted across the PsycNet, PubMed, Embase, Web of Science, and VHS databasesand relevant grayliterature. The primary search descriptors were executive functions, neuropsychological assessment, executive function assessment, healthy adults, and psychometric properties. Risk-of-bias assessment was performed for all reviewed studies.

Results: This review encompasses 29 studies. Thirteen studies introduced novel EFATs with promising psychometrics, whereas16 evaluated preexisting EFATs. Three studies adapted tools into Brazilian Portuguese. The most targeted constructs were inhibition, working memory, and cognitive flexibility. Additionally, processing speed, attentional control, verbal fluency, planning, episodic memory, and psychomotor abilities were also addressed. The most employed tools were the Stroop test, the digit span test, and the trail-making test. Most studies focused on digital tools, but only four had versions adapted to Brazilian Portuguese.

Conclusion: The findings revealed that the currently available EFATs are strongly based on the theory that EFs are composed ofthree primary components. Notably, online tools have emerged as a significant area of interest in neuropsychological research, as numerous studies are delving into online EF batteries and serious games for EF assessment.

Systematic review registration: This review was previously registered on the Prospective Register of Systematic Reviews (PROSPERO; CRD42023459513).

healthy adults

executive function assessment

neuropsychological assessment

digital tools

executive functions

psychometric properties

Executive functions (EFs) appear to be the most studied construct in neuropsychology (1) since the Luria’s model (2), which recognized cognition as occurring in different functional areas of the brain (primary, secondary, and tertiary). However, their understanding is far from a consensus, and several distinct models have been proposed in the literature since the concept of EF was first introduced (1,3,4).

A previous study compiled EF models from the literature, categorizing them into seven distinct approaches (3,4), including the supervisory attention system (SAS;5,6), working memory models (7–9), and the componential model (10–12). They suggested that the most commonly used approach is the componential approach, which considers EF as goal-directed, context-dependent controlled processes that include domain-general components (inhibition, working memory, and cognitive flexibility).

Although EF remains an ambiguous concept in the scientific community (1,3,4,13), it can be understood as high-order cognitive abilities responsible for regulating an individual's behavior, aiming to achieve the most appropriate response to environmental demands through a balance between automatic and controlled responses (4,14–16).

As EFs are important for most daily activities, their good functioning is highly important for both clinical practice and research endeavors. Indeed, mental and physical health, academic and professional performance, social welfare and quality of life are some facets of an individual’s life that are highly dependent on their EF integrity (10,12,17). In this context, the assessment of EF is fundamental and is considered one of the most important components of cognitive assessment in neuroscience (18).

EF assessment is a subset of neuropsychological evaluation, which is a specialized assessment designed to measure cognitive ability and function across various domains, including EF, memory, attention, language, and praxis (19,20). With a predominant emphasis on clinical treatment and therapeutic intervention, EF assessment is also becoming common for evaluating both strengths and weaknesses in cognitive capabilities across developmental and aging populations (21,22).

In the developmental phase (children and adolescents), EF assessment is important for detecting any anomaly in executive development that may jeopardize quality of life in adulthood (10,22,23). On the other hand, in the natural aging process, EF assessment is also essential for detecting deficits in critical cognitive domains that may compromise the autonomy of elderly individuals (24–26). In both cases, EF assessment is paramount to guide treatment planning.

Moreover, in previous studies, EF assessment tools (EFATs) were employed to assess the cognitive abilities of neurotypical adults, aiming to identify areas for potential enhancement and improve their professional performance (beyond expectations), particularly in leadership positions (27–29). However, few studies in the literature have focused on healthy individuals, particularly those in the economically active age bracket¹, with the aim of mapping out the need to enhance executive skills for the purpose of improving professional performance. In Brazil, more specifically, studies with this focus are lacking, whereas studies addressing developmental and aging cohorts are plentiful.

Investigating the executive abilities of healthy and economically active adult populations seems to be quite challenging. Among the reasons for this, the most important is that this population has time constraints to participate in scientific research. Furthermore, if they are willing to participate, they may resist the constraints of being assessed in person, along with the associated logistical disruptions. With respect to this issue, just as computerized and online tests, including serious digital games, have proven to be more favorable² for older adults (21,30–32), they would most likely be equally suitable for this segment of the population.

Considering this scenario, the present review aimed to examine the available data on executive function assessment tools (EFATs) for healthy adults with respect to the following: assessed constructs; reliability; validity; and other psychometric properties. Additionally, this review seeks to pinpoint the most employed EFATs per EF component in research, specifying those already adapted for the Brazilian population and those suitable for online administration. Therefore, the two main research questions are as follows: 1) What data are available on the digital EFAT for healthy adults regarding the following criteria: assessed construct, feasibility, reliability, validity, and psychometric properties? and 2) Which EFATs have been most commonly used in recent research, have Brazilian versions, and are suitable for online administration?

Finally, the answers to these questions will enable the author to select a suitable EFAT for a future evaluation of executive abilities in a cohort of federal public managers. This evaluation is a critical component of a larger project at the Laboratory for Cognitive and Behavioral Neuroscience at the Federal University of Rio Grande do Sul in Brazil.

The current review was meticulously organized and structured according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; 33). The authors aimed to provide a smoother and more comfortable reading experience.

Transparency and openness

To reduce bias and promote methodological quality, this review was registered with the International Prospective Register of Systematic Reviews (PROSPERO) before database searches were conducted (registration number CRD42023459513³). Please note that the title of this study differs slightly from the one registered on PROSPERO, as it was modified to accommodate the reviewers' suggestions.

Eligibility criteria

Inclusion criteria

The inclusion criteria for the database search results considered studies demonstrating statistically significant psychometric results for the investigated EFAT, specifically regarding validity, reliability, and factor analysis. Only studies providing evidence of validity—through correlation coefficients or effect sizes—were included. Reliability was assessed via intraclass correlation coefficients, Cronbach’s alpha, and correlation coefficients. For factor analysis, studies were evaluated on the basis of chi-square indices, the root mean square error of approximation (RMSEA), the confirmatory factor index (CFI), and the standardized root mean square residual (SRMR). Additionally, studies utilizing item response theory (IRT) were considered if they reported parameters of discrimination, difficulty, guessing, and ability.

Exclusion criteria

The reasons for exclusions were classified into four categories: wrong population (Studies targeting normative psychometric analyses of non-Brazilian populations, those with samples primarily composed of individuals with illnesses, and those involving children or adolescents); non-EF tests (Studies involving tests measuring cognitive functions other than executive functions); no psychometrics (Studies lacking psychometric property assessments); and wrong publication type (Studies such as systematic reviews/meta-analyses, case reports/case series, experiential/opinion pieces, and those published in nonindexed journals). Studies published in languages other than English or Portuguese were also excluded.

Data synthesis

The extracted data were evaluated for consistency, followed by tabulation and narrative discussion. A qualitative synthesis focused on cognitive domains associated with EF, organizing evidence from included studies in a tabular format, detailing the investigated EFAT, cognitive domains, sample characteristics, assessed psychometric properties, and availability of Brazilian-adapted or online versions of the tools.

Information source and search strategy

The search was performed on September 13, 2023, covering the period from 2013--2023, across the following databases: Embase, PsycNet, PubMed, Web of Science, and VHL (Virtual Health Library). This time frame was chosen because of the existence of a similar review in 2016 (1) and significant advancements in the field over the past decade. Additionally, to ensure comprehensive coverage of the literature, the reference lists of relevant systematic reviews and other studies for which full-text articles were retrieved were manually searched to identify any additional relevant studies without time restrictions.

The search strategy was structured as follows: ('executive function assessment' OR 'neuropsychological assessment') AND 'executive function' AND 'healthy adults' AND 'psychometric properties'. The search across all the databases included relevant synonyms for each term (sourced from Thesaurus, Emtree, and MeSH), with detailed information available in Appendix A. Notably, various descriptors from different databases were consolidated for general applicability across all the databases queried.

Study selection process

After the search was conducted across the databases, the studies were exported to the web app RAYYAN (34) for data analysis and synthesis of the primary studies. The following workflow was adopted: 1) Duplicate studies were removed by the primary author (T.N.T.); 2) titles and abstracts were reviewed (in a blinded manner) by the primary author (T.N.T.) and an independent reviewer (Gabriel Dias Curra) to exclude noneligible studies; 3) initial conflicts were resolved between the primary and second authors; and 4) the third author (R.M.M.A.) 5) full-text assessments were conducted by the primary author to exclude studies not meeting the criteria (reasons for exclusions were documented at this stage for the PRISMA flow diagram); 6) the second author (J.B.S.) reviewed exclusions at the full-text stage (in a blinded manner); 7) the third author (R.M.M.A.) resolved any disagreements; and 8) the PRISMA flowchart was generated to visually represent the study selection process.

For the manual search of gray literature, the same workflow was adopted. RAYYAN was employed to facilitate the categorization and interpretation of the included studies and generate contextually relevant summaries (34).

Data collection process and synthesis methods

The data were collected from the included studies by the primary author (T.N.T.). In cases of doubt or inconsistencies, clarification was sought from coauthors (J.B.S. and R.M.M.A.). NVivo software (version 1.7.1) was used to streamline the data organization, coding, annotation, querying, and visualization.

After completing data collection, a synthesis spreadsheet was generated in NVivo, including fields for authors, publication year, sample characteristics, assessment tool, assessed construct, validity (Boolean Y/N), reliability (Boolean Y/N), availability of a Brazilian version (Boolean Y/N), and availability of an online version (Boolean Y/N). This spreadsheet was shared with coauthors for assessment and suggestions.

NVivo also facilitated the narrative synthesis of findings through thematic analysis, covering aspects such as title and objective, theoretical discussions on EF and EFAT, statistical methods, psychometric results, critical reflections, and limitations. This enabled qualitative comparisons among the included studies.

In addition, another spreadsheet was manually created via Excel to register all EFATs utilized in the included studies (available in the supplementary material⁴). For each article, the spreadsheet recorded the following data: name of the EFAT, associated batterie, the assessed construct(s), and the complete reference of each test. This dataset served to provide insights into the frequency of test/battery usage relative to the construct assessed.

The heterogeneity among the reviewed studies is heterogeneous by their very nature, as they evaluated different EFATs. However, there was heterogeneity in the methodology used in each study for the evaluation of similar psychometric properties (e.g., studies that evaluated the test-retest reliability of an instrument opted for different methods, such as split-half and Parson’s and Spearman’s correlations). In this context, with respect to the issue of heterogeneity, these considerations will be considered in the results section.

Study risk-of-bias assessment

The initial evaluation of study quality was conducted by the primary author (T.N.T.) via the QUADAS-II tool (35), which was adapted for the specific focus of this systematic review. To tailor the QUADAS-II to the subject of this review, the signaling questions in each domain were adjusted accordingly (see Appendix B for further details).

Each included study was evaluated on the following criteria: low risk, assigned when all signaling questions within a domain are clearly answered and no concerns about bias are identified; high risk, given that the study’s methods or procedures may introduce bias, such as inappropriate participant selection; unclear risk, assigned when there is insufficient information to make a judgment about the risk of bias; and not applicable, applied when the domain or criterion does not pertain to the study being assessed.

After the initial assessment, the results were reviewed by the second author (J.B.S.). Any discrepancies were resolved with the assistance of the third author (R.M.M.A.). A continuous check-and-balance process was followed to ensure the accuracy of the study assessments.

Regarding the potential risk of bias from missing results, the authors carefully evaluated studies that omitted results to contextualize the missing data. Findings from this analysis are presented in subsequent sections.

Study selection

The PRISMA flow diagram for this review is presented in Fig. 1. The search across the five databases identified 3330 records. After deleting duplicate records, 2737 titles and abstracts were independently screened. A total of 2695 exclusions resulted from nonconformity with the eligibility criteria. The remaining 42 studies were retrieved, most directly from open sources and some from the primary authors (through e-mail). After the full-text assessment of these 42 selected studies, 17 were additionally excluded because of exclusion criteria (see Fig. 1), leaving 25 studies from the searched databases to be included in the current review.

Additionally, the manual search of other sources identified 20 records, all of which were retrieved directly from open sources and primary authors. After screening the abstracts and results, 16 studies were excluded according to the exclusion criteria, resulting in 4 studies for this review. Therefore, a total of 29 studies were included in this review.

Table 1 Study characteristics

Study	Country	EFAT	N (F)	M_AGE(SD)	M_EDU(SD)
Kruger et al. (42)	Brazil	Executive Function Scale for Adults (EFSA)	484 (376)	37.4 (16.4)	-
Scarfo et al. (13)	Australia	Andersons’s pediatric model.	133 (91)	29.7 (7.5)	-
Wang et al. (30)	China	“Fisherman” serious game.	108 (73)	66.8 (4.1)	11.9 (2.8)
Arioli et al. (40)	United States	Remote Characterization Module (RCM).	40 (23)	74.5 (6.5)	17.3 (1.3)
Karlsen et al. (43)	Norway	Cambridge Neuropsychological Test Automated Battery (CANTAB).	75 (30)	32.2 (13.1)	14.0 (2.4)
Ott et al. (44)	United States	National Institute of Health Cognitive Battery (NIH-CB).	83 (19)	44.3 (13.4)	16.4 (2.1)
Park & Schott. (31)	Germany	Digital version of the Trail Making Test (dTMT).	53 (31)	42.2 (22.8)	17.7 (4.7)
Wahyuningrum et al. (18)	Indonesian	Indonesian Neuropsychological Test Battery (INTB).	490 (294)	-	14.0 (2.8)
Heled et al. (45)	Israel	“Tactual Span”.	140 (70)	24.8 (3.0)	13.9 (1.3)
Pires et al. (46)	Portugal	Various EF tests (Set 1)	90 (76)	19.8 (2.8)	13.1 (1.5)
White et al. (47)	New Zealand	Various EF tests (Set 2).	30 (0)	68.1 (1.7)	-
Cotrena et al. (48)	Brazil	Melbourne Decision Making Questionnaire (MDMQ).	234 (101)	28.7 (11.2)	14.9 (4.4)
Feenstra et al. (49)	Netherlands	Amsterdam Cognition Scale.	248 (157)	49.1 (12.9)	-
Parsons & Barnett (39)	United States	Virtual apartment-based Stroop test.	89 (72)	44.2 (28.1)	-
Rijnen et al. (50)	Netherlands	Central Nervous System Vital Signs.	158 (90)	45.9 (14.4)	16.9 (3.3)
Soveri et al. (51)	Finland	Various EF tests (Set 3).	37 (-)	23.0 (2.4)	-
Ishigami et al. (37)	Canada	Attention Network Test-Interaction (ANT-I).	173 (91)	65.4 (6.5)	15.5 (2.7)
Kaller et al. (38)	Germany	Tower of London Freiburg version (TOL-F).	4600 (2292)	57.0 (13.6)	-
Cardoso et al. (52)	Brazil	Hotel Task (HT)	-	-	-
Godoy et al. (53)	Brazil	Barkley Deficits in Executive Functioning Scale (BDEFS).	60 (39)	27.3 (12.3)	-
Köstering et al. (54)	Germany	Tower of London Freiburg version (TOL-F)	27 (15)	22.6 (1.8)	-
Malloy-Diniz et al. (55)	Brazil	Brazilian version of the Barratt Impulsiveness Scale BIS-11_BV.	3053 (1852)	31.7 (11.8)	-
Heaton et al. (56)	United States	National Institute of Health Cognitive Battery (NIH-CB).	268 (149)	52.3 (21.0)	13.4 (2.9)
Troyer et al. (32)	Canada	Online EF tests tool (Set 4).	396 (202)	65.0 (8.2)	-
Aalbers et al. (57)	Netherlands	Brain Aging Monitor-Cognitive Assessment Battery (BAN-COG).	397 (312)	54.9 (9.6)	-
Brunner et al. (58)	Norway	P3 No-Go wave.	26 (10)	27.5 -med	-
Kang et al. (41)	United States	Kaplan version of the Stroop test.	153 (63)	70.2 (8.0)	16.8 (2.6)
Beato et al. (59)	Brazil	Brazilian version of the Frontal Assessment Battery (FAB_BV).	275 (163)	66.4 (10.6)	8.9 (5.1)
Dubois et al. (36)	France	Frontal Assessment Battery (FAB).	163 (-)	62.8 (11.1)	-

Note. F = female; M_AGE = mean age (years); M_EDU = mean education (years); SD = standard deviation; med = median; Set 1: Working Memory, Tower, Divided Attention, Stroop, Verbal Fluency, Word List, Confrontation Naming tests, Coding and Telephone tests; Set 2 = Simon, Visuoverbal and Visuospatial N-back, Letter-memory and the Number–letter tasks; Set 3: Pro, Anti and Pro/Anti, Simon, Flanker, Forward and Backward Spatial, and 2-back tests; Set 4: Spatial Working Memory, Stroop Interference, Face-Name Association, and Number-Letter Alternation tasks.

Finally, it is also important to highlight that in both the database and manual search processes, after conducting a thorough review of the articles (42 from the databases and 20 from the manual search), a discussion among the authors was imperative before deciding on each exclusion and the reason for it.

Study characteristics

The characteristics of each study are presented in Table 1. They comprise the study reference, the country of origin, the evaluated EFAT (single test or test battery), and the sample characteristics.

The studies were conducted on nearly every continent except Africa, with the majority in Europe (10 studies), North America (7 studies), and South America (6 studies in Brazil). Asia, with 4 studies, and Oceania, with 2 studies, are also represented in this review.

Most studies detailed sample characteristics, including the number of participants, mean age, and mean education level, along with their respective standard deviations (see Table 1). For studies that involved more than one sample (36–39), the weighted means and standard deviations were calculated and are presented in Table 1.

Furthermore, the samples utilized in all studies comprised healthy, economically active participants, with mean ages ranging from 19.8 to 74.5 years. Although only six studies specifically targeted older individuals (mean age ≥ 65 years), even within these studies, a portion of the participants belonged to the economically active age bracket. Hence, all the studies precisely met the criteria for the target population of the current review.

Risk of bias in the reviewed studies

Biases in each reviewed study were assessed via the adapted version of the QUADAS-II tool (35), which is provided in Appendix B. The detailed results for each study can be found in Table 2, whereas the results per domain are summarized in Figure 2.

In the domain of participant selection, almost all studies did not employ random selection methods. Additionally, in six studies, information regarding participant selection was inadequately documented. Convenience sampling emerged as a prevalent method in these studies, posing a significant risk of bias.

Apart from three out of 29 studies, the risk of bias for the index test domain (referred to as the EFAT in this review) appeared to be acceptable. However, in the study by Arioli et al. (40), the risk of bias was deemed high for two reasons. First, the results of the novel EFAT (Remote Characterization Module) were evaluated with prior knowledge of the results from conventional tests (reference standard) administered to the same participants before recruitment. Second, thresholds for reliability and validity were not predefined, which is crucial when assessing novel EFATs. Similarly, in the study by Kang et al. (41), no thresholds or reference values were provided for assessing the half-split reliability of the Kaplan version of the Stroop test. Nevertheless, the author classified the results for split-half reliability as good or high (without citing the reference to the literature).

In the domain of the reference standard, 11 out of 29 studies focused exclusively on assessing the test‒retest reliability of the EFAT under investigation, thus not utilizing a reference standard. In these instances, the risk of bias was categorized as not applicable (N/A).

Furthermore, two studies (37,42) lacked sufficient information regarding the use of a reference standard. Ishigami et al (37) failed to provide details on when and how participants underwent EF tests and measures, which served as the reference standard. Similarly, Kruger et al. (42) did not adequately justify their selection of the Dysexecutive Questionnaire (DEX) as the reference standard for evaluating the validity of their novel Executive Function Scale for Adults (EFSA). Clarifying these aspects would enhance the comprehensiveness and validity of their studies.

Table 2 Risk of bias in the reviewed studies

In the domain of flow and timing, 20 out of 29 studies were assessed as having a low risk of bias, as there was no delay observed between the administration of the index test (EFAT) and the reference standard (conventional gold standard EFAT). Conversely, four studies (18,36,55,56) lacked sufficient information to assess the risk of bias. This deficiency stemmed from the inadequate provision of details concerning the guidance provided to participants during test execution, as well as the uniformity of procedures across all participants, including whether they underwent both the index test and reference standard. Finally, five studies (32,38,47,50,51) were deemed to have a high risk of bias due to inconsistencies in the testing procedures administered to participants.

Concerning applicability, within the domain of the index test (the investigated EFAT), all studies were evaluated as having a low risk of bias. In contrast, regarding participant selection, six studies lacked sufficient information for bias assessment. Finally, within the domain of the reference standard, no studies were determined to exhibit a high level of bias. Moreover, 11 studies were deemed not applicable for bias assessment because they did not utilize reference standards, whereas the remaining studies demonstrated a low level of bias.

Finally, in the final study selection process, during the comprehensive article review stage, rigorous quality control (considering the robustness of each study's method and results, as well as the presentation of their limitations) was employed by both independent reviewers, resulting in a significant reduction in the original number of records (see Figure 1). Therefore, the outcome of the analysis of the included studies' quality (Tables 2 and Figure 3) demonstrated a relatively acceptable level of bias.

Results of the reviewed studies

The 29 studies reviewed in this analysis examined the psychometric properties of various EFATs for assessing EF, including both existing and newly developed measures. The respective EF components evaluated by each tool are outlined in Table 3a, while Table 3b presents an overview of the investigated psychometric properties.

Table 3a Assessed Components for each EFAT

Study	EFAT	Assessed EF components
Kruger et al. (42)	Executive Function Scale for Adults (EFSA)	Inhibition, working memory, and cognitive flexibility.
Scarfo et al. (13)	Andersons’s pediatric model.	Attentional Control, Cognitive Flexibility, Information Processing, and Goal Setting.
Wang et al. (30)	“Fisherman” serious game.	Inhibition, shifting, and working memory.
Arioli et al. (40)	Remote Characterization Module (RCM).	Verbal memory, language fluency, working memory span, and set shifting.
Karlsen et al. (43)	Cambridge Neuropsychological Test Automated Battery (CANTAB).	Visual learning and memory, executive function, and visual attention.
Ott et al. (44)	National Institute of Health Cognitive Battery (NIH-CB).	Cognitive flexibility, attention and executive function, episodic memory, working memory, processing speed, language, motor dexterity, and fluid composite
Park & Schott. (31)	Digital version of the Trail Making Test (dTMT).	Perceptual speed, visual sequencing ability, mental flexibility, visual working memory, inhibition.
Wahyuningrum et al. (18)	Indonesian Neuropsychological Test Battery (INTB).	Working memory, cognitive flexibility, divided attention, inhibition, verbal fluency, short-term visual-spatial memory, and sustained attention.
Heled et al. (45)	“Tactual Span”.	Working memory.
Pires et al. (46)	Various EF tests (Set 1)	Inhibition, planning, working memory, divided attention, verbal fluency, episodic memory, confrontation naming, and processing speed.
White et al. (47)	Various EF tests (Set 2).	Inhibition (Anti), response switching (Pro/Anti), selective attention (Simon and Flanker), and working memory (2-back).
Cotrena et al. (48)	Melbourne Decision Making Questionnaire (MDMQ).	Decision making (vigilance, hypervigilance, buck-passing and procrastination).
Feenstra et al. (49)	Amsterdam Cognition Scale.	Attention, information processing, learning and memory, executive functioning, and psychomotor speed.
Parsons & Barnett (39)	Virtual apartment-based Stroop test.	Inhibition, cognitive flexibility, and attentional control.
Rijnen et al. (50)	Central Nervous System Vital Signs.	Verbal memory, visual memory, processing speed, psychomotor speed, reaction time, complex attention, and cognitive flexibility.
Soveri et al. (51)	Various EF tests (Set 3).	Inhibition, working memory, and set shifting (cognitive flexibility).
Ishigami et al. (37)	Attention Network Test-Interaction (ANT-I).	Attentional Control (alerting, orienting and executive control network).
Kaller et al. (38)	Tower of London Freiburg version (TOL-F).	Planning ability.
Cardoso et al. (52)	Hotel Task (HT)	Planning, organization, self-monitoring and cognitive flexibility.
Godoy et al. (53)	Barkley Deficits in Executive Functioning Scale (BDEFS).	Self-management of time, self-organization/problem solving, self-restraint, self-motivation, and self-regulation of emotion.
Köstering et al. (54)	Tower of London Freiburg version (TOL-F)	Planning ability.
Malloy-Diniz et al. (55)	Brazilian version of the Barratt Impulsiveness Scale BIS-11_BV.	Attention, inhibition, motor and nonplanning impulsiveness.
Heaton et al. (56)	National Institute of Health Cognitive Battery (NIH-CB).	Cognitive flexibility, attention and executive function, episodic memory, working memory, processing speed, language, motor dexterity, and fluid composite.
Troyer et al. (32)	Online EF tests tool (Set 4).	Working memory, inhibition, cognitive flexibility and processing speed.
Aalbers et al. (57)	Brain Aging Monitor-Cognitive Assessment Battery (BAN-COG).	Working memory (“Conveyer Belt”), visuospatial short-term memory (“Sunshine”), episodic recognition memory.
Brunner et al. (58)	P3 No-Go wave.	Inhibition (P3NoGo Early) and monitoring of actions (P3 NoGo Late).
Kang et al. (41)	Kaplan version of the Stroop test.	Inhibition, selective attention, and processing speed.
Beato et al. (59)	Brazilian version of the Frontal Assessment Battery (FAB_BV).	Conceptualization, mental flexibility, motor programming, sensitivity to interference, inhibitory control, and environmental autonomy.
Dubois et al. (36)	Frontal Assessment Battery (FAB).	Conceptualization, mental flexibility, motor programming, sensitivity to interference, inhibitory control, and environmental autonomy.

Note. EFAT: Executive Function Assessment Tools; Set 1: Working Memory, Tower, Divided Attention, Stroop, Verbal Fluency, Word List, Confrontation Naming, Coding and Telephone tests; Set 2: Pro, Anti and Pro/Anti, Simon, Flanker, Forward and Backward Spatial, and 2-back tests; Set 3 = Simon, Visuoverbal and Visuospatial N-back, Letter-memory and the Number–letter tasks; Set 4: Spatial Working Memory, Stroop Interference, Face-Name Association, and Number-Letter Alternation tasks.

Table 3b Assessment of Psychometric Properties in the Reviewed Studies

Study	EFAT	Validity	Reliability	FA	Norms
Kruger et al. (42)	Executive Function Scale for Adults (EFSA)	Crit.	IC	CFA, EFA	No
Scarfo et al. (13)	Andersons’s pediatric model.	Predictive	No	No	No
Wang et al. (30)	“Fisherman” serious game.	Crit.	IC, S-H	CFA, EFA	No
Arioli et al. (40)	Remote Characterization Module (RCM).	Constr.	Intermeth	No	No
Karlsen et al. (43)	Cambridge Neuropsychological Test Automated Battery (CANTAB).	No	T-Rt	No	No
Ott et al. (44)	National Institute of Health Cognitive Battery (NIH-CB).	Constr.	No	No	No
Park & Schott. (31)	Digital version of the Trail Making Test (dTMT).	Constr.	IC	No	No
Wahyuningrum et al. (18)	Indonesian Neuropsychological Test Battery (INTB).	No	IC, T-Rt	PCA	Age-Edu related
Heled et al. (45)	“Tactual Span”.	Constr.	IC	EFA	No
Pires et al. (46)	Various EF tests (Set 1)	No	No	CFA	No
White et al. (47)	Various EF tests (Set 2).	No	T-Rt	No	No
Cotrena et al. (48)	Melbourne Decision Making Questionnaire (MDMQ).	Constr., Crit.	IC	CFA	No
Feenstra et al. (49)	Amsterdam Cognition Scale.	No	T-Rt	No	Equating
Parsons & Barnett (39)	Virtual apartment-based Stroop test.	No	No	No	No
Rijnen et al. (50)	Central Nervous System Vital Signs.	No	T-Rt	No	No
Soveri et al. (51)	Various EF tests (Set 3).	No	T-Rt	No	No
Ishigami et al. (37)	Attention Network Test-Interaction (ANT-I).	Constr., Crit.	S-H	No	No
Kaller et al. (38)	Tower of London Freiburg version (TOL-F).	No	S-H, T-Rt	No	No
Cardoso et al. (52)	Hotel Task (HT)	No	No	No	No
Godoy et al. (53)	Barkley Deficits in Executive Functioning Scale (BDEFS).	Constr., Crit.	IC	No	No
Köstering et al. (54)	Tower of London Freiburg version (TOL-F)	No	T-Rt	No	No
Malloy-Diniz et al. (55)	Brazilian version of the Barratt Impulsiveness Scale (BIS-11_BV).	No	IC	No	Percentiles
Heaton et al. (56)	National Institute of Health Cognitive Battery (NIH-CB).	Constr., Crit.	IC, T-Rt	No	No
Troyer et al. (32)	Online EF tests tool (Set 4).	Constr.	IC, S-H, T-Rt	PCA	Age-related
Aalbers et al. (57)	Brain Aging Monitor-Cognitive Assessment Battery (BAN-COG).	Constr.	IC, AFR	No	No
Brunner et al. (58)	P3 No-Go wave.	No	T-Rt	No	No
Kang et al. (41)	Kaplan version of the Stroop test.	Constr.	S-H	No	Age-related
Beato et al. (59)	Brazilian version of the Frontal Assessment Battery (FAB_BV).	No	No	No	Percentiles
Dubois et al. (36)	Frontal Assessment Battery (FAB).	Constr., Crit.	Interrater	No	No

Note. EFAT: Executive Function Assessment Tools; FA: Factor Analysis; Crit. : criterion; IC: internal consistency; CFA: confirmatory factor analysis; EFA: exploratory factor analysis; S-H: split-half; Constr. : construct; Intermeth: intermethod; T-Rt: test-retest; PCA: principal component analysis; Age-Edu: Age and education; Set 1: Working Memory, Tower, Divided Attention, Stroop, Verbal Fluency, Word List, Confrontation Naming, Coding and Telephone tests; Set 2: Pro, Anti and Pro/Anti, Simon, Flanker, Forward and Backward Spatial, and 2-back tests; Set 3: Simon, Visuoverbal and Visuospatial N-back, Letter-memory and the Number–letter tasks; Set 4: Spatial Working Memory, Stroop Interference, Face-Name Association, and Number-Letter Alternation tasks; AFR: Alternate Form Reliability.

Notably, two studies (39,52) diverge from this focus. Parsons and Barnett (39) investigated a virtual reality adaptation of the Stroop test (referred to as the virtual apartment-based Stroop), comparing its performance with both the Automated Neuropsychological Assessment Metrics (ANAM) Stroop test and the Paper-and-Pencil (P&P) color-word interference test from the Delis-Kaplan Executive System battery (D-KEFS). Conversely, Cardoso et al. (2015) conducted only a transcultural adaptation of the hotel task into Brazilian Portuguese. Despite not specifically examining psychometric properties, these studies were included in the current review because of their relevance to the objectives of the present study.

Regarding the frequency of tests administered in the studies, a total of 199 administrations were conducted in the 29 reviewed studies, covering a total of 11,386 healthy adult participants. Most tests employed, which serve to assess the validity of the target instruments investigated in the articles, belong to established batteries of neuropsychological assessments, including the Wechsler scales (Wechsler Adult Intelligence Scale and Wechsler Memory Scale, the Delis-Kaplan Function System (D-KEFS), and the Cambridge Neuropsychological Test Automated Battery (CANTAB), among others).

In the following sections, the most assessed EF components and associated EFATs, as well as the frequency of batteries of EFATs employed in the reviewed studies, are presented. The main findings from the 29 studies are subsequently presented and organized into two distinct categories for clarity in data presentation: novel EFATs (13 studies in total) and existing EFATs (16 studies in total). In the first section, the new tools (or batteries) are introduced, followed by the presentation of the results from their psychometric analyses. In the second section, the results of the psychometric analysis of the existing EFAT are presented exclusively since they are well renowned by the scientific community. Finally, considerations about results bias are also presented for each study.

The most commonly assessed EF components and associated EFATs in the reviewed studies

Table 4 highlights the most employed tests as well as the EF components, which most represented the targets in the reviewed studies. The EFAT investigated in the reviewed studies focused on assessing the three primary components of EF: inhibition (21 studies), working memory (18 studies), and cognitive flexibility (16 studies). The associated tests most employed were the Stroop test in several versions: the digit span test, which is mostly from the Wechsler scale (60), and the Trail Making Test.

Table 4 Frequency of Research on EF Components and Associated EFATs in the Reviewed Studies

EF Component	No. of studies (% of the total)	Most EFAT (No. of administrations)
Inhibition	20 (69.0%)	Stroop (10), TMT-B (4), Go/NoGo (3)
Working memory	18 (62.1%)	WAIS Digit Span (11), WAIS Word list (7), WAIS Number-Letter (6)
Cognitive flexibility	16 (55.2%)	TMT (8), WAIS Number-Letter (6), Stroop (2), WCST (2),
Processing speed	15 (51.7%)	Stroop (10), WAIS Digit Span (11), WAIS Digit Symbol Coding (5)
Attentional control	13 (44.8%)	WAIS Digit Symbol Coding (5)
Verbal Fluency	12 (41,4%)	Verbal Fluency (12)
Planning	10 (34.5%)	Tower of London (3)
Episodic memory	8 (27.6%)	WAIS Word list (7)
Psychomotor abilities	4 (20.7%)	FAB Motor Series (2)

Note. EFAT: Executive function assessment tool; most studies administered more than one test for the same component. EF: Executive Functions; TMT-B: Trail Making Test Part B; TMT: Trail Making Test; WCST: Wisconsin Cards Sorting Test; FAB: Frontal Assessment Battery; WAIS: Wechsler Adult Intelligence Scale.

Some studies have examined the role of these components with more complex executive processes, such as planning and problem-solving (18,38,49,52,54,57,58), decision-making (18,41,48), and abstract reasoning (13,37). Interestingly, the study by Scarfo et al. (2023) examined the influence of EF components in goal setting, a construct that comprises initiative, abstract reasoning, planning, and strategic organization.

Furthermore, several batteries were administered in the reviewed studies, as described in the next section.

Batteries of EFAT employed in the reviewed studies

In the reviewed studies, researchers utilized tests (or subscales) from various standardized batteries, demonstrating the significance of these batteries for EF assessment in neurotypical adults. Table 5 illustrates the results. The spreadsheet containing detailed data from all tests conducted in the reviewed studies is available in the supplementary material.

The most frequently employed test battery in the reviewed studies was the Wechsler Adult Intelligence Scales (WAIS), which was used in eight studies (13,30,40,41,45,46,56,57). The second and third most frequently used batteries were tied, with both cited in four studies: the Wechsler Memory Scales (WMS; 41,44,46,57) and the Delis–Kaplan Executive Function System (D-KEFS; 37,39,46,56).

Table 5 Frequency of test batteries used in the reviewed studies

Test battery	Frequency (n° studies)
Wechsler Adult Intelligence Scales (WAIS-R, WAIS-III, WAIS-IV, and WASI)	8
Wechsler Memory Scales (WMS-R, and WMS-III)	4
Delis-Kaplan Executive Function System (D-KEFS)	4
Behavioral Assessment of Dysexecutive Syndrome measures (BADS)	2
Cambridge Neuropsychological Test Automated Battery (CANTAB)	2
Frontal Assessment Battery (FAB)	2
National Institute of Healthy Toolbox Cognition Battery (NIHTB-CB)	2
Amsterdam Cognition Scan (ACS)	1
Army Individual Test Battery	1
Automated Neuropsychological Assessment Metrics (ANAM)	1
Brain Aging Monitor–Cognitive Assessment Battery (BAM-COG)	1
Central Nervous System Vital Signs (CNS VS)	1
Cognitive Test Battery (CTB)	1
FISHERMAN Serious Game	1
Indonesian Neuropsychological Test Battery (INTB)	1
Psycholinguistic Assessment of Language battery (PAL 09)	1
Test of Everyday Attention battery (TEA)	1

Note. WAIS-R, WAIS-III, and WAIS-IV are different versions of the Wechler Adult Intelligence Scale; WASI: Wechsler Abbreviated Scale of Intelligence; WMS-R and WMS-III are different versions of the Wechsler Memory Scale.

Other batteries, including the Behavioral Assessment of Dysexecutive Syndrome (BADS), the Cambridge Neuropsychological Test Automated Battery (CANTAB), the Frontal Assessment Battery (FAB), and the National Institutes of Health Toolbox Cognition Battery (NIHT-CB), were each employed in two studies. The remaining batteries, some of which are newly developed (described in the section below), were used in only one study each.

Novel EFAT

Among the 29 studies, 13 presented new EFATs, two of which focused on different psychometric analyses of the same tool. These new tools are briefly described below. Table 6 shows the results of their investigated psychometric properties.

Table 6 Psychometric Properties of Studies on Novel EFATs

Study	Tool	Psychometric properties
Kruger et al. (42)	EFSA	Reliability: acceptable to excellent internal consistency for two components, WM and IC (Cronbach’s α = .90 and .79, respectively), and low for CF (α = .62.). Validity: significant correlation with the DEX self-report scale, with strong correlation for WM, moderate for IC and weak for CF (ρ = .71, .59, and .17, respectively). CFA/EFA: moderate correlation between factors, with strong correlation between WM e IC, and weak correlation between WM e CF and IC e CF as well.
Wang et al. (30)	Fisherman	Reliability: revealed good internal consistency of the Cautions, Agile, and Wise subgames (Cronbach’s α = .83, .89, and .80, respectively). The split-half reliability of all subgames was acceptable to good (.83, .88, and .77, respectively). Validity: all three subgames showed significant moderate correlations: 1) Cautions Fisherman with stop signal (r = .40), and Trail Making Test Part A (TMT– A; r = .38); 2) Agile Fisherman with Number switch task (r =.51), and TMT-A (r = .40); and 3) Wiseman Fisherman with Corsi-Block-Tapping task (r = .75), and Digit backward span (r =.41). Among the subgames, the Cautions Fisherman and the Agile Fisherman were modestly correlated (r = .45).
Arioli et al. (40)	RCM	Reliability: intermethod revealed significant correlations (p<.05) for verbal memory, language fluency, and set shifting (r = .45 to .61, indicating low reliability), but not for working memory span (p = .19 to .21). Validity: no significant differences between RCM and P&P in four tasks (Short- and Long-Delay Free Recall, Lexical fluency, and Modified Trail Making B); significant differences for the other four tasks (Total Immediate Recall, Semantic Fluency, and Verbal Digit Span Forward and Backward), being modest for Total Immediate Recall (d = .35), whereas the remaining tasks exhibited robust differences (d = 1.28, 1.19, and .53, respectively).
Park and Schott (31)	dTMT	Reliability: good to excellent (ICC between .90 e .95) for all three conditions TMT-M, TMT-A, and TMT-B. Validity: showed good agreement (ρ = .82 to .90) and equivalence as well between the P&P and digital TMT, since no significant difference was found for both, the difference (B-A) e ratio (B/A).
Wahyuningrum et al. (18)	INTB	Reliability: low to excellent reliability (ICC = .44 to .91), PCA revealed a well-fitting seven-factors: Factor 1 - speed of visuospatial information processing and planning, Factor 2 - attention, auditory short-term, and working memory, Factor 3 - executive internal language, Factor 4 - visual cued semantic processing, Factor 5 speed and inhibitory control, Factor 6 - recall ability, and Factor 7 - learning ability. Normative: age demonstrated a significant effect (p > .01) on six out of the seven factors, except for factor 3 (p = .754); education showed significant influence on five factors, but not on factors 6 and 7.
Heled et al. (45)	Tactual Span	Reliability: Cronbach’s α for correct scores in all Tactual Span trials was .77 for the forward stage and .80 for the backward stage, indicating acceptable reliability. Validity: low to moderate convergent validity (ρ = .19 to .47) and good discriminant validity (no significant correlations between Tactual Span and selective attention and semantic fluency tasks). EFA: revealed two factors for the forward stage (F1: visual and auditory spans and F2: visuospatial and actual spans) and just one factor for the backward stage comprising all span task. This is aligned with the Baddeley’s model (8).
Feenstra et al. (49)	ACS	Reliability: low to acceptable test-retest reliability (ICC = .45 to .80 and .83 for the ACS total score). Multiregression analysis: significant effects of age, gender, and education on total ACS scores, with standardized β coefficients of -.597, .111, and -.112, respectively.
Parsons and Barnet (39)	VABS	Convergent validity: the test proved to be valid for evaluating cognitive interference control, with much more challenging than in traditional P&P tests without distractors.
Ishigami et al. (37)	ANT-I	Reliability: all three networks was significant (p<,05): alerting (r = .29), orienting (r = .70), and executive control (r = .68), but indicating low to acceptable reliability. Construct validity: each network score was significant (p < .01) and independent from each other. Criterion validity: executive network was a significant predictor of performance on conflict resolution and verbal memory and retrieval (β = −.165 and −.184, p’s < .05, respectively).
Kaller et al. (38)	TOL-F	Reliability: acceptable reliability for all estimates (λ₂, λ₃, λ₄, ω_tot, and glb) in both samples, Mainz (.713 - .755) and Vienna (.656 - .730). The estimate glb converged for both samples (.755 for Mainz and .730 for Vienna), as did the estimate λ4 (.743 for Mainz e .716 for Vienna).
Köstering et al. (54)	TOL-F	Reliability: planning accuracy test-retest for relative consistency (r = .734 to .739) as well as absolute agreement (r = .690) were acceptable; planning latencies, however, showed low reliability for both, relative consistency, and absolute agreement (r = .274 to .519).
Troyer et al. (32)	Online EF tests tool (Set 4)	Feasibility: 87% of participants successfully completed the tool, with 94% of completed tests yielding results within the anticipated range. Reliability: split-half reliability was low (0.62); high internal consistency (Cronbach’s α = .96); low to moderate test-retest reliability (r = .49 to .83 for individual tasks, and r = .72 for overall score); and finally, low to acceptable alternate version reliability (r = .48 to .82 for individual tasks, and r = .69 for overall score). Validity: for construct, significant correlation between age and target measures was low to moderate, with r₍₃₉₄₎ = -.20 to .31; for convergent, the intertask correlations of the target measures were low to moderate, with r₍₃₉₄₎ = .27 to .30. PCA: conservatively identified a single factor solution, but there was evidence also for two-factor solution in the context of speeded responding.
Aalbers et al. (57)	BAN-COG	Reliability: AFR was low with ICC = .420, .426, and .645 for the Conveyer Belt, Sunshine and Papyrinth games, respectively, but not for the Viewpoint game (ICC = .167). Convergent construct validity: moderate (ρ = .400 to .669), but not for the Viewpoint subgame. Divergent construct validity: good for all subgames (ρ < .200 with unrelated cognitive measure).

Note. EFSA = Executive Function Scale for Adults; WM = Working Memory; IC = Inhibitory Control (or inhibition); CF = Cognitive Flexibility; DEX = Dysexecutive Questionnaire; ρ = Spearman correlation coefficient; CFA = Confirmatory Factor Analysis; EFA = Exploratory Factor Analysis; r = Pearson correlation coefficient; TMT-A = Trail Making Test Part A; RCM = Remote Characterization Module; P&P = Paper-and-Pencil; d = Cohen’s effect size coefficient; dTMT = Digital Trail Making Test; TMT-M = Tral Making Test Motor Speed; TMT-B = Trail Making Test Part B; INTB = Indonesian Neuropsychological Test Battery; ICC = Intraclass Coefficient; PCA = Principal Component Analysis; ACS = Remote Characterization Module; β = Standardized Regression Coefficient; VABS = Virtual Apartment-Based Stroop; ANT-I = Attention Network Test-Interaction; TOL-F = Tower of London Freiburg version; λ₂, λ₃, λ₄, glb, and ω_tot = reliability indicators to assess internal consistency; Set 4: Spatial Working Memory, Stroop Interference, Face-Name Association, and Number-Letter Alternation tasks; BANCOG = Brain Aging Monitor-Cognitive Assessment Battery. AFR: Alternate Form Reliability.

The Executive Function Scale for Adults (EFSA) is a 27-item scale that assesses three EF components: working memory (WM), inhibitory control (IC) and cognitive flexibility (CF). Kruger et al. (42) investigated preliminary evidence of the validity of the EFSA and convergent validity on the basis of the relationship of the scale with external variables. Although the EFSA is the first developed exclusively for this population in the country, to the best of the authors’ knowledge, online implementation on a convenience scale may represent bias in the results.

The Fisherman game is a serious game consisting of three subgames: Cautions Fisherman, Agile Fisherman, and Wise Fisherman, each measuring inhibition, shifting, and working memory, respectively. Wang et al. (30) investigated the reliability, construct validity and criterion validity of the Fisherman and its subgames. The Fisherman method exhibits great promise as a tool for self-administered assessment and has shown high reliability. However, the absence of an explanation regarding the treatment of outliers in the results could pose a potential risk of bias.

The remote characterization module (RCM) is a digital cognitive screening tool devised by Arioli et al. (40). It comprises an application featuring eight tasks designed to assess verbal memory, language fluency, working memory span, and set shifting. Arioli et al. (2022) assessed RCM construct validity and intermethod reliability. They acknowledged that the lack of participants with lower levels of education limits the test's reliability. Additionally, the assumption of a normal distribution in the scores, without providing justification, raises concerns about potential bias in the findings, particularly given the small sample size.

The digital TMT (dTMT) is a computerized version of the Trail Making Test (TMT), which is a commonly used neuropsychological test for the assessment of visuomotor ability and mental flexibility, in addition to working memory and inhibition (61). In their study, Park and Schott (31) evaluated and compared the psychometric properties (reliability, equivalence, and agreement) of the P&P version of the TMT and the dTMT under three conditions: TMT-M (motor speed), TMT-A (Part A of the test), and TMT-B (part B of the test). Although the TMT displayed strong psychometric properties on the basis of thorough statistical analysis, the small sample size may have introduced bias into the results.

The Indonesian Neuropsychological Test Battery (INTB) consists of ten P&P tests targeting executive functions; reception and production of language; various types of learning and memory, both verbal and visuospatial; and attention and concentration. Wahyuningrum et al. (18) investigated the test-retest reliability of the INTB and performed a principal component analysis (PCA) to identify its cognitive construct factor structure and its respective age and education effects. While this study yields promising results for the performance of the INTB, it is important to acknowledge the overlooked aspect of how practice effects could influence test-retest reliability outcomes.

The Tactual Span, as a new modality of the WM test, measures working memory capacity; however, unlike the traditional WM test, it considers a third storage modality beyond the verbal and visuospatial storage proposed by (7,8). Heled et al. (45) investigated the reliability and construct validity of the Tactual Span. They also performed exploratory factor analysis (EFA) to examine whether Baddeley’s working memory (WM) model aligns with the multimodal span tests employed in the study. The results indicated that the Tactual Span emerged as a promising modality for assessing WM. However, it is important to note a potential risk of bias in the findings, given the homogeneous nature of the sample (composed mostly of undergraduate students).

The Amsterdam Cognition Scan (ACS) is an online self-administered neuropsychological test battery developed by Feenstra et al. (49) that measures attention, information processing, learning and memory, executive functioning, and psychomotor speed. They investigated the test-retest reliability of the RCM and established its regression-based demographically corrected normative data. The findings demonstrated a range of reliability from moderate to high. However, the presence of nonuniform conditions among the participants could introduce bias. For example, during the test, 40 out of the 235 participants were not alone during the test, which may have influenced the results.

The Virtual Apartment-Based Stroop (VABS) test is a VR variant of the Stroop test, where the Stroop stimulus is presented on television inside a virtual apartment (containing a kitchen and a window) with various distractors (auditory and visual). These distractors are positioned in different fields of view in the environment, such as a school bus passing by the street and visible from the window, a robot vacuum moving and generating noise on the floor, and a phone ringing on the table in front of the participant. Parsons and Barnet (39) evaluated the performance of the VABS compared with that of the ANAM and D-KEFS Stroop tests in older and young adults. Since the ANAM and D-KEFS Stroop tests do not include environmental distractors, there is a potential risk of bias in comparing the scores of interference inhibition tests where participants are under different conditions.

The attention network test-interaction (ANT-I) is a computerized variant of the attention network test (ANT; 62) proposed by Callejas et al. (63). The ANT target measures are the three attention networks (alerting, orienting and executive control) of the theoretical model by Petersen and Posner (64). Ishigami et al. (37) investigated the construct and criterion validity and split-half reliability of the ANT-I. This author did not include RT in the regression analyses of criterion validity, indicating that this variable could strengthen the identified relationship. However, the author could have used mediation/moderation analysis to determine the influence level of the RT.

The Tower of London Freiburg version (TOL-F) represents a computerized variant of the classic Tower of London task (TOL; 5), where participants engage with digital representations of colored balls on a computer screen to solve planning problems. Köstering et al. (54) delved into the test‒retest reliability of the TOL-F, employing three Iso-versions that maintained structural consistency while permuting ball colors to ensure novelty across repeated assessments. One potential concern regarding bias in the findings of this study is that the authors relied on a relatively small and homogenous sample consisting primarily of university students. Furthermore, Kaller et al. (38) provided insights into the test-retest and split-half reliability, as well as construct validity estimates of the TOL-F, drawing from large samples in Mainz, Germany, and Vienna, Austria.

The online EF test tool is a self-administered internet-based screening tool tailored for middle-aged and elderly individuals. Devised by Troyer et al. (32), the tool consists of four tests: spatial working memory, Stroop interference, face-name association, and number-letter alternation tasks. The authors investigated its feasibility, reliability, and construct validity and presented normative data. The authors aimed to create a self-assessment tool capable of informing an individual's decision regarding the need to seek healthcare services to evaluate potential memory deficits. They also recognized the risk of bias in their findings, given the myriad factors that can influence outcomes in an uncontrolled environment.

Finally, the Brain Aging Monitor-Cognitive Assessment Battery (BAM-COG) is an online tool consisting of four puzzle games: Conveyer Belt, Sunshine, Viewpoint, and Papyrinth, each of which assesses WM, visuospatial short-term memory, episodic recognition memory, and planning ability, respectively. Aalbers et al. (57) examined the reliability and construct validity of the BAM-COG. The author pointed very well to the risk of bias in the discussion section.

Preexisting EFAT

Among the 16 studies that focused on neuropsychological tests commonly used in the assessment of executive functions, three addressed cross-cultural adaptation of the Hotel Task (HT), the Melbourne Decision-Making Questionnaire (MDMQ), and the Barkley Deficits in Executive Functioning Scale (BDEFS) in Brazilian Portuguese (48,52,53). These studies are briefly described below.

Compared with the original version, the Brazilian-adapted version of the HT underwent several modifications to accommodate the cultural nuances of the Brazilian population. In both healthy adults and patients with traumatic brain injury, the adapted version exhibited no floor or ceiling effects and demonstrated significant score variability among participants. These findings suggest the preliminary suitability of the adapted HT for use within the Brazilian population, encompassing both healthy individuals and those with neurological conditions. Nonetheless, the authors emphasize the need for future research to establish evidence of reliability, construct and criterion validity, as well as sensitivity and specificity (52).

With respect to the Brazilian adaptation of the MDMQ, from the 22 statements of the original version, four items were excluded (2,4, 8, and 14) because of lower judges’ agreement (between 50% and 75%), and one (5) was substituted for a new item in the hypervigilance subscale. A confirmatory factor analysis (CFA) revealed a similar factor structure between the adapted and original MDMQ versions. Additionally, an acceptable overall internal consistency (Cronbach’s α = .824) was found, with α = .857, .853, .664, and .791 for the vigilance, buck-passing, hypervigilance, and procrastination subscales, respectively (48).

The Brazilian adapted version of the BDEFS showed semantic correspondence and satisfactory agreement with the original version, with 85 of the 89 items showing moderate to good correlation (53). Moreover, the five subscales were strongly correlated (ρ > .7) between the two versions. Additionally, the adapted version showed good internal consistency (Cronbach’s α = .961) and marginally moderate construct validity (ρ > .30) with the Brazilian versions of the Barrat Impulsiveness Scale (BIS-11) and the Adult Self-Report Scale (ASRS-18).

The results of the remaining 13 studies that focused on the psychometric analysis of existing EFATs are shown in Table 7. Among these studies, six investigated batteries frequently used in serial assessment (NIHT, CNS, CANTAB, and FAB); four studies investigated sets of numerous EF tests (Set 2: Simon, Visuoverbal and Visuospatial N-back, Letter-memory and the Number–letter tasks; and Set 3: Pro, Anti and Pro/Anti, Simon, Flanker, Forward and Backward Spatial, and 2-back tests); and three studies investigated commonly used EF tests (BIS, NoGo, and Stroop). Additionally, many reviewed studies tested gold standard subtests of other well-known assessment batteries, such as the Wechsler scales, D-KEFS (three studies), and BADS (see Table 5), which serve as reference standards to support the psychometric analysis of their investigated EFATs.

Table 7 Psychometric Properties of Studies on Existing EFATs

Study	Tool	Psychometric properties
Scarfo et al. (13)	Andersons’s pediatric model.	Predictive validity: good fitting model for AC (χ2(2) = 1.61, p = .447, RMSEA = 0.000, CFI = 1.000); CF (χ2(8) = 2.90, p = .940, RMSEA = 0.000, CFI = 1.000); IP (χ2(4) = 1.15, p = .886, RMSEA = 0.000, CFI = 1.000); and GS (χ2(8) = 7.22, p = .513, RMSEA = 0.000, CFI = 1.000).
Karlsen et al. (43)	CANTAB.	Test-retest reliability: low to moderate for most tests (r = .39 to .79). Acceptable test-retest reliability was observed just for SWM between errors (r = .71) and strategy (r = .79), AST percent correct (.75), and RVP (r = .75)
Ott et al. (44)	NIHTB-CB.	Construct validity: poor-to-adequate for attention and executive function, episodic memory, and processing speed domains (ICC = -.029 to .517); poor-to-good for working memory and motor dexterity domains, and the fluid composite (ICC = .215 to .801); adequate-to-good for the language (ICC = 0.408–0.829).
Pires et al. (46)	EF tests (Set 1)	CFA: good fitting model for a three-correlated factor model, with EF, VA, and PS factors (CFI = .992, RMSEA 90%CI = .018 (.000-.088), and SRMR = .65). PS is more related to EF (r = .64) than to VA (r = .41) and EF and VA showed no correlation with each other (r = .06).
White et al. (47)	EF tests (Set 2)	Test-retest reliability: low to excellent for RT (ICC = .34 to .93) in inhibition (Anti), response switching (Pro/Anti), selective attention (Simon and Flanker), and working memory (2-back) Lower reliability was observed between the first two-time points (within the first testing-day), indicating practice effects.
Rijnen et al. (50)	CNS VS.	Test-retest Reliability: low to good for consistency (ICC = -40 to .89) and agreement (ICC = .17 to .88); Correlations coefficients: psychomotor speed (.88), processing speed (.81), RT (.78), CF (.74), complex attention (.55), verbal memory (.43), and visual memory (.41).
Soveri et al. (51)	EF tests (Set 3)	Test-retest reliability (RT and Accuracy): 1) RT: low reliability for Simon test (r = .611 to .636), low to acceptable for Visuoverbal (r = .647 to .854), Visuospatial (r = .701 to .823), Letter-Memory (r = .827 to .847), and Number-Letter tests (r = .428 to .773); 2) Accuracy: low reliability for Visuoverbal N-back (r = -.221 to .687) and acceptable for Letter-memory test (r = .827 to .847).
Malloy-Diniz et al. (55)	BIS-11_BV.	Reliability of factor structure: Overall score (one-factor) presented acceptable internal consistency (Cronbach’s α = .790), with subscales ranging between .147 and .789; the two-factors (inhibition and nonplanning) presented the best internal consistency (Cronbach’s α = .789 and .618 for inhibition and nonplanning, respectively), considering cutoff value ≥ .600. Normative: no significant impact of age, sex, or education.
Heaton et al. (56)	NIHTB-CB.	Test-retest reliability: acceptable to good internal consistency (Cronbach’s α = 0.84, .83, and .77) and strong to very strong test-retest correlations (r = .92, .86, and .90) for Crystallized, Fluid, and Total Cognition Composite scores, respectively. Convergent construct validity: good correlations between NIHTB-CB and Gold Standard Measures on the Crystallized (r = .90), Fluid (r = .78), and Total (r = .89) Cognition Composite scores. Discriminant construct validity: moderate correlations between NIHTB-CB Crystallized and Gold Standard Fluid Cognition Composite scores (r = .39), and low between NIHTB-CB Fluid and Gold Standard Crystallized Cognition Composite scores (r = .19). Between NIHTB-CB Crystallized and Fluid Cognition Composite scores was also low (r = .17).
Brunner et al. (58)	P3 No-Go wave.	Test-retest reliability: acceptable for amplitude (ICC > .75), and excellent for latency (ICC > .90).
Kang et al. (41)	Kaplan Stroop	Split-half reliability: excellent for Stroop A (r = .907) and B (r = .911), and acceptable for Stroop C (r = .797). Construct validity: correlation coefficients between demographic variables and Stroop full-time scores did not differ significantly from similar correlations with half-time scores (Hotelling’s t tests, p > .05).
Beato et al. (59)	FAB_BV.	Normative: FAB scores significantly correlated with education (r = .47, p < .0001), and MMSE score (r = .39, p < .001). No significant correlations with age or gender (p = .13 and .09, respectively).
Dubois et al. (36)	FAB.	Interrater reliability: good (κ = .87), and good internal consistency (Cronbach’s α = .78). Concurrent criterion validity: moderate association between FAB and WCST for perseverative errors (ρ = .68, p < .001), and strong for the number of criteria (ρ = .77, p < .001); strong correlation between FAB and DRS (ρ = .82, p < .001). Discriminant construct validity: good between healthy participants and patients (analysis of covariance: F[1,131] 17. 24; p < 0.001).

Note. AC = attentional control; χ2 = Pearson's chi-square; RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index; CF = cognitive flexibility; IP = information processing; GS = goal setting; CANTAB = Cambridge Neuropsychological Test Automated Battery; r = Pearson’s correlation coefficient; SWM = Spatial Working Memory; AST = Attention Switching Task; RVP = Rapid Visual Processing; NIHTB-CB = National Institute of Health Cognitive Battery; ICC = Intraclass Coefficient; EF = Executive Functions; Set 1: Working Memory, Tower, Divided Attention, Stroop, Verbal Fluency, Word List, Confrontation Naming tests, Coding and Telephone tests; CFA = Confirmatory Factor Analysis; VA = Verbal Abilities; PS = Processing Speed; CI = Confidence Interval; SRMR = Standardized Root Mean Square Residual; Set 2: Pro, Anti and Pro/Anti, Simon, Flanker, Forward and Backward Spatial, and 2-back tests; CNS VS = Central Nervous System Vital Signs; RT = Reaction Time; Set 3 = Simon, Visuoverbal and Visuospatial N-back, Letter-memory and the Number–letter tasks; BIS-11_BV = Brazilian version of the Barratt Impulsiveness Scale; FAB_BV = Brazilian version of the Frontal Assessment Battery; MMSE = Mini-Mental State Examination; FAB = Frontal Assessment Battery; WCST = Wisconsin Card Sorting Test; ρ = Spearman’s correlation coefficients; DRS = Dementia Rating Scale.

With respect to the psychometric properties investigated in the reviewed studies, reliability (23 studies) was the most commonly evaluated, followed by validity (15 studies), factor analysis (7 studies), and normative data (6 studies). Test-retest reliability and construct validity are the most employed methods for reliability and validity analysis, respectively.

One critical aspect of psychometric analysis lies in its noteworthy susceptibility to bias (65). One kind of frequently observed bias in the reviewed studies was the sampling method, and most studies used convenience samples. Other equally relevant aspects were also observed: homogeneity of samples in terms of participants' educational level and little mention of the requirements for the parametric analysis (for example, some studies chose to use Pearson correlation with small samples, when Spearman’s correlation might have been more appropriate).

Furthermore, in the context of the observed methods for assessing reliability in the reviewed studies, biases may also have been raised from a) an exaggerated preference for using Cronbach's alpha as a measure of internal consistency without considering other more appropriate forms, such as λ₄ and ω_TOT (especially when different factorial loads are present), and b) the use of Parson’s or Spearman’s correlation when intraclass correlation might have been more appropriate (due to considering systematic error), among others posing a high risk that must be carefully addressed (66).

Finally, it is important to underscore the diverse approaches employed by authors in the reviewed studies regarding the establishment of thresholds for assessing reliability and validity (67–69). The absence of consensus on interpreting such thresholds, particularly concerning correlation indices (Pearson and Spearman) and interclass coefficients (ICCs), complicates efforts toward standardization (65,66).

The present review aimed to examine the available data on EFATs for healthy adults, focusing on 29 studies that covered a broad spectrum of cognitive tests, including some innovative ones. Additionally, the review sought to identify the EFATs most commonly used for each EF component in research, with particular emphasis on those already adapted for the Brazilian population and those suitable for online administration. Notably, the selected studies were conducted in diverse regions across the globe, including North and South America; North, Western, and Southern Europe; the Middle East; East Asia; and Oceania.

The 29 studies included in this review considered certain thematic topics that were frequently discussed by various authors. On the basis of these topics, this discussion section was organized to highlight the most common subjects in studies on EF assessment tools. According to these topics, the following subsections explore the results of this review: 1) EF theoretical models that most studies are based on; 2) The most assessed constructs and the associated assessment tools employed; 3) The importance of standardized national batteries for cognitive assessment in adults; 4) The international growth of digital test usage; 5) The lack of standardization of digital assessment tools for adults in Brazil; and 6) Strengths and limitations.

EF theoretical models on which studies were mostly based

The included studies were based on the variability of the EF theoretical models, such as the attention model, working memory model, and unity and diverse model. They seem to reflect the ongoing absence of a solid and uniform understanding in the scientific community regarding executive functioning (4,13).

Nevertheless, studies have consistently revealed a strong tendency among authors to interpret executive function (EF) as a collection of fundamental components associated with controlled cognitive processes, notably inhibition, working memory, and cognitive flexibility. This interpretation is in line with the prevailing theory positing EF as comprising three primary components (10,11), each exhibiting both unity (constituting a general EF domain) and diversity (operating independently). However, recent research has highlighted the context-dependent nature of executive abilities (70), thus perpetuating the ongoing debate surrounding the true essence of EF, which oscillates between its domain-generality and context dependency (71).

Furthermore, there appears to be a convergent understanding of attention as a strongly executive component, supported by the theoretical model that recognizes EF as related to controlled attention (5,64).

The most commonly assessed EF components and associated EFATs in the reviewed studies

Not surprisingly, the EFATs investigated in the reviewed studies focused predominantly on assessing the three primary components of EF: inhibition, working memory, and cognitive flexibility. The most commonly administered assessment tools were the Stroop test (distinct versions), the digit span test (Wechsler scale), and the trail-making test. This finding aligns with a previous study that identified the three primary EF components as the most frequently assessed as well as Stroop tests as the most commonly administered across 106 empirical studies (1).

Additionally, a considerable number of studies have assessed other components involved in executive control, including processing speed, attentional control, verbal fluency, planning, episodic memory, and psychomotor abilities (see Table 5). For these constructs, the most employed assessment tools were Stroop, digit symbol coding, verbal fluency, Tower of London, word list, and FAB motor series. The ongoing assumption of these constructs as executive components has been evidenced in some previous studies (3,72).

The importance of standardized batteries for EF assessment in adults

EF assessment batteries have been widely used in the international context, and it is highly probable that the components of EF represent distinct aspects of higher-order cognition while also sharing similar neurological circuits (4). Consequently, an assessment of the entire set of components appears to enable a more comprehensive interpretation of the cognitive abilities of the individual (38,44,51).

The present review revealed that six studies focused on EF assessment batteries, including the NIHTB-CB and CANTAB. The NIHTB-CG has been a strong effort of the EUA government (commissioned by 16 NIH institutes) to develop a standardized battery capable of providing brief and highly accessible measures for broad use in epidemiology and practice (56). On the other hand, the CANTAB is likely the most common assessment tool across European countries for both research and practice and has been used in projects to collect data from large populations (43). Many reviewed studies used gold standard subtests of other well-known assessment batteries, such as the Wechsler scales, the Delis-Kaplan Executive Function System (D-KEFS), the Behavioral Assessment of Dysexecutive Syndrome (BADS), and the Frontal Assessment Batterie (FAB).

One important reason for this international trend is the fast growth of the aging bracket of the population in most developed and developing countries (73). Standardized EF assessment batteries are crucial for enabling continuous monitoring of cognitive evolution in individuals during the aging process, for example, evaluating the course of an impairment or the impact of an intervention (47,50). Such data are invaluable for epidemiology, aiming to delineate prevention strategies against the pronounced cognitive decline in this increasing segment of the global population.

Additionally, the use of the standardized EF assessment batteries ensures consistency and reliability in the evaluation process, making it useful for tracking clinically and epidemiologically relevant data among different populations across the life span (56).

The international growth of digital and online EFATs

Of all 29 revised studies, 21 performed the administration of digital and online assessment tools, two of which investigated serious games (30,57). Many of these studies focused on emphasizing the growing importance of using digital technology for EF assessment, highlighting numerous reasons for this (30–32,47,49,57).

The three most acknowledged reasons for the advantages of digital EFATs are the ease of administration, higher score accuracy, and the potential to reach larger and more representative samples of the social diversity of the population, the latter being particularly important for normative studies (40,49,57)

Other frequently cited reasons in the studies included standardized administration, flexibility in timing and location for test administration, and ease of generating alternate forms of testing to mitigate practice effects (31,47). The less frequently cited reasons, though not less important, were the potential for acquiring additional data on cognitive performance, enjoyment, appeal, real-time feedback, and rewards (30,31).

The digital and online EFATs have emerged as a significant area of interest in neuropsychological research. Numerous studies are delving into online batteries and serious games for cognitive assessment, including EF assessments. A recent PubMed search conducted on March 20, 2024, using the terms ‘cognitive assessment’ and ‘online’, yielded 6897 results, of which 6119 were published in the last decade. Interestingly, when the term ‘online’ was substituted with ‘serious game’, the number of results narrowed to 206, with 203 appearing in the last decade. These data suggest a substantial and rapid expansion in the availability of online cognitive assessment tools.

The lack of standardization of digital and online EFATs for adults in Brazil

In Brazil, the availability of digital and online EFATs for the healthy adult population appears to be very limited, and what does exist is more focused on patients. A review conducted in 2014 revealed that out of 15 available EFATs in Brazil, 14 were aimed at children, 9 were aimed at adults, and only 5 were aimed at elderly individuals (74). Another review in 2020 identified 44 tools available in Brazil, of which 29 were for the adult population, indicating reasonable growth in availability (3). However, few tools have been developed or adapted for healthy adults in the economically active age range, as most focus on the detection of pathological deficits.

Nevertheless, executive deficits may represent a major problem for healthy adults because they can impact social and professional performance (3,10,75). A lack of self-management can lead, for example, to emotional disbalance, reducing the ability to maintain healthy social relationships (4). In particular, for adults in leadership positions, the impairment is even greater, as, in addition to the quality of interpersonal relationships, the leader needs to be assertive in their decisions, requiring attentional focus, inhibitory control, and cognitive flexibility, among other cognitive abilities (29,76).

Furthermore, contrary to the global trend, it appears that the growth of research on digital and online cognitive assessment tools in Brazil has been rather modest. Among the 29 reviewed studies, only four EFATs appear to have a similar digital version available in Brazil, one of which, the EFSA, was recently developed in Brazil (42).

This review presented several novel EFATs, many of which have the potential to be adapted into Brazilian Portuguese. However, it is important to highlight that availability should always be associated with the quality of the assessment instrument in terms of reliability and validity (65). Although many reviewed studies have shown modest reliability and validity results, others have shown very promising results, such as Kaller et al. (38), with the TOL-F, Kruger et al. (42), with the EFSA, Park and Schott (31), with the dTMT, Wang et al. (30), with the Fisherman, Troyer et al. (32), with the online EF tests tool, Arioli et al. (40), with the RCM, and Feenstra et al. (49), with the ACS.

Strengths and Limitations

The present work has several strengths. The review has been duly registered in PROSPERO, thereby guaranteeing accessibility and transparency. This study strictly adhered to the PRISMA guidelines, with special attention given to potential bias, which significantly contributed to enhancing the quality of the included studies. Additionally, it addressed research questions with evidence and considerations on the basis of an extensive body of literature on the topic. It also provides a critical analysis of the current state of EFATs in Brazil while proposing significant measures to increase the availability of such instruments for both clinical practice and research involving healthy adults in the country.

Nevertheless, it is important to consider several limitations: 1) most studies employ convenience sampling, which may limit diversity; consequently, many studies might not be adequately representative of the target population or generalizable to other populations; 2) some studies use small and demographically homogeneous samples; 3) although basic computer skills are crucial criteria for sample selection in research on digital assessment tools, very few studies perform assessments of such skills among participants; 4) many studies fail to provide clear and organized demographic data; 5) the test‒retest interval varies among participants in some studies, which might have compromised the accuracy of the reliability indices; 6) some studies do not provide a conflict of interest (perhaps this requirement may be beneficial as part of the inclusion criteria for future reviews); 7) all studies involve performance‒based EFATs and do not consider indirect measures to avoid confusion between low- and high-order cognitive processes; and 8) in the study selection process, the use of the English language as an inclusion criterion may have limited access to interesting research from around the world, although this review included studies from various regions on almost every continent; and 9) in several of the reviewed studies, the authors adopted different reference thresholds for reliability and validity, making it somewhat difficult to perform methodological comparisons among them.

This review provides data for the scientific community regarding the current availability of digital and online EFATs. It highlights the new trends of developing online tools to improve score precision, administration standardization and scalability, real-time feedback, appeal, enjoyability, and numerous other advantages.

The current findings may contribute to sparking the debate regarding the importance of advancing the adoption of digital and online instruments in Brazil. Additionally, it highlights an interesting study detailing how a national public consortium in Indonesia facilitated the development of a standardized neuropsychological test battery in that country (18). This study holds potential significance for Brazilian neuropsychological assessment research groups, as it offers a compelling example of a public initiative that could inform future efforts to establish a national (or regional) standardized battery of EF tests in the country.

The current work has several implications for future research on EF assessment of healthy adults. It focused on assessment instruments in general, revealing the prevalence of digital and online instruments. Considering this trend, subsequent reviews could focus exclusively on investigating studies of such instruments to map the stock already available internationally.

Within this review, several studies have compared performance between digital and traditional pen-and-paper (P&P) versions of the EFAT, such as the VBAS X P&P Stroop test and the dTMT X P&P TMT. Future research endeavors may advance these comparisons, encompassing a broader array of established P&P tests alongside their digital counterparts.

Furthermore, 12 novel assessment tools were presented in this review. Future research could expand investigations into the validity and reliability of the novel and promising assessment instruments outlined in this review, making use of larger samples and more robust methods of psychometric evaluation.

Concerning EF assessment batteries, future reviews could exclusively concentrate on the established standardized batteries (NIHTB, CANTAB, D-KEFS, WAIS, etc.), undertaking a comparison of their psychometric properties. This could trigger a medium-term project initiative in Brazil to either cross-culturally adapt an existing gold standard battery or build a national battery of its own.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

The data that support the findings of this study are available at http://doi.org/10.6084/m9.figshare.25833406.

Competing interests

The authors declare that they have no competing interests.

Funding

The authors reported there is no funding associated with the work featured in this article.

Authors’ contributions

TNT spearheaded the systematic review, including conception and design, database searches, study selection, data collection and synthesis, risk-of-bias analysis, and drafting the initial manuscript. JBS played a pivotal role in data collection, the risk-of-bias analysis, and commented on previous versions of the manuscript. RMMA played a crucial role in the systematic review concept and design. She also independently facilitated conflict resolution between the primary author and the external independent reviewer during the study selection process. All authors read and approved the final manuscript.

Acknowledgements

The authors extend their sincere gratitude to Mr. Celvio Derbi Casal for his invaluable assistance in developing the search strategy, and to Mr. Gabriel Dias Curra for his meticulous independent review of the records obtained through database searches. Their expertise and dedication significantly contributed to the quality and rigor of this research.

Baggetta P, Alexander PA. Conceptualization and Operationalization of Executive Function. Mind Brain Educ. março de 2016;10(1):10–33.
Luria, A. R. The working brain: an introduction to neuropsychology. New York: Basic Books; 1973. 398 p.
Dias NM, Malloy-Diniz LF. Funções executivas: modelos e aplicações. São Paulo: Pearson; 2020. 480 p.
Dias NM, Malloy-Diniz LF, organizadores. Tratado de funções executivas: Modelos teóricos, construtos associados e desenvolvimento. 1^a. Belo Horizonte: Editora Ampla; 2023.
Norman D, Shallice T. Attention to action: Willed and automatic control of behavior. Cognitive Neuroscience: A reader. 1986.
Shallice T, Burgess P. The domain of supervisory processes and temporal organization of behaviour. Philos Trans R Soc Lond B Biol Sci [Internet]. 29 de outubro de 1996 [citado 5 de março de 2024];351(1346). Disponível em: https://pubmed.ncbi.nlm.nih.gov/8941952/
Baddeley A. The episodic buffer: a new component of working memory? Trends Cogn Sci [Internet]. 11 de janeiro de 2000 [citado 5 de março de 2024];4(11). Disponível em: https://pubmed.ncbi.nlm.nih.gov/11058819/
Baddeley A. Working memory: theories, models, and controversies. Annu Rev Psychol. 2012;63:1–29.
Cowan N. Working Memory Underpins Cognitive Development, Learning, and Education. Educ Psychol Rev. 6 de junho de 2014;26(2):197.
Diamond A. Executive Functions. Annu Rev Psychol. 3 de janeiro de 2013;64(1):135–68.
Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A, Wager TD. The unity and diversity of executive functions and their contributions to complex “Frontal Lobe” tasks: a latent variable analysis. Cognit Psychol. agosto de 2000;41(1):49–100.
Zelazo PD. Executive function: Reflection, iterative reprocessing, complexity, and the developing brain. Dev Rev. 2015;38:55–68.
Scarfo J, Suleyman| E, Ball M. 20 years on: Confirmation of P. Anderson’s (2002) paediatric model of executive functioning in a healthy adult sample. Heliyon. 2023;9(5):e15504.
Damásio, A. R. O erro de Descartes: emoção, razão e o cérebro humano. 3^a. São Paulo: Companhia das Letras; 2012.
Kahneman, D. Rápido e Devagar: duas formas de pensar [Thinking, fast and slow]. 1^a. Rio de Janeiro: Objetiva; 2012. 608 p.
Rosales KP, Wong EH, Looney L. The Psychometric Structure of Executive Functions: A Satisfactory Measurement Model? An Examination Using Meta-Analysis and Network Modeling. Behav Sci Basel Switz. 8 de dezembro de 2023;13(12):1003.
Ferguson H, Brunsdon V, Bradford E. The developmental trajectories of executive function from adolescence to old age. Sci Rep. 14 de janeiro de 2021;11.
Wahyuningrum SE, Sulastri A, Hendriks MPH, van Luijtelaar G. The Indonesian Neuropsychological Test Battery (INTB): Psychometric properties, preliminary normative scores, the underlying cognitive constructs, and the effects of age and education. Acta Neuropsychol. 2022;20(4):445–70.
Kessels RPC, Hendriks MPH. Neuropsychological assessment. Em: Friedman HS, Markey CH, organizadores. Encyclopedia of Mental Health (Third Edition) [Internet]. Oxford: Academic Press; 2023 [citado 2 de março de 2024]. p. 622–8. Disponível em: https://www.sciencedirect.com/science/article/pii/B9780323914970000175
Zucchella C, Federico A, Martini A, Tinazzi M, Bartolo M, Tamburin S. Neuropsychological testing. Pract Neurol. junho de 2018;18(3):227–37.
Baars MAE, Rini, J., Anguera-Singla, R., Gazzaley, A., Wais PE. Validation of At-Home Application of a Digital Cognitive Screener for Older Adults. Front Aging Neurosci. 2022;14:907496.
Seabra AG, Dias NM, organizadores. Avaliação neuropsicológica cognitiva: atenção e funções executivas. Vol. 1. São Paulo: Memnom; 2012.
Happaney K, Zelazo PD. Development of Executive Function Skills in Childhood: Relevance for Important Life Outcomes. Em 2022 [citado 2 de março de 2024]. p. 427–51. Disponível em: https://www.cambridge.org/core/books/cambridge-handbook-of-cognitive-development/development-of-executive-function-skills-in-childhood/7347F27553E62F14958E25C03BB2BDFA
Köhler S, Soons LM, Tange H, Deckers K, van Boxtel MPJ. Sleep Quality and Cognitive Decline Across the Adult Age Range: Findings from the Maastricht Aging Study (MAAS). J Alzheimers Dis. 2022;96(3):1041–9.
Satoh M, Tabei KI, Fujita S, Ota Y. Online Tool (Brain Assessment) for the Detection of Cognitive Function Changes during Aging. Dement Geriatr Cogn Disord. 2021;50(1):85–95.
Statsenko Y, Habuza T, Smetanina D, Simiyu GL, Uzianbaeva L, Neidl-Van Gorkom K, et al. Brain Morphometry and Cognitive Performance in Normal Brain Aging: Age- and Sex-Related Structural and Functional Changes. Front Aging Neurosci. 26 de janeiro de 2022;13:713680.
Becker WJ, Cropanzano R, Sanfey AG. Organizational Neuroscience: Taking Organizational Theory Inside the Neural Black Box. J Manag. julho de 2011;37(4):933–61.
Ramchandran K. The neuropsychological correlates of leadership effectiveness. Em 2011 [citado 2 de março de 2024]. Disponível em: https://iro.uiowa.edu/esploro/outputs/doctoral/9983777073202771
Ramchandran K, Colbert AE, Brown KG, Denburg NL, Tranel D. Exploring the Neuropsychological Antecedents of Transformational Leadership: the Role of Executive Function. Adapt Hum Behav Physiol. dezembro de 2016;2(4):325–43.
Wang P, Fang Y, Qi JY, Li HJ. FISHERMAN: A Serious Game for Executive Function Assessment of Older Adults. Assessment. 2023;30(5):1499–513.
Park SY, Schott N. The trail-making-test: Comparison between paper-and-pencil and computerized versions in young and healthy older adults. Appl Neuropsychol Adult. 2022;29(5):1208–20.
Troyer AK, Rowe G, Murphy KJ, Levine B, Leach L, Hasher L. Development and evaluation of a self-administered on-line test of memory and attention for middle-aged and older adults. Front Aging Neurosci. 2014;6:335.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 29 de março de 2021;372:n71.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev [Internet]. 2016 [citado 29 de março de 2024];5(1). Disponível em: https://link.springer.com/epdf/10.1186/s13643-016-0384-4
Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 18 de outubro de 2011;155(8):529–36.
Dubois B, Slachevsky A, Litvan I, Pillon B. The FAB: a Frontal Assessment Battery at bedside. Neurology. 12 de dezembro de 2000;55(11):1621–6.
Ishigami Y, Eskes GA, Tyndall AV, Longman RS, Drogos LL, Poulin MJ. The Attention Network Test-Interaction (ANT-I): reliability and validity in healthy older adults. Exp Brain Res. 2016;234(3):815–27.
Kaller CP, Debelak R, Köstering L, Egle J, Rahm B, Wild PS, et al. Assessing planning ability across the adult life Span: Population-representative and age-adjusted reliability estimates for the Tower of London (TOL-F). Arch Clin Neuropsychol. 2016;31(2):148–64.
Parsons T, Barnett M. Virtual apartment stroop task: Comparison with computerized and traditional stroop tasks. J Neurosci METHODS. 2018;309:35–40.
Arioli M, Rini J, Anguera-Singla R, Gazzaley A, Wais PE. Validation of At-Home Application of a Digital Cognitive Screener for Older Adults. Front Aging Neurosci [Internet]. 1^o de julho de 2022 [citado 14 de março de 2024];14. Disponível em: https://www.frontiersin.org/articles/10.3389/fnagi.2022.907496
Kang C, Lee GJ, Yi D, McPherson S, Rogers S, Tingus K, et al. Normative data for healthy older adults and an abbreviated version of the Stroop test. Clin Neuropsychol. 2013;27(2):276–89.
Kruger OE, Zibetti MR, Schlindwein R, Lopes FM. Preliminary validity evidence for the Executive Function Scale for Adults (EFSA). Psychol Neurosci. 2023;No Pagination Specified-No Pagination Specified.
Karlsen RH, Karr JE, Saksvik SB, Lundervold AJ, Hjemdal O, Olsen A, et al. Examining 3-month test-retest reliability and reliable change using the Cambridge Neuropsychological Test Automated Battery. Appl Neuropsychol Adult. 2022;29(2):146–54.
Ott LR, Schantell M, Willett MP, Johnson HJ, Eastman JA, Okelberry HJ, et al. Construct Validity of the NIH Toolbox Cognitive Domains: A Comparison With Conventional Neuropsychological Assessments. Neuropsychology. 2022;36(5):468–81.
Heled E, Rotberg S, Yavich R, Hoofien AD. Introducing the Tactual Span: A New Task for Assessing Working Memory in the Teactile Modality. Assessment. 2021;28(3):1018–31.
Pires L, Moura O, Guerrini C, Buekenhout I, Simões MR, Leitão J. Confirmatory factor analysis of neurocognitive measures in healthy young adults: The relation of executive functions with other neurocognitive functions. Arch Clin Neuropsychol. 2019;34(3):350–65.
White N, Flannery L, McClintock A, Machado L. Repeated computerized cognitive testing: Performance shifts and test–retest reliability in healthy older adults. J Clin Exp Neuropsychol. 2019;41(2):179–91.
Cotrena C, Branco LD, Fonseca RP. Adaptation and validation of the Melbourne Decision Making Questionnaire to Brazilian Portuguese. Trends Psychiatry Psychother. 2018;40(1):29–37.
Feenstra HEM, Vermeulen IE, Murre JMJ, Schagen SB. Online self-administered cognitive testing using the Amsterdam Cognition Scan: Establishing psychometric properties and normative data. J Med Internet Res. 2018;20(5).
Rijnen SJM, van der Linden SD, Emons WHM, Sitskoorn MM, Gehring K. Test-retest reliability and practice effects of a computerized neuropsychological battery: A solution-oriented approach. Psychol Assess. 2018;30(12):1652–62.
Soveri A, Lehtonen M, Karlsson LC, Lukasik K, Antfolk J, Laine M. Test-retest reliability of five frequently used executive tasks in healthy adults. Appl Neuropsychol Adult. 2018;25(2):155–65.
Cardoso CDO, Zimmermann N, Paraná CB, Gindri G, de Pereira APA, Fonseca RP. Brazilian adaptation of the Hotel Task: A tool for the ecological assessment of executive functions. Dement Neuropsychol. 2015;9(2):156–64.
Godoy VP, Mata FGD, Conde BR, Souza CA de O e, Martins ALG, Mattos P, et al. Brazilian Portuguese transcultural adaptation of Barkley Deficits in Executive Functioning Scale (BDEFS). Arch Clin Psychiatry São Paulo. dezembro de 2015;42:147–52.
Köstering L, Nitschke K, Schumacher FK, Weiller C, Kaller CP. Test-retest reliability of the Tower of London Planning Task (TOL-F). Psychol Assess. 2015;27(3):925–31.
Malloy-Diniz LF, Paula JJ de, Vasconcelos AG, Almondes KM de, Pessoa R, Faria L, et al. Normative data of the Barratt Impulsiveness Scale 11 (BIS-11) for Brazilian adults. Rev Bras Psiquiatr Sao Paulo Braz 1999. 2015;37(3):245–8.
Heaton RK, Akshoomoff N, Tulsky D, Mungas D, Weintraub S, Dikmen S, et al. Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults. J Int Neuropsychol Soc. 2014;20(6):588–98.
Aalbers T, Baars MAE, Rikkert MGMO, Kessels RPC. Puzzling with online games (BAM-COG): Reliability, validity, and feasibility of an online self-monitor for cognitive performance in aging adults. J Med Internet Res. 2013;15(12):183–93.
Brunner JF, Hansen TI, Olsen A, Skandsen T, Håberg A, Kropotov J. Long-term test-retest reliability of the P3 NoGo wave and two independent components decomposed from the P3 NoGo wave in a visual Go/NoGo task. Int J Psychophysiol. 2013;89(1):106–14.
Beato R, Amaral-Carvalho, V., Guimarães, H. C., Tumas, V., Souza, C. P., De Oliveira, G. N., et al. Frontal assessment battery in a Brazilian sample of healthy controls: normative data. Arq Neuropsiquiatr. abril de 2012;70(4):278–80.
Wechsler D. WMS-III Administration and Scoring Manual. San Antonio, TX: The Psychological Corporation. Harcourt Brace & Co.; 1997.
Reitan RM. The relation of the Trail Making Test to organic brain damage. J Consult Psychol. 1955;19(5):393–4.
Fan J, McCandliss BD, Sommer T, Raz A, Posner MI. Testing the efficiency and independence of attentional networks. J Cogn Neurosci. 1^o de abril de 2002;14(3):340–7.
Callejas A, Lupiàñez J, Funes MJ, Tudela P. Modulations among the alerting, orienting and executive control networks. Exp Brain Res. 1^o de novembro de 2005;167(1):27–37.
Petersen SE, Posner MI. The Attention System of the Human Brain: 20 Years After. Annu Rev Neurosci. 7 de julho de 2012;35:73.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. https://www.apa.org. 2014 [citado 20 de março de 2024]. The Standards for Educational and Psychological Testing. Disponível em: https://www.apa.org/science/programs/testing/standards
Revelle W, Condon DM. Reliability from α to ω: A tutorial. Psychol Assess. 2019;31(12):1395–411.
Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Análise multivariada de dados [Multivariate data analysis]. 6. Porto Alegre: Bookman; 2009.
Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 1^o de junho de 2016;15(2):155–63.
McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30–46.
Doebel S. Rethinking Executive Function and Its Development. Perspect Psychol Sci. 1^o de julho de 2020;15(4):942–56.
Zelazo PD, Carlson SM. Reconciling the Context-Dependency and Domain-Generality of Executive Function Skills from a Developmental Systems Perspective. J Cogn Dev. 15 de março de 2023;24(2):205–22.
Tirapu-Ustárroz J, Bausela-Herreras E, Cordero-Andrés P. Modelo de funciones ejecutivas basado en análisis factoriales en población infantil y escolar: metaanálisis. Rev Neurol. 2018;67(6):215–25.
United Nations Department of Economic and Social Affairs. World Population Ageing 2023: Challenges and Opportunities of Population Ageing in the Least Developed Countries. [Internet]. 2023 [citado 22 de março de 2024]. Disponível em: https://www.un-ilibrary.org/content/books/9789213586747
Seabra AG, Laros JA, Macedo EC, Abreu N, organizadores. Inteligência e funções executivas: avanços e desafios para avaliação neuiropsicológica. São Paulo: Memnom; 2014.
Zelazo PD, Cunningham W. Executive Function: Mechanisms Underlying Emotion Regulation. Handb Emot Regul. 1^o de janeiro de 2007;
Diamond A, Ling DS. Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Dev Cogn Neurosci. 2016;18:34.

Economic active age brackets refer to segments of the population categorized by age that are actively employed or of working age.
In this review, 'more favorable' refers to assessment tools or methods that exhibit enhanced effectiveness, user engagement, accessibility, or overall satisfaction compared to traditional approaches. For example, digital assessment tools that incorporate gamification elements have been shown to increase participant motivation and performance outcomes (30). Additionally, online assessments provide greater accessibility for economically active individuals, allowing them to complete tests at their convenience, which can lead to higher participation rates (21). Furthermore, tools that offer immediate feedback and adaptive testing features contribute to a more personalized user experience, which is often preferred by participants (31).
Available at https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42023459513
The supplementary material is available at http://doi.org/10.6084/m9.figshare.25833406.

Digital and Traditional Executive Function Assessment Tools (EFATs) for Healthy Adults: A Systematic Review of Psychometric Properties

Status:

Version 1

Abstract

Figures

Introduction

Methods

Transparency and openness

Eligibility criteria

Inclusion criteria

Exclusion criteria

Data synthesis

Information source and search strategy

Study selection process

Data collection process and synthesis methods

Study risk-of-bias assessment

Results

Study selection

Study characteristics

Risk of bias in the reviewed studies

Results of the reviewed studies

The most commonly assessed EF components and associated EFATs in the reviewed studies

Batteries of EFAT employed in the reviewed studies

Novel EFAT

Preexisting EFAT

Discussion

EF theoretical models on which studies were mostly based

The most commonly assessed EF components and associated EFATs in the reviewed studies

The international growth of digital and online EFATs

The lack of standardization of digital and online EFATs for adults in Brazil

Strengths and Limitations

Conclusion

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors’ contributions

Acknowledgements

References

Footnotes

Supplementary Files

Status:

Version 1