No other systematic review seeking to identify instruments to evaluate the performance of the clinical pharmacist in different pharmaceutical practice environments was found during this study. Thirty-two studies were included, and 32 instruments were identified to be used by pharmacists acting in various pharmaceutical practice scenarios. The instruments varied in terms of type (general and specific, according to the approached constructs), methods used (from literature review to mixed methods), items, dimensions and domains, and psychometric properties (validity and reliability).
A different approach was taken in this review compared to that in other studies, in which all instruments, regardless of the practice scenario, were analyzed. In another study, in which the objective was to evaluate instruments in clinical pharmaceutical practice, the authors Alshakrah, Steinke, & Lewis [47] limited the practice environment by selecting the instruments for the hospital environment. However, considering that clinical pharmacy covers not only hospital environments, our study sought to identify the tools for other environments, such as community and outpatient pharmacies.
In addition, the diversity of approaches to measuring pharmaceutical practice was broad for most analyzed criteria, particularly concerning the constructs and practice scenarios of interest for each study. This may be related to the different levels of knowledge and experience of pharmacists in each region of the country. This depends on the obtained training, additional qualifications in clinical pharmacy, and their individual fields of interest. For example, some pharmacists are active in the field of acute illness treatment, whereas others prefer to focus on chronic diseases, in addition to the different types of clinical services offered [48].
Regarding the construct, in the included studies, some instruments were aimed at evaluating the pharmaceutical performance in specific chronic diseases. Considering these instruments, studies that evaluated the performance of the pharmacist in the management of diabetes [25, 26, 27], cardiovascular diseases [28], and mental illnesses [29, 31] prevailed. In some cases, the country’s legislation emphasizes the necessity of involving pharmacists in the follow-up of chronic diseases [33].
Owing to this extensive approach, many of the instruments included were developed by the authors themselves as a secondary objective of the study. Therefore, 27 of them were classified as methodological and cross-sectional. In the analysis of the method, it was found that some authors did not describe the method and others classified the studies only as cross-sectional. Considering that some of these studies developed instruments and followed criteria of validity and reliability, these were classified as methodological and cross-sectional.
For the development of these instruments, different methodologies were used, and the literature review with the adaptation of items from other studies was the most used type for the creation of the instruments’ items. Besides to this technique, the literature review combined with expert’s consensus techniques that was used for the development of some instruments in this study, was also reported in another review [49].
Some studies adapted validated instruments available in the literature, including the Behavioral Pharmaceutical Care Scale (BPCS), Pharmacy Survey on Patient Safety Culture (PSOPSC), and Diabetes Attitude Scale (DAS). The BPCS instrument [15] was the most extensively examined regarding the psychometric properties and the one offering more robust statistical results, being used and mentioned in many other studies. However, this instrument is to be applied only in the community pharmaceutical practice environment, which limits its use. The advantages of utilizing validated instruments include psychometric measures like validity and reliability, allowing the comparison of different studies. However, the limitations of the use of these instruments refer to the difficulties in finding an instrument for local use and meeting the proposed objectives. Furthermore, the adapted instruments did not follow the cross-cultural adaptation methodology.
Regarding the structure and the instruments dimension, the sample size in psychometric studies is based on the number of items in the instrument and aims to provide greater assurance in their analysis and quality [50, 51]. A sample of ten participants per item is considered acceptable, but there are studies that prove that twenty or more can significantly reduce error and inaccuracies in the solution of psychometric models, such as percentage of samples with correct factorial structure, average number of items classified incorrectly in the wrong factor, average error in eigenvalues, average error in factorial loads, the percentage of analyzes that do not converge after 250 interactions, and percentage with Heywood cases [52].
The limitation in the sample requires that initial minimum parameters of adequacy, such as factorial loads, commonality and the goodness of fit indexes are greater than in studies with larger samples. In only two of the 32 articles analyzed, the ratio between the number of participants for each item of the instrument was greater than 20: 1 and in five articles the ratio was 10: 1. However, no study has reported whether the sample size was determined and whether this fact also guided the establishment of the minimum parameters model.
As for the reliability measure, most studies used Cronbach’s Alpha with acceptable values within the established criteria. This coefficient depends on the magnitude of the correlation between the items and the number of items on the instrument [53]. Many studies have criticized the use of Alpha without considering the nature and distribution of the data and the sample size, especially in samples involving more than 1000 participants [54, 55]. The use of McDonald’s Omega and Greatest Lower Bound are preferable when there is data asymmetry, even in small samples and where high alpha values do not necessarily mean greater reliability and quality of the scales or tests, because they can be the result of long scales with parallel and redundant items or generate a restriction in the construct under study [56, 57].
Also, of the 32 studies analyzed, three performed test-retest reliability and did not validate the instrument’s construct. Bertchold 2016 [58] questions the use of the reliability term in the test-retest, reinforcing that Pearson’s correlation is a measure of association and not of reliability. Another way to clarify the reliability of an instrument and the possibility of guaranteeing its quality in different contexts is through invariance tests.
In this review, different instruments were identified to measure the performance of the clinical pharmacist, but several weaknesses were detected in the available instruments. According to the parameters and evidence criteria, few have undergone validation procedures with satisfactory results. Many authors refer to evidence from only one or two sources, such as reliability or correlation with the scores of another instrument, to support the validity of interpretations and, therefore, should be used with caution [59].
Concerning the methodological quality of the included cross-sectional studies, 55% of the studies had a high risk of bias. The JBI checklist addresses critical issues of internal and external validity that should be considered when assessing the validity of study prevalence data [60]. Therefore, the high risk of bias in these studies allows us to have more consistent and reliable conclusions from the data obtained, with important implications for data comparison.
The strengths of this study include the non- restrictions of language and time; high agreement between reviewers in the title and in the screening for inclusion of the studies; psychometric assessment of the instruments for methodological studies. As for the limitations, some studies may not have been found, as they are not indexed in the selected databases, or because they were published by institutions, foundations, or societies.