This section reports and discusses the experimental explainability performance scores obtained with the proposed FVX-PAP protocol using the subjective test specifications described in Section 4.
5.1 Subject Outlier Detection
The subject outlier detection process described in Subsection 3.7.1 was followed to detect subjects who have not focused adequately or were answering more randomly during the FVX-PAP experiment. Given the present FVX-PAP experiment only considered two FV explainability tools, i.e., FV-RISE and CorrRISE, the violation of the transitivity rule has not been considered to check the reliability of subjects. Consequently, the determination of inconsistent subject ranks was conducted based on swapping pairs and repeated pairs as detailed in Table 1.
Given Eq. (5), the maximum number of inconsistent rankings allowed by a subject before exclusion has been established as \(IRThresh=3\). This has led to screening out one subject in the Acceptance Decisions Test Sub-Experiment and no subjects in the Rejection Decisions Test Sub-Experiment. Consequently, the rankings from 22 subjects (Acceptance Decisions Test Sub-Experiment) and 23 subjects (Rejection Decisions Test Sub-Experiment) will be used in the following to estimate the eXplainability Performance Scores, \(XPS\), for the evaluated explainability tools.
5.2 Explainability Performance Score Estimation
Once the subjects’ ranks were collected, the main goal was to estimate the explainability performance scores, \(XPS\), for the assessed explainability tools. The following subsections report and analyze the estimated explainability performance scores obtained for each type of FV decision, notably acceptance (TA and FA) and rejection (TR and FR) as well as for all FV decisions together.
Accumulative Pairwise Comparison Matrices Generation
The subject preferences obtained with the FVX-PAP protocol appear as winning and ties frequencies between each pair of face heatmaps. Consequently, to estimate the explainability performance scores of the assessed tools, the subject preferences are translated into numerical rankings to simplify the estimation process leading to a numerical score. More specifically, after each trial ranking, a score of “+1” is attributed to the explainability tool with the preferred face heatmap, a score of “0” is attributed to the explainability tool with the non-preferred face heatmap, while a score of “+0.5” is attributed to both explainability tools when the face heatmaps of the pair are in a tie, i.e., they are equivalent in the explainability power. After, the APCM matrix is constructed as defined in Sub-section 3.7.2 by accumulating the preference matrices of all participating subjects, thereby capturing the frequency with which the face heatmaps of each assessed explainability tool are preferred over those of the other explainability tool.
The subjects’ preference matrices and the APCM matrix generation processes are performed independently for each type of FV decision, i.e., TA, FA, TR, and FR, thereby providing insights into the efficacy of the assessed explainability tools across the four FV decision types. \({APCM }_{TA}, {APCM }_{FA}, {APCM }_{TR},\) and \({APCM }_{FR}\) refer to the APCM matrices associated to each type of FV decision, wherein the matrices entries correspond to the number of times the face heatmaps of one explainability tool are preferred over the other for the TA, FA, TR, and FR FV decisions, respectively. For the performed experiments, the counting of the ranks led to the following matrices:
\(\begin{array}{c} \\ \begin{array}{c} \\ \begin{array}{c} \\ \end{array}\\ \end{array}\\ \begin{array}{c} {\varvec{A}\varvec{P}\varvec{C}\varvec{M} }_{\varvec{T}\varvec{A}}= \\ \end{array}\end{array}\begin{array}{c} \begin{array}{c} FV\_RISE \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \\ \end{array}\\ \begin{array}{c}FV\_RISE\\ \\ \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \end{array}\left(\begin{array}{ccc}0& & 310.5\\ & & \\ 349.5& & 0\end{array}\right)\\ \end{array}\) \(\begin{array}{c} \\ \begin{array}{c} \\ \begin{array}{c} \\ \end{array}\\ \end{array}\\ \begin{array}{c} {\varvec{A}\varvec{P}\varvec{C}\varvec{M} }_{\varvec{F}\varvec{A}}= \\ \end{array}\end{array}\begin{array}{c} \begin{array}{c} FV\_RISE \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \\ \end{array}\\ \begin{array}{c}FV\_RISE\\ \\ \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \end{array}\left(\begin{array}{ccc}0& & 326\\ & & \\ 334& & 0\end{array}\right)\\ \end{array}\)
\(\begin{array}{c} \\ \begin{array}{c} \\ \begin{array}{c} \\ \end{array}\\ \end{array}\\ \begin{array}{c} {\varvec{A}\varvec{P}\varvec{C}\varvec{M} }_{\varvec{T}\varvec{R}}= \\ \end{array}\end{array}\begin{array}{c} \begin{array}{c} FV\_RISE \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \\ \end{array}\\ \begin{array}{c}FV\_RISE\\ \\ \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \end{array}\left(\begin{array}{ccc}0& & 348.5\\ & & \\ 341.5& & 0\end{array}\right)\\ \end{array}\) \(\begin{array}{c} \\ \begin{array}{c} \\ \begin{array}{c} \\ \end{array}\\ \end{array}\\ \begin{array}{c} {\varvec{A}\varvec{P}\varvec{C}\varvec{M} }_{\varvec{F}\varvec{R}}= \\ \end{array}\end{array}\begin{array}{c} \begin{array}{c} FV\_RISE \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \\ \end{array}\\ \begin{array}{c}FV\_RISE\\ \\ \text{C}\text{o}\text{r}\text{r}\text{R}\text{I}\text{S}\text{E} \end{array}\left(\begin{array}{ccc}0& & 347.5\\ & & \\ 342.5& & 0\end{array}\right)\\ \end{array}\)
Table 2 provides a detailed distribution of the subjects’ ranks collected for the different types of FV decisions, i.e., acceptance (TA and FA) and rejection (TR and FR), according to the FVX-PAP protocol. Specifically, \({Count}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)and \({Count}_{\varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E} }\)indicate the number of times the FV-RISE face heatmaps are preferred over the CorrRISE face heatmaps, excluding tied ranks, and vice versa, respectively. Moreover The \({Ties}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\) indicates the number of times the FV-RISE and CorrRISE face heatmaps are in a tie. Finally, \({Count\_Ties}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)and \({Count\_Ties}_{\varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\) indicate the number of times the FV-RISE face heatmaps are preferred over the CorrRISE face heatmaps, incorporating the tied ranks, and vice versa, respectively.
Table 2
Subjects’ rankings across the FVX-PAP protocol experiments
|
Acceptance Decisions Test Sub-Experiment
|
Rejection Decisions Test Sub-Experiment
|
True Acceptance
|
False Acceptance
|
True Rejection
|
False Rejection
|
\({Count}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)
|
215
|
251
|
277
|
286
|
\({Count}_{\varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)
|
254
|
259
|
270
|
281
|
\({Ties}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)
|
191
|
150
|
143
|
123
|
\({Count\_Ties}_{\varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)
|
310.5
|
326
|
348.5
|
347.5
|
\({Count\_Ties}_{\varvec{C}\varvec{o}\varvec{r}\varvec{r}\varvec{R}\varvec{I}\varvec{S}\varvec{E}, \varvec{F}\varvec{V}\_\varvec{R}\varvec{I}\varvec{S}\varvec{E}}\)
|
349.5
|
334
|
341.5
|
342.5
|
5.2.2 Explainability Performance Scores
To effectively assess with some detail the efficacy of the FV-RISE and CorrRISE explainability tools, a comprehensive approach is followed, which involves assessing their performance across the four FV decision types, notably TA, FA, TR, and FR. Consequently, the \({APCM }_{TA}, {APCM }_{FA}, {APCM }_{TR},\) and \({APCM }_{FR}\) matrices are used as described in Subsection 3.7.2 to estimate the corresponding explainability performance scores.
The FV-RISE and CorrRISE explainability performance scores for the different types of FV decisions e.g., acceptance (TA, and FA) and rejection (TR and FR) as well as the overall explainability performance scores, considering all cases together are depicted in Fig. 7. A higher explainability performance score indicates superior explainability performance for one tool regarding the other and vice-versa for the relevant FV decision-making.
The results in Fig. 7 allow deriving the following observations and conclusions:
• True acceptance decisions – CorrRISE outperforms FV-RISE by achieving an explainability performance score of 0.530 compared to a FV-RISE score of 0.470, see left side of Fig. 7(a), which indicates the ability of CorrRISE to provide better visual explanations when processing genuine faces and taking acceptance decisions.
• False acceptance decisions – The right side of Fig. 7(a) shows that CorrRISE and FV-RISE offer an equivalent explainability performance for false acceptance decisions, notably 0.506 versus 0.494, which proves that both explainability tools offer nearly equivalent visual explanations when an impostor is erroneously accepted as a genuine user.
• True rejection decisions – FV-RISE and CorrRISE achieve equivalent explainability performance score for true rejection decisions, see left side of Fig. 7(b). This behavior indicates that both explainability tools generate nearly similar dissimilarity visual explanations to explain rejection decisions when impostor faces are processed.
• False rejection decisions – For false rejections, the two tools offer almost equal explainability performance scores, see the right side of Fig. 7(b). This reinforces the conclusion that both explainability tools generate nearly similar dissimilarity visual explanations, but this time when genuine faces are processed and rejection decisions are made.
To gain a better understanding of the assessed explainability tools performance, the comparison of tools may be also carried out with respect to the type of FV decision, i.e., acceptance or rejection, regardless of whether genuine or impostor faces are processed, and all decision together, with the following conclusions:
• Acceptance decisions – The explainability scores reveal that CorrRISE performs slightly better than FV-RISE when acceptance decisions are made by exhibiting an explainability performance score of 0.52 compared to 0.48 for FV-RISE, see the left side of Fig. 7(c).
• Rejection decisions – For rejection decisions, FV-RISE exhibits slightly superior performance to CorrRISE with an explainability performance score of 0.504 versus 0.496 for CorrRISE. This result reinforces the observation from Fig. 7(b), indicating that both explainability tools offer nearly equivalent dissimilar visual explanations to explain FV rejection decisions.
• Overall decisions – The explainability performance scores are also estimated for FV-RISE and CorrRISE considering all the cases, regardless of whether genuine or impostor faces are processed and irrespective of acceptance or rejection decisions. Figure 7(d) allows concluding that both FV-RISE and CorrRISE offer a rather similar explainability performance with CorrRISE showing a very slight advantage over FV-RISE. This similar performance was somehow expected since the two explainability tools are technically rather similar as they share the adoption of the RISE random-masking technique to produce similarity/dissimilarity visual explanations, thus highlighting rather similar/dissimilar face regions in their explanations. Unfortunately, it was not possible to perform this FVX-PAP proof-of-concept experiment with other, technically mode different, FV explainability tools since no other tools generating heatmap-based visual explanations for the four types of FV decisions i.e., TA, FA, TR, and FR, are available, specially the false FV decisions i.e., FA and FR, while the target was to validate the proposed protocol across the four types of FV decisions.
In summary, the proposed FVX-PAP protocol offers a comprehensive approach for evaluating and comparing FV explainability tools generating visual explanations; this has demonstrated with an experiment involving heatmaps. The FVX-PAP protocol facilitates the quantification of human subjective evaluations pertaining to these tools by comparing their visual explanation outputs in pairs. Through this structured approach, the protocol enables the estimation of explainability performance scores, thereby facilitating a relative assessment of the explainability performance offered by the evaluated FV explainability tools.