The prediction accuracies of the NN model are 100% for training set and 88.2% (15/17) for test set. The confusion matrix of the test set is [[4, 1, 0], [0, 9, 0], [0, 1, 2]]. It is important to note that although a certain level of overfitting is expected, the test set accuracy is reasonable, as only two data points are incorrectly classified. From the test set confusion matrix, 1 out of 5 spallation cases and 1 out of 3 non-oxidation resistances are classified incorrectly, which can be due to lack of training data for those two specific classes.
The average SHAP contribution of all the features are provided in Fig. 1(a). As mentioned earlier, SHAP values explain how each of the features locally impact the selection of three classes of oxidation resistance (both directionally and quantitatively). The SHAP value of each of the feature quantifies the numerical influence on the final prediction. A positive SHAP value indicates that the feature pushes the prediction towards the particular class and negative value deters the selection of the class. Apart from the directional information, the quantitative estimation of the SHAP values can precisely estimation how much the positive or negative influence is. Mo has the highest contribution in classifying a composition as a good oxidation resistant alloy while Al has the most influence on spallation and non-oxidation resistant alloys. This may come as counter-intuitive if analyzed from a stable oxide forming element perspective. As the feature importance plot provides very little directional information, one gets no indication about the explainability of results from such plot. A scatter plot for the data-points for all the classes are shown in Fig. 1(b-d) which indicate the directional feature importance. For good oxidation resistant alloys, Mo concentration always contributes negatively while Al primarily contributes positively, i.e., increases the chance of forming oxidation resistant alloy (Fig. 1(b)). As the blue and red dots are intertwined for Cr in Fig. 1(b), its contribution is inconclusive. Figure 1 (c) clearly demonstrate the effect of Cr and Al in predicting non-oxidation resistant alloy. Alloys with low Al and Cr wt. % (blue dots) increases the chances of forming non-oxidation resistant alloy, e.g., alloys with thick oxide scale. On the other hand, high Mo containing alloys have higher chances of forming thick oxide scale as the red dots lie towards the right side in Fig. 1(c). In the experimental literature of FeCrAl oxidation, the addition of Al and Cr has been proven to form thin oxide scale at both high and low temperatures[26]. If Mo is present, it forms a think oxide scale at low temperature, making the alloy un-protective of oxidation[26]. Such understandings are drawn from expensive experimental characterization like Scanning Electron Microscopy (SEM), Transmission Electron Microscopy (TEM) of different FeCrAl compositions. Gaining similar insight from specific mass change data only is new to the community and can provide meaningful insight from inexpensive and high throughput tests if XAI is used correctly. In terms of spallation (Fig. 1(d)), the absence of Mo (blue dots) increases the change of oxide layer falling off. This is somewhat controversial, and no such direct evidence is obtained so far in the literature. That being said, FeCrAl alloys in absence of reactive elements and Mo, especially with zero to very low Al content, is susceptible to spallation at high temperature oxidation as reported in[26]. The presence of high Ni concentration in FeCrAl will generally encourage low oxidation resistance and more changes of spallation, therefore, will not form good oxidation resistant alloys. Ni is kinetically favored to form an oxide, but NiO is not a passivating layer as shown in the previous literature[27].
Next, we pull two alloys, one predicted correctly by the model (Fig. 2(a)) while the other was an incorrect prediction (Fig. 2(b)) to further analyze the SHAP contributions for each deeply. According to the original dataset and the threshold of good oxidation resistant alloy, they both form thin oxide scale as the mass gain is small (below 5.0×10− 4 g/cm2). Steam oxidation Fe-21Cr-5Al-3Mo at 900°C after 4 hours is predicted correctly with 84.9% chance for forming protective scale while the same compassion tested at 1000°C for 2 hours is predicted incorrectly with 78.8% chance of forming thick oxide scale. The SHAP contribution of all the components remain within ± 10% except for Mo. Mo reduces the chances for both the cases. For the high temperature short duration test (Fig. 2(b)), SHAP negative contribution of Mo increases by 30%, but it has not decreased high enough to classify it as good oxidation material. Looking back at the experimental specific mass gain data (i.e., 5.75×10− 4 g/cm2) for the wrong classification, we found it to be very close to the threshold (5.0×10− 4 g/cm2) as well. This emphasizes the importance of careful selection of threshold and care should be taken to select the threshold if a problem statement is changed from regression to classification.
The two-way interaction plot of two variables is important to understand the interaction of features, as shown for Al and temperature of steam oxidation in Fig. 3. Alloys without Al at high temperature tend to be poor oxidation resistant as absence of Al contributes positively towards classification of unprotective oxide formation which can be observed from Fig. 3(e). As our dataset is not well distributed in the temperature and Aluminum range, we stay away from drawing further conclusive information. The purpose is to demonstrate the ability of SHAP to inform what the model is doing and not treat it as a black box.
Provided all the features remaining unaltered, effect of one parameter change is helpful to understand the effect of single feature. The effect of Al wt. % changes from 0 to 7% for Fe-21Cr alloy for oxidation at 1300° C for 4 hours, e.g., regular operating scenario (Fig. 4(a)) and 600° C for 100 hours, e.g., accidental scenario (Fig. 4(b)). At 600°C, all the alloys fall into good oxidation resistant alloy category, primarily due to the presence of high Cr (21 wt. %) forming protective oxide. At high temperature, however, low Al alloys form thick oxide scale, and more than 6 wt. % Al alloys tend to indicate spallation. At high temperature (> 1000°C), only Al oxide is protective, and its absence will create thick oxide scale, hence oxidation non-protective alloy. On the other hand, as the model alloy does not have Mo or any other reactive element, high Al containing alloys are susceptible to spallation which has been experimentally reported[26]. The optimal performance is seen for 3.5 to 6 wt. % of Al. At low temperature, the Cr oxide is protective enough to make the alloy oxidation resistant even in the absence of Al.