Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals

doi:10.21203/rs.3.rs-3969238/v1

Download PDF

Article

Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals

https://doi.org/10.21203/rs.3.rs-3969238/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 07 Aug, 2024

Read the published version in Scientific Reports →

You are reading this latest preprint version

Growing restrictions and bans on animal testing for chemical safety assessment under different regulations have led to an increasing use of alternative methods. Read-across is one of the major approaches used for this purpose, which relies on the identification of toxicological hazards of a data-poor or untested (target) chemical from data on other already-tested (source) similar chemicals. This requires the target substance to be first assigned to a group or category of ‘similar’ chemicals. The ‘similarity’ may be in terms of structural features alone, or in combination with certain rules that are based on mechanistic and/or toxicological aspects. In this regard, the OECD QSAR Toolbox - a major free-access in silico platform - is widely used to derive toxicity predictions for a range of (eco) toxicological endpoints. The Toolbox allows the user to identify a set of similar chemicals (analogues) by computational ‘profilers’ that incorporate different structural alerts, or a combination of structural alerts and physicochemical and/or toxicokinetic rules relevant to a specific toxicological endpoint.

The overall aim of this study was to assess the performance of the in silico profilers provided in the OECD QSAR Toolbox for reliability for identifying chemical analogues for category formation in a number of high-quality databases on mutagenicity, carcinogenicity, and skin sensitisation. The study also aimed to identify the reasons for any limitations in the performance of the profilers, and propose ways to improve their overall accuracy. The results showed that whilst some structural alerts are fit-for-purpose as such within the acceptable limits, others need refinement or a consideration for their possible exclusion from the profiler. Such refinements are imperative for a reliable use of the profilers in read-across and grouping/categorisation for classification, labelling and risk assessment of chemicals.

Biological sciences/Computational biology and bioinformatics

Biological sciences/Computational biology and bioinformatics/Computational models

Chemical grouping based on category formation allows risk assessment of a data-poor substance through inference from the data on other members of the same category. The concept behind the approach is based on the notion that similar substances generally have similar properties. At the physicochemical level, the factors considered for grouping can be a similarity in chemical structures, functional group(s); metabolic/ degradation profiles, or additional parameters, such as log P, protein binding, etc. Where sufficient 'similarity' criteria are met in a set of chemical substances that follow a regular pattern, it can be considered a 'category'. According to the OECD grouping guidance (OECD, 2014): ‘A chemical category is a group of chemicals whose physicochemical and human health and/or ecotoxicological properties and/or environmental fate properties are likely to be similar or follow a regular pattern as a result of structural similarity. The similarities may be based on the following:

a common functional grouping
common constituents or chemical classes
an incremental and constant changes across the category
The likelihood of common precursors and/or breakdown products, via physical or biological processes, which result in structurally similar chemicals.’

The European Chemicals Regulation “REACH” encourages the use of grouping and categorisation of chemicals for classification, risk assessment and labelling purposes. Some of the data gaps in this regard can be filled using read-across within a given category, which is based on interpolation of experimental data from the tests conducted on 'similar' (source) substances to the untested (target) substance. The data for the endpoint in question for the target substance are predicted using the experimental data for the same endpoint of the source substance(s). Since each endpoint has a different set of complexities, e.g. in regard to mode of action against a biological target site, a read-across needs to be considered on an endpoint-by-endpoint basis (EChA, 2017).

There has been an increasing emphasis on applying the 3Rs principles to refine, reduce and replace the use of animals in chemical safety testing under most regulatory frameworks in Europe. A complete ban on animal testing has been implemented under the EU Cosmetic Regulation since March 2013. Such developments have made the use of read-across in risk assessments ever more important, with emphasis on the outcomes to be transparent and reliable. In this regard, the definition of 'similarity' between the target and source substances has also evolved to include similarity not only in the structural features, but also in other physicochemical, mechanistic and/or metabolic aspects (Schultz and Cronin, 2017).

For read-across to be valid, a robust category of analogues must be derived from high quality datasets. A valuable tool for achieving this is the OECD QSAR Toolbox, which is a freely available multifunctional in silico platform that allows the users to make informed decisions about toxicity predictions for a range of (eco)toxicological endpoints (Schultz et al, 2018). As part of the process of identifying a set of analogues for read-across, the Toolbox allows the user to apply structural alerts in the form of computational “profilers”. One or more profilers are available for many of the toxicological endpoints that are based around chemistry; e.g. covalent binding to proteins or DNA, and/or other mechanistic or toxicological aspects. These profilers incorporate structural alerts, and in some cases a combination of structural alerts and physicochemical rules, that are relevant to a specific toxicological endpoint. The target compound is first subjected to profiling, and then the profile is used to screen for compounds in the databases with the same or similar structural, mechanistic, and/or toxicological profiles. The analogues found this way are reduced to those that have measured values for the specific endpoint(s) of interest and therefore provide a basis for prediction of the endpoint value of the target compound.

The OECD QSAR Toolbox (referred to herein as the “Toolbox”) incorporates databases on chemical (e.g. properties), toxicological and ecotoxicological data, as well as estimated values from in silico QSAR models. The system also incorporates QSAR models built within an informatics chassis, that are designed for in silico prediction of toxicological hazard. The Toolbox therefore allows the user to perform a number of functions (OECD 2008):

Identification of analogues for a chemical, retrieval if experimental results available for those analogues and data gap filling by read-across or trend analysis;
Categorisation of large inventories of chemicals according to mechanisms or modes of action;
Filling of data gaps for a chemical by using appropriate model(s) from the collection of QSAR models;
Evaluation of the robustness of a potential analogue for read-across;
Evaluation of the appropriateness of a (Q)SAR model for filling a data gap for a particular target chemical; and
The capability of building QSAR models.

Over the years, the Toolbox has become a widely used platform for chemical grouping/categorisation and estimation of chemical toxicity by in silico methods and read-across. Despite the increasing reliance of risk assessors on the Toolbox, attempts have only recently started to assess the reliability and limitations of the profilers provided in the system (Devillers et al., 2011; Mombelli, 2012; Yordanova et al., 2019). As part of the current study, it was felt that an in-depth assessment of the profilers was necessary to determine the usefulness and limitations for their use in screening databases for analogous compounds for further use in read-across or development of (Q)SAR models. This study, therefore, investigated the performance of the OECD QSAR Toolbox profilers for mutagenicity, carcinogenicity and skin sensitisation potential of chemical substances, to assess how reliably they identified both positive and negative compounds contained within the databases. The analysis also aimed at investigating any factors that would limit the usefulness of a profiler, and to propose ways to improving their performance.

This study used version 3.1 of the Toolbox. As part of the appraisal of the available profilers, a number of databases integral to the Toolbox as well as a few additional databases from publicly available sources were examined. These are listed in Table 1.

Table 1

Toxicological databases used in this study (details given for each database are as retrieved at the time of this study in 2020)
Database/ source	Description
Bacterial mutagenicity ISSSTY (donated to the Toolbox by the Istituto Superiore di Sanità (ISS), Rome, Italy)	41,634 Ames test data points for 7,367 compounds. The overall endpoint value (positive, negative, equivocal, inconclusive) outcome is determined as: - Positive: at least one strain is positive (with or without Metabolic activation); - Equivocal: no strain is positive, and at least one equivocal result is present in one of the following strains (with or without Metabolic activation): TA1535, TA100, TA98, TA1538, TA1535, TA97; - Negative: no positive or equivocal results are present in any strain, and negative outcomes exist for: a) at least one strain from among TA1535 or TA100 or TA97 (with and without Metabolic activation); and b) at least one strain from among TA1538 or TA98 or TA1537 (with and without Metabolic activation); - Inconclusive: If none of the above criteria are fulfilled. When more than one experiment in one strain was available, the number of reported positive and negative studies was counted, and the strain overall outcome was determined as follows: if the percentage of Positive studies is lower than 40%, then outcome = Negative; if the percentage of Positive studies is between 40 − 60%, then outcome = Equivocal; if the percentage of Positive studies is higher than 60%, then outcome = Positive (Aljallal,2020).
Carcinogenicity and mutagenicity (ISSCAN) (donated to the Toolbox by the Istituto Superiore di Sanità (ISS), Rome, Italy)	6,979 data points for 1,150 compounds. There are three endpoints for which data are presented; gene mutation, summary carcinogenicity, and TD50. The TD50 data are not used in this current study. Gene mutation data in Ames reported in the same way as in the previous database, and a single datum point is available for each of the 832 compounds. Summary carcinogenicity data are represented as “positive”, “negative” or “equivocal”. Positives are carcinogenic in at least one experimental group; equivocal results are given to chemicals with equivocal results in at least one experimental group, together with negative results in the other experimental groups, and negatives are non-carcinogenic in all tests (Aljallal,2020).
Genotoxicity OASIS (donated to the Toolbox by the Laboratory of Mathematical Chemistry, Bourgas, Bulgaria)	7,500 compounds collected from seven sources. It contains data for mutagenic determined by the Ames test with and without metabolic activation. The database also includes chromosomal aberrations determined by in vitro tests using Chinese hamster lung cells (CHL, with and without S9). Micronucleus (MN) and mouse lymphoma gene mutation assay (MLA) are evaluated by Chinese hamster lung cells (CHL / IU) and by in vitro T-lymphoma cell lines, respectively. All endpoints were evaluated on a dichotomic scale (YES/NO). Data used in this study were the Ames test data, and chromosomal aberration as the Toolbox has profilers for these endpoints (Aljallal,2020).
CRD-AGES (Data from the UK Chemicals Regulatory Directorate and the Austrian Agency for Health and Food Safety)	179 pesticides with mutagenicity data, and 100 with carcinogenicity data (Worth et al 2010). Mutagenicity data are binary active/inactive from Ames tests and carcinogenicity also binary active/inactive from a range of tests
DSS Pesticide carcinogenicity (the US EPA National Center for Computational Toxicology, www.epa.gov/ncct/dsstox/index.html)	Summary carcinogenicity data for 1,282 pesticide compounds
SAR carcinogenicity and genotoxicity databases	These are high-quality data databases used by Leadscope® program to build in silico predictive models. Both databases contain summarised in vitro and in vivo cancer and mutagenicity endpoints along with chemical structures. To ensure the high quality data for SAR, structure form, analysis of various salt forms of chemical compounds and their respective toxicity data have also been carried out to derive an overall endpoint for the active portion of the chemical. Several sources of experimental test results have been included in these databases, such as from FDA, NTP, CCRIS, CPDB and other primary sources. All chemical structures have been provided in SAR, neutral and tested form, and confirmed for accuracy. The SAR carcinogenicity database includes 3,598 compounds with 11,538 test results and provides carcinogenicity study endpoint for male and female rat (1,774, 1,725 compounds respectively) and male and female mouse (1,640, 1,675 compounds respectively). The SAR genotoxicity database provides compound-level calls for 46 genetic toxicity endpoints for 10,534 compounds. These include 32 bacterial mutagenicity endpoints, four in vitro mammalian, five in vitro chromosomal aberration and six in vivo micronucleus results (Aljallal,2020).
EFSA Pesticides Mutagenicity	Ames test active/inactive classification for 741 pesticides, compiled by, and downloaded from the European Food Safety Authority.
NISS mutagenicity database	A database of 1,863 compounds with binary active/inactive Ames test data. Downloaded from the US National Institute of Statistical Sciences, (www.niss.org/)
Inchemicotox Skin Sensitisation	A version of the Cronin and Basketter dataset (Cronin and Basketter 1994) from the Inchemicotox project (www.inchemicotox.org/), comprising 322 compounds, with results taken from guinea pig maximisation tests. The classification is derived from the percentage of animals sensitised in the test: non-sensitiser = 0–9%, weak sensitiser = 10–29%, moderate sensitiser = 30–79%, strong sensitiser = 80–100%.
CAESAR Skin Sensitisation	209 compounds from the EU CAESAR project (www.caesar-project.eu). For developing classification models, this data set was subdivided in two classes, sensitiser (S) and non-sensitisers (N), which gave a good distribution of the numbers of compounds in each class. The class S merges the first four ranges established by ECETOC: Extreme (EC3 < 0.1%), Strong (0.1%<EC3 < 1%), and Moderate (1%<EC3 < 10%) and Weak (EC3 > 10%) ranges; the class N regroups all compounds belonging to the non sensitisers
ECETOX Skin sensitisation	39 compounds with experimental results on skin and respiratory sensitisation. The compounds were selected as known sensitisers and non-sensitisers for the assessment of novel test techniques. (www.ecetoc.org/technical-reports)
OECD Skin Sensitisation	1,036 compounds from 2 databases and includes chemicals tested by Local Lymph Node Assay,(LLNA) or Guinea Pig Maximization Test, (GPMT). Based on the observed skin sensitisation effect the chemicals are classified in three classes: - strong sensitisers, weak sensitisers or non-sensitisers
Lazar Opentox Rat Carcinogenicity	Lazy Structure-Activity Relationships (LAZAR) is an open-source tool for the prediction of complex toxicological endpoints such as carcinogenicity (female/male, hamster/ mouse/rat/rodent) and Salmonella mutagenicity. The compounds were selected from database of experimental toxicity data. Carcinogenicity models are based on CPDB, while the Salmonella mutagenicity model uses a dataset of 3895 compounds determined in vitro. (https://lazar.in-silico.de/predict)
VEGA carcinogenicity	The VEGA platform serves to access a number of QSAR models for predicting mutagenicity and carcinogenicity. The compounds were selected from a set of 4225 molecules tested with the Ames bacterial test. And for carcinogenicity were selected from set of 805 chemicals from the Carcinogenic Potency Database (CPDB)
The Carcinogenic Potency Database (CPDB)	Data relating to cancer causing chemicals were compiled from the Carcinogenic Potency Database (CPDB), which is freely available at http://toxnet.nlm.nih.gov/cpdb/cpdb.html. This database is a widely used and unique international resource comprising the results of 6,540 chronic, long-term animal carcinogenicity tests on 1,547 chemicals in rats, mice, dogs, hamsters and non-human primates (Aljallal,2020).

The profilers assessed in this study are listed in Table 2. These were used to assess the compounds that had experimental values against specific endpoints from the databases listed in Table 1.

Table 2

The OECD QSAR Toolbox profilers assessed in this study
Profiler/ toxicological endpoint	Description
Mutagenicity/ genotoxicity
DNA binding by OASIS v1.1	This profiler is a mechanistic profiler developed from an analysis of Ames mutagenicity data. It contains a number of structural alerts that have been shown to be related to established electrophilic reaction chemistry known to be important in covalent DNA binding (Mekenyan et al 2004; Serafimova et al. 2007)
DNA binding by OECD.	This profiler is based on structural alerts for the electrophilic reaction chemistry associated with covalent DNA binding (Enoch and Cronin 2010). The profiler returns a range of structural alerts that contain electrophilic centres or those that can be metabolically activated to electrophiles.
Carcinogenicity (genotox and nongenotox) alerts by ISS.	This profiler is based on a list of 55 structural alerts from the software Toxtree (http://toxtree.sourceforge.net/). About 20 of the alerts are for non-genotoxic carcinogenicity, and the remainder for genotoxic carcinogenicity (mutagenicity)
DNA alerts for AMES, MN and CA by OASIS v.1.1	a refinement of above
mutagenicity (Ames test) alerts by ISS.	The present list of SAs is a subset of the original Toxtree list, obtained by eliminating the SAs for nongenotoxic carcinogenicity and is a refinement of above
In vivo mutagenicity (Micronucleus) alerts by ISS	. This profiler is based on the ToxMic rule-base of the software Toxtree. This rule-base provides a list of 35 structural alerts (SAs) for a preliminary screening of potentially in vivo mutagens. These SAs are molecular functional groups or substructures that are known to be linked to the induction of effects in the in vivo micronucleus assay
Carcinogenicity
DNA binding by OASIS v1.1	As above
DNA binding by OECD	As above
Carcinogenicity (genotox and nongenotox) alerts by ISS	The SAs for carcinogenicity are molecular functional groups or substructures known to be linked to the carcinogenic activity of chemicals. As one or more SAs embedded in a molecular structure are recognised, the system flags the potential carcinogenicity of the chemical (Aljallal,2020).
OncoLogic Primary Classifier	This profiler consists of molecular definitions derived by the Toolbox developers to mimic the structural criteria of chemical classes of potential carcinogens covered by the U.S. Environmental Protection Agency’s OncoLogic™ Cancer Expert System for Predicting the Carcinogenicity Potential (www.epa.gov/oppt/sf/pubs/oncologic.htm).
Skin Sensitisation
Protein binding by OASIS	These profilers have been developed to indicative of skin sensitisation potential and consist of 85 structural alerts relating to 11 reactions, or chemical interactions, which are known to be associated with skin sensitisers
Protein Binding by OECD	The protein binding by OECD profiler contains 16 mechanistic alerts covering 52 structural alerts. These data are supported by mechanistic chemistry and references to the scientific literature (the meta data). They represent a parallel approach to those of the OASIS profiler and capture mechanistic features of target compounds
Protein binding potency	This profiler is developed on the basis of empirical data for thiol reactivity expressed by the in chemico RC50 value. All the chemicals have two common electrophilic mechanisms of interaction with GSH – interaction via SN2 and interaction via Michael addition (MA) mechanism. The profiler contains 49 MA and 46 SN2 categories
Keratinocyte gene expression	This profile is built in relation to the database derived from the KeratinoSens assay, which examines the potential for chemicals to induce the expression of a luciferase reporter gene under control of a single copy of the ARE element of the human AKR1C2 gene stably inserted into immortalised human keratinocytes. Relevance to skin sensitisation is inferred from the relationship of Keap1-Nrf2-ARE regulatory pathway and its detection of electrophilic chemicals to sensitisation. The profiler contains 22 categories
Protein binding alerts for skin sensitisation by OASIS	This profiler seems to be much the same as the one at 2.2.3.1 above though there are some minor differences
DPRA Lysine peptide depletion	This profile is built on the basis of data derived from Direct Peptide Reactivity Assay (DPRA). The DPRA is a reactivity assay which evaluates the ability of chemicals to react with proteins. Model synthetic peptides containing either lysine or cysteine are used. The remaining concentration of cysteine- or lysine-containing peptide is measure after 24 hours incubation with the test chemical at 25 ± 2.5ºC. The peptide reactivity is reported as percent peptide depletion. The relevance with skin sensitisation is the presence of cysteine and lysine residuals in the skin proteins. The profiler contains 24 structural alerts extracted from about 110 chemicals with experimentally measured lysine depletion values
DPRA Cysteine peptide depletion	As described above, this profiler contains 32 categories of alerts

Further information is provided in the supplementary file on the total number of substances identified with experimental data against each endpoint in different databases, and that used in the assessment of the profilers performance.

Data analysis

The compounds identified in the databases for the specific endpoint were profiled using appropriate profilers. For each of the compounds, if an alert was triggered, the compound was allocated a score of 1, if no alerts were triggered, a score of 0 was allocated. The results were compared with the assigned binary activities from the original database (positive = 1; negative = 0).

Cooper statistics (Cooper et al, 1979) were used to assess the results against the experimental values given in the databases, by calculating the sensitivity, specificity and accuracy of the alert triggers as follows:

Sensitivity (True positive rate) = TP/ TP + FN
Specificity (True negative rate) = TN/TN + FP
Accuracy = (TN + TP)/(TN + FP + FN + TP)
PPV (Positive predictive value) or (precision) = TP/ TP + FP
MCC = (TPxTN)-(FPxFN)/√(TP + FN)(TP + FP)(TN + FN)(TN + FP)

Where TP = True positive, TN = True negative, FP = False positive, FN = False negative

Sensitivity is defined as the percentage of correctly classified positive predictions among the total number of positive instances.

Specificity is the percentage of correct negative predictions compared to the total number of negatives.

Accuracy (concordance or “Q”) is defined as the total number both positive and negatives correctly predicted among the total number of compounds.

PPV (positive predictive value) is defined as the total number of correctly classified positive predictions among the total number of both negative and positive instance.

MCC (Matthews correlation coefficient) is a weighted value that overcomes any imbalance in the data classes which might lead to over optimistic values of Q (Matthews, 1975). An MCC value of 1 indicates that the model can predict the data classes of unknown compounds perfectly, whilst a MCC value of 0 indicates that the predictions are no better than random guessing, and a MCC value of -1 indicates total disagreement between the predicted data and the actual data.

The results of the assessment of the profilers against the experimental data for mutagenicity, carcinogenicity and skin sensitisation are shown in Tables 3, 4 and 5 respectively. Further detailed analysis was undertaken to identify over-predicting structural alerts in the carcinogenicity profilers i.e. with the Precision or PPV (positive predictive value) lower than 0.5. This analysis was performed to determine structural alerts with little information or predictive capability to increase the sensitivity and overall accuracy of the profiler. Detailed analysis of 13 non-genotoxic carcinogenicity structural alerts was conducted and results are presented in Table 6.1 and 6.2.

An additional analysis for the Oncologic Primary Classification carcinogenicity profiler was performed for 30 structural alerts incorporated in the profiler. The purpose of this analysis was to assess which of the structural alert(s) had a precision PPV (positive predictive value) lower than 0.5. This analysis is presented in Table 6.3.

The cutoff value was set to be 0.5 to ensure that none of these profilers had a lower predictive power compared to bacterial Ames test, which has also been applied to predict carcinogenicity of genotoxic substances in rodents. The high predictive power of positive Ames, which ranges from 77–90% depending on various factors, makes it superior to any other in vitro genotoxicity assay, all of which have a relatively lower performance in terms of predicting genotoxicity (Kazius et al., 2006).

Mutagenicity profilers

The results shown in Tables 3 indicated that the accuracy (percentage of positives and negatives correctly predicted) of the mutagenicity profilers varies across the datasets from 51–76%. Clearly, whilst 76% can be accepted because it is in line with the level of error generally seen in the experimental data in most databases, 51% is barely better than chance. The micronucleus alerts appear to be general and, as such, significantly over-predict mutagenicity. The most common alert triggered by this profiler is “Hacceptor-path3-Hacceptor”. This alert indicates the non-covalent binding of the target chemical to DNA via two bonded atoms connecting two H bond acceptors (Snyder et al. 2006). However, it appears that such a functional grouping is common in both mutagens and non-mutagens. It is likely that the performance of this profiler would improve if this specific alert was omitted.

As expected, both DNA binding profilers work best with the data obtained from Ames type tests (due to the availability of large databases for this test), but do not perform well for chromosome aberration or micronucleus data.

The genotoxicity and non-genotoxicity alerts (ISS) have acceptable true positive results but fail to distinguish the negatives. Overall, these alerts perform best with Ames type data.

The OASIS DNA alerts for Ames, micronucleus and chromosomal aberration predict the results in Ames datasets fairly well but for both micronucleus data and chromosomal aberration data, these profilers underpredict positive compounds with sensitivity rates ranging from 36 to 44%. The ISS Ames test alerts have accuracies over 70% for Ames datasets, which together with MCC values greater than 0.5, indicate that the performance is independent of skewed sample categories. It, however, needs to be noted that the micronucleus/ CA alerts may not be suitable predictors of Ames. They need to be considered separately and used, along with Ames, to develop the overall weight of evidence.

The receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as the y coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the quality or performance of diagnostic tests which in this case is an in silico profiler. As it is shown in Fig. 1, the ROC curve analysis showed that Ames ISS profiler achieved the highest balanced accuracy for both true positive rate and low false positive rate values.

Carcinogenicity profilers

Both DNA binding profilers performed equally poor with carcinogens and non carcinogens from all datasets, with accuracy values rarely above 60% and, MCC values indicating a performance barely better than chance.

The ISS carcinogenicity alerts fared a little better in predicting carcinogens, but showed a poor segregation of non-carcinogens reduced the overall effectiveness of this profiler with accuracy levels between 57% and 68% for the sample datasets, and modest to poor performance on skewed datasets as indicated by the MCC values of 0.17 to 0.36. The ROC analysis, shown in Fig. 2, indicate that ISS carcinogenicity profiler was the highest quality performance profiler compared to the other 3 profilers in terms of both true positive rate and false positive rate.

13 Non-genotoxic carcinogenicity (NGC) structural alerts among ISS carcinogenicity profiler were analysed individually to test their performance by PPV (positive predictive value). As shown in Table 6.1, the overall PPV of ISS non-genotoxic carcinogen structural alerts was 57%, where 326 substances are truly predicted as nongenotoxic carcinogen out of total 570 substances that had been identified to contain one of the 13 NGC structural alerts.

The precision value (PPV) for non-genotoxic carcinogenicity structural alert ranged from 0.92 for Trichloro (or fluoro) ethylene and Tetrachloro (or fluoro) ethylene as a highest PPV to 0.39 for Quercetin type flavonoids. Four out of 13 NGC structural alerts show over-prediction. These are thiocarbonyl, substituted n-alkylcarboxylic acids, quercetin type flavonoids, and phtalate (or butyl) diesters and monoesters. All of these four structural alerts predict non carcinogenic substances as carcinogens in more than 50% of the total substances that contain this structural alert, which lowers the total accuracy of the ISS carcinogenicity profiler.

Thiocarbonyl NGC structural alert was flagged in 106 substances as the only structural alert. Any substance that contained more than one NGC structural alert was not counted in this analysis to avoid any interference. Sixty two non-carcinogenic substances were falsely predicted as carcinogenic substances by thiocarbonyl structural alerts with sensitivity rate of 0.42. Due to this over-prediction of thiocarbonyl alert, it can be suggested that taking this alert out could increase sensitivity of the total NGC structural alerts and the ISS carcinogenicity profiler. This would, however, not be the ideal solution, as any thiocarbonyl NGCs would then be completely out of the scope of the profiler. Instead, it is proposed that further research be carried out to see whether performance of this alert can be improved using a larger database of thiocarbonyl substances.

Likewise, the other three NGC structural alerts; i.e. substituted n-alkylcarboxylic acids, quercetin type flavonoids and phthalate (or butyl) diesters and monoesters also showed a precision value lower than 0.5 with a 0.42, 0.39 and 0.38 respectively. Again ignoring these four structural alerts increased the total precision value of NGC structural alerts and consequently of the performance of the ISS carcinogenicity profiler. The results (Table 6.2) showed that the precision value of ISS non-genotoxic structural alert was improved from 0.57 to 0.64 by ignoring the 4 structural alerts. However, for the reasons mentioned before, it is proposed that further research should be carried out to improve the performance of these alerts within the profiler.

The Oncologic primary classification alert over-predicted carcinogens with sensitivity rates of 66–73% at the expense of poor prediction of non-carcinogens (30–46%), resulting in overall performance, which is barely better than chance for most of the datasets. All 30 structural alerts in Oncologic primary classification profiler were individually analysed for their precision as shown in Table 6.3. The four structural alerts showed over-prediction of non-carcinogenic substances as carcinogenic in more than 50% of the total substances containing this structural alert. These four alerts were carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone reactive functional groups. Carbamate type compounds structural alert for carcinogenicity was triggered in 63 substances with only 22 true positive carcinogenic substances. The other 43 substances that were flagged by this alert to be carcinogenic were non-carcinogenic in real experimental tests. This gave a low precision value for carbamate type compounds of 0.35. The second structural alert in oncologic primary classification profiler with low precision value (lower than 0.36) was organophosphorus type compounds structural alert, where 80 substances out of 126 flagged by this alert were wrongly predicted as carcinogenic substances. Peroxide type compounds structural alert showed only 0.27 precision (positive predictive value) rate with only 3 correctly predicted carcinogenic substances out of 11 substances flagged by the alert. The lowest structural alert in precision within oncologic primary classification profiler was for reactive ketone functional groups with only 0.1 where there was an over-prediction for 19 out of total 21 flagged substances. This mean that only 2 substances that were flagged by this alert were correctly predicted as carcinogenic substances out of the total 21 substances.

It can therefore be suggested that all four of these structural alerts could be ignored from the oncologic primary classification profiler to increase the total sensitivity and accuracy of the profiler. Indeed, as shown in Table 6.3, the total precision of the profiler was improved by 0.023 which is nearly 3% improvement in the overall performance of the profiler.

This would, however, not be the ideal solution, as this will miss carcinogenic compounds and would be completely out of the scope of the profiler. Therefore, it is proposed that further research be carried out to see whether performance of these four alerts can be improved using a larger database of thiocarbonyl substances.

Skin sensitisation profilers

The performance of the protein binding profilers was not found to be consistent across the sampled datasets. For the CAESAR dataset, these profilers tend to have a low predictivity for non-sensitisers, whilst for the other datasets it is the sensitisers that are not well predicted. Overall, the performance of these predictors is moderate to poor for all the datasets.

The DPRA lysine peptide depletion profilers showed a similar pattern, with performance being uniformly not better than chance for the CAESAR dataset but highly under-predictive for sensitisers in the other datasets. The protein binding potency profiler is also uniformly poor across all datasets, failing to detect the majority of sensitisers, with sensitivity rates between 12% and 29%. A similar pattern is seen with the keratinocyte gene expression profiler, where sensitivity rates were 19–43%.

The overall ROC analysis for all seven profiler shown in Fig. 3 indicated that protein binding OASIS has a relatively better performance compared to the other skin sensitization profilers in OECD Toolbox.

The study also carried out further analysis to investigate the contribution of different alerts within a given profiler on its overall performance. For this purpose, non-genotoxic carcinogenicity profilers were investigated as an example. This was because, under most EU regulatory frameworks, tests for carcinogenicity are only required when there is either a positive in vitro mutagenicity/genotoxicity test, or there are indications of carcinogenic effects from long term in vivo studies. This means that, whilst the current risk assessments framework would identify genotoxic carcinogens, it is possible that the carcinogenic effects of a nongenotoxic mechanism will not be identified without in vivo tests that are increasingly restricted or banned under different regulatory frameworks.

This study has assessed and evaluated the structural alerts and profilers provided in the OECD QSAR Toolbox for mutagenicity, carcinogenicity, and skin sensitisation. These profilers are provided for the purpose of constructing categories of mechanistic and identifying structural analogues of a target compound. Apart from the OECD QSAR Toolbox, profilers are also used by a number of other applications for indicating certain toxicity endpoints, such as Toxtree (http://toxtree.sourceforge.net/), and Oncologic (www.epa.gov/oppt/sf/pubs/oncologic.htm). Although not intended for making toxicological predictions in themselves, these profilers provide a useful means for read-across from experimental data on analogous compounds to estimate a property or biological activity of an untested compound. The accuracy and reliability of a profiler in terms of predicting a target compound is, therefore, important for defining the structural and functional features so that it is placed in the correct category/group of analogous substances for the purpose of read-across. This is also important because, in practice, safety of chemical substances is often based on (Q)SAR Toolbox profilers alone in the dossiers submitted for regulatory risk assessment, where the absence of an alert alone is (wrongly) argued as an evidence for the absence of toxicity.

In this regard, those profilers that are equally likely to select endpoint-positive and endpoint-negative compounds into a grouping of analogues to be used in a read across, will by definition give rise to equivocal predictions for the target compound. This makes it essential to know how accurately different profilers perform in terms of sensitivity, specificity and accuracy, and to investigate possibilities for their improvement.

This also requires an understanding of the role and merits of each individual alert within a profiler so that only the most relevant and reliable ones can be relied upon as indicators of the particular endpoint. The results of this study have shown that many structural alerts within each profiler have unacceptably low predictivity, which has a bearing on the performance of the relevant profiler.

As an example, this study has found that the alert “Hacceptor-path3-Hacceptor” in the micronucleus profiler is too ubiquitous to be a useful indicator of mutagenicity. Similarly, in three structural alerts for non-genotoxic carcinogenicity within the ISS carcinogenicity profiler (thiocarbonyl, sub alkyl carboxylic acid, quercetin and phthalate), and the four structural alerts in oncologic primary classification profiler (carbamate type compounds, organophosphorus type compounds, peroxide type compounds and reactive ketone functional groups) were identified to have extremely low precision values. As shown in the examples investigated in this study, the omission or substitution of those alerts that unduly draw predictions towards equivocal outcomes within a profiler, can improve the overall performance of the profilers. This, however, would bring the drawback of making some of the toxicity alerts out of the scope of the profiler, and therefore further research is needed to find out whether they can be substituted with other more efficient alerts, or their performance can be enhanced by other means, e.g. by exploring larger datasets of relevant substances.

Another factor influencing the segregation of compounds by the alerts could be the way in which the categorical data found in the 3 endpoint datasets studied here is derived. Very often the binary categorisation of data is achieved by manipulation of the continuous data in some way to provide “cut off” points for positive or negative assignment. The way in which this is done may affect the flagging of an alert in an essentially “negative” compound, or vice-versa. By definition, most alerts have been derived from datasets that are rich in endpoint-positive compounds, because deriving “negative alerts”, like proving a negative hypothesis, is generally not feasible, and negative results are not published so often. This study, therefore, has also given an insight into those alerts that may be found equally in endpoint-positive or negative compounds, and those which may be more effectively utilised to form groups of analogues for read-across predictions.

Further research in this area is necessary to study the suitability and merits of each of the alerts within the profilers, both within the OECD Toolbox and other in silico toxicity platforms, to identify the root causes of the inadequacies and to investigate the possibilities for improvement in the profilers’ performance. This will, by implication, also improve the reliability of chemical read-across and grouping/categorisation for use in classification, labelling and risk assessment.

The main goal of the study was to inform the scientific community about any limitations of the Toolkit profiles, and the structural alerts contained within the profilers, that are otherwise useful in understanding the underlying mechanisms of toxicity. The in-depth analysis has indicated that many of the profilers have quite poor performance when tested against a large number of compounds with experimental data. This casts doubts over their usefulness in category formation for read-across for which they are commonly employed for subsequent use in regulatory risk assessments. There is therefore a need for caution when using the profilers for chemical category formation and in read-across, because reliability of the latter would be dependent on the accuracy and specificity of grouping/category formation in the first place. It has also highlighted that toxicity predictions should not be performed blindly by relying on structural alerts alone, and that other in silico tools, such as QSAR models, should be employed to develop a more reliable weight of evidence for that purpose. Another important point is that the influence of any part of a compound on its biological effect(s) is not constant and strongly depends on its structural environment. Thus, any alert, even when derived by mechanistic interpretation of statistically significant QSAR models does not have automatic predictive power. Alerts should therefore be viewed as structural hypotheses of chemical action only, and their true predictive power should be confirmed by QSAR predictions and, if possible, through experimental validation. In this context, this study is positioned as a pragmatic statement that could change the thinking of both regulators and researchers.

Funding statement

The authors did not receive support from any organization for the submitted work.

No funding was received to assist with the preparation of this manuscript.

No funding was received for conducting this study.

No funds, grants, or other support was received.

Author Contribution

M.J., Q.S. and N.P designed the study,M.J filtered and did the calculation for the profilers. M.J wrote the manuscript with support from Q.S and N.P All authors discussed the results and contributed to the final manuscript. All authors provided critical feedback and helped shape the research, analysis and manuscript.

Aljallal, M (2020) Investigation of in Silico Modelling to Predict the Human Health Effects of Cosmetics Ingredients. Doctoral thesis, Liverpool John Moores University.
Balls, M. (2019). Chapter 1.1 - The Introduction and Influence of the Concept of Humane Experimental Technique. In: M. Balls, R. Combes and A. Worth, ed., The History of Alternative Test Methods in Toxicology. pp.3-6.
Benigni, R. (2008). The Benigni/Bossa rulebase for mutagenicity and carcinogenicity–a module of Toxtree. EUR, pp.23241:1–70.
Cronin, M., Madden, J., Enoch, S. and Roberts, D. (2013). Chemical Toxicity Prediction: Category Formation and Read-Across. Royal society of chemistry (RSC), pp.1-29.
Devillers, J., Mombelli, E. and Samserà, R. (2011). Structural alerts for estimating the carcinogenicity of pesticides and biocides. SAR and QSAR in Environmental Research, 22(1-2), pp.89-106.
European chemical agency (Echa) (2017). Grouping of substances and read-across - ECHA. [online] Echa.europa.eu. Available at: https://echa.europa.eu/documents/10162/13628/read_across_introductory_note_en.pdf [Accessed 18 Sep. 2018].
Enoch, S.J. et al. (2011) ‘A review of the electrophilic reaction chemistry involved in covalent protein binding relevant to toxicity’, Critical Reviews in Toxicology, 41(9), pp. 783–802. doi:10.3109/10408444.2011.598141.
Mekenyan, O., Dimitrov, S., Serafimova, R., Thompson, E., Kotov, S., Dimitrova, N. and Walker, J. (2004). Identification of the Structural Requirements for Mutagenicity by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals I: TA100 Model. Chemical Research in Toxicology, 17(6), pp.753-766.
Mombelli, E. (2012). Evaluation of the OECD (Q)SAR Application Toolbox for the profiling of estrogen receptor binding affinities. SAR and QSAR in Environmental Research, 23(1-2), pp.37-57.
OECD (2008) (Q)SAR Application Toolbox version 1.1: Getting Started, Organisation for Economic Cooperation and Development, Paris, France.
Schultz, T. and Cronin, M. (2017). Lessons learned from read-across case studies for repeated-dose toxicity. Regulatory Toxicology and Pharmacology, 88, pp.185-191.
Schultz, T., Diderich, R., Kuseva, C. and Mekenyan, O. (2018). The OECD QSAR Toolbox Starts Its Second Decade. In: O. Nicolotti, ed., Computational Toxicology. New York, NY: Humana Press, pp.55-77.
Serafimova, R., Todorov, M., Pavlov, T., Kotov, S., Jacob, E., Aptula, A. and Mekenyan, O. (2007). Identification of the Structural Requirements for Mutagencitiy, by Incorporating Molecular Flexibility and Metabolic Activation of Chemicals. II. General Ames Mutagenicity Model. Chemical Research in Toxicology, 20(4), pp.662-676.
Worth A., Lapenna S., Lo Piparo E., Mostrag-Szlichty A., Serafimova R. (2010) The Applicability of Software Tools for Genotoxicity and Carcinogenicity Prediction: Case Studies relevant to the Assessment of Pesticides JRC Scientific and Technical Report. 2010 EUR 24640.
Yordanova, D., Schultz, T., Kuseva, C., Ivanova, H., Pavlov, T., Chankov, G., Karakolev, Y., Gissi, A., Sobanski, T. and Mekenyan, O. (2019). Alert performance: A new functionality in the OECD QSAR Toolbox. Computational Toxicology, 10, pp.26-37.

Tables 3 to 6 are available in the Supplementary Files section.

No competing interests reported.

Download PDF

Journal Publication

published 07 Aug, 2024

Read the published version in Scientific Reports →

Editorial decision: Revision requested
12 Jun, 2024
Reviews received at journal
07 Jun, 2024
Reviewers agreed at journal
29 May, 2024
Reviews received at journal
02 May, 2024
Reviewers agreed at journal
30 Apr, 2024
Reviewers agreed at journal
25 Apr, 2024
Reviewers invited by journal
15 Mar, 2024
Editor assigned by journal
11 Mar, 2024
Editor invited by journal
11 Mar, 2024
Submission checks completed at journal
11 Mar, 2024
First submitted to journal
19 Feb, 2024

You are reading this latest preprint version

Assessment of Performance of the Profilers Provided in the OECD QSAR Toolbox for Category Formation of Chemicals

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Materials and Methods

Data analysis

Results

Mutagenicity profilers

Carcinogenicity profilers

Skin sensitisation profilers

Discussion & Conclusions

Declarations

Author Contribution

References

Tables

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1