Modeling and the use of surrogate endpoints: Is this a valid approach?

doi:10.21203/rs.3.rs-5073551/v1

Download PDF

Research Article

Modeling and the use of surrogate endpoints: Is this a valid approach?

https://doi.org/10.21203/rs.3.rs-5073551/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Development of novel colorectal cancer (CRC) screening tests is a dynamic field. Decision analytic modeling based on inputs derived from rigorous prospective studies informs CRC screening guidelines. Exploratory modeling may have a place in early phases of test development. We explored whether 1) surrogate endpoints for long-term, programmatic effectiveness and cost-effectiveness could be potentially informative in early stages of test development, and 2) whether rapid exploratory modeling with a web-based tool would be feasible. First, based on comparisons with published estimates for reductions in CRC mortality with various screening tests in four established decision analytic models of CRC screening, the surrogate endpoint of the number needed to colonoscope to detect one CRC or advanced precancerous lesion (APL) in round 1 of screening appears to hold promise as a measure of clinical effectiveness. Similarly, based on comparisons with published estimates for cost/quality-adjusted life-year gained with screening in the four models, the surrogate endpoint of cost to detect one CRC or APL in round 1 of screening appears to hold promise as a measure of cost-effectiveness. Second, exploration of the impact of lowering the screening initiation age in Israel from age 50 to 45 with the web-based EU-TOPIA tool, compared with the results of a recently published comprehensive modeling study, suggests that exploratory modeling of programmatic screening may be feasible with relatively low time demand vs. that required for typical full-length modeling publications. Further validation will be needed before surrogate endpoints or rapid modeling are incorporated into the novel test development process.

Despite the solid evidence that screening can decrease colorectal cancer (CRC) incidence and CRC mortality [1, 2], and the availability of multiple screening test options with a range of attributes [3], screening for CRC and its precursors remains underutilized around the world among persons who are eligible for screening. Research and development of novel tests to screen for CRC and its precursors is a dynamic field [4–8], fueled by the goals to improve the sensitivity and specificity of available tests, and to address patient concerns over the attributes of existing tests (e.g. the stool handling that is required by fecal test-based strategies, or the invasive nature of colonoscopy and the required colon cleansing before colonoscopy), and by the commercial opportunities for industry in the large CRC screening market. From the public health perspective, novel tests should ideally deliver improved outcomes at acceptable incremental costs and ease of use.

The recently published update to the World Endoscopy Organization (WEO) CRC Screening Committee Expert Working Group (EWG) guiding principles on the evaluation of novel non-invasive screening tests proposes a four-phased approach, and considers where decision analytic modeling may be useful [9]. Phases I and II consist, respectively, of small studies assessing the test’s ability to discriminate between CRC and non-cancer states, and the test’s accuracy across the continuum of neoplastic lesions in neoplasia-enriched populations. Progress to Phase III requires promising results in Phases I and II. Phase III consists of prospective studies to determine single round intention-to-screen outcomes, and confirmation of the test positivity threshold. The ambitious Phase IV involves evaluation over repeated screening rounds with monitoring for missed lesions. The consensus document states that modeling studies that mimic randomized controlled trials and are based on high-quality observational data can be informative, but there is no explicit recommendation on how they might be incorporated in the early phases of test development.

The rich published literature on the cost-effectiveness of CRC screening typically focuses on screening tests with solidly ascertained test performance characteristics, usually in randomized controlled trials or large prospective studies (i.e. usually Phase III studies) [10]. While Phases III and IV provide the most reliable data to model test impact on CRC incidence and CRC mortality in scenarios that are not addressed directly in the real-world trials, exploratory modeling can contribute to research and development efforts at earlier stages in the development of novel tests.

During the planning and preparation of the recent WEO consensus document, one of its lead authors, Graeme Young, asked whether early-stage proxies/surrogates for long-term programmatic effectiveness and cost-effectiveness could serve as a form of rudimentary early-stage modeling to inform test development. The aim of developing such early-stage proxies/surrogates would be to incorporate a quantitative estimation of the potential clinical and economic impact of emerging tests during the early phases of test development (Phases 1 and II), as opposed to only during later phases.

Preliminary exploration of two potential surrogates, the number needed to colonoscope to detect one CRC or advanced precancerous lesion (APL), and the cost to detect one CRC or APL in one (the initial) round of screening, was presented by one of the authors (UL) at the WEO CRC Screening Committee 2022 meeting in San Diego. During the discussion, another author (ILV) asked whether proxies/surrogates were necessary if comprehensive modeling could be undertaken instead. The aim of the current paper is to explore these two related questions.

The paper consists of two distinct parts. First, we explored two potential early-stage surrogates of effectiveness and cost-effectiveness by comparing them to the long-term predictions of four established decision analytic models of CRC screening in the US population (MOSAIC [11], and three models of the CISNET consortium, MISCAN, CRC-SPIN and SIMCRC [12]). In order to provide the greatest transparency, we used recently published estimates from the four models in two publications [11, 12] that explored the potential effectiveness and cost-effectiveness of novel blood-based biomarkers of colorectal neoplasia. These publications focused on the potential long-term effectiveness and cost-effectiveness of a hypothetical test that meets the minimum thresholds set by the Centers for Medicare and Medicaid Services (CMS) of CRC sensitivity ≥74% and specificity ≥90%, but they also examined other existing or hypothetical tests [13]. In the current study, to reflect clinical effectiveness, we compared the models’ published predictions for long-term CRC mortality reduction to the newly calculated number needed to colonoscope (NNC) to detect one CRC or one APL in round 1 of screening for a series of tests, based on test performance characteristics. To reflect cost-effectiveness, we compared to the models’ published predictions for long-term cost-effectiveness to the newly calculated cost to detect one CRC or one APL in round 1 of screening for the series of tests, based on test performance characteristics and associated costs. The model’s results for discounted cost/quality-adjusted life-year [QALY] gained vs. no screening were used as the measure of cost-effectiveness.

Second, as a first exploration of the feasibility of performing comprehensive modeling during early stages of test development using existing tools, we performed first-pass modeling with a web-based tool of the EU-TOPIA project (https://eu-topia.org) [14], which is based on MISCAN, and compared the results to those recently published for a model that was adapted from the previous version of MOSAIC [15]. The case study for this exercise was the potential effectiveness of screening with annual fecal immunochemical testing (FIT) with various starting and stopping ages in Israel, with a focus on the potential incremental benefit of lowering the screening initiation age from 50 to 45 years. Given that we were interested in the feasibility of two different methods (i.e. use of surrogate markers or rapid modeling), it was not necessary for the two parts of the current study to be based on the same scenarios. Time accounting for modeling with the EU-TOPIA tool was performed in order to reflect the potential effort required to carry out exploratory modeling.

Comprehensive models

Exploration of surrogate outcomes was based on comparison to the published estimates of four established decision analytic models, MOSAIC [11], and three models of the CISNET consortium, MISCAN, CRC-SPIN and SIMCRC [12]. Table 1 shows the inputs used by each model for test performance characteristics, screening interval and cost for a variety of screening tests. The long-term programmatic estimates that were used as comparators for the surrogate endpoints were published recently [11, 12], reflecting screening from ages 45 through 75 in an average-risk US population.

Table 1

Test performance characteristics, screening interval and cost assumptions, by model
Strategy	Sensitivity for colorectal cancer (CRC)	Sensitivity for advanced precancerous lesion (APL)	Sensitivity for non-advanced adenoma (NAA)	Specificity (equal to 1minus the false positive rate in normal colon)	Screening interval (years)	Cost (age 65+), US dollars	Cost (age < 65), US dollars
MOSAIC*
FIT, 1y	0.733	0.238	0.076	0.964	1	$18.05	$18.05
FIT, 2y	0.733	0.238	0.076	0.964	2	$18.05	$18.05
CMS, 3y	0.74	0.1	0.1	0.9	3	$509	$681
CMS, 2y	0.74	0.1	0.1	0.9	2	$509	$681
CMS, 1y	0.74	0.1	0.1	0.9	1	$509	$681
FIT-DNA	0.933	0.424	0.172	0.898	3	$509	$681
Shield	0.83	0.13	0.1	0.9	3	$509	$681
mSep9	0.64	0.22	0.22	0.79	3	$192	$498
CMS (high sens)	0.9	0.7	0.3	0.9	1	$509	$681
CMS (adjusted sens)	0.75	0.35	0.15	0.95	1	$509	$681
MISCAN, CRC-SPIN, SIMCRC**
FIT	0.738	0.238	0.076	0.964	1	$23.42	$25.53
CMS	0.74	0.1	0.1	0.9	3	$500	$675
FIT-DNA	0.94	0.42	0.15	0.91	3	$550	$550
Shield	0.83	0.13	0.1	0.9	3	$500	$675
mSep9	0.7	0.2	0.2	0.804	3	$500	$675
FIT (stage-specific sens)***	0.738	0.238	0.076	0.964	1	$23.42	$25.53
CMS (stage-specific sens)***	0.74	0.1	0.1	0.9	3	$500	$675
* Colonoscopy (diagnostic/with intervention) costs $786/1,106 and $1,415/$1,991, for ages 65 + and < 65, respectively
** Colonoscopy (diagnostic/with intervention) costs $950/1,312 and $1,428/$1,890, for ages 65 + and < 65, respectively
*** Colorectal cancer stage I sensitivity assumed to be 0.8-fold the sensitivity for colorectal cancer stages II–IV

Surrogate endpoints

A simple calculator was created in Excel (Microsoft, Redmond WA) to calculate the surrogate endpoints. Table 2 shows the elements of the calculator, which are an assumed underlying prevalence of CRC, APL and non-advanced adenoma (NAA) in the population; the test performance characteristics of the test being evaluated; the cost of the test; and the cost of colonoscopy without intervention (i.e. colonoscopy without biopsy or polypectomy in persons with a normal colon with a false-positive screening test) or with intervention (i.e. with biopsy or polypectomy in persons with CRC, APL or NAA).

Table 2

Calculator for surrogate markers, with illustration for the fecal immunochemical test (FIT) based on assumptions in modeling with MOSAIC
	Test performance characteristics (TPC)				Screening interval (years)	Cost, US dollars
	Sensitivity for colorectal cancer (CRC)	Sensitivity for advanced precancerous lesion (APL)	Sensitivity for non-advanced adenoma (NAA)	Specificity (equal to 1 minus the false positive rate in normal colon)	Screening interval (years)	Cost, US dollars
Screening test = FIT	0.733	0.238	0.076	0.964	1	$18.05
	Prevalence (Prev) of colon health states
	Prevalence CRC, out of 10,000	Prevalence APL, out of 10,000	Prevalence NAA, out of 10,000	Prevalence normal colon, out of 10,000	Total cohort, n	--
Lesion prevalence based on Imperiale et al. [16]	65	758	2,896	6,281	10,000	--
	Positive screen (calculated from TPC x Prev)
	Among those with CRC out of 10,000	Among those with APL, out of 10,000	Among those with NAA, out of 10,000	Among those with normal colon, out of 10,000	Not referred to colonoscopy, out of 10,000	Total referred to colonoscopy	CRC or APL referred to colonoscopy
Positive screen and referred to colonoscopy (based on test sensitivities and specificity)	0.733 x 65 = 48	0.238 x 758 = 180	0.076 x 2,896 = 220	(1-0.964) x 6,281 = 226	9,325	674	228
	Costs *
	Cost (screening test plus colonoscopy with intervention)	Cost (screening test plus colonoscopy with intervention)	Cost (screening test plus colonoscopy with intervention)	Cost (screening test plus colonoscopy without intervention)	Cost (screening test only)	Total cost for 10,000
Cost (screening test cost plus colonoscopy cost, as applicable)	$95,721	$362,441	$442,184	$324,036	$168,330	$1,392,711
Surrogate markers
Number needed to colonoscope to detect one CRC or APL						674/228 = 3.0
Cost to detect one CRC or APL with 1 test round						$1,392,711/228 = $6,108
* Colonoscopy (diagnostic/with intervention) costs $1,415/$1,991; intervention reflects biopsy and/or polypectomy
CRC, colorectal cancer; APL, advanced precancerous lesion; NAA, non-advanced adenoma; FIT, fecal immunochemical test

The calculator is based on an assumed prevalence of CRC, APL and NAA as observed in the 2014 prospective study of FIT-DNA vs. FIT by Imperiale et al. [16]. In a hypothetical cohort of 10,000 persons with such an underlying spectrum of colorectal neoplasia, the number of positive results with a novel test can be calculated based on the novel test’s assumed sensitivity for CRC, APL, and NAA, and its specificity (defined as 1 minus the false-positive rate in persons with a normal colon). For simplicity, the assumption is embedded into the calculator that CRCs and APLs are detected at follow-up colonoscopy if the screening test was positive. A potential correction that assumes a small fraction of missed CRCs and APLs at follow-up colonoscopy could be built into a future version of the calculator, but it is unlikely to meaningfully change the calculator’s estimates of the relative impact of alternative screening tests.

As shown in Table 2, when a novel test with a given set of test performance characteristics (TPC) interacts with an assumed spectrum of underlying neoplasia prevalence (Prev) in the population, it yields a group of persons with a positive test result who have CRC, APL, NAA or a normal colon (four colon “health states”). These person with a positive test result then undergo colonoscopy. Conceptually, this group is represented by the sum of TPC x Prev across the various colon health states. Costs can then be counted, consisting of screening test costs for the entire cohort, and colonoscopy costs (with or without intervention) for those with a positive screening test. The NNC to detect one CRC or one APL in round 1 of screening, and the cost to detect one CRC or one APL in round 1 of screening, are calculated based on these numbers for clinical outcomes and costs (Table 2).

Surrogate endpoints vs comprehensive models

In order to compare the surrogate endpoints with the published estimates of comprehensive models, we created figures for each model that include the strategies that were modeled in each model’s recent publications.

The core strategies evaluated with MOSAIC [11] and the three CISNET models MISCAN, CRC-SPIN and SIMCRC [12] consisted of FIT (annual); a hypothetical test that meets the minimum CMS thresholds (CMS) every 3 years; FIT-DNA every 3 years; methylated Septin9 DNA every 3 years; and Guardant Shield (based on preliminary data available at the time) every 3 years.

Additional strategies evaluated with MOSAIC [11] consisted of FIT every 2 years; CMS every 2 or 1 years; a hypothetical high-sensitivity blood-based test (CMS high sens); and a hypothetical high-sensitivity blood-based test with sensitivities that were down-adjusted in order to meet the CMS requirement for CRC (not CRC or APL [17]) specificity ≥90% [13].

Additional strategies evaluated with the three CISNET models MISCAN, CRC-SPIN and SIMCRC [12] consisted of FIT (annual) with CRC stage I sensitivity 0.8-fold that of stages II-IV (FIT stage-specific sens); and CMS with CRC stage I sensitivity 0.8-fold that of stages II-IV (CMS stage-specific sens).

Exploratory comprehensive modeling vs. recently published model

The web-based tool of the EU-TOPIA project (https://eu-topia.org) [14], which is based on MISCAN, aims to facilitate modeling the potential impact of colorectal, breast and cervical screening interventions in European countries. After logging into the online environment, users can supply demographic, epidemiologic and screening data from their country. The webtool automatically adjusts model parameters based on these inputs. In case of missing or unavailable data, the parameters from one of four benchmark models can be chosen as a substitute. For CRC, benchmark models have been developed for countries with available CRC screening trial data representative of four European regions: Italy for the south, The Netherlands for the west, Finland for the north and Slovenia for the east of Europe [18]. After setting up the data, users can evaluate screening strategies in terms of, for example, CRC incidence and mortality, number of tests and number of complications.

To adjust the webtool for Israel, authors based in Israel (EH, BS and ZL) provided data on population size forecasts for the period 2015–2050, CRC incidence and CRC location for the years before screening introduction in Israel (2001–2005), CRC survival, and all-cause mortality by age from 2022 in Israel, as well as the parameters of the screening program (ages, and interval and characteristics of FIT used). The same data for CRC incidence before screening introduction and for all-cause mortality were used in the recent modeling study with a model that was constructed for Israel based on the previous version of MOSAIC for the US [15]. Data on FIT characteristics and CRC survival in Israel could not be used in the webtool due to incompatibility with the webtool input options, mainly because of differences in CRC staging methods. Based on the totality of the data for Israel, including CRC epidemiology and FIT characteristics, we elected Italy among the benchmark models already programmed in the EU-TOPIA tool as the reference country that is most similar to Israel. Consequently, a 75% colonoscopy follow-up rate after abnormal FIT was assumed, in line with Italian adherence statistics.

The webtool was set up such that screening started in 2018, and was populated with a cohort of 45–49 year-old individuals of the same size as in the population of Israel in 2018. Consequently, initial screening took place at ages 45–49 for the strategy of screening at age 45. Since the webtool yielded estimates of CRC cases and CRC deaths between 2018 and 2050, ages 45 to 81 are covered. All outcomes are reported per 10,000 individuals alive in 2018.

The fully published model based on MOSAIC assumed entry into the model at age 45, with screening at exactly age 45 for the strategy of screening initiation at age 45, and a 100% colonoscopy follow-up rate after abnormal FIT. Therefore, trends and patterns in the relative reductions in CRC cases and CRC deaths are more informative for comparison across methods than the absolute estimates for CRC cases and CRC deaths.

Exploratory comprehensive modeling: Time requirements

In order to capture an important dimension of feasibility, the authors kept track of the time required to obtain the necessary inputs for modeling in EU-TOPIA, as well as the time required to produce the exploratory estimates.

Surrogate endpoints of clinical effectiveness: Number needed to colonoscope (NNC)

Figure 1 shows the estimated reduction (%) in CRC deaths vs. the NNC to detect one CRC or one APL in round 1 of screening for a group of strategies for each of the four models. To a first approximation, there is an inverse relationship between the NNC and the predicted long-term CRC mortality reduction with programmatic screening. This is as expected if the surrogate marker were to prove useful: the lower the NNC, the higher the potential long-term benefit.

However, the modeled strategies that do not fit neatly into the overall trend of an inverse relationship highlight the potential limitations of the surrogate marker. First, FIT at 1 vs. 2 years, and CMS at 1 vs. 2 vs. 3 years in MOSAIC demonstrate how the surrogate marker, which is calculated for only the first round of screening, cannot reflect that more frequent testing is generally expected to deliver better long-term outcomes. A shorter testing interval moves up the long-term outcome of effectiveness along the Y-axis of Reduction in CRC deaths, while not changing along the X-axis of NNC (Fig. 1).

Second, mSep9, which is an outlier in all four models, represents the interesting case of a test with relatively low sensitivity for CRC as well as low specificity (Table 1). The low specificity means that many persons with NAA and a normal colon are referred for colonoscopy – and this makes the NNC relatively high. However, at the same time, a relatively high number of persons with APL are also referred for colonoscopy – poor specificity leads to a relatively high reduction in CRC deaths because of “chance” detection of APLs. Screening colonoscopy (not shown in Fig. 1) would be the extreme example for which the apparent inverse relationship no longer holds for tests with low specificity and therefore a high colonoscopy referral rate, because all persons undergo colonoscopy in a primary screening colonoscopy strategy (screening colonoscopy has a high NNC and also a very high estimated reduction in CRC deaths, placing it in the top right of the graphs in Fig. 1).

The results above suggest that the NNC surrogate endpoint might be useful as long as test intervals are generally consistent with test performance (the highest performing test is recommended at the longest interval, and the lowest performing test is recommended at the shortest interval), and only for tests with relatively high specificity (i.e. ≥90%) and a low rate of colonoscopy in persons without CRC or APL. One cautionary point is that, even if there appears to be an inverse relationship between the long-term estimate from comprehensive models and the surrogate endpoint of clinical effectiveness in Fig. 1, that relationship is not strictly linear.

Surrogate endpoint of cost-effectiveness: Cost per CRC or APL detected

Figure 2 shows the estimated cost/QALY gained vs. the cost to detect one CRC or one APL in round 1 of screening for a group of strategies for each of the four models. To a first approximation, there is a direct relationship between the cost to detect one CRC or APL and the estimated long-term CRC cost/QALY gained with programmatic screening. This is as expected if the surrogate marker were to prove useful: the higher the cost to detect a relevant lesion at round 1, the higher the potential long-term cost/QALY gained (i.e. higher the cost per lesion at round 1, the worse the long-term cost-effectiveness).

As for the surrogate endpoint of NNC, the strategies of FIT at 1 vs. 2 years, and CMS at 1 vs. 2 vs. 3 years in MOSAIC demonstrate how the surrogate marker for cost-effectiveness, which is also calculated for only the first round of screening, cannot reflect the impact of more frequent testing on effectiveness and cost (Fig. 2). In contrast, mSep9 does not appear as an obvious outlier for the surrogate measure of cost-effectiveness. This suggests that the cost consequences of poor specificity are reflected in both the surrogate endpoint as well as the long-term estimates from comprehensive models.

The results above suggest that the surrogate endpoint of cost per relevant lesion detected might be useful as long as test intervals are generally consistent with test performance, as noted above for NNC. The same cautionary point applies as for NNC -- even if there appears to be a direct relationship between the long-term estimate from comprehensive models and the surrogate endpoint of cost-effectiveness in Fig. 2, that relationship is not strictly linear.

Exploratory comprehensive modeling vs. recently published model

Table 3 shows the exploratory comparison between the fully published modeling analysis for Israel based on a previous version of MOSAIC and preliminary modeling with the EU-TOPIA web-based tool based on MISCAN, assuming that Italy’s CRC epidemiology and screening implementation best reflect the case for Israel. As explained in the Methods, direct comparisons of specific CRC outcomes are not strictly possible between the two methods, given the fundamental differences in how model outputs are expressed, and the differences in underlying assumptions, most importantly a colonoscopy follow-up rate of 100% vs. 75% after abnormal FIT in the published modeling paper vs. the exploratory modeling with the EU-TOPIA tool, and differences in the time horizon through older ages.

Table 3

Exploratory comparison between fully published modeling analysis (previous version of MOSAIC) and preliminary modeling with web-based tool (EU-TOPIA)* based on MISCAN
	No screening		Fecal immunochemical testing yearly, ages 50–74		Fecal immunochemical testing yearly, ages 45–74		Fecal immunochemical testing yearly, ages 45–75
	Full model (previous version of MOSAIC)	Web tool (EU-TOPIA, based on MISCAN)	Full model (previous version of MOSAIC); colonoscopy follow-up 100% after FIT-positive	Web tool (EU-TOPIA, based on MISCAN); colonoscopy follow-up 75% after FIT-positive	Full model (previous version of MOSAIC); colonoscopy follow-up 100% after FIT-positive	Web tool (EU-TOPIA, based on MISCAN); follow-up 75% after FIT-positive	Full model (previous version of MOSAIC); colonoscopy follow-up 100% after FIT-positive	Web tool (EU-TOPIA, based on MISCAN); follow-up 75% after FIT-positive
Colorectal cancer cases	804 per 10,000 45 year-olds until death or age 100	568 per 10,000 persons alive in 2018 until death or year 2050	349 per 10,000 45 year-olds until death or age 100	434 per 10,000 persons alive in 2018 until death or year 2050	325 per 10,000 45 year-olds until death or age 100	410 per 10,000 persons alive in 2018 until death or year 2050	317 per 10,000 45 year-olds until death or age 100	418 per 10,000 persons alive in 2018 until death or year 2050
Reduction vs. no screening (%)	--	--	56.6%	23.7%	59.6%	27.9%	60.6%	26.5%
Colorectal cancer deaths	268 per 10,000 45 year-olds until death or age 100	198 per 10,000 persons alive in 2018 until death or year 2050	79 per 10,000 45 year-olds until death or age 100	82 per 10,000 persons alive in 2018 until death or year 2050	73 per 10,000 45 year-olds until death or age 100	76 per 10,000 persons alive in 2018 until death or year 2050	69 per 10,000 45 year-olds until death or age 100	76 per 10,000 persons alive in 2018 until death or year 2050
Reduction vs. no screening (%)	--	--	70.5%	58.6%	72.8%	61.6%	74.3%	61.6%
*The webtool is populated with a cohort of 45–49 year-olds that is representative for the Israeli population in 2018. The outcomes are reported until 2050. As such, in the webtool, the oldest person in 2050 is aged 81 years.

Focusing on the most relevant outputs in Table 3 that lend themselves to comparison, namely the estimated reductions in CRC cases and CRC deaths for the various FIT scenarios compared with no screening, two patterns emerge. First, the estimated impact is higher for the full model for Israel that is adapted from a previous US version of MOSAIC. This is not necessarily surprising, since EU-TOPIA is based on MISCAN, and MISCAN tends to predict somewhat lower effectiveness than MOSAIC, CRC-SPIN, and SIM-CRC for CRC incidence, CRC mortality, and for the surrogate markers explored above (Fig. 1). Furthermore, the full modeling study reported idealized results assuming full participation, including a 100% colonoscopy follow-up rate, and the EU-TOPIA web-tool exercise assumed a 75% colonoscopy follow-up rate. Second, the relative magnitude of changes in the estimated impact when the screening initiation age was lowered from age 50 to 45 is similar between the two methods. However, when the screening cessation age was increased from 74 to 75, CRC incidence increased in the webtool whereas it decreased in the full modeling study. This is because the cohort in the webtool is aged 77–81 in 2050, which is the end of the simulation in the webtool. Consequently, the impact of the final FIT round at age 75 on CRC incidence and mortality at ages > 81 is not captured in the webtool’s simulation.

Exploratory comprehensive modeling: Time requirements

The time required to obtain the necessary inputs for modeling in the EU-TOPIA tool was approximately 10 hours, including initial virtual meetings among investigators, email correspondence, and gathering of data (approximately 8 hours to extract and format data from the Israel National Bureau of Statistics and Israel National Cancer Registry databases). The time required to carry out the modeling work in the EU-TOPIA tool and preparing results for presentation was approximately 19 hours. An additional approximate 12 hours were invested into exploring other possible modeling outputs with the EU-TOPIA tool (e.g. colonoscopy strategies), but when unforeseen issues were encountered, this aspect of the modeling was deferred.

The initial exploratory exercises described here for surrogate markers and for rapid modeling are encouraging. If these approaches are validated further, it is possible that one or both of them could be incorporated into the early phases of novel test development.

First, it appears that the NNC to detect one CRC or one APL in round 1 of screening may be a useful surrogate for a novel test’s clinical effectiveness, with the limitation that testing interval is not captured, and that specificity must be high in order to avoid outlier results. Similarly, it appears that the cost to detect one CRC or one APL in round 1 of screening may be a useful surrogate for a novel test’s long-term cost-effectiveness, again with the limitation that testing interval is not captured. For this surrogate of cost-effectiveness, low test specificity may not be a major limitation.

Second, our experience with the EU-TOPIA tool suggests that informative exploratory modeling may be feasible with a much smaller time commitment than is required for comprehensive modeling that leads to a full-length peer-reviewed publication. It remains to be determined how exactly such exploratory modeling would be used.

The explorations presented here must be considered preliminary. Further exploration of the surrogate endpoints for clinical effectiveness and for cost-effectiveness, such as with dedicated modeling of a larger set of emerging or hypothetical non-invasive tests in the four comprehensive models, can be contemplated, but these were beyond the scope of the current paper. It remains to be determined how relevant is the limitation of failing to capture testing interval, and whether the limitation with respect to test specificity matters if no future tests with relatively low specificity are seriously considered for clinical adoption.

Moreover, the EU-TOPIA webtool is designed and intended for evaluation of screening in European contexts. Caution should be taken when extrapolating results to other continents with different lifestyle, genetic and environmental risk factors. Given the strict format of the webtool’s input, it might be worth developing tools for other contexts with other parameters that can be easily adjusted, such as the assumption of 75% adherence to follow-up colonoscopy. However, if more options increase time burden, then analyses would not be performed as quickly.

Potential future collaborative efforts might address multiple open questions. These include: 1) whether the proposed surrogate endpoints also perform well when compared to the projections of other validated, long time-horizon decision analytic models; 2) whether the surrogates are more useful than a general impression or “gestalt” based on initial estimates of test performance characteristics, assumed interval, and anticipated cost; 3) whether comprehensive exploratory analyses in established models that account for the early-stage uncertainties are feasible and preferable [11, 12, 19]; 4) whether relatively rapid modeling with tools such as the EU-TOPIA tool is feasible, and whether such tools will be accepted to inform progress in the field; 5) whether “look-up” tables [20] can capture all the relevant parameters and permutations (e.g. sensitivity for CRC [possibly stage-specific], advanced adenoma, APL, NAA and serrated lesions; specificity; interval; cost), and how often they may need updating; and 6) which audiences would find which type of early-stage modeling estimates useful during which phases of test development (e.g. test developers/industry, screening program directors, budget managers).

If, in the future, the surrogate endpoints or the type of analyses that we explored here are applied in the early stages of test development, the results should not be over-interpreted. Most importantly, these initial modeling explorations should never stifle scientific innovation. It is crucial to appreciate that many of the critical long-term considerations (potential sensitivity vs. specificity trade-offs, actual test cost, test interval, permutations of performance/cost/interval, participation rates, outreach costs) are not yet determined during early phases of a screening test’s development. Furthermore, it is anticipated that first-generation versions of new technologies might lead to refined next-generation versions based on scientific advances. The ultimate test of an early-stage surrogate endpoint would be its contribution to the development of screening tests that can improve patient outcomes at the population level.

Competing Interests

UL: advisor Universal Dx, Cylinder, Kohler Ventures, Lean Medical; consultant Freenome, Guardant, Neptune Medical, Medtronic, WilliamBlair, Medacorp

Author Contribution

Concept and design: UL, ILV; acquisition of data: UL, LvD, BS; analysis: UL, LvD, ILV; writing and final approval of manuscript: all authors

Data Availability

Data are included in the manuscript

Lin JS, Perdue LA, Henrikson NB, Bean SI, Blasi PR. Screening for Colorectal Cancer: Updated Evidence Report and Systematic Review for the US Preventive Services Task Force Jama. 2021;325:1978–1998.
Ladabaum U, Dominitz JA, Kahi C, Schoen RE. Strategies for Colorectal Cancer Screening Gastroenterology. 2020;158:418–432.
Davidson KW, Barry MJ, Mangione CMet al.. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement Jama. 2021;325:1965–1977.
Shaukat A, Levin TR. Current and future colorectal cancer screening strategies Nature reviews. Gastroenterology & hepatology. 2022;19:521–531.
Imperiale TF, Porter K, Zella Jet al.. Next-Generation Multitarget Stool DNA Test for Colorectal Cancer Screening N Engl J Med. 2024;390:984–993.
Chung DC, Gray DM, 2nd, Singh Het al.. A Cell-free DNA Blood-Based Test for Colorectal Cancer Screening N Engl J Med. 2024;390:973–983.
Barnell EK, Wurtzler EM, La Rocca Jet al.. Multitarget Stool RNA Test for Colorectal Cancer Screening Jama. 2023;330:1760–1768.
Shaukat A, Meng Z, Sun C-Ket al.. Clinical evaluation of a blood-based screening test for the early detection of colorectal cancer Gastroenterology. 2024;166:1057d.
Bresalier RS, Senore C, Young GPet al.. An efficient strategy for evaluating new non-invasive screening tests for colorectal cancer: the guiding principles Gut. 2023;72:1904–1918.
Ladabaum U. Cost-Effectiveness of Current Colorectal Cancer Screening Tests Gastrointestinal endoscopy clinics of North America. 2020;30:479–497.
Ladabaum U, Mannalithara A, Weng Yet al.. Comparative Effectiveness and Cost-Effectiveness of Colorectal Cancer Screening With Blood-Based Biomarkers (Liquid Biopsy) vs Fecal Tests or Colonoscopy Gastroenterology. 2024;167:378–391.
van den Puttelaar R, Nascimento de Lima P, Knudsen ABet al.. Effectiveness and Cost-Effectiveness of Colorectal Cancer Screening With a Blood Test That Meets the Centers for Medicare & Medicaid Services Coverage Decision Gastroenterology. 2024;167:368–377.
Centers for Medicare & Medicaid Services. Decision Memo for Screening for Colorectal Cancer - Blood-Based Biomarker Tests (CAG-00454N). Accessed on August 15, 2024 at https://www.cms.gov/medicare-coverage-database/details/nca-decision-memo.aspx?NCAId=299&bc=AAAAAAAAAAQA&.
Gini A, van Ravesteyn NT, Jansen EELet al.. The EU-TOPIA evaluation tool: An online modelling-based tool for informing breast, cervical, and colorectal cancer screening decisions in Europe Prev Med Rep. 2021;22:101392.
Half EE, Levi Z, Mannalithara Aet al.. Colorectal cancer screening at age 45 years in Israel: Cost-effectiveness and global implications Cancer. 2024;130:901–912.
Imperiale TF, Ransohoff DF, Itzkowitz SHet al.. Multitarget stool DNA testing for colorectal-cancer screening N Engl J Med. 2014;370:1287–1297.
Ladabaum U, Church TR, Feng Z, Ransohoff DF, Schoen RE. Counting Advanced Precancerous Lesions as True Positives When Determining Colorectal Cancer Screening Test Specificity J Natl Cancer Inst. 2022;114:1040–1043.
Gini A, Buskermolen M, Senore Cet al.. Development and Validation of Three Regional Microsimulation Models for Predicting Colorectal Cancer Screening Benefits in Europe MDM Policy Pract. 2021;6:2381468320984974.
Lieberman DA. Commentary: Liquid Biopsy for Average-Risk Colorectal Cancer Screening Clin Gastroenterol Hepatol. 2024;22:1160–1164.e1161.
Lansdorp-Vogelaar I, Goede SL, Bosch LJWet al.. Cost-effectiveness of High-performance Biomarker Tests vs Fecal Immunochemical Test for Noninvasive Colorectal Cancer Screening Clin Gastroenterol Hepatol. 2018;16:504–512.e511.

Competing interest reported. UL: advisor Universal Dx, Cylinder, Kohler Ventures, Lean Medical; consultant Freenome, Guardant, Neptune Medical, Medtronic, WilliamBlair, Medacorp

Download PDF

Editorial decision: Revision requested
30 Sep, 2024
Reviewers agreed at journal
21 Sep, 2024
Reviews received at journal
19 Sep, 2024
Reviewers agreed at journal
19 Sep, 2024
Reviewers invited by journal
19 Sep, 2024
Editor assigned by journal
12 Sep, 2024
Submission checks completed at journal
12 Sep, 2024
First submitted to journal
11 Sep, 2024

You are reading this latest preprint version

Modeling and the use of surrogate endpoints: Is this a valid approach?

Status:

Version 1

Abstract

Figures

Introduction

Methods

Comprehensive models

Surrogate endpoints

Surrogate endpoints vs comprehensive models

Exploratory comprehensive modeling vs. recently published model

Exploratory comprehensive modeling: Time requirements

Results

Surrogate endpoints of clinical effectiveness: Number needed to colonoscope (NNC)

Surrogate endpoint of cost-effectiveness: Cost per CRC or APL detected

Exploratory comprehensive modeling vs. recently published model

Exploratory comprehensive modeling: Time requirements

Discussion

Declarations

Author Contribution

Data Availability

References

Additional Declarations

Status:

Version 1