Systematic review of health research using internet search data

doi:10.21203/rs.3.rs-4456499/v1

Download PDF

Article

Systematic review of health research using internet search data

https://doi.org/10.21203/rs.3.rs-4456499/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Novel types of digital data, including internet search data, have potential to improve understanding of early predictors of serious health conditions and enable timely management. While many studies have used aggregate anonymized search trends in this way, what is less clear is the predictive or diagnostic value of online searches at the individual level. While an increasing number of studies have used these kinds of data, this research method is still emerging. We therefore undertook a systematic review of published research that has assessed the predictive or diagnostic value of individual internet search data. MEDLINE and Embase were searched through March 2024 for studies utilising individual internet search data to predict or diagnose patient disease status. Due to the heterogeneous nature of the design, methodology and reported outcomes of included studies, a narrative synthesis of studies and pre-specified outcomes was performed. Study quality was assessed with the Newcastle-Ottawa Scale and PROBAST tool. Twenty-three studies met the inclusion criteria. Conditions of interest encompassed mental health, neurological conditions, malignancies, and miscellaneous healthcare presentations. Data on individuals’ search history were obtained from search engines using anonymous search queries (Bing, Yahoo!) or from consented participants (Google) where consent rates ranged from 20–70%. Wide variability in AUROC (range: <0.53 to > 0.99), sensitivity (range: 0.44 to 0.81) and F1 score (0.36 to 0.80) were reported. Studies noted a range of predictive linguistic, temporal, and other features (e.g., spelling error frequency). This review demonstrated that the use of individual internet search data holds diagnostic and predictive potential, with evidence of strong associative features. However, there was significant variability regarding conditions of interest, methodology, and predictive models used. Given the common use of internet searches by patients as part of their healthcare journeys, individual search data holds significant potential, and justifies further research, including the use of established diagnoses.

Health sciences/Health care/Diagnosis

Health sciences/Signs and symptoms

Health sciences/Health care/Public health

Early disease detection facilitates timely intervention, monitoring and exercising of patient autonomy. Long asymptomatic periods pose challenges in early diagnosis of many serious medical conditions, including malignancy, cardiovascular disease and mental health conditions.¹ Non-specific early symptoms present further diagnostic challenges in primary and emergency care settings. Increasing numbers of medical malpractice claims for delayed diagnosis underline its impact.²

A large body of medical research has attempted to identify early or pre-clinical markers to allow early diagnosis and disease prediction. These have typically identified key diagnostic and prognostic factors using medical records and insurance claims databases.^3,4 However, these data are inherently limited by their date of collection, failing to capture symptoms or concerns experienced by individuals prior to healthcare contact. Other research methods have used interviews with patients or caregivers, which can provide a rich source of pre-diagnosis information, but are limited by recall bias and are challenging to scale.^5–8 At a population level, large cohort studies have provided valuable information on risk prediction, such as for cardiovascular disease.⁹

Digital data potentially provide a novel approach to understanding the earliest features and predictors of serious health conditions. Current evidence suggests that individuals with some health conditions may exhibit changes in symptoms, behaviour and activity for long periods of time prior to first contact with healthcare services.^10,11 Furthermore, digital health resources are increasingly important as a first port-of-call to patients accessing healthcare advice.¹² Indeed, previous work has noted that half to three-quarters of patients attending outpatient appointments describe searching for information about their health conditions prior to their visit.^13,14 Online information resources commonly used as part of the healthcare decision-making process include search engines, online content (e.g., YouTube videos), and social media.¹⁵

There is a large existing body of evidence supporting the use of aggregate online search trends for predicting infectious disease outbreaks at the national or population level (e.g., predicting rates of COVID-19 infection using trends in internet searchers for flu-like symptoms).^16,17 However, what is less clear is the predictive or diagnostic value of online searches at the individual level. An increasing number of studies have explored the use of internet search data both from consented individuals and from aggregated anonymous search data, focussed on predicting a range of health outcomes. However, there have been no efforts to consolidate existing studies that have utilised individuals’ online search history to identify and predict presentation or progression of health conditions. The aim of this systematic review is to assess evidence for (1) the predictive or diagnostic value of personal internet search history for health outcomes and (2) associations between features of internet search history and health outcomes.

Literature search

The literature search yielded 23 studies that met the inclusion criteria. Figure 1 presents the flow of studies through the screening process. Of note, one study was excluded as it explored internet search use in patients after diagnosis during a period of treatment.¹⁸

An overview of the risk of bias assessments with the Newcastle-Ottawa Scale and PROBAST can be found in Figs. 2 and 3, respectively. All diagnostic and predictive studies were rated to have ‘high’ or ‘unclear’ overall risk of bias or applicability concerns by the Prediction model Risk Of Bias Assessment Tool.

Characteristics of included studies

Study participants

Characteristics of included studies can be found in Table 1. The number of participants with a condition of interest ranged from 20 to 11050.^19,20 Most studies involved US-based participants, with the exception of one study in Japan, one in the UK, and one involving international participation.^21–23 Age of participants in reported studies ranged from 20.6 to 53 years.^23,24 The proportion of males in reported studies ranged from 0–81%.^22,25

A wide range of conditions of interest was examined. Broadly, included conditions encompassed mental health (i.e., schizophrenia spectrum disorder, suicidality, mood disorders, psychosis, eating disorders),^{19,22,24,26–30} neurologic conditions (i.e., Parkinson’s disease, amyotrophic lateral sclerosis, stroke, sleeping disorders),^31–34 malignancies (i.e., pancreatic cancer, lung cancer, paediatric oncology, breast cancer, colon cancer, gynaecological malignancies),^{23,25,35–37} miscellaneous conditions (i.e., diabetes, coeliac disease, biliary atresia, pyloric stenosis),^20,21,38,39 and a variety of primary and secondary care presentations.^40,41

Ten studies recruited participants from inpatient or outpatient settings, and consented participants face-to-face.^{19,23–27,29,30,38,40} Three studies recruited participants from online research crowdsourcing platforms or advocacy groups.^22,28,34 Two studies recruited participants with targeted advertisements on search engines.^33,37 Nine studies extracted search data from unique, non-identifiable users from back-end databases (8 Bing, 1 Yahoo! Japan).^{20,21,31,32,34–36,39,41} Investigators then identified users who searched for a ‘diagnosis ascertaining query’ (e.g., ‘I have lung cancer’, ‘I have been diagnosed with diabetes’) and extracted their search data from a period of time prior to the diagnosis ascertaining query.

The consent rate for the use of participant personal search data in studies from individuals providing explicit consent ranged from 20–70%.^25,34

Search engines and extraction

Search engine, data extraction, ground truth determination and outcomes from included studies are detailed in Table 2.

Ten studies used Bing (Microsoft Corporation),^{20,31–37,39,41} nine studies used Google Search (Google LLC),^{19,24–26,26,29,30,38,40} one study used Yahoo! Search Japan (Yahoo Inc.),²¹ one study utilised data from both Google and Bing,²³ and two studies used unspecified general web browsing history, including search history data.^22,28 All Bing search data was extracted using proprietary backend techniques by Microsoft employees or investigators. All Google data was downloaded by research participants themselves using Google Takeout and then provided to academic investigators with participants’ explicit informed consent. Two studies extracted general web browsing data using a commercial executable downloaded and installed by participants on their computers (Browsing History View, Nirsoft);^22,28 participants then uploaded the extracted data onto a study website.

The total number of search queries per study ranged from > 30,000 to 591,421.^19,40 The duration of time over which individuals’ search data was examined ranged from seven days to two years.^23,26,40

Use of machine learning algorithms

Machine learning was utilised in the included studies to (1) develop diagnostic or predictive classifiers, based on participant internet search data or (2) natural language processing (NLP) for categorisation of search queries. 15 studies developed machine learning-driven diagnostic or predictive classifiers.^{20–23,27,28,31–39} A variety of machine learning algorithms were used, including decision tree, gradient boosted, linear regression, logistic regression, random forest, and support vector machine. Two studies utilised NLP for categorisation and linguistic thematic analysis of search queries .^29,38

Ground truth and control

The “ground truth” (i.e., disease status) for included diagnostic or predictive studies was determined by three distinct methods. Six studies ascertained diagnoses from patient health records.^{19,23,24,26,27,40} Ten studies ascertained a “ground truth” from a user’s diagnosis ascertaining query.^{20,21,23,31,32,34–36,39,41} Finally, seven studies ascertained ground truth from participant survey scores.^{22,28–30,33,37,38}

Regarding control populations, two studies recruited healthy volunteers as a control group for studies of participants with schizophrenia, spectrum disorders, and/or mood disorders.^26,27 Four studies which included unique, non-identifiable Bing users for conditions of interest (stroke, Parkinson’s disease, amyotrophic lateral sclerosis and gynaecological malignancies) also recruited matched control populations.^23,31,32,34

Outcomes

Measures of predictive model performance were heterogeneous, and studies reported area under the receiver-operator characteristic curve values (AUC), positive predictive value (PPV, i.e., diagnostic precision), sensitivity, and F1 score.

Table 2

Search engine, data extraction, ground truth determination and outcomes of included studies. Note: API - application programming interface; AUC - area under the receiver operator characteristic curve; LIWC - linguistic inquiry and word count; PPV - positive predictive value.
Study	Search Engine	Extraction	Total no. of search queries	Duration of search queries	Ground truth	Use of machine learning	Data extraction and labelling of content	Outcomes
Paparrizos (2016)³⁵	Bing	Bing proprietary backend extraction	479787	Mean 109 days	Diagnosis ascertaining query for pancreatic cancer	Gradient boosted statistical classifier.	Predictive model trained on extracted features including demographics, search characteristics, symptom characteristics, temporal characteristics and risk factors.	AUC.
White (2017)³⁶	Bing	Bing proprietary backend extraction	Not stated.	1 to 52 weeks	Diagnosis ascertaining query for lung carcinoma	Statistical classifier.	Predictive model trained on extracted features including risk factors and symptoms. Symptoms identified via query terms matching a symptom set defined in literature review.	AUC.
Asch (2019)⁴⁰	Google	Google Takeout	591421	7 days	Diagnosis from medical records	No	Manual thematic analysis of search queries.	Health-relatedness of searches and associations between clinical and demographic characteristics and internet search volume and search content.
Hochberg (2019)²⁰	Bing	Bing proprietary backend extraction	Not stated.	Up to 1 year	Diagnosis ascertaining query for diabetes	Four prediction models: linear regression, logistic regression, decision trees, and random forest.	Prediction model trained on extracted queries related to 24 diabetes symptoms.	AUC and PPV.
Lebwohl (2019)³⁹	Bing	Bing proprietary backend extraction	Not stated.	10 months	Diagnosis ascertaining query for coeliac disease	Two predictive models: linear regression and random forest.	Extracted queries related to a list of 195 symptoms and their synonyms.	AUC.
Phillips (2019)²⁵	Google	Google Takeout	81725	6 months	Not stated.	No	Manual thematic analysis of search queries.	Description of health-related queries prior to diagnosis.
Youngmann (2019)³¹	Bing	Bing proprietary backend extraction	327492	Not stated.	Diagnosis ascertaining query for Parkinson’s disease	Automated supervised long short-term memory model and boosted decision forest classification model.	Manual and automatic feature extraction	AUC.
Birnbaum (2020)²⁷	Google	Google Takeout	Not stated.	4 weeks	Psychosis hospitalisation from medical records	Three prediction models: support vector machine, random forest and gradient boosting.	LIWC extraction of linguistic features. Predictive model trained on 123 extracted linguistic and search behaviour features.	AUC.
Birnbaum (2020)²⁶	Google	Google Takeout	405523	52 weeks	First psychiatric hospitalisation from medical records	No	LIWC extracted 93 linguistic features.	Between-group and within-group comparisons of linguistic and search behaviour features.
Hochberg (2020)⁴¹	Bing	Bing proprietary backend extraction	Not stated.	Not stated.	Diagnosis ascertaining query for 20 medical conditions	No	Extracted queries related to relevant symptoms before DAQ. List of relevant symptoms defined by investigators.	Correlation between queries of symptoms and DAQ.
Kirschenbaum (2020)¹⁹	Google	Google Takeout	> 30000	6 months	First episode psychosis from medical records	No	Manual thematic analysis of search queries.	Qualitative description of search queries
Sadeh-Sharvit (2020)²²	General browsing history	Nirsoft BrowsingHistoryView	Not stated.	Up to 6 months	Stanford-Washington University Eating Disorder Screen Questionnaire	Three prediction models: multiclass predictor, logistic regression and random forest.	Feature extraction in conjunction with model generation. Extracted features include keywords, demographic data and search behaviour.	Sensitivity.
Schueller (2020)²⁸	General browsing history	Nirsoft BrowsingHistoryView	Not stated.	At least 10 days	Perceived Barriers to Psychological Treatment Scale Patient Health Questionnaire	Three predictive models: linear, classification decision tree and random forest.	Predictive model trained on extracted features including online search behaviour, demographics and keywords.	AUC.
Yom-Tov (2020)³⁷	Bing	Bing proprietary backend extraction	Not stated.	3 months	Suspected Cancer Recognition and Referral Questionnaire	Random forest predictive model.	Predictive model trained on extracted features including query terms, frequency of medical symptoms. Symptoms defined by a list of 195 symptoms and layperson descriptions.	AUC.
Zhang (2020)²⁹	Google	Google Takeout	115493 in first round 122353 in second round	2 rounds of 2 months	Patient Health Questionnaire-9 General Anxiety Disorder-7 Questionnaire	Google natural language processing API to categorise search queries.	LIWC extraction of linguistic features from search queries and YouTube video titles. Other features extracted included online behaviours and distribution of online activity over a day.	Changes in online behaviours, LIWC attributes, search categories.
Arean (2021)³⁰	Google	Google Takeout	349922	Not stated.	Suicide Attempt and Self-Injury Count Questionnaire	No	Vector representations of queries derived. Cue terms were manually iteratively derived.	Changes in online search behavior prior to a suicide attempt.
Moon (2021)²⁴	Google	Google Takeout	37738	3 months	Mood disorder diagnosis from medical records	No	Manual thematic analysis of search queries.	Qualitative description of search queries.
Shaklai (2021)³²	Bing	Bing proprietary backend extraction	Not stated.	30 days	Diagnosis ascertaining query for stroke	Random forest predictive model	Predictive model trained on extracted features chosen to represent cognitive ability including search behaviour, frequency, spelling and mouse positioning.	AUC.
Yamaguchi (2021)²¹	Yahoo! Japan	Not stated.	Not stated.	Not stated.	Diagnosis ascertaining query for biliary atresia or pyloric stenosis	Logistic regression predictive model.	Predictive model trained on extracted search queries.	Sensitivity.
Zaman (2021)³⁸	Google	Google Takeout	Not stated.	Not stated.	Promote Health Survey	Two predictive models: logistic regression and support vector machine Google natural language processing API to classify queries.	LIWC extraction of linguistic features. Predictive models trained on extracted linguistic features and search category features.	F1 score and differences in linguistic attributes and search behaviours.
CohenZion (2022)³³	Bing	Bing proprietary backend extraction	Not stated.	Up to 1 year	Digital Sleep Questionnaire	Random forest predictive model.	Predictive model trained on search activity and keywords and demographics.	AUC.³⁴
Yom-Tov (2023)³⁴	Bing	Bing proprietary backend extraction	Not stated.	Not stated.	Diagnosis ascertaining query for ALS. Online individuals who self-identified to have ALS.	Random forest predictive model.	Predictive model trained on session duration, use of autocorrect and search query characteristics.	AUC.
Barcroft (2024)²³	Google	Google Takeout Bing proprietary backend extraction	519,048	2 years	Malignant or benign diagnosis from medical records.	Gradient boosting predictive model.	Predictive models trained on number of search terms in each keyword category or vector-space model of words and word pairs.	AUC.

Diagnostic or predictive value

The reported predictive or diagnostic value of included studies can be found in Table 3.

AUC was reported by ten studies with a wide variability in values, ranging from < 0.53 (identifying users with sustained interest in coeliac disease) to > 0.99 (detection of impending stroke).^32,39 Three studies noted that classifier AUC increased with search data closer to time of diagnosis of malignancies, including pancreatic cancer (0.832 to 0.911 from 21 weeks to 1 week prior to the diagnosis ascertaining query),³⁵ lung cancer (0.855 to 0.942 from 52 weeks to 1 week prior to the diagnosis ascertaining query),³⁶ and differentiating between malignant and benign gynaecological disease (0.64 to 0.74 from 1 year to 60 days prior to specialist referral).²³

Sensitivity was reported by three studies, and ranged from 0.44 for predicting eating disorder risk status to 0.81 for the detection of pyloric stenosis in infants from their parents’ search data.^21,22 F1 score was reported by two studies, and ranged from 0.36 for the detection of relapse of schizophrenia spectrum disorders to 0.80 for the likelihood of the presence of intimate partner violence.^27,38

Seven studies utilised more than one machine learning algorithm for the same dataset, noting variability in classification performance.^{20,22,27,28,31,38,39} For example, Birnbaum et al. noted differences in performance of support vector machine, gradient boosted, and random forest classifier models for both diagnosis and relapse prediction of schizophrenia spectrum disorders (AUC 0.66 to 0.74 and 0.69 to 0.71, for diagnosis and relapse, respectively).²⁷

Associations and predictive features

Results of the linguistic, temporal and other associations in the included studies can be found in Supplementary Table 1.

Linguistic associations

20 studies observed specific linguistic themes among study participants' health- and symptom-related searches prior to diagnosis. Asch et al. noted that 63% of people presenting to the emergency department searched proportionally more health-related queries prior to presentation compared to baseline.⁴⁰ Similarly, Philips et al. found that 13% of total Google searches by parents of paediatric cancer patients were health-related (of which 31% was symptom or disease related, 29% was hospital logistics related, and 18% was cancer-specific).²⁵ Additionally, the study authors noted that parents’ health-related searches increased prior to diagnosis, from mean 0.2 searches/day six months prior to diagnosis to 1.2 searches/day at diagnosis.

Search queries for physical symptoms appeared related to conditions of interest. Hochberg et al. noted search keywords related to symptoms (”impotence”, ”malaise”, ”polyuria” and ”thirst”) were significantly associated with a diagnosis of diabetes.²⁰ Similarly, Lebwohl et al. found that search queries related to the symptom ”diarrhoea” were most strongly associated with coeliac disease.³⁹ Finally, Barcroft et al. noted that for patients with suspected gynaecological malignancies, gastrointestinal and pain-related symptoms were apparent up to one year prior to specialist referral and further investigation, whereas urinary, bleeding-related, bloating and other gynaecological and menopausal symptoms became apparent later, up to 70 days prior to referral and diagnosis.²³

Qualitative thematic assessment of search queries were also noted in studies of individuals with mental health conditions. Kirschenbaum et al. noted that certain search keyword themes, including ”delusions”, ”negative symptoms”, ”thought process”, ”mental health”, ”illicit drugs”, ”help seeking”, and ”suicide”, were searched by a significant proportion of patients prior to a first presentation of psychosis.¹⁹ Moon et al. identified themes in the searches of participants hospitalised with suicidal thoughts and behaviours which included “suicide”, “help seeking”, “substance abuse”. and “mood and anxiety symptoms”.²⁴ Furthermore, Birnbaum et al. demonstrated keyword themes most significantly predictive for diagnosis and relapse of schizophrenia spectrum disorders included “inhibition”, “positive affect”, “anxiety”, “sexual”, “health”, “hear”, “anger”, “sadness”, and “perception”.²⁷ Additionally, Sadeh-Sharvit et al. noted search queries containing themes related to ”mental health”, ”treatment”, ”eating disorder”, “diet”, “body”, “imagery”, ”food”, ”sex”, ”clothing”, “allergy”, and ”stimulant drugs” were queried by between 12.2–98.6% of users classified with clinical or subclinical eating disorders.²² Finally, Arean et al. noted that searches related to suicide were found in some individuals, but with very wide ranging proximity to their suicide attempts.³⁰

Finally, significant linguistic shifts were noted in several included studies. Birnbaum et al. found significant differences in the use of pronouns and lack of punctuation prior to hospitalisation for participants with schizophrenia spectrum disorders and mood disorders, compared to healthy volunteers.²⁶ Similarly, Zaman et al. noted significantly higher “I” and “pronoun” usage for participants experiencing intimate partner violence compared to participants that showed no signs of intimate partner violence.³⁸

Temporal associations

Differences in frequency of search activity were also noted by studies. Birnbaum et al. found significantly lower overall search frequency for participants with schizophrenia and non-psychotic mood disorders up to one year prior to hospitalisation, compared to healthy volunteers.²⁶ Sadeh-Sharvit et al. also noted median web browsing frequency of participants as a predictive feature for eating disorders.²²

Changes in timing of search activity were also noted by studies. Birnbaum et al. noted significant shifts in timing of search activity in participants with schizophrenia spectrum disorders and mood disorders prior to hospitalisation compared to healthy volunteers.²⁶ Similarly, Zhang et al. found significant correlations between increased night-time search activity and both PHQ-9 and GAD-7 scores for depression and anxiety, respectively.²⁹ Additionally, Schueller et al. noted diurnal activity was a predictive feature for increased PHQ-9 score.²⁸ Finally, Zaman et al. noted increased search frequency during the weekends was associated with participants at higher risk of experiencing intimate partner violence compared to participants who showed no signs of intimate partner violence.³⁸

Other associations

Shaklai et al. noted the most predictive features for impending stroke were associated with cognitive function, including the number of queries per session, number of spelling mistakes and the number of repeated queries.³²

Table 3

Findings of predictive or diagnostic value of included studies. Note. DAQ - diagnosis ascertaining query; DT - decision tree; FPR - false positive rate; GB - gradient boosted; PBPT - perceived barriers to psychological treatment; PHQ - patient health questionnaire; RF - random forest; SVM - support vector machine.
Study	Condition of interest	Area Under the Receiver Operator Characteristic Curve	Positive Predictive Value	Sensitivity	F1 score
Paparrizos (2016)³⁵	Pancreatic cancer	0.91 (1 week before DAQ) 0.83 (21 weeks before DAQ)	Not stated.	Not stated.	Not stated.
White (2017)³⁶	Lung cancer	0.94 (1 week before DAQ) 0.87 (52 weeks before DAQ)	Not stated.	Not stated.	Not stated.
Hochberg (2019)²⁰	Diabetes	0.93 (RF) 0.92 (logistic regression) 0.89 (linear regression) 0.86 (DT)	0.16–0.28 (at FPR 0.01)	Not stated.	Not stated.
Lebwohl (2019)³⁹	Coeliac disease	< 0.53 (both linear regression and RF)	Not stated.	Not stated.	Not stated.
Youngmann (2019)³¹	Parkinson's disease	0.93	0.25	Not stated.	Not stated.
Birnbaum (2020)²⁷	Schizophrenia spectrum disorders	0.74 (diagnostic RF) 0.68 (diagnostic GB) 0.66 (diagnostic SVM) 0.69 (relapse RF) 0.71 (relapse GB) 0.71 (relapse SVM)	Not stated.	0.73 (diagnostic RF) 0.65 (diagnostic GB) 0.65 (diagnostic SVM) 0.61 (relapse RF) 0.65 (relapse GB) 0.63 (relapse SVM)	0.54 (diagnostic RF) 0.47 (diagnostic GB) 0.49 (diagnostic SVM) 0.53 (relapse RF) 0.57 (relapse GB) 0.36 (relapse SVM)
Sadeh-Sharvit (2020)²²	Eating disorders	Not stated.	Not stated.	0.526 (multiclass) 0.44 (logistic regression)	Not stated.
Schueller (2020)²⁸	Perceived barriers to psychological treatment	0.86 (PBPT score) 0.65 (PHQ score)	Not stated.	Not stated.	Not stated.
Yom-Tov (2020)³⁷	Lung cancer Breast cancer Colon cancer	0.66 (total) 0.74 (colon cancer) 0.56 (lung cancer) 0.50 (breast cancer)	Not stated.	Not stated.	Not stated.
Shaklai (2021)³²	Impending stroke	> 0.99	Not stated.	Not stated.	Not stated.
Yamaguchi (2021)²¹	Biliary atresia and pyloric stenosis	Not stated.	Not stated.	0.79 (biliary atresia) 0.81 (pyloric stenosis)	Not stated.
Zaman (2021)³⁸	Intimate partner violence	Not stated.	Not stated.	Not stated.	0.66 (logistic regression) 0.80 (SVM)
CohenZion (2022)³³	Sleep disorders	0.62 (insufficient sleep syndrome) 0.65 (delayed sleep phase syndrome) 0.69 (obstructive sleep apnoea) 0.69 (insomnia)	Not stated.	Not stated.	Not stated.
Yom-Tov (2023)³⁴	Amyotrophic lateral sclerosis	0.69 (90 days before DAQ) 0.73 (30 days before DAQ) 0.81 (all data) 0.74 (validation with prospective data)	Not stated.	Not stated.	Not stated.
Barcroft (2024)²³	Gynaecological cancer	0.64 (1 year) 0.74 (60 days) 0.82 (sample size-adjusted)	Not stated.	Not stated.

This systematic review highlighted that models derived using individual internet search data appear to have predictive value for a range of health conditions. However, we found considerable variability in predictive performance across studies, depending on the condition of interest, study characteristics, and predictive model used. Our findings also highlighted that researchers have found that significant associations may exist linking search data to eventual diagnosis. These findings suggest individuals’ internet search data holds diagnostic or predictive potential. To our knowledge, this is the first systematic review examining research that has focused on the diagnostic and predictive value of internet search data. This is important and timely, as internet searches serve an increasingly prominent role in patients’ healthcare journeys, are currently clinically under-utilised given the wide availability of internet access and give potentially novel insight into preclinical signals of disease. As such, internet search data holds great potential for further research efforts to describe and quantify diagnostic or predictive value and identify the potential for interventions to improve diagnosis and risk prediction.

This review captured a mix of small and large-scale studies, with heterogeneous methodologies, participants, conditions of interests, and outcomes. Included studies also illustrated the range of research tools involved in data extraction and analyses. Some investigators utilised proprietary backend extraction methods from anonymised Bing search data. Others, requested participants download their own Google internet search data and provide it with explicit consent to the research team. Several studies demonstrated impressive AUC values for their predictive models trained on search data. Interestingly, this review revealed not only the use of collected keyword and temporal search features, but also user interaction data. For example, features most predictive of an impending stroke event found by Shaklai et al. included query frequency, spelling error frequency, and frequency of repeated queries.³² Thus, combinations of different data from users, including non-linguistic or temporal features, may hold greater value for predictive models than single sources (e.g., semantics of queries) alone, allowing investigators unique insights into the asymptomatic disease period.

Consent rates reported in included studies varied between 20 to 70%.^25,34 Interestingly, only 7% of potential participants approached by Asch et al. cited privacy concerns as their reason for not participating.⁴⁰ Differences in attitudes may vary due to demographic differences and exacerbate selection bias. Thus, further work examining the acceptability of the use of personal search data is warranted, particularly for individuals and communities who may have lower digital literacy or higher levels of concern with the use of their data for health research.

Although several studies demonstrated potentially valuable levels of diagnostic accuracy, it is inevitable that diagnostic tools utilising internet search data will have imperfect predictive capability. Consequences of false positives from any such future diagnostic tools could lead to undue patient anxiety and healthcare resource drain.^23,42 Conversely, false negatives could lead to inappropriate reassurance, leading to potentially late or missed diagnosis. Thus, research on the potential diagnostic or predictive value of search will need to balance factors including the severity of the condition of interest, accuracy of diagnostic models (specifically, the added value that models provide on top of current care pathways and diagnostic tests), benefits of early detection, and acceptability of any potential interventions prompted by internet searches by users.

Limitations

Our review highlighted several limitations in the current body of evidence. First, there was significant heterogeneity in methodology and outcome measures in the included studies. Heterogeneity and incomplete reporting not only precluded meta-analysis in this review but will also potentially limit downstream evaluation of these models and any real-world benefits. Second, the applicability of machine learning models is inherently dependent on the quality and representativeness of the data inputted;⁴³ our review highlighted several aspects limiting generalisability. The majority of studies included US-based participants, most of whom were younger in age. Thus, the representativeness of study findings for other populations is not known, given cultural and linguistic differences in lower-economically developed populations or different age groups, who may exhibit different health-seeking behaviours, digital literacy rates, and disease burdens.⁴⁴ Furthermore, several studies recruited participants from higher risk groups (e.g., individuals admitted to hospital, or those referred from primary care to specialists), therefore, findings may be less generalisable to general populations given differences in disease spectrum and prevalence. Future work on developing machine learning-based search data algorithms and their accompanying datasets must consider the work’s inclusivity and diversity, supported by the consensus-built recommendations from the STANDING Together project.⁴⁵ Third, ascertaining presence of disease or condition of interest is fundamental in studies of diagnosis and prediction. Only a small number of captured studies linked internet search data with documented clinical diagnosis as ground truth. While studies using anonymous search data attempted to establish ground truth using ‘diagnosis ascertaining queries’, the validity of this method is currently unproven. Fourth, given that many of the conditions assessed in the included studies are rare, even predictive models that have high sensitivity and specificity will typically yield a high ratio of false positives to true positives, limiting their clinical utility. Therefore, researchers should consider combining diagnostic models derived from search data to other signals to provide diagnostic information that is clinically meaningful for low prevalence conditions.

Future directions

The findings of this review highlighted several methodological issues that need to be addressed in future research efforts using internet search data (Table 4). First, while initial evidence of diagnostic and predictive value is promising, validation of performance is needed using independent, prospective studies. In addition, given the importance of ascertaining disease or risk status in studies of diagnosis and prediction, it will be important to both validate the use of diagnosis ascertaining queries, and also to ensure disease or outcome truth are established using the most valid methods feasible. Data linkage is complicated by the different data infrastructure and security and governance structures involved in health record data compared to internet search data. Second, the clinical utility of internet search data has not been established. Evidence of utility will be needed at the individual level, where for example searches might trigger personalised health insights or information (similar to how current internet searches related to self-harm trigger location-specific helplines and online resources⁴⁶). At the clinic or health system level the utility of internet search data as part of routine health care is also currently unknown but is starting to be evaluated in pilot programs as an additional source of patient history within clinical care.⁴⁷ As additional user-held data emerges (e.g. from wearables), the relative contributions of these novel data types to health care will be a key factor. Third, to improve generalisability, research is needed across wider geographies, clinical settings, age groups and cultural settings. In particular, current lack of digital literacy particularly around data portability may act as a major barrier to wider participation. It will also be important to understand individuals’ concerns and preferences around use of internet search data. Although prior research has explored the public’s attitudes and willingness to share digital data for health research,^48,49 there still remains barriers to participating in research.

Finally, a number of reporting standards apply to research in this area, it is essential that future research efforts adopt these to help ensure quality and rigour of research. For example, complete reporting in future studies can be ensured by current guideline initiatives such as STARD-AI,⁵⁰ which will mandate the reporting of key study characteristics and outcomes for AI-centred diagnostic accuracy studies.

Table 4

Methodological considerations needed to strengthen research involving internet search data.
Methodological considerations	Rationale and future needs
Optimising and validating accuracy	● Identifying the range of potential predictors in search data, including health-related linguistic variables, as well as non-health-related linguistic variables, and non-linguistic search variables. ● Validation of diagnostic models using independent, prospective data. ● Tools to efficiently and securely link internet search data to established disease or outcome ground truth, involving medical record data, with participant consent. ● Validation of ‘diagnosis ascertaining queries’ as ground truth.
Determining utility	● Exploring methods to share back information and findings based on internet search queries at an individual level prior to any real-world deployment. ● Demonstrating the diagnostic or predictive value of internet search data (and other user-held data) over existing clinical tools. ● Exploring methods of integrating internet search data within routine healthcare, and impact of intended and unintended consequences.
Improving generalisability	● Studies from wider geographies and individuals with a range of ages, health and digital literacies, and cultures. ● Understand and address individual and community concerns and benefits to participating in research using internet search data as part of healthcare research. ● Research from clinical settings and populations with broader disease prevalence and spectrum
Using reporting guidelines	● Reporting studies using current guidelines such as STARD-AI,⁵⁰ for AI-centred diagnostic accuracy studies.

Technologists and developers

Several technical developments could facilitate use of internet search data as part of healthcare research. Many existing studies used prospective informed consent, which involved participants downloading their data and sharing with researchers. However, the technical process for doing may be challenging for many individuals, and likely creates a barrier to participation. A secure, modern way of doing this where users are in complete control of sharing, but is easier than access-download-access another service-upload, would lower the barriers to participating in research. Second, enabling users to share data from multiple sources could provide a more complete picture of online activity than internet searches alone (particularly for certain age groups), including for example videos, shopping searches, chatbot interactions. Third, a challenge researchers face with the influx of data from multiple sources is standardisation and understanding when ingesting it into their analytics platforms. Initiatives such as the Data Transfer Initiative,⁵¹ an open source portability project which aims to standardise different data into a common model regardless of the source of the data, could potentially decrease the effort and costs associated with data ingestion and handling. Fourth, although data portability tools can make it easier for users to move a copy of their data from one service to another, additional efforts are needed to understand post processing features required to preserve a user's privacy. For example, a user may want to share only some of their internet search or other online data, but not material they consider sensitive. Finally, some research groups are better technically equipped than others to adopt and integrate these internet search data and new portability tools. Facilitating access through open source codelabs, tools, and end-to-end systems could lead to more involvement from the research community and more inclusive studies.

This systematic review demonstrated that researchers have found that individual internet search data can be successfully used in diagnostic and predictive studies. In some medical conditions, researchers have found strong associations between search queries associations and features and conditions of interest. However, there was significant variability regarding conditions of interest, methodology, and predictive models used across the included studies. Nevertheless, the review suggests that internet search data holds potential as a tool for researchers, given its use before patients engage with the healthcare system. Research here is early, and further large-scale prospective validation work is required, with established diagnostic ground-truths. Future applications using internet search data as diagnostic adjuncts or for health information interventions must be balanced with inclusivity and bias considerations, privacy concerns, and robust informed consent.

This systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 recommendations.⁵² This review was registered on PROSPERO (CRD42023468760).

Eligibility criteria

All retrospective and prospective studies utilising individual internet search data to predict or diagnose patient disease status were included. The study inclusion and exclusion criteria are detailed in Table 5. Of note, studies that utilised internet search data and trends to predict disease status on a national or population-level (e.g., whether trends in online searches for cough or anosmia can predict a COVID-19 outbreak) were excluded.

Table 5

Study inclusion and exclusion criteria for this systematic review.
Inclusion criteria	Exclusion criteria
• Internet search history from a search engine. • Use of internet search history for disease risk prediction, diagnosis prediction or disease association. • Studies of any duration. • Data obtained from consented individuals or from anonymous (aggregated) search records. • No age restriction of searchers. • Study outcomes include diagnosis, risk prediction or association with a specific health condition. • Study type: case control, cohort, cross sectional. • Any language. • Any country of origin. • Published between 2013 and 2024.	• Associations of national or population-level temporal trends in search terms and health outcomes. • Study type: qualitative studies, case reports, study protocols, abstracts, conference proceedings. • Study interventions including use of geolocation, telemetric or wearable information. • Published as conference abstract only.

An electronic database search was conducted using Ovid (Wolters Kluwer, Alphen aan den Rijn). The databases MEDLINE and Embase were searched to include articles from January 1 2013 to March 22 2024. A search strategy was devised with the aid of a medical librarian. Search terms included both subject headings and keywords with proximity and wild-card operators. The search strategy can be found in Supplementary Table 2. Searches were not limited by language or country of origin.

Search results were imported into Covidence (Covidence.org, Melbourne) for the screening process and duplicate removal. References were screened by two independent reviewers for potential inclusion using title and abstract review. Full text review of potentially eligible studies was conducted by two independent reviewers, with a third reviewer used to resolve differences.

Data extraction

Key data were extracted and tabulated (Google Sheets, Google LLC) from the included studies by one investigator. This included details of study design, participants characteristics, search engine used, extraction technique, use of analytic techniques, including machine learning, outcomes, findings and funding sources. The location of the study was determined by the location of the recruited participants.

Outcomes

The primary outcomes of this systematic review were (1) the predictive or diagnostic value of personal internet search data for detecting a condition of interest and (2) associations found between features of internet search data (i.e., internet search attributes, such as keywords, search themes, or search frequency) and a condition of interest.

Due to the heterogeneous nature of the design, methodology and reported outcomes of the included studies, a meta-analysis was not performed. A narrative synthesis of the included studies and pre-specified outcomes was therefore conducted.

Risk of Bias

Included non-randomised, non-diagnostic studies were assessed for risk of bias with the Newcastle-Ottawa Scale and the included diagnostic and predictive studies were assessed with the Prediction model risk of bias assessment tool (PROBAST).^53,54 The risk of bias was assessed across the four domains by two independent investigators.

Data availability

The search strategy is detailed in the Supplementary Material. Any additional data is available from the corresponding author on reasonable request.

Acknowledgements

This study received no funding. Infrastructure support was provided by the Center for Health Policy, Institute of Global Health Innovation, Imperial College London and the NIHR Imperial Biomedical Research Centre.

Author information

Contributions

MT and VS contributed to the initial study conception. CC, ED and MT contributed to the data collection and analysis. CC, MT, VS contributed to the manuscript writing. KO, KG, JT, RS, JF and AD contributed to the critical revision of the manuscript. All authors read and critically commented on drafts of the study, including the latest version, and jointly take responsibility for the decision to submit this work for publication.

Corresponding author

Correspondence to Matthew Thompson.

Competing interests

Authors MT, JT, VS, KO, RS, JF are all employed by Google, and have stock in Alphabet. Authors CC, ED, KG, AD declare no financial or non-financial competing interests.

Murru, A. & Carpiniello, B. Duration of untreated illness as a key to early intervention in schizophrenia: A review. Neurosci. Lett. 669, 59–67 (2018).
Wallace, E., Lowry, J., Smith, S. M. & Fahey, T. The epidemiology of malpractice claims in primary care: a systematic review. BMJ Open 3, e002929 (2013).
White, B. et al. Does changing healthcare use signal opportunities for earlier detection of cancer? A review of studies using information from electronic patient records. Cancer Epidemiol. 76, 102072 (2022).
Viscidi, E. et al. Clinical Features of Patients With Progressive Supranuclear Palsy in an US Insurance Claims Database. Front. Neurol. 12, 571800 (2021).
Rasmussen, H., Hellzen, O., Stordal, E. & Enmarker, I. Family caregivers experiences of the pre-diagnostic stage in frontotemporal dementia. Geriatr. Nur. (Lond.) 40, 246–251 (2019).
Tavares, T. P. et al. Early symptoms in symptomatic and preclinical genetic frontotemporal lobar degeneration. J. Neurol. Neurosurg. Psychiatry 91, 975–984 (2020).
Bangerter, A. et al. Caregiver daily reporting of symptoms in autism spectrum disorder: Observational study using web and mobile apps. JMIR Ment. Health 6, e11365 (2019).
Auning, E. et al. Early and presenting symptoms of dementia with Lewy bodies. Dement. Geriatr. Cogn. Disord. 32, 202–208 (2011).
Mahmood, S. S., Levy, D., Vasan, R. S. & Wang, T. J. The Framingham Heart Study and the epidemiology of cardiovascular disease: a historical perspective. The Lancet 383, 999–1008 (2014).
Schalkamp, A. K., Peall, K. J., Harrison, N. A. & Sandor, C. Wearable movement-tracking data identify Parkinson’s disease years before clinical diagnosis. Nat. Med. 29, 2048–2056 (2023).
Brewer, H. R. et al. Association Between Purchase of Over-the-Counter Medications and Ovarian Cancer Diagnosis in the Cancer Loyalty Card Study (CLOCS): Observational Case-Control Study. JMIR Public Health Surveill. 9, e41762 (2023).
Wallace, W. et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. Npj Digit. Med. 5, 118 (2022).
AlGhamdi, K. M. & Moussa, N. A. Internet use by the public to search for health-related information. Int. J. Med. Inf. 81, 363–373 (2012).
Hertling, S., Hertling, D., Schleußner, E., Loos, F. & Graul, I. E-health – The importance of the internet as an informative digital health application for gynecological patients in times of SARs-CoV2: A national cross-sectional survey. Inform. Med. Unlocked 30, 100942 (2022).
Fox, S. The social life of health information. Pew Research Center https://www.pewresearch.org/short-reads/2014/01/15/the-social-life-of-health-information/ (2014).
Lampos, V. et al. Tracking COVID-19 using online search. Npj Digit. Med. 2021 41 4, 1–11 (2021).
Higgins, T. S. et al. Correlations of Online Search Engine Trends With Coronavirus Disease (COVID-19) Incidence: Infodemiology Study. JMIR Public Health Surveill. 6, e19702 (2020).
Li, Y. et al. Web-based medical information searching by Chinese patients with breast cancer and its influence on survival: Observational study. J. Med. Internet Res. 22, e16768 (2020).
Kirschenbaum, M. A. et al. Google search activity in early psychosis: A qualitative analysis of internet search query content in first episode psychosis. Early Interv. Psychiatry 14, 606–612 (2020).
Hochberg, I., Daoud, D., Shehadeh, N. & Yom-Tov, E. Can internet search engine queries be used to diagnose diabetes? Analysis of archival search data. Acta Diabetol. 56, 1149–1154 (2019).
Yamaguchi, S. et al. Usefulness of Web Search Queries for Early Detection of Diseases in Infants. Nagoya J. Med. Sci. 83, 107–111 (2021).
Sadeh-Sharvit, S., Fitzsimmons-Craft, E. E., Taylor, C. B. & Yom-Tov, E. Predicting eating disorders from Internet activity. Int. J. Eat. Disord. 53, 1526–1533 (2020).
Barcroft, J. F. et al. Using online search activity for earlier detection of gynaecological malignancy. BMC Public Health 24, 608 (2024).
Moon, K. C. et al. Internet Search Activity of Young People with Mood Disorders Who Are Hospitalized for Suicidal Thoughts and Behaviors: Qualitative Study of Google Search Activity. JMIR Ment. Health 8, (2021).
Phillips, C. A. et al. Health-related Google searches performed by parents of pediatric oncology patients. Pediatr. Blood Cancer 66, 1–7 (2019).
Birnbaum, M. L. et al. Identifying emerging mental illness utilizing search engine activity: A feasibility study. PLoS ONE 15, 1–15 (2020).
Birnbaum, M. L. et al. Utilizing machine learning on internet search activity to support the diagnostic process and relapse detection in young individuals with early psychosis: Feasibility study. JMIR Ment. Health 7, 1–14 (2020).
Schueller, S. M., Steakley-Freeman, D. M., Mohr, D. C. & Yom-Tov, E. Understanding perceived barriers to treatment from web browsing behavior. J. Affect. Disord. 267, 63–66 (2020).
Zhang, B., Zaman, A., Silenzio, V., Kautz, H. & Hoque, E. The relationships of deteriorating depression and anxiety with longitudinal behavioral changes in google and youtube use during COVID-19: Observational study. JMIR Ment. Health 7, 1–21 (2020).
Areán, P. A. et al. Perceived utility and characterization of personal google search histories to detect data patterns proximal to a suicide attempt in individuals who previously attempted suicide: Pilot cohort study. J. Med. Internet Res. 23, (2021).
Youngmann, B., Allerhand, L., Paltiel, O., Yom-Tov, E. & Arkadir, D. A machine learning algorithm successfully screens for Parkinson’s in web users. Ann. Clin. Transl. Neurol. 6, 2503–2509 (2019).
Shaklai, S., Gilad-Bachrach, R., Tov, E. Y. & Stern, N. Detecting impending stroke from cognitive traits evident in internet searches: Analysis of archival data. J. Med. Internet Res. 23, 1–10 (2021).
Cohen Zion, M., Gescheit, I., Levy, N. & Yom-Tov, E. Identifying Sleep Disorders From Search Engine Activity: Combining User-Generated Data With a Clinically Validated Questionnaire. J. Med. Internet Res. 24, 1–8 (2022).
Yom-Tov, E., Navar, I., Fraenkel, E. & Berry, J. D. Identifying amyotrophic lateral sclerosis through interactions with an internet search engine. Muscle Nerve 69, 40–47 (2024).
Paparrizos, J., White, R. W. & Horvitz, E. Screening for pancreatic adenocarcinoma using signals from web search logs: Feasibility study and results. J. Oncol. Pract. 12, 737–744 (2016).
White, R. W. & Horvitz, E. Evaluation of the feasibility of screening patients for early signs of lung carcinoma in web search logs. JAMA Oncol. 3, 398–401 (2017).
Yom-Tov, E. Screening for Cancer Using a Learning Internet Advertising System. ACM Trans. Comput. Healthc. 1, 1–13 (2020).
Zaman, A. et al. Discovering intimate partner violence from web search history. Smart Health 19, 100161 (2021).
Lebwohl, B. & Yom-Tov, E. Symptoms prompting interest in celiac disease and the gluten-free diet: Analysis of internet search term data. J. Med. Internet Res. 21, 1–8 (2019).
Asch, J. M. et al. Google search histories of patients presenting to an emergency department: An observational study. BMJ Open 9, 1–6 (2019).
Hochberg, I., Allon, R. & Yom-Tov, E. Assessment of the frequency of online searches for symptoms before diagnosis: Analysis of archival data. J. Med. Internet Res. 22, 1–7 (2020).
Yom-Tov, E., Cherlow, Y., & Rabbi. Ethical challenges and opportunities associated with the ability to perform medical screening from interactions with search engines: Viewpoint. Journal of Medical Internet Research vol. 22 e21922 (2020).
Siebert, J. et al. Construction of a quality model for machine learning systems. Softw. Qual. J. 30, 307–335 (2022).
Jia, X., Pang, Y. & Liu, L. S. Online Health Information Seeking Behavior: A Systematic Review. Healthcare 9, 1740 (2021).
STANDING Together Working Group. STANDING Together. STANDING Together https://www.datadiversity.org/ (2022).
Get help & support for suicide. Google Search Help https://support.google.com/websearch/answer/11181469.
Macrynikola, N., Nguyen, N., Lane, E., Yen, S. & Torous, J. The Digital Clinic: An Innovative Mental Health Care Delivery Model Utilizing Hybrid Synchronous and Asynchronous Treatment. NEJM Catal. Innov. Care Deliv. (2023) doi:10.1056/CAT.23.0100.
Shiells, K. et al. Participant acceptability of digital footprint data collection strategies: an exemplar approach to participant engagement and involvement in the ALSPAC birth cohort study. Int. J. Popul. Data Sci. 5, 1728.
Hirst, Y. et al. Understanding Public Attitudes and Willingness to Share Commercial Data for Health Research: Survey Study in the United Kingdom. JMIR Public Health Surveill. 9, e40814 (2023).
Sounderajah, V. et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat. Med. 26, 807–808 (2020).
Data Transfer Initiative. Data Transfer Initiative https://dtinit.org/.
Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, (2021).
Wells, G. et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
Wolff, R. F. et al. PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Ann. Intern. Med. 170, 51–58 (2019).

Table 1 is available in the Supplementary Files section.

There is a conflict of interest Authors MT, JT, VS, KO, RS, JF are all employed by Google, and have stock in Alphabet. Authors CC, ED, KG, AD declare no financial or non-financial competing interests.

Download PDF

Review #2 received at journal
30 Jul, 2024
Reviewer #2 agreed at journal
16 Jul, 2024
Review #1 received at journal
05 Jul, 2024
Reviewer #1 agreed at journal
26 Jun, 2024
Reviewers invited by journal
31 May, 2024
Editor assigned by journal
22 May, 2024
Submission checks completed at journal
22 May, 2024
First submitted to journal
21 May, 2024

You are reading this latest preprint version

Systematic review of health research using internet search data

Status:

Version 1

Abstract

Figures

Background

Results

Study participants

Search engines and extraction

Use of machine learning algorithms

Ground truth and control

Outcomes

Linguistic associations

Temporal associations

Other associations

Discussion

Technologists and developers

Conclusions

Methods

Declarations

Data availability

Acknowledgements

Author information

Contributions

Corresponding author

Competing interests

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1