The Delphi method is an accepted method of gathering quantitative and qualitative data from subject area experts. The process consists of discussion and the administration of a series of questionnaires aimed at generating consensus20. It has been described as the only systematic method of combining expert opinion and evidence21. Quantitative data collected via questionnaires are aggregated and re-introduced to participants for further discussion and feedback. Multiple iterations lead to consensus. Delphi methods are appropriate when there is inadequate empirical evidence and/or the research question has no answer that has been agreed upon22. The Delphi approach was applied to refine, substantiate and finalise a list of nutritional risk indicators, from 42 indicators found to be associated with malnutrition and health outcomes through a program of literature reviews. The methods and results of the current research are illustrated in Fig. 1. The list of the 42 indicators identified via the literature reviews is shown in Table A1 in Additional data file 1. In summary, these covered areas of disease status or condition; eating, appetite and digestion; type of diet; cognition, psychology and social factors; and polypharmacy.
Recruitment of Delphi participants
A Project Advisory Committee including managers from New South Wales (NSW) Ministry of Health and NSW Local Health Networks in Australia invited organisations across NSW to nominate appropriate employees to participate as key stakeholders. From this process, 54 key stakeholders agreed to participate, including clinical staff, food service staff and managers from the disciplines of Nutrition and Dietetics, Food Service, Speech Pathology, Nursing, Clinical Governance and Data Systems.
In addition, Australian and International experts in the areas of Nutrition and Dietetics, Nursing, Speech Pathology and Medicine were invited to form part of an expert panel to provide feedback during consultation. We sought to create a panel of multi-disciplinary experts with evidence of expertise and research track record in the assessment and management of adults at risk of malnutrition. Of the 18 experts invited, 11 agreed to participate, one of which withdrew before providing feedback.
Generation of preliminary list of malnutrition risk factors
To inform Delphi round 1, we conducted a review of reviews using a systematic search methodology across six electronic databases [Medline (Ovid), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Database of Systematic Reviews, Joanna Briggs Institute Database, Embase (Ovid) and Scopus] generating 5,889 citations for screening. Following screening, 59 reviews summarising original studies reporting on indicators of nutritional risk were identified. After quality appraisal, the data from seven high quality reviews identified a list of 57 unique indicators of nutritional risk. Two indicators (organ failure and infectious diseases) were grouped together to form one indicator: critical illness. Therefore, 56 indicators formed the basis for 56 separate literature reviews of associations between each indicator and health outcomes, i.e. morbidity, mortality, length of hospital stay and complications. Those indicators that were found to have no evidence for an association with outcomes of interest (n = 10) were removed from consideration in round 1 consultation as part of the Delphi method. Of the remaining 46 indicators, 34 were confirmed to be supported by a significant body of evidence (> 10,000 articles). Twelve indicators were determined to have evidence of a relationship with health outcomes from a limited number of studies. Therefore, 46 indicators were considered for progressing through to Round 1 of the Delphi process. The project team then clarified the list of indicators by removing or grouping similar indicators, e.g. anorexia and altered intake; and expanding indicators such as surgery into minor and major surgery, and organ failure into renal failure, hepatic failure, respiratory failure and heart failure. After these changes, a total of 42 indicators (see Table A1 in Additional data file 1 for the list of these indicators).
The Delphi approach
Our iteration of the Delphi method was conducted between August and November 2018 and included four rounds of consultation with stakeholders and an expert panel. Each of the rounds is described in more detail below. A project website was used to provide contact with the participants throughout the Delphi process. A face-to-face (either in-person or videoconference) workshop was also conducted in Round 3. Multiple electronic questionnaires were conducted throughout all four rounds to collect participant feedback. Each questionnaire was tested for accuracy, clarity and consistency with members of the Project Advisory Committee before being distributed to the key stakeholder and expert panels. Results of each questionnaire were communicated to participants via the project website so that subsequent consultation and feedback were informed by the findings of previous questionnaires.
Round 1
Video and written summaries of the reviews to identify nutritional risk indicators were made available via the project website at the commencement of Round 1. In part 1 of Round 1, key stakeholders were asked to complete an online questionnaire to rate each of the 42 nutritional risk indicators on a Likert scale from 1 (extremely low importance) to 5 (extremely high importance). The results of the part 1 questionnaire informed the separation of the indicators into quartiles (quartile 1 contained the most important indicators, quartile 4: least important). In part 2 of Round 1, the stakeholders completed a second questionnaire in which they selected if they agreed or disagreed with the quartile each indicator had been allocated to.
Round 2
Members of the expert panel accessed the video and written summaries of the literature reviews on the project website at the commencement of Round 2. The quartile groups of nutritional risk indicators generated through Round 1 were made available to the experts via the Project Website. Via a questionnaire, the expert panel selected if they agreed/disagreed with the quartile that each indicator had been allocated to. The questionnaire also allowed expert panel members to suggest new indicators to be added to the list. Based on the findings of Round 2, the indicators were rearranged into new quartiles according to perceived importance.
Round 3: Face-to-face workshop
Key stakeholders were invited to attend one of two face-to-face half day workshops. The workshops were facilitated by the project team and were held local to the stakeholders to facilitate attendance. Every effort was made to ensure that the range of different stakeholder disciplines were equally represented on each day. There were two sessions per workshop in which group discussion was facilitated by researchers. Throughout the sessions, the facilitators summarised the important issues and aspects of the discussion and, where needed, prompted or guided the discussion to ensure the group continued to move toward consensus. At the end of each session, the participants completed an online questionnaire. Although groups discussed the findings of previous questionnaires openly, they independently and anonymously completed individual electronic questionnaires immediately after each session. In the first questionnaire, participants considered the allocation of indicators into quartiles informed by Round 2. They were asked to select if they agreed or disagreed with these allocations. Participants were also able to suggest indicators be added or removed and a revised list was generated. This list was fed back to the participants, leading to further discussion. Participants then completed a second questionnaire where they were asked to rate each indicator in the revised list from 1–5 (1: least important, 5: most important). In addition to the quantitative data collected in questionnaires, qualitative data were collected during workshop group discussions. After the workshops, a further revised list was generated based on qualitative and quantitative data, which consisted of 15 nutritional risk indicators (listed in Table 1).
Round 4
Finally, key stakeholders and members of the expert panel completed an electronic questionnaire to rate the importance of each of the 15 indicators from 1–5 (1: least important; 5: most important).
Statistical Analysis
Data generated from surveys in Rounds 1–4 were exported to Stata 15 software for analysis23. Mean and variance scores for each indicator were tested for randomness, the standardised mean score for each indicator (item) was used as a group effect size of the response, the standard deviation (SD) as the dispersion of scores, the median score as the direction of the rating. Consistency was tested with intra class correlation coefficient (ICC), interpreted as follows: ≤0.40, poor consistency or large variation in opinion; 0.41–0.74, acceptable consistency; and ≥ 0.75, good consistency24. There is no accepted, set standard for the target percentage of agreement, with thresholds and definitions of consensus ranging between 51% and 80%24. We conservatively defined consensus as when ≥ 80% of participants rated each individual statement as very important or extremely important on the five-point Likert scale. Statements not meeting 80% agreement were modified according to feedback provided and redistributed to the panellists for the next round of consultation25. Cronbach's alpha (α) was used during each round of the Delphi process to determine the reliability of the indicators in the developing instrument. An a priori α of 0.7–0.9 was used to define moderate to high reliability26.
To ensure further reliability and validity of the refinement process, the final list of indicators was subjected to an exploratory principal component analysis (PCA)27 to examine the interrelationships among the indicators and identify the shared proportion of variance to summarise and validate the number of indicators while maximising the amount of information retained. It was guided by the correlations of the indicators’ importance ratings with each other, typically, variables which correlate highly with each other will be combined into a single component28. This study complied with the guide for the conducting and reporting of Delphi studies (CREDES)29.