Development of the first version of the rating reference guide
Table 1 shows the first version of the rating reference guide for body function categories of the ICF Generic-30 Set. In developing the rating reference guide, two key topics emerged from the qualitative content analysis of the discussion notes: What to rate in each category, and how to frame the response options.
Table 1
|
First version of the rating reference guide
|
|
|
|
What aspect should be scored
|
What is the complete problem
|
b130
|
Energy and drive functions
|
・The extent of the problem
|
May include: Having no motivation or appetite at any time
|
・The frequency of the problem
|
|
|
|
|
b134
|
Sleep functions
|
・The extent of the problem
|
May include: Incapable of sleep at all, or the sleeping schedule has collapsed
|
・The frequency of the problem
|
|
|
|
|
b152
|
Emotional functions
|
・The extent of the problem
|
May include: Complete loss of control of emotions every day, or incapable of expressing emotions
|
・The frequency of the problem
|
|
|
|
|
b280
|
Sensation of pain
|
・The extent of the problem
|
May include: Suffering from continuous intolerable pain at any time
|
・The frequency of the problem
|
|
|
|
|
b455
|
Exercise tolerance functions
|
・The extent of the problem
|
May include: Incapable of bearing any single activity of daily living at any time due to cardiorespiratory problems
|
・The frequency of the problem
|
|
|
|
|
b620
|
Urination functions
|
・The extent of the problem
|
May include: Complete urinary retention or continuous incontinence at any time
|
・The frequency of the problem
|
|
|
|
|
b640
|
Sexual functions
|
・The extent of the problem
|
May include: Having no mental and/or physical ability to perform sexual activity, or complete loss of control in sexual desire at any time
|
・The frequency of the problem
|
|
|
|
|
b710
|
Mobility of joint functions
|
・The extent of the problem
|
May include: Complete joint contracture in all major joints
|
・The ratio of the joint with the problem
|
|
|
|
|
b730
|
Muscle power functions
|
・The extent of the problem
|
May include: Complete loss of muscle power in all major muscles
|
・The ratio of the joint with the problem
|
|
|
|
|
Ratings
|
*Note that the rating should reflect the body function without the help of devices
|
0
|
No problem
|
|
|
1
|
Mild problem
|
|
|
2
|
Moderate problem
|
|
|
3
|
Severe problem
|
|
|
4
|
Complete problem
|
|
|
8
|
Not specified
|
|
|
9
|
Not applicable
|
|
|
What to rate in each category
The challenges posed by having several aspects (e.g. severity, frequency, location) to consider in rating a given category were discussed. For example, for the category “b134 sleep functions” the problem could be understood as a combination of the frequency and the extent of problems in sleep functions. To address these challenges, working group members highlighted specific aspects to focus on when evaluating each category. The overall sentiment was that having too many aspects to focus on would complicate the rating process and that the rating reference guide should be kept as simple as possible. Accordingly, the working group members identified two main aspects that should be considered when rating the body function categories of the ICF Generic-30 Set – the extent and frequency of the problem in the given category.
How to frame the response options
The working group members also proposed that concrete examples should be provided to improve the clarity of the guide, and that these examples should include information on clinical instruments that are commonly used to assess the given category. This idea was first adopted during the development of the initial draft; however, during the review process by ICF experts, there was a concern raised that this approach may result in overly complicated manuals. Since every category would highlight several aspects that should be considered during rating, the manuals would have to include specific descriptions of those aspects as well as define how to consider a combination of problems. For example, if we rate the status of muscle functions with manual muscle testing, then we should also consider how many and which muscles are impaired. Several reviewers raised concerns that this would make the use of the rating reference guide in the clinic too complicated. Accordingly, concrete descriptions based on the clinical instruments in the reference guide were removed from the initial draft.
The first version of the rating reference guide was simplified as follows: Two major aspects were specified and should be considered when assigning ratings (see Table 1) for each category . For example, for “d620 urination function,” the guide instructs raters to consider the frequency and extent of the problem when rating this category from 0 to 4. No further explanation regarding each response option was added.
Field test to gain feedback on the rating reference guide in practical use
A total of 60 individuals (52 inpatients and 8 community-dwelling elderly) participated in this study. The 52 patients underwent rehabilitation during April 2017 to November 2018 at Fujita Health University Hospital and Fujita Health University Nanakuri Memorial Hospital. Among these 60 individuals (38 males and 22 females), 31 had neurological disease; 9 had musculoskeletal disease; 9 had cardiopulmonary disease; and 4 had other health issues. The mean age of the patients was 64.5 ± 17.7 years.
The results of the interrater reliability calculation of the ratings using the initial version of the rating reference guide is shown in Table 2. The weighted kappa statistics ranged from 0.25 to 0.92, indicating low interrater reliability for several categories, while moderate to excellent for other categories. For example, the weighted kappa coefficients for b620 Urination functions indicated an excellent interrater reliability, while the kappa for “b152 emotional functions” a low or fair interrater reliability [22].
Table 2
|
Interrater reliability of the rating using the first version of the rating reference guide
|
|
|
|
|
|
Categories
|
Weighted κ
|
|
b130
|
Energy and drive functions
|
0.56
|
|
b134
|
Sleep functions
|
0.62
|
|
b152
|
Emotional functions
|
0.25
|
|
b280
|
Sensation of pain
|
0.44
|
|
b455
|
Exercise tolerance functions
|
0.55
|
|
b620
|
Urination functions
|
0.92
|
|
b640
|
Sexual functions
|
0.80
|
|
b710
|
Mobility of joint functions
|
0.58
|
|
b730
|
Muscle power functions
|
0.65
|
|
|
|
|
|
Investigation of the potential problems with the first version of the rating reference guide
According to the results of the first field test, the guidance information for the following four categories with the lowest interrater reliability were identified needing improvement: “b130 energy and drive functions,” “b152 emotional functions”, “b280 sensation of pain,” and “b455 exercise tolerance functions”. The following problems raised during the field test were discussed in an effort to improve the interrater reliability of these categories: Difficulty distinguishing between mild problems and moderate problems, difficulty rating patients who cannot express their emotions (“b152 emotional functions”), and lack of consideration of the number of pain sites (“b280 sensation of pain”).
Two of the raters indicated that it was difficult to distinguish mild problems from moderate problems. The ICF published by the World Health Organization states that a moderate problem is “generally up to half of the scale of the total problem” [1]; thus, the raters felt that differences between moderate and severe problems were relatively easy to distinguish. However, other than this clarifying statement about the interpretation of a moderate problem, there are no clarifications about mild problems, only the presentation of the corresponding percentages of mild problems and moderate problems (5%–24% and 25%–49%, respectively). This makes it difficult to differentiate between mild and moderate problems. This lack of guidance was seen as problematic for rating patients in the clinic.
The second point raised by the raters was the complexity of the first version of the rating reference guide. The first version of the rating reference guide outlined specifically for each category various aspects to be rated and an example of what a complete problem would encompass. The raters were required to take this information into account when rating each category, but without a concrete guide for each response option (0 to 4). Several raters stated that these rating instructions were confusing and made the rating difficult.
An issue related to "b152 emotional functions" was also raised. Specifically, a rater highlighted the difficulty in evaluating the emotional functions of patients with problems expressing emotions. The first version of the guide indicated that a “complete problem” in b152 is exemplified by completely losing control of emotions every day. However, there are patients who do not lose control of emotions but rather are unable to express emotions. Three of the raters agreed that the inability to express emotions should also be recognized as a problem in b152.
With regard to rating “b280 sensation of pain” the first version of the rating reference guide instructed that frequency and the extent of pain should be considered when rating. However, two of the raters indicated that the number of pain sites also has an influence on the degree of the problem.
Modification of the reference guide
The reference guide was modified by a multidisciplinary panel according to the feedback from the raters in the field test. As the feedback from the raters were focused on the issues in the rating and did not always include concrete suggestion for improvement, the investigators decided to revisit the records of the first cognitive interviews that resulted in the initial draft of the guide (see figure 1). This helped to address the difficulty in distinguishing mild problems from moderate problems. According to the interview records, several raters mentioned that assigning a rating of 1 (mild problem) for five of the nine categories (“b130 energy and drive functions,” “b134 sleep functions,” “b280 sensation of pain,” “b710 mobility of joint functions,” and “b730 muscle power functions”) was due to the lack of impact these body functions had on daily activities. Given this, the members of the panel added text to the guide that describes a mild problem in a particular ICF category as a problem that does not affect daily activities. To clarify the difference between a rating of 2 (moderate problem) and a rating of 3 (severe problem), the following explanations were given: A rating of 2 “may include a problem that exceeds a rating of 1, but still remains a relatively minor problem (< 50%) in the given category”, and a rating of 3 “may include a problem that is a major problem (≧ 50%) in the given category”. The percentages (< 50% and ≧ 50%) were added to emphasize that a “moderate problem” is “generally up to half of the scale of the total problem” [1]; this further distinguishes the ratings 2 and 3. The percentage was set to describe how much is the problem is relative to a complete problem (100% as the amount of the problem). For example, in scoring "b130 energy and drive functions", a complete problem is described as “having no motivation nor appetite at any time”, and this is regarded to be 100% of the problem. Raters then consider the amount of problem a person has in b130 by considering the extent and the frequency of the lack of motivation or appetite.
The rating reference guide descriptions for “b152 emotional functions” and “b280 sensation of pain” were also modified. For “b152 emotional functions,” the following explanation describing a complete problem in this category was added: “being incapable of expressing emotions at any time”. For “b280 sensation of pain,” the pain site was added as an aspect to be considered prior to assigning a rating score. The modified rating reference guide (from now on referred to as “final version of the rating reference guide”) is shown in Table 3.
Table 3 Second version of the rating reference guide
|
|
Category
|
Aspect to be scored
|
Description of each response option
* The percentage describes the severity of the problem, if 100% means it is a complete problem. **The rating should reflect the body’s function without the help of devices.
|
b130
|
Energy and drive functions
|
Extent and frequency of the problem, such as loss of motivation or appetite
|
0: No problem
|
|
1: Mild problem: May include problems with energy and drive functions that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem in energy and drive functions that exceeds 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) in energy and drive functions
|
|
4: Complete problem: May include a complete problem with energy and drive functions, such as having no motivation or appetite at any time
|
|
|
|
|
b134
|
Sleep functions
|
Extent and frequency of the problem, such as shortage of sleep or irregular sleep schedules
|
0: No problem
|
|
1: Mild problem: May include problems with sleep that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem with sleep that exceeds 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with sleep
|
|
Complete problem: May include a complete problem with sleep, such as being incapable of sleeping, or a complete day–night reversal every day
|
|
|
|
|
b152
|
Emotional functions
|
Extent and frequency of the problem, such as loss of emotional control or lack of emotional expression
|
0: No problem
|
|
1: Mild problem: May include problems with emotions that do not affect the patient's daily activities
|
|
2: Moderate problem: May include problems with emotions that exceed 1, but remains relatively minor (<50%)
|
|
|
3: Severe problem: May include a major problem (≧ 50%) with emotions
|
|
|
4: Complete problem: May include a complete problem with emotions, such as complete loss of control of emotions, or being incapable of expressing emotions at any time
|
|
|
|
|
b280
|
Sensation of pain
|
Extent, frequency, and number of sites with pain
|
0: No problem
|
|
|
1: Mild problem: May include problems with sensations of pain, but does not affect the patient's daily activities
|
|
|
2: Moderate problem: May include a problem with sensations of pain that exceeds 1, but remains a relatively minor problem (<50%)
|
|
|
3: Severe problem: May include a major problem (≧ 50%) with sensations of pain
|
|
|
4: Complete problem: May include a complete problem with sensations of pain, such as continuous, intolerable pain
|
|
|
|
|
b455
|
Exercise tolerance functions
|
Extent and frequency of the problem, such as decline in respiratory and cardiovascular capacity that is required to perform daily activities
|
0: No problem
|
|
1: Mild problem: May include problems with exercise tolerance that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem with exercise tolerance that exceeds Level 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with exercise tolerance
|
|
|
4: Complete problem: May include a complete problem with exercise tolerance, such as being incapable of bearing any single activity of daily living at any time due to cardiorespiratory problems
|
b620
|
Urination functions
|
Extent and frequency of the problem, such as difficulty urinating or urinary incontinence
|
0: No problem
|
|
1: Mild problem: May include problems with urination that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem with urination that exceeds 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with urination
|
|
4: Complete problem: May include a complete problem with urination, such as complete urinary retention or continuous incontinence at any time
|
|
|
|
|
b640
|
Sexual functions
|
Extent and frequency of the problem, such as loss of sexual desire and/or physical ability to engage in sexual activity
|
0: No problem
|
|
1: Mild problem: May include problems with sexual functions that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem with sexual functions that exceeds 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with sexual functions
|
|
4: Complete problem: May include a complete problem with sexual functions, such as a complete loss of sexual desire and/or physical ability to engage in sexual activity at any time
|
|
|
|
|
b710
|
Mobility of joint functions
|
Extent of the problem, such as joint contracture, or limitations in range of motion and percentage of joints with mobility problems
|
0: No problem
|
|
1: Mild problem: May include problems with joint mobility functions that do not affect the patient's daily activities
|
|
2: Moderate problem: May include problems with joint mobility that exceed 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with joint mobility
|
|
4: Complete problem: May include a complete problem with joint mobility, such as complete joint contracture in all of the major joints
|
|
|
|
|
b730
|
Muscle power functions
|
Extent of the problem, and percentage of joints with muscle power problems
|
0: No problem
|
|
1: Mild problem: May include problems with muscle power that do not affect the patient's daily activities
|
|
2: Moderate problem: May include a problem with muscle power functions that exceeds 1, but remains a relatively minor problem (<50%)
|
|
3: Severe problem: May include a major problem (≧ 50%) with muscle power
|
|
4: Complete problem: May include a complete problem with muscle power, such as a complete loss of muscle power in all of the major muscles
|
Interrater reliability using the final version of the rating reference guide
A total of 123 patients who underwent rehabilitation from April to June 2020 at Fujita Health University Hospital participated in this study. Among these individuals (78 males and 45 females), 93 had neurological disease; 17 had musculoskeletal disease; 9 had cardiopulmonary disease; and 4 had various other health issues. The mean age of the patients was 69.1 ± 15.1 years. The results of the interrater reliability study by four raters (two physical therapists and two occupational therapists) using the final version of the rating reference guide are shown in Table 4. The weighted kappa coefficient ranged from 0.54 to 0.85, indicating that all of the categories had moderate to excellent interrater reliability. A high rate of missing data (41.5%) was observed for “b640 sexual functions”.
Table 4
|
Interrater reliability of the rating using the second version of the rating reference guide
|
|
|
|
|
|
|
Categories
|
Weighted κ
|
Missing values
|
|
b130
|
Energy and drive functions
|
0.80
|
0
|
|
b134
|
Sleep functions
|
0.69
|
0
|
|
b152
|
Emotional functions
|
0.69
|
0
|
|
b280
|
Sensation of pain
|
0.82
|
1 (0.8%)
|
|
b455
|
Exercise tolerance functions
|
0.67
|
0
|
|
b620
|
Urination functions
|
0.79
|
2 (1.6%)
|
|
b640
|
Sexual functions
|
0.54
|
51 (41.5%)
|
|
b710
|
Mobility of joint functions
|
0.77
|
0
|
|
b730
|
Muscle power functions
|
0.85
|
0
|
|