System assessment:
In September 2019, a prototype of EPR-Youth was launched that was not finished yet. All CJG-professionals could write their reports in the system. The client-portal, however, contained only basic functionalities: clients could manage appointments and ask questions. They had access to vaccination status and growth data. Full access to visit notes and to a viewlog, showing which professionals had accessed their record, followed in 2020. Automatic portal access was limited to parents of preschool children. Adolescents and parents of children aged 4 years and older were given access manually upon their request.
In June 2020, most members of the steering committee and project group said that EPR-Youth was performing well for now. However, the system was still under development: 'Proud that we have something working, which is not yet finished, but is developing in the intended direction'. [109:153] 'that we have actually delivered something that our professionals can work with' [109:154].
Implementation process:
Based on qualitative analysis of focus group interviews and project meetings, we defined barriers and facilitators in the implementation process.
Barriers:
Complexity of co-creative process: The project proceeded slower than expected, partly due to underestimation of the complexity of a co-creative process of change: 'We are actually going to use one EPR-system with three organizations. So we’re going to merge virtually….I think we underestimated the complexity behind it....'[31:63]. Underlying to this complexity were differences between organizations. Whereas the members of the steering committee shared one vision about care for youth, this vision appeared not to be embraced yet throughout all three organizations. A steering committee member said: ‘You share a vision….but it also has to fit the organization you’re in. And then, along the way…..,you discover how incredibly difficult that can be'[109:10] In the project group, someone said: ‘.... that we have had a great deal of trouble pretending to be one organization, while we are not at all. And that it does not come to the surface, .... that we are not working on one vision now’ [107:28] In the project group, the differences in vision and working method induced recurring substantive and legal discussions, creating interpersonal tensions: 'At the end, I saw the eyes turning like: there they are again'. [107:6]
Leadership: The project group felt the project was lacking a plan with clear division of responsibilities: 'It was very ad hoc...I missed a plan of action...clarity about everyone's role'. [107:3] Although the project group was expected to show shared ownership of the process, they did not at first experience ownership, nor did the project leader require ownership from them: '...since the project leader was constantly saying "we have to ask the steering committee about that" ....'[107:152] Maintaining balance between self-organizing as a project group and receiving directions from the steering committee was a recurring theme. Whereas the project leader called for directing professionals’ attitude and behavior, the steering committee held back, trusting the change process to continue organically: ‘setting up the vision is an ongoing process....this is a change of culture’ [82:24]
The steering committee acknowledged that, as a result ‘the project group …might felt thrown in at the deep end sometimes’[109:24]. They could have intervened earlier, especially in recurring discussions about vision: ‘The steering committee ... including myself…. assumed too easily that everyone knew the ultimate objective.’[108:9]
Facilitators:
Structuring the process: The project group appreciated that, at a certain point, clear deadlines were set: 'That deadline had to be met at all costs. Then, we were able to make rigorous decisions.’ [107:13] The steering committee agreed with that: 'Yes, that deadline was helpful. As far as I'm concerned, we should have set many more deadlines.’ [108:96] At the same point, the project group was divided in smaller groups, working on specific tasks. This was also perceived as helpful and contributed to the sense of ownership: 'This is what people had suggested for a long time: one group working on the content, and another working on other issues, like the newsletter, the user’s manual and things like that' [107:13].
Clarifying the vision: To the project group, a clear explanation of the vision beforehand would have been helpful: ‘I would have liked to dwell more on the vision, really share it with each other...'. [107:7] Simultaneously, they valued clarifying the vision through their discussions. After the process of clarification, they felt that decision-making became easier: '...somehow, we needed that process to be able to take decisions...... once everyone clearly understood the 'why', decisions were taken quickly'. [107:11] The steering committee said the process of jointly clarifying the vision had also reduced resistance: 'the resistance among professionals is, eh, much less than it was.'[109:156]
Pioneering spirit: Everybody was aware of the pioneering character of this project, realizing that there was no earlier example that could be copied: 'That made it more tolerable, because we were just taking up this challenge together that no one had tackled before.'[107:38] and '.... at a birthday party, it's nice to be able to tell...this project is unique in the Netherlands.’ [107:15]. This awareness created a feeling of pride, both in the steering committee and in the project group: 'I'm also proud...we've had some pretty tough discussions, some people have even left.... but we are still getting along very well'. [107:14]
Implementation Outcomes
Acceptability:
The steering committee considered the initial implementation a success. In their focus group interview, one member said: ‘We noticed few interruptions where it comes to... user adoption... In terms of... user acceptance, the process went quite smoothly’ [109:63]. An application manager said: 'We have had relatively few start-up problems......I do get questions, but nothing like "what a miserable system" or “we can't work with this”.’ [107:149]
From the 914 parents and 89 adolescents that completed the clients’ questionnaire, The client questionnaire 511 parents and 15 adolescents reported they had logged on to the client-portal and responded on the user experience questions (figure 3). Clients were predominantly positive about easy access, comprehensibility, and clear overview. No difference in scores were found according to educational level or native country. In the focus groups, clients reported that the client-portal was easy to use. However, adolescents considered the obligatory two-factor authentication time-consuming: ‘You have to enter the website and you need to verify in, like, four steps..., so you have to take your time,…..to check things’ [5:146].
Professionals differed in their opinion about ease-of-use. One participant in a focus group said ‘It’s easy to fill in. (EPR-Youth)' [112:19], whereas someone else said: 'it's a nuisance’. [112:45]. The professionals’ questionnaire was completed by 66 of the 92 (72%) invited CJG professionals (figure 3). A statistically significant difference for ease-of-use was found between different organizations, and between age groups. Professionals delivering preventive child healthcare to pre-school children scored significantly more positive in this domain (M=3,36, SD=1.0) than youth care professionals (M=2.6, SD=0.86), as determined by Tukey HSD post-hoc test (p<0.001). Professionals aged 60 years and older scored significantly more positive in this domain (M=3.8, SD=0.72) than professionals between 40 and 50 years of age (M=2.4, SD=0.88), as determined by Tukey HSD post-hoc test (p=0.02). Professionals reported both in focus groups and in the questionnaire that they particularly appreciated new time-saving functionalities, like automatically generating referrals or indication statements. However, many professionals appeared not to be familiar with all possibilities of EPR-Youth.
Answers were given on a 5-point Likerts scale, ranging from ‘absolutely’ (1) to ‘not at all’ (5) for clients, and ranging from ‘totally agree (1)’ to ‘totally disagree’ (5) for professionals. Scores were reversed for all questions, except the ones marked with an asterisk, resulting in low scores representing a negative opinion and high scores representing a positive opinion. In the clients’ questionnaire, Q2-4 was answered only by parents and adolescents that had logged in on the client-portal.
Adoption:
In de North Veluwe region live approximately 38.000 children, aged 0 to 18 year, and 52.800 parents. As CJG-clients, they all have access to the client-portal. In the period September 2019 until December 2020, 5174 clients logged in to the portal. In the first five months, the monthly number of portal users slowly increased, then stabilized around 1200 portal users monthly. Table 2 shows that most portal users were women (87.1%) and most users were aged between 30 and 44 years (76.6 %). Compared to the average local population, few portal users were of non-Dutch nativity. The average number of logons per person was 3.7 (SD 4.1) in 15 months, ranging from 1 to 47 logons. No differences were found in logon frequency, according to sex, age and native country.
Table 2
Mean number of logons to client-portal, in relation to demographic characteristics of portal users; distribution of portal user’s demographic characteristics is compared with general population of the North Veluwe region, aged 15 to 65.
Source: Central Bureau for Statistics Netherlands.
| | Mean logons per user (SD) | Number of users (percentage) | Inhabitants North Veluwe (percentage) |
TOTAL | | 3.68 (SD 4.1) | 5174 | |
Sexa | Male | 3.72 (SD 4,4) | 659 12.7%) | 50.5% |
Female | 3.67 (SD 4.0) | 4509 (87.1%) | 49.5% |
Unknown | | 6 (0.1%) | |
Ageb | 15-29 years | 3.57 (SD 4,0) | 1107 (21.4%) | 28.4% |
30-44 years | 3.69 (SD 4.3) | 3964 (76.6%) | 27.6% |
45-64 years | 3.87 (SD 4.0) | 102 (2.0%) | 44.0% |
Native countryc | Netherlands | 3.68 (SD 4.1) | 4895 (94.6%) | 92.4% |
Surinam | 2.00 (SD 1.4) | 5 (0.1%) | 0.2% |
Netherlands Antilles | 4.40 (SD 3.4) | 5 (0.1%) | 0.1% |
Turkey | 3.75 (SD 3.5) | 28 (0.5%) | 1.0% |
Morocco | 4.13 (SD 5.3) | 15 (0.3%) | 0.5% |
Europe, North America, Oceania, Indonesia, Japan | 3.55 (SD 3.7) | 105 (2.0%) | 2.8% |
Other countries | 3.68 (SD 3.9) | 121 (2.3%) | 3.1% |
a Significantly different from North Veluwe population (Chi2-test (1) 2945.942, 2-sided p<0.001) |
b Significantly different from North Veluwe population (Chi2-test (2) 6671.353, 2-sided p<0.001) |
c Significantly different from North Veluwe population (Chi2-test (6) 41.644, 2-sided p<0.001) |
The clients’ questionnaire showed that logon percentages among parents differed, according to educational level, family composition and children’s age (Table 3). These differences did not appear among adolescents. Parents of children aged 0-3 years, reported most often that they had logged on to the client-portal. As reasons to logon to the portal, clients mentioned: checking or managing appointments (72.4%), reading what was discussed (54.3%), asking a question (16.7%) and adding or changing information (2.5%). In the client focus groups, most participants reported they had not been aware of the existence of the client-portal until they were invited to complete the clients’ questionnaire.
Table 3
Comparison of login percentages among respondents of the client questionnaire, according to socio-demographic characteristics. Percentages are given between brackets.
| Parents | Adolescents |
| Logged in N=490 (53,8) | Total N=911 | Statistic | 2-sided p-value | Logged in N=14 (16,1) | Total N=87 | Statistic | 2-sided p-value |
Educational level | | | | | | | | |
Low | 32 (50,8) | 63 | c2 = (2) 9.285 | 0.01 | 6 (15,0) | 40 | c2 = (1) 0.671 | 0.413 |
Middle | 207 (50) | 414 | NA | |
High | 230 (60,5) | 380 | 7 (22,6) | 31 |
Missing | 21 | 54 | | | 1 | 16 | | |
Family composition | | | | | | | | |
2-Parent family | 452 (58,9) | 768 | c2 = (1) 49.523 | <0.001 | 5 (16,1) | 31 | Fisher’s exact | 1.000 |
Other situation | 16 (18,8) | 85 | 7 (15,9) | 44 |
Missing | 22 | 58 | | | 2 | 12 | | |
Native country | | | | | | | | |
Netherlands | 431 (54,5) | 791 | c2 = (1) 0.447 | 0.504 | 12 (17,4) | 69 | Fisher’s exact | 0.581 |
Other country | 15 (48,4) | 31 | 0 (0,0) | 6 |
Missing | 44 | 89 | | | 2 | 12 | | |
Age children | | | | | | | | |
Children 0-3 y | 401 (77,0) | 521 | c2 = (1) 264.065 | <0.001 | | | | |
Children 4-18 y | 67 (20,2) | 332 | | | | |
Missing | 22 | 58 | | | | | | |
Appropriateness
In 2019, a prototype was launched that was considered 'good enough': the system supported the most important working processes of CJG professionals. In the focus groups, youth care workers, as opposed to preventive child health professionals, reported that EPR-Youth not fully matched their working processes. As a result, youth care workers sometimes felt as if they were 'visiting' someone else's system: 'As a youth care professional, it feels as if I am allowed to report in a preventive child health system [111:34]’. Administrative assistants on the other hand, felt that EPR-Youth supported their working processes better than before. However, one assistant reported that she missed a crucial functionality to plan ahead. ‘During the pandemic, I had to deliver numbers concerning delay in our regular work. Honestly, I don't know how...' [111:43] A few professionals used to report information about parents that was not to be read by the child, for instance in case of a divorce: ‘I don't think a 12-year-old should have to read that.'[112:242] In EPR-Youth, however, adolescents aged 12 and older have full access to their own EPR. Information in an adolescents’ EPR can be shielded from parents, but not from the adolescents themselves. For professionals who frequently must deal with difficult divorce situations, this felt as an obstacle in their working processes.
Fidelity
To test fidelity, we discussed with professionals in the focus groups whether the system was being used and implemented as intended. Professionals reported that EPR-Youth facilitated interdisciplinary collaboration: 'If you know that other colleagues are involved as well, you can read what they have written'[112:21] and: '...when a child starts at school, for example, that you don't have to transfer the record to the other preventive healthcare organization anymore.' [111:2] Moreover, professionals noticed that the ability to read and to plan appointments enhanced parent’s involvement. ‘Yes, managing appointments is nice, for parents as well. Parents can also have a look in the portal and become more involved’. [112:16]. One member of the steering committee considered transparency for clients the most important achievement: 'We are breaking .....(an unwritten law), that it is no longer possible, at least, to report things...that clients are unaware of.' [108:93].
On the other hand, some functionalities were not utilized by all CJG-organizations: the possibility to manage appointments, for example, was offered by only one CJG-organization. Furthermore, professionals appeared to lack knowledge of some functionalities and how to use them as intended: ‘I have a lot of colleagues who still write their indication statements in MS Word.’ [112:171] and 'Actually, everything you just said is new to me’. [112:32] Ultimately, some professionals struggled to make EPR-Youth work for them, concordant with the regional vision, and requested additional training. ‘Of course, this is about doing your job in line with our vision, and about the moments of tension that arise from that.'[112:507]