In total, 34 (68%) male and 16 (32%) female clinical supervisors participated in the study. The work experience as an RN ranged from four months to 19 years, with a mean of 7.95 years. The work experience from the ambulance services ranged from 2.5 months to 35 years with a mean of 6.63 years (missing data n=5). All participants had experience of clinical supervision and assessing nursing students during clinical education. The logistic binary regression analysis showed that there was no significant difference in 23 out of 24 LOs as displayed in table 1-4. One significant difference was found in the ACIEd LO 2 (p= 0.021) when the length of experience among the clinical supervisors was used as a dependent variable (table 4). LO 2 assessed the students’ knowledge of the technical equipment and how they interacted with the patient while using the equipment.
Table 1 Effect of the dependent variable ‘Genders’ on grading related to learning objectives using a manikin for both the AAI and ACIEd with 1 degree of freedom.
Independent variable LOs
|
Beta coefficient
|
Standard error
|
Wald chi-square test
|
P-Value
|
Exponentiation of the B coefficient
|
AAI* LO1
|
.451
|
.866
|
.271
|
.602
|
1.571
|
AAI LO 2
|
-.932
|
.813
|
1.314
|
.252
|
.394
|
AAI LO 3
|
.277
|
1.024
|
.073
|
.787
|
1.319
|
ACIEd** LO 1
|
-.478
|
.960
|
.247
|
.619
|
.620
|
ACIEd LO 2
|
.969
|
.965
|
1.009
|
.315
|
2.636
|
ACIEd LO 3
|
-.003
|
.965
|
.000
|
.998
|
.997
|
*AAI – Ambulance Assessment Instrument
**ACIEd – Assessment of Clinical Education
Table 2 Effect of the dependent variable ‘Experience’ on grading related to learning objectives using a manikin for both the AAI and ACIEd with 1 degree of freedom.
Independent variable LOs
|
Beta coefficient
|
Standard error
|
Wald chi- square test
|
P-Value
|
Exponentiation of the B coefficient
|
AAI* LO 1
|
.600
|
1.117
|
.288
|
.591
|
1.822
|
AAI LO 2
|
2.418
|
1.279
|
3.574
|
.059
|
11.219
|
AAI LO 3
|
.167
|
1.306
|
.016
|
.898
|
1.182
|
ACIEd** LO 1
|
1.346
|
1.186
|
1.288
|
.256
|
3.841
|
ACIEd LO 2
|
-2.193
|
1.546
|
2.012
|
.156
|
.112
|
ACIEd LO 3
|
-1.955
|
1.220
|
2.568
|
.109
|
.142
|
*AAI – Ambulance Assessment Instrument
**ACIEd – Assessment of Clinical Education
Table 3 Effect of the dependent variable ‘Gender’ on grading related to learning objectives using the Standardized patient for both the AAI and ACIEd with 1 degree of freedom.
Independent variable LOs
|
Beta coefficient
|
Standard error
|
Wald chi-square test
|
P-Value
|
Exponentiation of the B coefficient
|
AAI* LO 1
|
.499
|
1.021
|
.239
|
.625
|
1.647
|
AAI LO 2
|
.561
|
.597
|
.883
|
.347
|
1.752
|
AAI LO 3
|
.540
|
.825
|
.428
|
.513
|
1.716
|
ACIEd** LO 1
|
-.992
|
.987
|
1.010
|
.315
|
.371
|
ACIEd LO 2
|
.328
|
.725
|
.205
|
.651
|
1.388
|
ACIEd LO 3
|
-.329
|
1.092
|
.091
|
.763
|
.720
|
*AAI – Ambulance Assessment Instrument
**ACIEd – Assessment of Clinical Education
Table 4 Effect of the dependent variable ‘Experience’ on grading related to Learning objectives using the Standardized patient for both the AAI and ACIEd with 1 degree of freedom.
Independent variable LOs
|
Beta coefficient
|
Standard error
|
Wald chi-square test
|
P-Value
|
Exponentiation of the B coefficient
|
AAI* LO 1
|
-.405
|
.970
|
.174
|
.676
|
.667
|
AAI LO 2
|
.662
|
.562
|
1.388
|
.239
|
1.939
|
AAI LO 3
|
-.527
|
.788
|
.447
|
.504
|
.590
|
ACIEd** LO 1
|
-.204
|
.972
|
.044
|
.834
|
.815
|
ACIEd LO 2
|
1.918
|
.834
|
5.289
|
.021*
|
6.805
|
ACIEd LO 3
|
-1.260
|
1.160
|
1.181
|
.277
|
.284
|
*AAI – Ambulance Assessment Instrument
**ACIEd – Assessment of Clinical Education
4.1 Qualitative Results
The thematic analysis used to explore clinical supervisors’ experience of assessment in relation to the two different assessment instruments resulted in three themes: Learning objectives, Assessment and Supervisorship.
4.1.1 Learning objectives
Supervisors described a variety of experiences relating to the LO`s in the two sub-themes: Phrasing of the LOs, subjectivity in the LOs, Complexity of the LO`s. The subthemes describe the supervisors’ view of the learning objectives in relation to how the LO`s are constructed and how they are interpreted as well as how they are used in the daily activities.
Phrasing of the learning objectives:
The supervisors discussed the language used in the ACIEd to describe the LO`s and that the academic writing created confusion and was complex and hard to understand. They also discussed the lack of clear definitions, and difficulties using the ACIEd with several sub-criteria embedded in one LO, which resulted in individual interpretation of the LO and a risk of subjective assessment. According to the supervisors’ reasoning, several sub-criteria in the same LO complicated the assessment when students performed well according to some of the sub-criteria and poorly according to others. The supervisors expressed that the LOs used in AAI had a clearer phrasing making it easier to use. LO`s related to communication and patient relations were considered complicated to assess using the ACIEd in contrast to LO`s regarding medical procedures where right and wrong was clearly defined. Medical procedures were considered easier to assess in both instruments.
Complexity of the learning objective
When assessing the students’ nursing and care skills in the simulations, the supervisors claimed that the complexity of ACIEd made it challenging to explaining why they assessed as they did. In the clinic setting the supervisors said that the complexity of the LO`s made the usage of ACIEd time-consuming and poorly adapted to everyday work, which resulted in limited use, and only used in a summative way right before mid and end assessments. Altogether, the ACIEd was considered by the supervisors to be time-consuming, and leading to subjective interpretations and sometimes, conflicting assessments. The AAI was considered more relevant due to the “simplified” LO`s. However, AAI was thought to be difficult to use due to the Likert scale ranging from 1-7 were lack of clear definitions of the grading steps increased the complexity of assessing the LO`s.
4.1.2 Assessment
Supervisors described assessment from different perspectives as presented below in the following sub-themes: Summative Vs Formative assessment, Subjectivity in assessment, Pass/Fail or Likert scale, and Supervisors experience related to assessments. Within the sub-themes there were variations in how the supervisors described their experiences.
Summative Vs Formative assessment
The supervisors discussed if and how the assessment instruments could be used for formative or summative assessment and concluded that both instruments could be used for both purposes. However, the supervisors discussed whether ACIEd was harder to use as a formative instrument due to the sub-criteria in the LO`s and that the pass/fail scale were less useful then the 7 graded Likert scale when it came to displaying progress. The supervisors discussed whether displaying the student’s progression had a positive pedagogical value and they concluded as preferable relating it to formative feedback. Supervisors also discussed the frequency of formative assessments. Some supervisors raised concerns related to the increased workload generated by daily documented, formative feedback.
Subjectivity in assessment
To decrease the subjectivity, the supervisors discussed involving the patients in the assessments of students’ performance. They argued that the only persons who could assess the LOs concerning the patients’ own perception were the patients themselves. The supervisors concluded that more care encounters, with assessments between every encounter would probably produce a more accurate assessment of the student’s nursing skills. The supervisors also discussed the risk that the students would only be assessed according to the supervisor’s interpretation of what is a pass performance in relation to the assessment instrument.
The supervisors found that the assessments with a Likert scale could be beneficial but that the grading steps needed to be carefully defined to avoid subjective assessments. The definitions could be made clearer by using examples in relations to the grading steps and the LO`s where requirements could be listed for each step.
Supervisors said that assessments using a Pass/Fail grading was too definite. Supervisors described that assessing the care encounters became complicated since student’s performances may contain good and bad performances and with Pass/Fail grading the nuances did not become clear.
Supervisors experience related to assessments.
Supervisors’ description of their relationship to the LO`s varied where some had more issues than others. Supervisors with less experience were more critical towards the LO`s then experienced supervisors. Furthermore, supervisors with less experience discussed that the student were obligated to display skills and knowledge and that supervisors were obligated to assess in accordance with the LO`s, meaning responsibility for demonstrating knowledge and skills fell on the student.
The more experienced supervisors argued that the LO`s were more like guidelines than specific goals to achieve, and that they used LO`s as a basis for discussions with the students. They argued that their clinical experience and understanding were the basis for the assessment, making the challenge with the LO`s less important, which contrasted with the view of the less experienced supervisors who interpreted the LO´s more literally. Furthermore, the more experienced supervisors discussed that complex care encounters offered scarce opportunities for the students to display knowledge and skills and that several care encounters were needed to assess knowledge and skills over time.
4.1.3 Supervisorship
The supervisorship was described by the supervisors in the following two sub-themes: Relationship with the students and Teams of supervisors.
Relationship with the students
All supervisors said that a relationship with the student could be a confounder when assessing students. They discussed whether a good relationship with the student would probably result in a more favourable assessment. The supervisors also claimed that a troubled relationship with the student might result in disinterest from their perspective which could result in diminishing feedback and lowered clinical education quality resulting in higher risk for failure. The supervisors discussed that formative feedback could help detect a lack of progression at an early stage and that measures could be taken to improve the situation for the student. Among the female supervisors a recurring statement was that it was difficult to assess the student negatively in the simulations because it felt harsh or even cruel to fail a student. Among the male supervisor’s similar feelings were described, but they related to the fact that assessments are not carried out on single occasions but over time, making the assessment instrument unfit for this kind of assessment. Male supervisors argued to a greater extent that it was hard to assess the student in the simulation due to lack of information about both patients and students. They argued that although some simulations were not as described in “textbook” examples, guidelines were bent daily and therefore it would be unfair to expect “textbook” care from the students. Hence, the male supervisors argued that the assessment instrument was a tool used for discussion and could not stand alone which it did in the simulations.
Teams of supervisors
The supervisors highlighted continuity as something important for the student’s learning, but it could also complicate their assessment since they may develop a relationship with the student. The desire for continuity in supervisorship could also create a problem when working schedules changed. This could result in the involvement of other supervisors, and disruption in the individual learning plan. The supervisors with longer experienced argued that it could be favourable with more than one supervisor involved in the assessment of the student, due to different perspectives. They argued that a supervisor team with different combinations of knowledge could be beneficial for the student, but that such teams must be coordinated and documented to ensure that the student’s learning progress was not hindered.