The following sections focus on Study 2 involving school children, including data collection methods, results, discussion and summary. A final summary is then provided highlighting the key findings across the studies.
Method
The biosecurity education kit
An educational biosecurity kit called ‘Invasion Busters’ was developed by the House of Science, a charitable trust in New Zealand which provides ‘hands-on’ science resource kits. The Invasion Busters kit was designed by the Resource Developer (previously a medical microbiologist), and informed by a population ecologist. The kit was targeted at children aged five to 12 years old (years 1 to 8) and included seven activities in total, varying in difficultly to cater to the broad age range. Six activities were related to different components of the biosecurity system, such as sorting and identifying seeds at the border, identifying pest threats (the brown marmorated stink bug, set in clear resin), and modelling insect population growth. The final activity in the kit is a board game where children collaborate to keep incoming pest threats under control, given the different ‘roles’ they are assigned, such as biosecurity officers, stevedores (who unload goods on port), insect trappers, and incursion investigators. Pests included in the game are real current pest threats and are accompanied by information such as the potential threat they pose, country of origin, and damage they can do to host plants. An example of the ‘role cards’ and ‘pest cards’ from the game are displayed in Figure 2.
The Invasion Busters biosecurity education kit was piloted in schools around the Tauranga area in March and April of 2018. The data for this evaluation were collected from the classrooms who piloted the kit, because these teachers attended a ‘launch’ of the kit, where they were informed of the evaluation. The teachers therefore had an understanding and appreciation of the purpose of the evaluation and would be more likely to administer the questionnaires using the requested method.
The biosecurity kit survey questionnaire
Questionnaires to evaluate the education kit were administered three times; once immediately before the kit was introduced (e.g. on a Monday morning), once immediately after use of the kit (e.g. on a Friday afternoon, after using the kit several days that week) and once six months later, to test retention of knowledge. These measures will be referred to as the pre, post, and follow-up measures hereafter. The questionnaires were sent out with the kit for the pre and post measures, and teachers were re-contacted and sent the questionnaires again for the follow-up measure. Instructions for administration were included with the survey, including a request for teachers to assist with survey comprehension and completion, without explaining the answers to the questions (including avoiding explaining what biosecurity meant, prior to using the kit). The follow-up measure packet included reminder instructions for administration, a recap purpose statement and return envelopes, in addition to a list of names of the children in the class who needed to complete the questionnaire (based on children who had informed consent from their parents and had completed both the pre and post measures).
The questionnaires included two types of questions, the first being a Likert-scale question using a 5-point smiley-face Likert scale combined with word based ‘degrees of agreement’ as anchors (see Figure 3). This scale was used because evidence shows that children have greater engagement with smiley face Likert scales than other evaluation instruments (e.g. scales solely using words as anchors), and 5-point smiley face Likerts are recommended for children (Hall & Hume 2016).
There were nine questions rated on the smiley face Likert scale. These statements were designed to be concrete rather than abstract (avoiding ‘feeling’ statements), as research suggests that both younger and older children (age range 6-12) understand graded scales when making judgements about more concrete concepts (Mellor & Moore 2014). The teacher also assisted children’s understanding by reading the questions aloud. These included “I know what biosecurity is” and “My family talk about stopping bad insects from hurting animals, plants, or people in New Zealand”.
There were a further eight short answer questions, designed to move beyond self-perceptions and test existing knowledge and retention of knowledge. These related to specific activities within the kit, for example “Can you name three bad insects we don’t want to come into New Zealand?”, and “Can you think of something which might make bad insects grow faster or have lots of babies?”. Children should have learnt the answers to these questions while using the kit, for example three current insect pest threats through the ‘Invasion Busters’ board game, and factors which affect insect population growth in the population modelling activity. Teachers were asked to identify which activities they completed (and did not complete) on a form delivered with the questionnaires, so this could be considered in the data analysis.
Data cleaning and analysis
Data from a total of 48 children were removed from the study, due to issues with data continuity and quality. This included 37 children who had completed either a pre or a post-kit survey but not both, and 11 students (one classroom) where the pre-kit survey was administered after use of the kit, rather than before. This was evident in children’s answers, which included direct quotes from the kit, such as ‘Catch it, snap it, report it’, a catch phrase from a local biosecurity initiative (Kiwifruit Vine Health). Removal of these students left a total sample size of 120 children.
For the Likert-scale questions, the scores were analysed using repeated measures fitted as a linear mixed model in Genstat 19. To account for correlation between measurements taken for the same child (pre, post and follow-up measure), an unstructured correlation model was used. Random effects (constrained to be positive) were included to account for school, teacher, age at pre-test (as a factor) and gender variation. The fixed term assessed was ‘survey session’, a factor with three levels (pre, post and follow-up measure). In addition, Fisher’s unprotected least significant differences at the 5% level were used to compare the predicted means. Residual plots were assessed to check that the assumptions of normality and constant variance broadly held and data for each question were analysed independently.
For the short answer questions, scoring was more difficult. Due to the diversity of the answers provided (some being very creative), the research team assembled to decide what constituted a ‘correct’ answer for each question. This was a difficult task, as often children identified ‘bad’ insects (such as wasps or tarantulas) as pests, however a majority are not considered ‘biosecurity pests’ because they may already be in New Zealand, or are not on an ‘unwanted’ pest list for New Zealand. Children also provided varying levels of detail in their responses, which demonstrated varying degrees of knowledge. For example, children were asked to identify the difference between a brown marmorated stink bug (key current pest threat for New Zealand), and a regular stink bug. This was the basis of one of the activities in the kit. Many children wrote ‘colour’ or ‘size’, which, while correct, does not provide sufficient evidence that they would be capable of successfully differentiating a brown marmorated stink bug from a regular stink bug in real life. Other children wrote “the brown marmorated stink bug has three white spots”, which is a much more specific, and correct, answer. This variance and subjectivity of responses was managed by creating an ‘inventory’ of all answers provided, for each question. Each inventory was then marked by two independent raters, as to whether answers were correct (one point given), partially correct (half point given, indicating understanding of the underlying premise), or incorrect (zero points given). Discrepancies in ratings were discussed and resolved, to ensure ratings were consistent. All responses were then given a numeric score, which allowed use of the same method of statistical analyses as for the Likert scale questions above.
Results
Demographics
One hundred and twenty children participated in the research, spread across three schools in the Tauranga area[1], with a total of six teachers across the children. The schools were deciles[2] 4 (32% of children), 6 (23% of children) and 9 (45% of children). Children ranged from six to ten years old, with the largest number aged 7-8 years old, as displayed in Figure 4. There were equal numbers of boys and girls (60 each).
Results for the smiley face Likert scale questions are reported in graph format (see Figure 5). For five of the eight questions children showed a statistically significant improvement over time, for two questions there was no difference, and for one question, children showed a decline in correct answers at the six-month follow up.
After using the kit, children demonstrated an increase in their self-rated understanding of biosecurity (t87=12.7; p<.001), and this was retained six months later (t87=11.1; p<.001). Children were also more likely to rate that they understood ‘that some insects can hurt other animals, plants or people’, however this was only significant between the pre-test and the follow-up (t87=11.1; p<.001). This finding triangulates with the first question as a proxy measure for biosecurity understanding, reinforcing that understanding did improve. Children appeared less confident in their understanding when the term ‘biosecurity’ was used, as opposed to the general statement that insects may cause harm, which is a simpler explanation of the premise of biosecurity.
Children demonstrated an increased and sustained recognition of the importance of border surveillance for biosecurity risks, acknowledging ‘it is important to stop new insects coming into New Zealand’ (pre vs post t80=5.9; p<.001) (post vs follow up t80=6.1; p<.001). On the other hand, children’s ratings for biosecurity behaviours did not significantly improve, with no change in likelihood of reporting potential biosecurity incursions or talking with family about biosecurity. Surprisingly, the third biosecurity behaviour, ‘looking for insects around home or school’ actually saw a significant decrease six months after completing the kit, relative to before using the kit, where a majority of children reported they did not look for insects at school or at home (t87=-3.3; p=.001).
The final measurements on the Likert scale relate to knowledge of insect pests (‘which insects might hurt animals, plants or people’), and knowledge of appropriate reporting protocols (‘I know what to do if I see a bad insect’). Immediately after completing the kit, there was weak evidence that self-rated knowledge of pest insects had increased (t90=1.7; p=.097), although this was not maintained at the follow-up. However, when asked to name current pest threats directly (rather than self-rating their confidence about naming pests), children demonstrated a significant improvement, both immediately after using the kit (t87=3.5; p<.001), and at the follow-up (t87=3.7; p<.001). This result is substantiated in that children were more than twice as likely to name the stink bug as a pest (featured in the kit), after using the kit. Children did demonstrate increased confidence in their knowledge of reporting protocols (t92=3.1; p=.002), however this was not maintained at follow-up. Finally, children were better at identifying incursion pathways, both after using the kit (t93=6.7; p<.001) and at the follow-up (t93=5.7; p<.001). These results are displayed in Figures 5 and 6.
Further questions where children demonstrated improved knowledge included differentiating the brown marmorated stink bug from regular stink bugs (pre vs post t91=5.6; p<.001) (post vs follow-up t91=5.1; p<.001), naming locations one might find insects (pre vs post t167=3.1; p=.002) (post vs follow-up t167=2.1; p=.04), and identifying potential biosecurity strategies for New Zealand (pre vs post t93=3.8, p<.001) (post vs follow-up t93=4.0; p<.001) as shown in Figure 7. Some of these questions were clearly more complex, with considerably lower baseline averages (differentiating stink bug and identifying biosecurity strategies), however demonstrated marked improvements over time. Children’s answers also demonstrated highly creative thinking, where some individuals (perhaps unknowingly) identified or alluded to sophisticated biosecurity control mechanisms currently used by New Zealand. This included “getting their DNA”, “spray smoke in their eyes so they can’t see” (fumigation), and “releasing a new species of insect and make it eat it” (biological control). Other humorous answers are displayed in Table 2. Overall, the children demonstrated a good awareness of current border biosecurity measures, such as checking bags, disposing of food before entering the country, and pest trapping.
Despite using the Likert scale to indicate they did not feel an increased confidence in knowing what to do if they saw a pest threat, children were significantly better at identifying correct biosecurity behaviours after using the kit. This included reporting (telling someone, such as an adult), trapping, or killing the pest (the least encouraged response, but marked as correct).
There were two final questions where children showed no improvement over time. These were ‘Can you name a trap we might use to catch a bad insect?’, and ‘Can you think of something which might make bad insects grow faster or have lots of babies?’. These questions were related to specific activities within the kit, where children got to build and use insect traps, and do population modelling, where changes in different variables in the environment affected population growth. They were therefore reliant on the extent to which they had completed these activities, which will have varied between classrooms, and may explain the absence of change. Many children did provide accurate answers to this question which indicated an understanding of the premise, particularly around providing a food source for the insects, inaction (“not killing”) and mating with other insects.
Table 2 Examples of children’s responses to various survey questions
Q.
|
Can you name a trap we could use to catch a bad insect?
|
A.
|
- When it is sleeping tie a string around it
- Sleep smoke
- A rat trap with fly spray in it
- A micronet
|
Q.
|
Can you name a way that bad insects could get into NZ?
|
A.
|
- Ride on fish
- They could walk in a group
- Camouflage
|
Q.
|
What should you do if you see a bad insect?
|
A.
|
- Call for help and tell the pound
- Tell your mum
- Step on it and say goodnight
|
Q.
|
Can you think of something which might make bad insects grow faster or have lots of babies?
|
A.
|
- Finding a wife or husband
- Hope
- Eating, killing and stealing
- Eat, eat, eat and eat
|
Q.
|
Can you think of something we could do to stop bad insects from coming into NZ?
|
A.
|
- Build a wall like Donald Trump did
- Releasing a new species of insect and make it eat it
- Make a force field around NZ
- Putting smoke in their eyes so they can’t see
- Make robots that search for them in NZ
- Make the government stop them
|
Discussion
The biosecurity kit was successful at improving children’s understanding of biosecurity, perceived importance of biosecurity, and their critical knowledge about biosecurity, including current pest threats, incursion pathways, and biosecurity control strategies. The kit was somewhat less successful at improving biosecurity behaviours, with no change in reporting behaviours, or discussion with family about biosecurity. Further, there was a decrease in the likelihood that children would search for insects at home or school, however one possible explanation for this finding may be seasonal differences – the six-month post measure was undertaken during early spring and therefore children may have spent less time outdoors during this period due to rain or colder temperatures. Overall, these results suggest the kit is an excellent learning resource, but is less effective as a behaviour change tool, as least without additional interventions, such as encouragement from parents to keep looking for and discussing insects or pests at home. With this additional support, it may be that the kit would have greater spill-over benefits, in generating interest and learnings for parents, siblings and others living in the home.
The findings also provided some key learnings about measurement of biosecurity learnings among children. The term biosecurity may be a barrier for some children, as children were more likely to rate that they understood the premise of biosecurity, rather than the term itself. This may emphasise a need to reduce jargon when communicating about biosecurity with children, and in particular when measuring children’s understanding of the concept, to gain an accurate measure. Similarly, there was some contradiction apparent in children’s self-rated knowledge about biosecurity, versus their actual performance when providing short answers. That is, while children rated that they did not have significantly greater knowledge after completing the kit, their answers indicated that they did in fact increase and retain biosecurity knowledge. This suggests that a short answer format may be more accurate than self-ratings, and that children underestimated the amount of learning they achieved through the kit.
[1] Schools will remain unnamed, for discretion and confidentiality purposes.
[2] School deciles are a measure of the socio-economic positions of a school’s student community, where decile 1 schools are the 10% with the highest proportion of students from low socio-economic communities.