“We Are Not Stray Leaves Blowing About In The Wind”: Summing Up The Impact of Family Wellbeing Empowerment Research, 1998-2021

doi:10.21203/rs.3.rs-779109/v1

Download PDF

Research

“We Are Not Stray Leaves Blowing About In The Wind”: Summing Up The Impact of Family Wellbeing Empowerment Research, 1998-2021

https://doi.org/10.21203/rs.3.rs-779109/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: An Aboriginal-developed empowerment and social and emotional wellbeing program, known as Family Wellbeing (FWB), has been found to strengthen the protective factors that help Indigenous Australians to deal with the devastating legacy of colonisation and intergenerational trauma. This article reviews the research that has accompanied the implementation of FWB over a 23 year period to assess the long-term impact of FWB research and the strengths and limitations of the impact data. This will inform more comprehensive monitoring of research impact.

Methods: The study took a theory of change approach, following the steps of a research program logic, to assess impact. A mixed methods evaluation of publicly available research outputs was conducted. We analysed FWB participation data and research funding using descriptive statistics. We conducted systematic scoping reviews of: 1) nine evaluative studies to show reported social and emotional wellbeing benefits of FWB; and 2) seven FWB research translation-related resources to show impact of research outputs on further reach and uptake.

Results: The investment of $2.3 million in research-related activities over 23 years resulted in a range of research outputs that evidenced social and emotional wellbeing benefits arising from participation in FWB. This evidence, in part, facilitated more demand for FWB. Overall the FWB program, and accompanying research, had a direct positive impact on the social and emotional wellbeing of approximately 7,500 participants.

Conclusions: The study raised a range of issues relevant to demonstrating research impact in the context of Indigenous health research including the value of a logic model that is informed by Indigenous research principles; the need to take a longer-term incremental approach to building evidence when dealing with complex or messy social health interventions; and the importance of collecting evidence, especially from the point of view of end users, about the extent to which research influenced their decisions and actions regarding the program. This lesson, and the impact logic, highlight that researchers need to create appropriate databases from the beginning and collect data well beyond the life of the project, even though we acknowledge that such endeavours are rarely funded.

Health Policy

Aboriginal and Torres Strait Islander health

Indigenous health

empowerment

research impact

social and emotional wellbeing

research evaluation

logic model

theory of change

Improving empowerment and social and emotional wellbeing (SEWB) among Aboriginal and Torres Strait Islander people is critical for addressing health and wellbeing inequity. In Australia, there exists a trans-generational legacy of complex trauma associated with colonisation, dispossession, inflexible systems and government policies, such as the removal of children from their families (1). This has resulted in many young Indigenous people living in situations of severe social adversity (2, 3). A broad range of indicators evidence legacies of trauma, with young Indigenous people reported to be: six times as likely to be teenage mothers; seven times as likely to receive child protection services; 26 times as likely to be in the juvenile justice system; 2.7 times as likely to be unemployed; and three times as likely to live in overcrowded housing compared to young non-Indigenous people (4–8).

The legacy is also seen in unemployment and economic disadvantage with 60% of Indigenous Australians living below the poverty line (Dudgeon & Hirvonen, 2019; Dudgeon & Holland, 2017; Korff, 2019). Furthermore, suicide impacts 95% of Indigenous Australians. The suicide rate for males is 40% higher than for non-Indigenous males (Korff, 2019). While Indigenous children make up less than 5% of the population, they constitute 25% of all youth suicide in Australia (Higgins, 2019). To overcome the effects of these problems, individuals and communities need to build protective factors such as connection to family, country, language and culture. Empowerment enables people to have more control over their lives and the social and economic environment in which they live (1); and has been found to improve education and employment outcomes for Aboriginal people in remote Australia (9).

The persistence of alarming inequities between Indigenous and non-Indigenous Australians places urgency on health research to help build the evidence base for protective factors; to be beneficial to the researched; to be accountable; and to address needs identified by Indigenous people (10). As a prominent leader in Indigenous health research, Dr Pat Anderson, powerfully expressed, “we need to have a sense of agency in our lives, that we are not stray leaves blowing about in the wind … we need empowerment” (11). Not only is there a need for interventions that support this goal, but their impact on SEWB must be monitored and the research findings must be transferred to the relevant parties so that these parties can make informed decisions. This has great potential to improve health equity.

However, reviews of SEWB (which included empowerment) interventions for Indigenous people have found that few programs are documented in the literature and few have been evaluated (12). Of those evaluated, the dearth of rigorous research designs and validated quantitative measures has attracted a certain stigma because of a perceived lack of specificity or quantification. In addition, a systematic review of Indigenous health research found little reporting of the impact of health research for Aboriginal and Torres Strait Islander people in Australia (13), and this is even less so when it comes to reporting impact on difficult-to-quantify constructs such as empowerment and wellbeing (14). What is important to know is how intervention research contributes to further uptake of interventions and the subsequent impact of such uptake.

Research impact is defined as “the demonstrable contribution that research makes to the economy, culture, national security, public policy or services, health, the environment, or quality of life, beyond contribution to academia” (15). The expectation of researchers to demonstrate this impact, in addition to the usual quality measures such as journal impact and citation frequency, is growing (14). This is evidenced by the UK government’s introduction of a Research Impact Assessment framework in 2014 and the Australian Research Council’s (ARC) three-yearly Engagement and Impact Assessment in 2018 (16, 17). While this focus is relatively new in academic and government settings, Indigenous Australian community leaders and organisations have been calling for scholars to be held accountable for their impact and to identify how their work enhances health equity for Indigenous people for some time (14, 18–21). In this article we explore the impact of a research program, based on an SEWB and empowerment intervention called Family Wellbeing (FWB), which has spanned 23 years and has involved the development of a national network of researchers and communities.

The task of reporting health research impact is complex with several methodological challenges. Firstly, there can be a significant time lag between conducting research and then using the results to translate into change for participants. It is also difficult to draw a clear cause and effect relationship due to the many factors that combine to produce change. Furthermore, researchers must mine retrospective data, that were collected for various purposes, to build an argument that impact occurred. Tracking societal impact also involves a time and resource cost that exceeds the relative immediacy of providing bibliometric measures such as citations (18, 21–24). However if researchers do not attempt to assess real impact, even through limited retrospective data, then there is risk that the impact agenda becomes no more than box-ticking rather than a genuinely reflexive exercise (14).

When the ARC piloted their research impact assessment framework in 2017, it was a catalyst for the FWB research network to systematically explore their research impact. We examined the ARC’s requirements to report impact according to one’s current institutional affiliation and within a 15-year time frame; and the constraints this put on assessing the 23-year FWB research program in which the key researchers changed institutional affiliations over time (14). We also explored the feasibility of using the framework to describe FWB research impact in one specific community and the challenges and opportunities involved (Whiteside et al 2021).

In this article, therefore, we build on the two aforementioned studies to take a long-term view of FWB research impact in order to identify potential impact domains for future research, and to enable planning and collection of appropriate data that can support such impact claims. We focus on how the research itself, and the dissemination and translation of the research findings, brings benefits or achieves impacts, mainly but not limited to, Aboriginal and Torres Strait Islander SEWB and empowerment.

The aims of this article are to: 1) use retrospective data to explore the impact of a 23-year long research program; 2) identify the strengths and limitations in the impact data and provide a way forward for improving the research impact evidence-base.

In the next two sections, we give a brief description of the FWB program and summarise associated research to date. Since the focus of the article is on exploring the impact of research associated with the FWB program, we distinguish between the impact of the empowerment intervention called FWB and the research that accompanied the journey of the intervention. Therefore, we refer to the former as “FWB program outcomes” and the latter as “FWB research impact”.

Family Wellbeing program description

FWB is an empowerment program developed by Aboriginal Australians (via the Aboriginal Education Development Branch of the South Australian Department of Education) in 1993. Its aim is to empower Indigenous people to deal with the after-effects of colonisation, and other problems associated with rapid social change, by helping individuals, families and communities to take greater control and responsibility over their lives. FWB is an accredited program through the Australian vocational educational training system, comprising 150 hours of facilitated small group learning (full version). It can also be modularised to comprise just one 30-hour introductory component (foundation version).

FWB workshops exemplify a trauma-informed approach to healing. Topics covered include human qualities; basic human needs (physical, emotional, mental and spiritual); human relationships; life journey; beliefs and attitudes; violence and abuse; addictions; crisis and emotions; loss and grief; conflict resolution; and caring for ourselves and others (11, 25, 26). The goal is to strengthen the protective factors needed to endure through the legacy of colonisation and health inequity.

Although FWB was developed primarily in response to the special needs of Indigenous Australians, it can be adaptable to the needs of other cultures and social groups in Australia and beyond (27–30).

Family Wellbeing research to date

This section summarises the journey of FWB research over three overlapping phases (Fig. 1).

Phase 1

Phase 1 encompassed the pilot program evaluation which took place between 1998 and 2001. In this phase, FWB addressed needs that were identified as a priority by the Indigenous participants in the program, namely empowerment and SEWB as protective factors against widespread trauma and disadvantage. The program’s explicit foci on suicide prevention, family violence, parenting, and employment readiness had long been identified as critical to improving Indigenous health, but there had been little reporting about practical interventions to address these needs (26).

The pilot evaluation adopted a narrative approach based on participants’ advice to “ask us to tell stories or ask us to write our stories of change in diaries” (26). This approach facilitated a strengths-based evaluation that prioritised Aboriginal people’s views and experiences. The Cooperative Research Centre for Aboriginal and Tropical Health (CRCATH) (1997–2002) distributed the evaluation report to Indigenous organisations widely, resulting in more interest and ultimately demand for the FWB program.

Phase 2

Phase 2 covered a 10-year program of research known as the Empowerment Research Program (ERP) (11, 31, 32). The ERP had a broad research agenda in which systematically exploring the potential contribution of FWB to empowerment and improving Indigenous SEWB was a key part. The research group drew upon a combination of Indigenous research principles (Indigenous leadership and participation; addressing priority needs; being beneficial; transferring benefits to others; enhancing capacity (10, 25)); and a “phased-approach”, including both short- and long-term objectives, to evaluating complex interventions (31, 33). Evaluation of the FWB program confirmed its applicability in a variety of contexts such as community wellbeing promotion, school health promotion, and workforce development; and that the consistent outcome of participation in FWB was individuals’ enhanced capacity to exert greater control over factors shaping their SEWB (34).

A close relationship with the Lowitja Institute, Australia’s national institute for Aboriginal and Torres Strait Islander health research, and the associated co-operative research centres that predated it (i.e. the CRCATH & the Cooperative Research Centre for Aboriginal Health), assisted the impact of FWB research. Indigenous researchers associated with the Institute provided expertise that was integrated into the research program (11). The Lowitja Institute supported FWB research translation activities including funding FWB research policy briefs, FWB knowledge sharing forums, discussion papers, and plain language community reports (35–37). By the end of Phase 2 in 2011, FWB had been introduced, with varying degrees of sustainability, in 57 sites around Australia (38).

Phase 3

In Phase 3, the research findings from Phases 1 and 2 enabled a focus on knowledge translation and impact assessment which included supporting further uptake of the program. We collaborated with Indigenous service providers to start embedding FWB within their core services and programs such as for Indigenous child protection and family support in north Queensland (1) and social and emotional wellbeing in rural Victoria (39).

The overall aims of this study were to assess the long-term impact of FWB research and the strengths and weaknesses of tracking such impact in order to provide a way forward for improving the research impact evidence-base. This involved the following methodological approach and data collection.

Approach

As a framework for assessing research impact, we adopted a theory of change approach. Thus, evaluating the research involved a systematic and cumulative study of its aims, context, activities and potential to create needed results (40). This monitoring approach meant trying to understand what had changed for people and the relative contribution of FWB research to that change (41). Importantly, we placed value on supporting Indigenous people to define their intended outcomes (42), and then we monitored progress towards achieving those outcomes over time.

We mapped out the theory of change in a logic model to provide, in diagrammatic form, the rationale that links research aims to outputs and outputs to impact. Importantly, the model links the community need for an intervention to the research services that are undertaken in response to that need (21, 24). The logic model is predictive and linear, positing that if the first block is in place, then subsequent blocks are likely to follow on from that and reach the desired outcome. Such a model is limited in its ability to reflect complex social conditions and research pathways (24), however it plays an important role in approximating the potential path for research translation and impact (21).

Data and Analysis

For data analysis, as previously noted, we first mapped out the FWB research impact theory of change, in the form of a logic model. This showed the pathways through which participation in FWB training workshops resulted in improved SEWB for participants, and the extent to which knowledge of research confirming this finding influenced further program uptake and spread among user organisations and communities. Based on the impact logic, we developed an overarching FWB research impact narrative showing the scope and extent of the SEWB impacts, the beneficiaries, and the relative contribution of the James Cook University (JCU)-led 23-year program of FWB research to the impact claim. In order to develop this narrative, we concentrated on four main areas for data collection: 1) participation; 2) funding; 3) program outcome evaluation; and 4) accounts of research utilisation.

Data collection deliberately drew on publicly available sources to avoid the cost of collecting new impact data and to ensure easy public access to verifiable data (See Additional file 1). We searched the online institutional repository for JCU (43) and the National Centre for FWB website (44). We also searched the Research Achievements, Empowerment Research Program 2001–2020 database, maintained by our network, which contains all available data about FWB research inputs, activities, outputs and evidence of research uptake and subsequent impacts (14). These sources provided details of all peer-reviewed papers (n = 49); research reports (n = 19); newsletters; data on participation numbers; grants and funding sources; DVDs; videos and documentary films; training packages; citations of FWB research in government and other policy documents; policy briefs; and FWB newsletters. Other outputs included one book, four book chapters and three doctoral theses.

From the search, we collated FWB participant data from the three FWB program training providers; Technical and Further Education, South Australia (TAFE SA), Batchelor Institute of Indigenous Tertiary Education (Northern Territory (NT)) and our JCU-led empowerment research network, up until the year 2011. Post-2011, the key participant data source has been the JCU-led research network reports on FWB program provision. The search also identified: a list of 16 FWB training and research grants; nine evaluative studies; and seven FWB research translation resources and other policy documents containing evidence of FWB research utilisation and impact.

To develop the FWB research impact narrative, we analysed the FWB participation data, using descriptive statistics, to show the growth in participation and the relative contribution of the JCU-led research network, as concurrently a program provider, vis-à-vis the other two program providers. We also analysed the funding data, using descriptive statistics, to show the scope and trends in funding or investment in FWB training and research over time. After that, we conducted a systematic scoping review of the nine FWB evaluative studies to show the reported SEWB benefits arising from individuals attending FWB training workshops. This was followed by a systematic scoping review of the seven FWB research translation resources to show the extent to which our research outputs influenced the decisions of service providers and other FWB user organisations to take up FWB and make it available to their clients and service communities.

The results section is divided into the five steps outlined in our approach to analysis. First, we present the research program impact logic and provide a brief impact narrative and statement of the relative contribution of the research. This leads to an overview of participation in the FWB program over 23 years. We then outline the trends in FWB research funding and investment that enabled us to undertake evaluative research to identify the SEWB impacts of participation in the FWB program. After that, we summarise the findings from the accounts of how FWB research was utilised by user organisations and public bodies. Following these results, a discussion section will highlight strengths and gaps in the impact data and the plan towards improving the FWB research impact evidence-base.

Family Wellbeing research impact logic model and overall impact claim

The FWB research impact logic model is presented in Fig. 2. It shows how 23 years of, and $2.3 million of funding investment in, FWB research-related activities have resulted in a range of research outputs and impacts. The evaluative studies found evidence of SEWB benefits arising from participation in the FWB program. The research outputs in turn facilitated more demand for the FWB program and hence more people accessed the program and experienced flow-on benefits. Overall, the FWB program has had a direct positive impact on the social and emotional wellbeing of 5,405 recorded participants, plus another estimated 2,000 who were not recorded, with ripple benefits for people in their social networks such as their families, friends and work colleagues.

There are two ways in which the FWB research contributed to achieving this impact claim. Firstly, the direct contribution was that researchers trained people in FWB, that is, the majority (59%) of the approximately 7,500 participants accessed the program through the JCU-led research network as a training provider. Secondly, the indirect contribution was that evidence of FWB social and emotional wellbeing benefits, collected via research, influenced more organisations and communities to use the program, thereby extending the uptake of FWB and hence its benefits. As the model shows, research impacts were made possible through a range of enabling factors including, but not limited to, embedding the research in Indigenous research principles.

In the next section we show how FWB program participation numbers reflect the increased program uptake and wider research impact.

Family Wellbeing program uptake and reach

Figure 3 shows the variable but increasing annual number of FWB participants over the years. Of the 5,405 total participants for which we have definitive data, 537 (10%) participated through overseas pilot studies in Papua New Guinea (2009–2012), Timor-Leste (2016) and China (2016–2019).

When FWB began in 1993, TAFE SA was the sole FWB training provider and 467 people participated in the program for the first seven years. Following the pilot program evaluation in 1998–2000, and the dissemination of the findings, there was an increase in numbers as more people learnt about FWB. In Phase 2, during the 10-year ERP, 2,300 people participated through TAFE SA and two additional new providers, namely, Batchelor Institute of Indigenous Tertiary Education (NT) and our JCU-led research network. During this phase, the research informed the network that participants were benefitting even from limited attendance without being present for the entire 150-hour course. As a result, we developed two-day and three-day short courses (with follow-up peer support) to make the program more accessible to people. Offering shortened courses further enabled uptake and spread of the program. In the third and final knowledge translation and impact phase, a total of 2,638 have participated through this network alone.

Although we do not have data for the other two providers during this period, based on anecdotal evidence, we estimate that more than 2,000 additional people participated. The workforce training during the knowledge translation and impact phase means there would be vastly increased deliveries to community members via those new trainers. However, with a few exceptions (45), we were unable to track the extent to which FWB-trained workers delivered FWB to their communities and the consequences of such deliveries.

What we can say though, looking at the increase in participation since 2001 (Fig. 3) is that FWB is unlikely to have achieved the levels of participation and impact that it did without the research and reporting of that research around Australia and the world. The dissemination of our research publications and knowledge translation activities and resources including community reports, knowledge sharing forums and importantly, word of mouth (the “Indigenous grapevine”), all helped to spread the message about FWB program outcomes. People working in community organisations then learnt about the intervention and proceeded to support individuals to participate in FWB through one of the above-mentioned providers.

In the following section we back up the statement of overall research impact claim and underpinning logic with an examination of the nature and extent of the research investment.

Family Wellbeing research related funding

Additional file 2 provides details of 16 FWB research-related grants totalling $2.308 million, received by the JCU-led FWB research network between 1998 and 2019. The initial small grant from the Tangentyere Council, an Aboriginal community-controlled organisation, to evaluate their FWB program, became the catalyst for 23 years of FWB research with significant outputs and impacts. Grant amounts ranged from the initial $3,000 to $465,000 with the median being $82,000. The majority of grants were for a single year (9/16), followed by two years (6/16), and only one lasted three years. There were diverse funding sources – four grants from the National Health and Medical Research Council (NHMRC); three from Queensland Health ; two from the Lowitja Institute and its predecessor Cooperative Research Centres; two from Apunipima Cape York Health Council; and several other health-related non-government organisations. The funding was used for a range of purposes, from wellbeing skills development to evaluation, and research translation and impact. The table also highlights that the research program relied on sourcing a variety of short-term funding in order to survive.

In the next section we provide qualitative and quantitative evidence of SEWB outcomes of FWB. We highlight FWB program outcomes as evidence of what the research found – and to argue that this evidence was of value to research users, and thus a vehicle to program uptake and wider benefit. The evidence in the following results speaks to the claim that FWB enhances the critical protective factors needed in contexts of intergenerational trauma and social disadvantage.

Evidence of Family Wellbeing benefits

This section presents a summary of the reported SEWB benefits based on the results of the scoping review of nine evaluative papers (one qualitative, one quantitative and seven mixed methods) involving a total of 1,010 FWB participants (see Additional file 3 for study characteristics). Findings are divided into qualitative and quantitative evidence and we focus on the benefits in Indigenous settings. Relevant qualitative and quantitative evaluative studies in non-Indigenous contexts are summarised in Additional file 3.

Reported qualitative social and emotional wellbeing outcomes

The evaluative studies brought to light some key SEWB benefits for Aboriginal and Torres Strait Islander participants of the program, and these findings resonate strongly with the range of protective factors outlined in the introduction. The findings are summarised into five themes below with the inclusion of quotes from the evaluations to demonstrate program outcomes. In addition, we mention some important population-level outcomes.

Personal growth

Participants reported on improved communication skills and the ability to better manage emotions. For example, a young Aboriginal man said; “I learned to use my voice, so I am a lot more confident to speak to people and crowds”; and a young Aboriginal woman reported that: “Before coming into this program, I bottled up all me emotions and feelings up but now I know that it’s better to talk to someone and know you’re not alone” (Central Coast NSW (46)).

Another woman had learned to understand and to manage her anger;

I feel very light… after bringing up some of the pains, you know? I think maybe all of us… It’s been quite a while since I’ve been angry, like after doing the course. I mean, I could get angry very easily… I thought I was nuts… whereas things I used to get angry about, I just don’t feel that anger any more (North Queensland (1)).

Increased confidence and resilience

These steps in personal growth contributed to increased self-confidence and esteem. An Aboriginal worker found that expressing her feelings gave her the space to think about how she could manage situations that arose for her; “…once you voice them then that gives the opportunity to think ‘well, what can I do about this?’ and maybe use the tools that are given to me through the program, that I can do something about it” (Victoria (39)). An Aboriginal community elder said she felt calmer and more confident; “I just sort of calmed down all the time… It just made you more confident in yourself, don’t sort of rush, panic or be frightened of things, making mistakes” (Victoria (39)).

Self-management

Participants took control of their SEWB in several ways including better management of mental health, alcohol use and gambling. This had flow-on benefits for other issues such as family relationships and financial stress;

Oh, yeah, things have changed. I guess I used to drink a lot and that and now I don’t drink that much. Now I’ve got money in the house, now that I’ve stopped drinking and I’ve got money to buy for the children. It’s because of the Family Wellbeing and the women’s group; it is those two things together (North Queensland (34)).

Building healthier and happier relationships

Participants spoke of building healthier and happier relationships with their families, teachers, and their communities. Participants learned how to better manage conflict in their families, and for some, this involved taking responsibility for their own behaviour;

Before when me and my wife used to fight, I used to get the rage and wanted to hit her…but now doing the FWB, I get to find other ways [of dealing with anger], plus [learning] how to deal with emotions and I find myself more at peace. (North Queensland (34)).

Also, participants learned to set boundaries and demand respect in their relationships; one woman told a story of her relationship with her nephew who;

…when drunk would come and punch walls and swear … I asked him to leave... he didn’t want to leave and argued he didn’t do or say anything wrong – so I wrote down the things that had upset me. [I] allowed him to come … once he had apologised… now he behaves sensibly because I stated ‘remember there’s the gate if you ever start your caper’ (Central Australia (26)).

Education and employment

Improved confidence and hope were evident as participants increasingly engaged in education, as demonstrated in this comment by a young Aboriginal man: “[the program] helped me get back into school. I'm now in a leadership program” (Central Coast NSW (46)). In Victoria, improvements in Aboriginal high school students’ school attendance were reported by both the students and the school principal in one region (39).

The effect of FWB participation was profound for this participant’s development;

I considered myself illiterate. I was pretty insecure. Once I did FWB I had more than I believed I had. Then I went to college and studied counselling. I had to write assignments. I hadn’t been to school since I was 14 (Central Australia (26)).

Participants also prepared for and engaged in employment. In Victoria, community members were reported to be getting “work ready”, for example going to a job agency or getting their drivers licence (39). Several people managed to gain employment; “...one of the ladies who lives in [name withheld] Community House, she’s got a job …it is three hours preparing some food but it’s extra cash” (Victoria (39)).

Population level outcomes

The previous five FWB program outcomes point to the flow-on effects that individual improvements in SEWB and empowerment have for families, communities and networks. The community of Yarrabah (Far North Queensland), in particular, has achieved notable population-level results through its involvement with FWB. It was found to be one of only two Indigenous communities across Australia (Tiwi Islands being the other) that reduced high suicide rates in the past 20 years (47). The evaluation noted that FWB research had made an essential contribution to this reduction.

Reported quantitative social and emotional wellbeing outcomes

The reported individual qualitative wellbeing benefits above are reflected in the results from the quantitative pre/three-month-post measures. In one of the five Indigenous studies in this group, Kinchin et al. (48) examined four measuring tools on their sensitivity to detecting changes in emotional development, including communication, conflict resolution, decision making and life skill development, in child protection agency staff. Of the four tools assessed, the Growth and Empowerment Measure (GEM) responses on self-capacity, inner peace, strength, happiness and connectedness (questions 1–14) indicated a 17% positive change in the mean scores for FWB participants. For the Australian Unity Wellbeing Index (AUWI), “the most satisfactory post-interventional response was provided on future security which was estimated even higher than the national benchmark” (48). With an r score of 17% and a p-value of < 0.001, the GEM proved to be the most sensitive tool in shedding light on how well participants coped with stress and demands on their time, as well as their perceptions of personal accomplishment and overall satisfaction with life (48).

Klieve et al. (46) reported improved participant wellbeing overall among two groups of vulnerable Aboriginal young men, with a highly significant reduction in psychological distress (t = 3.67, df = 12, p = .003) among the first group. The results were similar among the subsequent group with a significant decrease in the aggregated scores across the Kessler 5 scale (K5) items (t = 3.943, df = 47, p < .001, d = 0.5691). The study concluded that FWB might have given these young men an opportunity to enhance their SEWB and in turn had the potential to mitigate some of the costs associated with medical treatment and criminal interventions (46).

FWB delivered to Aboriginal health service workers who support users of methamphetamines was shown to significantly increase those worker’s GEM scores in life satisfaction (z = 2.25, p = .024) and inner peace (z = 2.25, p = .024). The effect sizes for all measures were large and positive (0.62–0.69), except for the self-capacity subscale where the effect size was small but still positive (0.16). In addition, participants reported improved wellbeing and feeling empowered in supporting users of methamphetamines and their families (39).

As stated in the introduction, there can be a stigma placed on qualitative research of SEWB, even though it is this kind of research that captures compelling narratives about the effectiveness of wellbeing and empowerment interventions. To support these accounts, the FWB research program sought to quantitatively measure program outcomes as a way of increasing its profile, strengthening the FWB outcome evidence base, and assisting the spread of the program.

Evidence of Family Wellbeing research utilisation or influence on program uptake

In this section we provide evidence of the utilisation of FWB research that led to decisions to implement the program in various contexts. The complete results of the review are presented in Additional file 4, however we highlight here the findings of three reflections by health and other human services managers on their respective decisions to use FWB in their organisations; and two national policy documents citing our research as evidence that FWB improves community functioning and SEWB, thereby helping to overcome Indigenous disadvantage.

In one of the three FWB user reflections, Baird (49) explained how a community-controlled health service formed a long-term relationship with the FWB research network to implement and evaluate the program in the community of Yarrabah. The research contributed to successful grant acquisitions which helped further the integration of the FWB program into the health service. Community members also became trained as researchers as part of empowering participants to conduct research in their own community.

Gabriel (50) described how, for a health service on the Central Coast in NSW, they first learnt of the program from reading the initial evaluative study (26). This led to uptake of FWB as part of their SEWB program for young Aboriginal men in the area. Back in Queensland, a child protection agency known as Act for Kids drew on FWB research to create a policy brief for senior management about the impact of the program (51). As a result, a relationship developed between the organisation and the FWB research network which led to FWB becoming a practice framework across all its sites in Far North Queensland.

FWB research was cited in two nationwide reports. The Social Justice report by the Australian Human Rights Commission (52) described FWB “as an example in how to support communities to address complex problems by drawing on holistic healing methods which blend cultural renewal and spirituality with conflict resolution and other problem-solving skills”. Two government agency reports on overcoming Indigenous disadvantage cited FWB research papers as an example of a healing program that works (53, 54).

The evidence in the seven publications shows how the FWB research program influenced the thinking of both the primary user group i.e. the frontline service providers and, to some degree, policy makers.

In this article we set out to explore the impact of the 23-year long FWB research program and to examine the strengths and limitations of the impact data, as a way of planning for more targeted impact data collection into the future. We presented the data to support an overall impact claim; that FWB research contributed to approximately 7,500 participants accessing the FWB program and experiencing SEWB benefits as a result. The following key points are pertinent.

The logic model facilitated a clear path for research impact and allowed for the challenges in showing cause and effect in research. We documented the complex and messy nature of research, knowledge translation and impact. While we attempted to distinguish between “FWB program outcomes” and “FWB research impact” in our results, we found that such delineation was not always straightforward. For example, the original funding for an evaluation from 1998 (26) led to initial uptake by some organisations and benefits for their participants, but it also led to the 10-year empowerment research program which created further outcomes and research impact. Therefore, a grant that is originally an input then becomes the facilitator of another grant which, in a sense, is an output. This output then becomes an input for subsequent outputs.

As Whiteside et al. (45) put it, where knowledge translation and research impact begin and end is neither linear nor easily measurable. Besides, while FWB research is definitely important, it is only one factor among many that lead to ultimate impact, for example, the fact that the program is Aboriginal-developed may also account for its wide reach (55). Therefore, it is not enough to talk about “research impact”, we should also be talking about the relative contribution of research vis-à-vis other contributory factors to the impact claim. This is where research utilisation data is valuable because it explains, from the point of view of users, not only the extent to which the research influenced their decisions and actions but importantly the impacts of such decisions. Without this information it is hard for researchers to show the pathways through which their research leads to particular impacts. Thus, collecting such data from the users’ point of view has important potential to bridge the research impact attribution gap.

This article also highlighted the nature and scope of the research funding or investment involved in the 23-year research program. Since most funding was relatively short-term (one to two years), this limited the capacity of the researchers to follow up with participants in the medium to long term. Despite the ad hoc short-term funding, a commitment to employing and supporting local Indigenous people as “community-based” researchers (in keeping with Indigenous research principles), not only increased employment and built research capacity but, importantly, it encouraged local ownership of the research and hence the capacity to use it to achieve desired outcomes (25, 49).

We argued a case for ensuring the quality of intervention research. The higher the scientific quality of research, the more likely it will be taken up and used to achieve impact. But research must also be fit-for-purpose. As Onnis et al. (56) point out, the quantity and type of evidence that is needed to demonstrate the effectiveness of an intervention is not universally agreed upon. While randomised control trials are considered gold standard, it is not always appropriate or viable to conduct such research. Clearly, in complex, varied, and small-scale research settings, alternative ways of incrementally demonstrating the effectiveness of an intervention using continuous quality improvement approaches are called for (56).

As discussed in this article, a key finding of FWB program outcomes was that it contributed to strengthening the protective factors needed for dealing with issues of intergenerational trauma and the legacy of colonisation. As part of the research program, researchers adopted a participatory, iterative, and continual approach to capture this data. Commencing with a narrative evaluation of FWB was very powerful in facilitating impact, especially in enhancing uptake among Indigenous community organisations. Frontline workers and managers of community SEWB organisations reported finding narrative qualitative evidence about the program’s effectiveness to be very compelling (49–51). Whilst we have built on this narrative research by implementing pre- and post-quantitative evaluations, such studies have been small in scale with limited participant numbers in discrete projects. However we are currently building new partnerships designed to generate policy-relevant evidence including the extent to which investing in FWB constitutes a social return on investment.

Whilst we point out the strengths of the retrospective data in being able to demonstrate meaningful research impact and uptake of the program, we also acknowledge a key limitation – that the data was originally collected for other research purposes. Having said that, the application of Indigenous research principles allowed us to collect data on issues that Indigenous people value or expect of research, so it is from these measures which we attempted to glean evidence of research impact. Into the future, what is required are clear evaluative indicators to measure the extent to which Indigenous research principles are applied in the research endeavour (14, 21). In conclusion, these lessons and the impact logic highlight that researchers need to create appropriate databases from the beginning and collect data well beyond the life of the project, even though we acknowledge that such endeavours are rarely funded (14, 21).

AUWI - Australian Unity Wellbeing Index

CRC – Cooperative Research Centre

ERP - Empowerment Research Program

FWB – Family Wellbeing

GEM - Growth and Empowerment Measure

IAD - Institute for Aboriginal Development

JCU - James Cook University

K5 - Kessler 5 scale

NHMRC - National Health and Medical Research Council

NSW – New South Wales

NT – Northern Territory

QLD - Queensland

SA – South Australia

SEWB – Social and Emotional Wellbeing

TAFE - Technical and Further Education

UK – United Kingdom

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Availability of data and materials

The datasets analysed during the current study are available in the James Cook University (ResearchOnline@JCU) https://researchonline.jcu.edu.au/ and the National Centre for FWB website https://family-wellbeing.squarespace.com/fwbprogram.

All data generated or analysed during this study are included in this published article [and its additional files].

Competing interests

The authors declare that they have no competing interests.

Funding

The authors would like to acknowledge three entities that have supported the publication of this article: Lowitja Institute; Centre of Research Excellence – Strengthening systems for Indigenous health care equity (CRE-STRIDE); and an anonymous philanthropic trust.

Authors' contributions

The first two authors (NP, KT) made a substantial contribution to the conception; analysis and interpretation of data; and drafted the work substantively. MH and MW contributed to data collection and analysis. All authors have read and approved the manuscript.

Acknowledgements

The authors thank all the Family Wellbeing research participants, researchers, and partner organisations in the research network for their contributions to this study.

Onnis L-A, Moylan R, Whiteside M, Klieve H, Smith K, Tsey K. Integrating the Family Wellbeing program into practice: A conceptual model. Aust Soc Work. 2020 Oct;73(4)(1):435–48.
Atkinson J. Educaring: A trauma informed approach to healing generational trauma for Aboriginal Australians [Internet]. Goolmangar, NSW: We Al-li; 2012. Available from: https://www.oics.wa.gov.au/wp-content/uploads/2017/07/Judy-Atkinson-Healing-From-Generational-Trauma-Workbook-We-Al-li.pdf.
Bamblett M, Lewis P. Detoxifying the child and family welfare system for Australian Indigenous peoples: Self-determination, rights and culture as the critical tools. First Peoples Child Fam Rev. 2007;3(3):43–56.
Australian Institute of Health and Welfare (AIHW). Young Australians: Their health and wellbeing 2011 [Internet]. ACT: Author; 2011 [cited 2021 May 11]. Available from: https://www.aihw.gov.au/reports/children-youth/young-australians-their-health-and-wellbeing-2011/summary.
Australian Institute of Health and Welfare (AIHW). Housing circumstances of Indigenous households: Tenure and overcrowding [Internet]. ACT: Author; 2014 [cited 2021 May 11]. Report No.: Cat.: IHW 132. Available from: https://www.aihw.gov.au/reports/indigenous-australians/housing-circumstances-of-indigenous-households/contents/table-of-contents.
Australian Institute of Health and Welfare (AIHW). Child protection Australia 2016–17 [Internet]. ACT: Author; 2018 [cited 2021 May 11]. (Child Welfare Series). Report No.: No. 68. Available from: https://www.aihw.gov.au/reports/child-protection/child-protection-australia-2016-17/contents/table-of-contents-print-report.
Australian Institute of Health and Welfare (AIHW). Youth detention population in Australia 2018, Table of contents [Internet]. ACT: Author; 2018 [cited 2021 May 11]. (Cat.: JUV128). Report No.: Bulletin No. 145. Available from: https://www.aihw.gov.au/reports/youth-justice/youth-detention-population-in-australia-2018/contents/table-of-contents.
Commonwealth of Australia, Department of the Prime Minister and Cabinet. Closing the Gap Prime Minister’s report [Internet]. 2018. Available from: https://www.pmc.gov.au/sites/default/files/reports/closing-the-gap-2018/sites/default/files/ctg-report-20183872.pdf?a=1.
Wilson B, Abbott T, Quinn SJ, Guenther J, McRae-Williams E, Cairney S. Empowerment is the basis for improving education and employment outcomes for Aboriginal people in remote Australia. Aust J Indig Educ. 2019 Dec;48(2):153–61.
Laycock A, Walker D, Harrison N, Brands J. Researching Indigenous health: A practical guide for researchers [Internet]. Melbourne: The Lowitja Institute; 2011. Available from: https://www.lowitja.org.au/content/Document/Lowitja-Publishing/Researchers-Guide_0.pdf.
Whiteside M, Tsey K, Cadet-James Y, Mccalman J. Promoting Aboriginal health: The Family Wellbeing empowerment approach. Cham: Springer International Publishing : Imprint: Adis; 2014.
Bainbridge R, McCalman J, Jongen C, Campbell S, Kinchin I, Langham E, et al. Improving social and emotional wellbeing for Aboriginal and Torres Strait Islander people: an Evidence Check rapid review brokered by the Sax Institute for Beyond Blue [Internet]. Beyond Blue; 2018. Available from: https://www.beyondblue.org.au/docs/default-source/policy-submissions/aboriginal-programs-for-sewb_final-4.pdf?sfvrsn=157bbfea_4.
Kinchin I, McCalman J, Bainbridge R, Tsey K, Lui FW. Does Indigenous health research have impact? A systematic review of reviews. Int J Equity Health. 2017 Mar 21;16(1):52.
Tsey K, Onnis L, Whiteside M, McCalman J, Williams M, Heyeres M, et al. Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research. Eval Program Plann. 2019 Apr;1:73:176–86.
Australian Research Council. Engagement and impact assessment pilot 2017 [Internet]. 2017. Available from: https://www.arc.gov.au/engagement-and-impact-assessment/ei-pilot-overview.
Heyeres M, Tsey K, Yang Y, Yan L, Jiang H. The characteristics and reporting quality of research impact case studies: A systematic review. Eval Program Plann. 2019;73:10–23.
Zuchowski I, Miles D, Gair S, Tsey K. Social work research with industry: a systematic literature review of engagement and impact. Br J Soc Work. 2019;49:2299–324.
Bainbridge R, Tsey K, McCalman J, Kinchin I, Saunders V, Watkin Lui F, et al. No one’s discussing the elephant in the room: contemplating questions of research impact and benefit in Aboriginal and Torres Strait Islander Australian health research. BMC Public Health. 2015 Jul 23;15(1):696.
Cheer K, Lui FW, Shibasaki S, Harvey A, Grainger D, Tsey K. The case for a Torres Strait Islander-driven, long-term research agenda for environment, health and wellbeing. Aust N Z J Public Health. 2020;44(3):177–9.
Shibasaki S, Sibthorpe B, Watkin Lui F, Harvey A, Grainger D, Hunter C, et al. Flipping the researcher knowledge translation perspective on knowledge use: a scoping study. Altern Int J Indig Peoples. 2019 Sep 1;15(3):271–80.
Tsey K, Lawson K, Kinchin I, Bainbridge R, McCalman J, Watkin F, et al. Evaluating research impact: The development of a “RESEARCH for IMPACT” TOOL. Front Public Health [Internet]. 2017 [cited 2020 Aug 24];4. Available from: http://dx.doi.org/10.3389/fpubh.2016.00160.
Greenhalgh T, Fahy N. Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK Research Excellence Framework. BMC Med. 2015 Sep 21;13(1):232.
Holt GD, Goulding JS, Akintoye A. Enablers, challenges and relationships between research impact and theory generation. Eng Constr Archit Manag. 2016 Jan 1;23(1):20–39.
Searles A, Doran C, Attia J, Knight D, Wiggers J, Deeming S, et al. An approach to measuring and encouraging research translation and research impact. Health Res Policy Syst. 2016 Aug 9;14(1):60.
Tsey K. Working on wicked problems: A strengths-based approach to research engagement and impact. Switzerland: Springer; 2019.
Tsey K, Every A. Evaluating Aboriginal empowerment programs: the case of Family WellBeing. Aust N Z J Public Health. 2000;24(5):509–14.
Kitau R, Kinchin I, Whiteside M, Hane-Nou G, Tsey K. Effectiveness of the uptake and implementation of an Aboriginal Australian empowerment program in the context of public health training in Papua New Guinea. Pac J Med Sci. 2016 Nov;16:16–34.
Tsey K, Lui SM (Carrie), Heyeres M, Pryce J, Yan L, Bauld S. Developing soft skills: Exploring the feasibility of an Australian well-being program for health managers and leaders in Timor-Leste. SAGE Open. 2018 Oct 1;8(4):2158244018811404.
Whiteside M, Bould E, Tsey K, Venville A, Cadet-James Y, Morris ME. Promoting Twenty-first-century Student Competencies: A Wellbeing Approach. Aust Soc Work. 2017 Jul;3(3):324–36. 70(.
Yan L, Yinghong Y, Lui SM (Carrie), Whiteside M, Tsey K. Teaching “soft skills” to university students in China: The feasibility of an Australian approach. Educ Stud. 2019 Mar 4;45(2):242–58.
Tsey K. A brighter future for life in the tropics: translating big picture academic vision into practical research. J Trop Psychol. 2015 ed;5.
Tsey K, Wilson A, Haswell-Elkins M, Whiteside M, McCalman J, Cadet-James Y, et al. Empowerment-based research methods: a 10-year approach to enhancing Indigenous social and emotional wellbeing. Australas Psychiatry. 2007;15:34–8.
Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000 Sep;16(7262):694–6. 321(.
Tsey K, Whiteside M, Haswell-Elkins M, Bainbridge R, Cadet-James Y, Wilson A. Empowerment and Indigenous Australian health: A synthesis of findings from Family Wellbeing formative research. Health Soc Care Community. 2009;18(2):169–79.
Daly B, Tsey K, Whiteside M, Baird B, Kingsburra S, Jackson K, et al. ‘We’re the talk of the town’: Facilitating mastery and control in Indigenous communities. An evaluation of a ‘Family Wellbeing personal and community development project in Yarrabah. Cairns: University of Queensland School of Population Health and Gurriny Yealamucka Health Service Aboriginal Corporation; 2004.
Every A, Williams E, Tsey K. “Caring and sharing. In: Arnte Arnte Aremele Antheme”: Family Wellbeing community report. Cairns and Alice Springs. University of Queensland School of Population Health and Gurriny Yealamucka Health Service Aboriginal Corporation and Tangentyere Council; 2002.
Tsey K, Every A. Taking control: A summary report for Family WellBeing graduates. Darwin and Alice Springs. Cooperative Research Centre for Aboriginal and Tropical Health and Tangentyere Council; 2000.
McCalman J. The transfer and implementation of an Aboriginal Australian wellbeing program: a grounded theory study. Implement Sci. 2013 Oct;31:8:129.
Whiteside M, MacLean S, Callinan S, Marshall P, Nolan S, Tsey K. Acceptability of an Aboriginal wellbeing intervention for supporters of people using methamphetamines. Aust Soc Work. 2018 Jul 3;71(3):358–66.
Connell JP, Kubisch AC. Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems. In: Fulbright-Anderson K, Kubisch AC, Connell JP, editors. New approaches to evaluating community initiatives: Theory, measurement, and analysis. Washington, DC: Aspen Institute; 1998. pp. 1–13.
Kelly L, Whiteside M, Barich H, Tsey K. Checking up to keep on track: An Aboriginal-led approach to monitoring well-being. Eval J Australas. 2021;1–14.
Yap M, Yu E. Community wellbeing from the ground up: a Yawuru example. 2016. Report No.: Bankwest Curtin Economics Centre Research Report 3/16 August.
James Cook University. ResearchOnline@JCU [Internet]. 2021 [cited 2021 Jun 25]. Available from: https://researchonline.jcu.edu.au/.
NCFWB. National Centre for Family Wellbeing [Internet]. 2017 [cited 2021 Jun 25]. Available from: https://family-wellbeing.squarespace.com/fwbprogram.
Whiteside M, Thomas D, Griffin T, Stephens R, Maltzahn K, Tsey K, et al. Capturing research impact: The case study of a community wellbeing research partnership. Aust Soc Work. 2021 Mar;17(0):1–12. 0(.
Klieve H, Cheer K, Whiteside M, Baird L, MacLean S, Tsey K. “A safe haven to support me”: An evaluation report on the Central Coast Family Wellbeing program [Internet]. Cairns, Queensland: Cairns Institute; 2019 [cited 2020 Aug 19]. Available from: https://researchonline.jcu.edu.au/59155/.
Prince J, Jeffrey N, Baird L, Kingsburra S, Tipiloura B, Dudgeon P. Stories from community - How suicide rates fell in two Indigenous communities. Canberra: Aboriginal & Torres Strait Islander Healing Foundation; 2018.
Kinchin I, Jacups S, Tsey K, Lines K. An empowerment intervention for Indigenous communities: An outcome assessment. BMC Psychol [Internet]. 2015 Aug 21 [cited 2021 May 11];3(1). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4546092/.
Baird L. The solution to Indigenous suicide crises lies in listening to Aboriginal people. Overland [Internet]. 2019 Jun 24 [cited 2020 May 11]; Available from: https://overland.org.au/2019/06/the-solution-to-indigenous-suicide-crises-lies-in-listening-to-aboriginal-people/.
https://www.cairnsinstitute.jcu.edu.au/fwb-newsletter/.
Moylan R, Drew V. It’s not about reinventing the wheel, it’s about making the wheel turn. Family Wellbeing newsletter [Internet]. 2017;(3). Available from: https://www.cairnsinstitute.jcu.edu.au/fwb-newsletter/.
Australian Human Rights Commission. Social Justice Report 2011 [Internet]. 2011. Available from: https://humanrights.gov.au/sites/ default/files/content/social_justice/sj_report/sjreport11/pdf/sjr2011.pdf.
Closing the Gap Clearinghouse (AIHW & AIFS). What works to overcome Indigenous disadvantage: key learnings and gaps in the evidence [Internet]. Canberra: AIHW; 2013 [cited 2021 May 14]. Available from: https://www.aihw.gov.au/reports/indigenous-australians/works-to-overcome-indigenous-disadvantage-2011/contents/table-of-contents.
Productivity Commission. Overcoming Indigenous Disadvantage: Key Indicators 2016 [Internet]. Canberra: Productivity Commission; 2016 [cited 2021 May 16]. Available from: https://www.pc.gov.au/research/ongoing/overcoming-indigenous-disadvantage/2016.
McCalman J, Bainbridge R, Brown C, Tsey K, Clarke A. The Aboriginal Australian Family Wellbeing program: A historical analysis of the conditions that enabled its spread. Front Public Health. 2018 Mar;1:6:26–6.
Onnis L-A, Klieve H, Tsey K. The evidence needed to demonstrate impact: A synthesis of the evidence from a phased social and emotional wellbeing intervention. Eval Program Plann. 2018;70:35–43.

Download PDF

Editorial decision: Major Revision
29 Oct, 2021
Review #1 received at journal
15 Oct, 2021
Review #2 received at journal
09 Oct, 2021
Reviewer #3 agreed at journal
07 Oct, 2021
Reviewer #2 agreed at journal
06 Oct, 2021
Reviews received at journal
24 Aug, 2021
Reviewer #1 agreed at journal
23 Aug, 2021
Reviewers invited by journal
10 Aug, 2021
Editor assigned by journal
05 Aug, 2021
Submission checks completed at journal
04 Aug, 2021
Editor invited by journal
04 Aug, 2021
First submitted to journal
04 Aug, 2021

You are reading this latest preprint version

“We Are Not Stray Leaves Blowing About In The Wind”: Summing Up The Impact of Family Wellbeing Empowerment Research, 1998-2021

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Reported qualitative social and emotional wellbeing outcomes

Reported quantitative social and emotional wellbeing outcomes

Discussion And Conclusion

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Supplementary Files

Status:

Version 1