Theory-informed evaluation of digital health interventions: case study reflections on the use of a ‘Theory of Change’ framework

Cooper, R.¹, Bell, S.², Pringle, E.¹, Edmundson, E.³, Nielsen, H.³, Roberts, S.³, Edelstein, M.¹, Mounier-Jack, S.² and Chantler, T.² (Author affiliations: 1. Public Health England, UK. 2. Department of Global Health and Development, Faculty of Public Health & Policy, London School of Hygiene and Tropical Medicine, UK. 3. Hounslow and Richmond Community Healthcare NHS Trust, UK.)

Correspondence to Rosie Cooper.

Abstract

Background

In 2018, a London NHS Trust piloted a digital health intervention in the form of an electronic consent intervention for the adolescent girls’ Human Papilloma Virus (HPV) vaccination programme. Prior to the pilot, a multidisciplinary team developed a bespoke Theory of Change (ToC) as a framework for the evaluation. This paper discusses the construction and use of the ToC to facilitate a mixed-methods evaluation of the digital intervention. The aim of our case study is to describe the development and use of the ToC including how it enabled examination of the usability and acceptability of the intervention and assessment of the impact on consent form returns and HPV vaccine uptake.

Methods

A multi-disciplinary team iteratively developed a bespoke ToC, which was used as an evaluative framework for the pilot study. This was used to apply a qualitative methodology (alongside a quantitative methodology) to examine the usability and acceptability of the intervention.

Results

Use of a ToC provided an effective framework for a mixed-methods evaluation resulting in consideration of each indicator on the causal pathway. This resulted in identification of an issue in the ‘Activities’ and ‘Pathways’ section, specifically with embedding the new intervention and parental accessibility.

Conclusions

Constructing and using our ToC demonstrates the importance of considering behavioural science in implementing novel digital health interventions. The use of a ToC facilitated the effective evaluation of a novel digital health intervention, enabling learning to be applied before the intervention was further scaled up. We advocate considering the use of a ToC in evaluations to aid understanding of barriers or enablers for incorporation of digital healthcare interventions into practice.

Keywords

Theory of Change, Digital Health, E-consent, Immunisation Consent, Immunisation.

Background

In England, the Human Papilloma Virus (HPV) vaccine is routinely offered to adolescents aged 12-14 to provide protection against high-risk types of HPV that cause cancer [1]. In 2019, the offer of the HPV vaccination was extended to adolescent boys as part of the School-Aged Immunisation Pathway [1]. At the time of the case study (2018-2019) discussed in this paper, the vaccine was only offered to adolescent girls. Current procedure for offering the HPV vaccine in England (in line with The Green Book [2]) involves disseminating information about the HPV vaccination (mainly via schools) and obtaining either consent from parents/ guardians or self-consent from Gillick competent adolescents. Assessing Gillick's competence for immunisation requires the health professional to assess and decide whether a child under 16 is competent and willing, able and mature enough to make a decision [3]. The status quo for delivering consent forms is a paper-based system which involves some logistical challenges and requires significant human resource [4], [5].

The UK National Health Service (NHS) Long Term Plan [6] highlights an expectation of digital transformation to safeguard health systems data and streamline communication between health facilities and the public. In this context, school-based immunisation programme managers and researchers have been evaluating the potential for digital interventions to improve the logistics of obtaining parental consent and adolescent self-consent. The case study we focus on in this paper is a usability and acceptability evaluation of an electronic consent (e-consent) intervention. The intervention was developed by a London Community Health NHS Trust and piloted as part of the Trusts school-based immunisation programme in 2018-19 [7]. The goal of the e-consent intervention was to encourage behavioural change at an individual and system level, with the objectives of increasing both parental consent form return, and uptake of the HPV vaccination programme.

Given the intended future direction of the NHS [6] the Community Health NHS Trust wanted to examine the following:

1. Whether and how the intervention achieved its objectives
2. How the intervention could become part of normalised routine practice
3. Scalability potential of the intervention.

Hence, a multidisciplinary evaluation team comprised of academics, public health clinicians and front-line immunisation nurses (implementers) was formed. This team designed a mixed-methods, theory-informed evaluation which used a Theory of Change (ToC) as an evaluation framework. Quantitative methods were used to assess whether the pilot intervention increased consent form return and the uptake of the first dose of the HPV vaccine in adolescent girls. The steps involved in implementing the e-consent system, people’s experiences of these, the interactions between inputs, activities, pathways and outputs and outcomes and impacts were investigated using qualitative methods [7].

The purpose of this paper is to share reflections on the use of a ToC as a framework in this pilot evaluation. The advantages and disadvantages will be discussed, alongside describing team involvement in the adaptation of the ToC over time in line with evaluation findings. Additionally, as strongly advocated by Davis et al., [8], the paper describes how other behavioural theories were employed to inform interpretation of data and maximise the usefulness of the intervention.

The e-consent intervention

The e-consent intervention comprised of an online platform with an e-consent form as well as a data platform and related implementation procedures (Figure 1). Functionally, the platform aimed to: 1) provide parents/ guardians with access to an online portal through which they could access relevant information and were able to register their adolescent and consent/decline HPV vaccination; 2) provide nurses with electronic portal access, facilitating screening and enabling records to be updated during immunisation sessions; 3) enable central immunisation record databases to be automatically updated. However, the online portal and data platform were not fully functioning prior to the introduction of the intervention in Year 1 (June/July 2018), which specifically affected aims 2 and 3. Therefore, the way in which students’ information and consent forms were screened prior to and during immunisation sessions had to be modified. Figure 1 details both the Year 1 modifications and, in blue, the use of the fully functioning intervention in Year 2 (June/July 2019).

The ToC Design and Conceptualisation Process

The development and use of a ToC formed an intentional part of the evaluation and was included in the protocol. To conceptualise and design the ToC, a small subset of the larger evaluation team worked together to draft the first version. It was decided at this stage that the ToC should be depicted as a model to aid the larger study team (comprising a diverse range of professionals) to reach a consensus on overall aims, and methods to achieve these and to assess success. The aim of the ToC model was to provide a comprehensible framework, which articulated the causal pathway within which change happens to achieve the intended outcomes and impact, resulting in increased HPV vaccine uptake. The ToC model is therefore depicted as a results chain which provides a theoretical model for defining each component. In developing the ToC it was ensured that each box was causally linked to the previous box and that the final model was as simple as possible, rendering it easily comprehensible to multidisciplinary professionals. The draft ToC (Figure 2) was considered to be flexible and was drafted with the aim of seeking input from key stakeholders (represented in the evaluation team) before being finalised.

The ToC Development Process

To plan and develop the evaluation, regular face-to-face investigator meetings were scheduled for the multidisciplinary evaluation team. The ToC was developed in an iterative manner and to ensure it was both credible and supported, the initial version of the ToC was presented to the evaluation team at each meeting and used as a tool to frame and facilitate open conversation about the causal mechanisms in the intervention. As a framework for the conversation, the team first considered the desired impact (increased HPV vaccination) and outcomes (increased consent form return) and then worked backwards to determine the logical steps that were needed to reach this goal. The investigator meetings considered key areas such as context analysis, mapping pathways and assumptions underlying the ToC. There was an inherent degree of optimism that the intervention would be beneficial for health outcomes and NHS resource use. The model described the key aspects of behaviour change required for the intervention to work, from inputs through to the final goal of increased HPV vaccine uptake. This became an iterative development and refinement cycle and formed a part of three investigator meetings over a 7-month period. In a cyclical process, the ToC was mainly revised by two investigators and then shared with the wider team for input. In the final investigator meeting, the team worked together to develop the final ToC, which was then reflected in a logic model.

Concurrent to developing the ToC for evaluation, the Trust began to implement the intervention. As part of its development, members of the evaluation team (RC, TC and SM-J) sought out a contextual understanding of the intervention by accompanying immunisation teams during seven e-consent school HPV immunisation sessions. Observations from the sessions, as well as insights from the implementers, were fed back into workshops which enabled us to further consider the ToC and ensure that it was robust.

Utilising the ToC

The ToC was used pragmatically to inform the evaluation and was utilised as a framework to explore the interactions between inputs, activities, pathways and outputs, outcomes and impacts. It was integrated into the thematic analytical approach of the evaluation which combined semi-deductive mapping of data to codes aligned with each part of the ToC, alongside inductive open coding to capture additional, unanticipated themes.

Results

The final ToC is depicted in Figure 3. The ToC describes the hypothesised change mechanisms occurring within the intervention. During the evaluation, each indicator was considered, which resulted in the identification of two specific problems: 1) Under ‘Activities’, it was found that the new electronic system usability was limited by the lack of fully operational IT functions, thus preventing the intervention from being embedded, and 2) Under ‘Pathways’, problems were identified from some parents accessing the e-consent system. Reflections on the process and lessons learnt follow.

Reflections

Use of the Toc during evaluation resulted in key learning around implementation and evaluation which enabled the Trust to address specific areas, enabling the intervention to be developed and improved before being scaled up. Although the ToC was used as a framework to evaluate a digital consent intervention for adolescent girls in schools, there is no apparent reason that this could not also be applied to the HPV immunisation programme for adolescent boys, or indeed as consent for other immunisations offered in schools.

What worked well

The principal strength of the method described in this paper is that the ToC framework used was able to succinctly translate complex behavioural change science into a framework that was readily comprehensible for different audiences, enabling understanding of how, why and whether the digital health intervention worked.

The ToC was used appropriately to articulate how change happens within the digital intervention [9]. The ToC was found to be helpful in providing a framework to structure discussion around how this novel digital health tool was theorised to result in improving the return of HPV vaccine consent forms and HPV vaccine uptake in the school immunisation programme of adolescent girls. Working with a diverse range of stakeholders, this was particularly helpful as it helped to communicate succinctly to multidisciplinary professionals, ensuring there was a shared understanding, and resulting in clarification of terminology. ToCs are proposed to be useful in such settings to avoid misunderstandings and help to clarify what the desired change is, why it is desired and how it could/should happen [10]. The ToC described here provided an effective method to describe the links between intervention and outcomes, and the chain of steps that lead to achieving an increased return of HPV vaccine consent forms and HPV vaccine uptake.

A further strength was the flexibility of the initial design of the ToC (and subsequent iterations), which allowed multiple cycles of iteration and refinement, as many times as needed. The use of the ToC in workshops provided opportunities for discussion about the evaluation, enabling stakeholders to unpack different aspects of the intervention, which may have otherwise been overlooked in the evaluation. It was felt that the final ToC was robust and served its purpose to guide the team in developing an evaluation of the digital health intervention.

Lessons Learnt

Stakeholder engagement and flexibility is key

Taking a multidisciplinary and pragmatic approach to constructing the ToC was key to developing an appropriate model. Identifying and engaging appropriate stakeholders is crucial and their insight is important for understanding the context of the ToC, which in turn is key to understanding behaviour change [11]. It is suggested that digital healthcare evaluations would benefit from collaborations across academia and the NHS, creating the potential for a more meaningful sharing of resources and knowledge across the system. It is also recommended that stakeholders should be involved in conceptualising and developing the ToC from the outset. In line with existing evidence, it is recommended that conceptualised ToCs are dynamic and viewed as an iterative process, able to be revised as further information becomes available [12]. The use of participatory methods which encourage input and ownership of the ToC should be considered. A joint ToC aids in laying the groundwork for collaboration and can be helpful to understand how each stakeholder understands the situation [10]. In developing the ToC, it is recommended that stakeholders are provided with sufficient space to consider and feedback on the theory (and support if they are less familiar with the model used) [13].

Consider both the positive and negative consequences of the intervention

Optimism bias was found to be inherent in the workshops, as many held the implicit assumption that the intervention would succeed. However, it may also be valuable to consider why an intervention may fail, for example through use of a pre-mortem analysis. Using a similar process to the one described above to identify the causal pathways of success, a pre-mortem analysis requires one to imagine the intervention has failed and to work backwards to imagine all of the possible reasons for this [14]. This technique can also be employed to help overcome groupthink [14].

New interventions take time to become embedded before behaviour change can happen

The behaviour change expected from this intervention was more parents returning the HPV vaccine consent form, resulting in greater uptake of the HPV vaccine. Constructing the ToC to identify what needed to change in behavioural terms is recommended as a crucial first step [15]. During the evaluation, it was concluded that the intervention did not result in the desired behaviour change in parents. However, one of the benefits of using our ToC was that it required the evaluation to consider each stage of the ToC, not just the outcomes. This enables the unpacking of the ‘black box’ of the intervention. Following consideration of each element of the ToC, alongside drawing on the existing evidence base around embedding complex healthcare interventions, it was evident that immediate behaviour change was unlikely to result from the intervention. Implementation of interventions is a complex process, which takes time to become embedded [16]. Evaluation data indicated that the new systems were hard to adapt to; suggesting implementing this intervention successfully will require an embedding process. This may suggest that to fully understand the impact of behaviour change a re-evaluation may be required once the process has become embedded.

Conclusion

This paper provides a report of the design and utilisation of a ToC in evaluating a digital intervention, and a reflection on using it pragmatically. Although the e-consent intervention did not increase the return of consent forms, developing and using a bespoke ToC acted as an explanatory framework, through which it was possible to identify challenges in transitioning to a new way of working, and recognise that more time is required to embed the intervention before the intended behaviour change can be expected to take place.

Digital health interventions have enormous potential to improve healthcare delivery but consideration of the possible benefits of such interventions requires a theory-based characterisation of the necessity of the intervention and the context within which it will be used. The development and use of the ToC described here demonstrate that behavioural science has a significant role to play in the evaluation of novel digital healthcare interventions.

Acknowledgements

The authors are grateful for the information shared freely by interview participants, schools and the Hounslow and Richmond Community Healthcare NHS Trust.

Ethics Approval

The research published in this manuscript was approved by the PHE Research Support & Governance Office (Ref: NR0131) and the London School of Hygiene and Tropical Medicine Observational/Interventions Research Ethics Committee (Ref: 15839).

References

1. HPV vaccine overview – NHS n.d.
2. Public Health England. Consent: The Green Book: Chapter 2. In: The Green Book: Immunisation against infectious disease. Public Health England, 2013.
3. Griffith R. What is Gillick competence? Human Vaccines and Immunotherapeutics 2016;12:244–7.
4. Boyce T, Holmes A. Addressing Health Inequalities in the Delivery of the Human Papillomavirus Vaccination Programme: Examining the Role of the School Nurse. PLoS ONE 2012;7:e43416.
5. Brabin L, Stretch R, Roberts SA, Elton P, Baxter D, McCann R. The school nurse, the school and HPV vaccination: A qualitative study of factors affecting HPV vaccine uptake. Vaccine 2011;29:3192–6.
6. The NHS Long Term Plan n.d.
7. Chantler T, Pringle E, Bell S, Cooper R, Edmundson E, Nielsen H, et al. Does electronic consent improve the logistics and uptake of HPV vaccination in adolescent girls? A mixed-methods theory informed evaluation of a pilot intervention. BMJ Open 2020;10:38963.
8. Davis R, Campbell R, Hildon Z, Hobbs L, Michie S. Theories of behaviour and behaviour change across the social and behavioural sciences: a scoping review. Health Psychology Review 2015;9:323–44.
9. Maini R, Mounier-Jack S, Borghi J. How to and how not to develop a theory of change to evaluate a complex intervention: Reflections on an experience in the Democratic Republic of Congo. BMJ Global Health 2018;3.
10. Hivos Theory of Change – Hivos n.d.
11. Michie S, van Stralen MM, West R. The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science 2011;6:1–12.
12. Paina L, Wilkinson A, Tetui M, Ekirapa-Kiracho E, Barman D, Ahmed T, et al. Using Theories of Change to inform implementation of health systems research and innovation: Experiences of Future Health Systems consortium partners in Bangladesh, India and Uganda. Health Research Policy and Systems 2017;15:29–38.
13. Scottish Government T. DESIGNING AND EVALUATING BEHAVIOUR CHANGE INTERVENTIONS The 5 Step Approach to Evaluation. n.d.
14. Behavioural Insights Ltd. A review of optimism bias, planning fallacy, sunk cost bias and groupthink in project delivery and organisational decision making 2017:1–30.
15. West R, Michie S, Atkins L, Chadwick P, Lorencatto F. Achieving behaviour change: A guide for local government and partners. PHE Publications 2019.
16. May C. A rational model for assessing and evaluating complex interventions in health care. BMC Health Services Research 2006;6:1–11.