PURPOSE. This thesis considers the best way to address the challenges faced by educators, institutions and funding bodies trying to not only develop and implement educational technology successfully but tackle the challenge of understanding and evidencing what works (and what does not) and why. The aim of the research was to find and validate an evaluation method that provided usable and useful evidence. APPROACH. A range of evaluations were undertaken to elicit the strengths and weaknesses of different approaches, augmented by drawing upon the experiences and outcomes published by others. An analysis of the issues was made and significance of the problem established. The problem being premature timing, unsuitable models, rapid change, complex implementation chains, inconsistent terminology, ideology and marketisation. A tailored realist evaluation framework was proposed as an alternative method and it was tested to evaluate an institutional lecture capture (LC) initiative. FINDINGS. The theory-driven realist approach provided a level of abstraction that helped gather evidence about wider influences and theories of potential future impact of the LC programme and its linked policy. It proved valuable in generating real and practical recommendations for the institution, including what more could be done to improve uptake and support embedding in teaching and learning, from practice, policy and technological points of view. It identified some unanticipated disadvantages of LC as well determining how and when it was most effective. PRACTICAL IMPLICATIONS. A Realist Evaluation of Technology Initiative (RETI) framework has been produced as tool to aid the rapid adoption of the approach. Recommendations for future research and seven guiding principles have been proposed to encourage the formation of a community of realist evaluative researchers in educational technology. ORIGINALITY/VALUE. The rigorous application of a tailored realist evaluation framework (RETI) for educational technology (including the development of two Domain Reference Models) is the primary contribution to new knowledge. This research is significance because it has potential to enable the synthesis of evaluation findings within the sector. This will enable an evidence-base of what works, for whom, in which contexts and why, ultimately benefiting policy-makers and practitioners to support better informed decision making and investment in education.

Yes / This article recounts our experience of eliciting, cataloguing and prioritizing conjectured Context-Mechanism-Outcome configurations at the outset of a realist evaluation, to provide new insight into how Context-Mechanism-Outcome configurations can be generated and theorized. Our construction of Context-Mechanism-Outcome configurations centred on how, why and in what circumstances teamwork was impacted by robotic surgery, rather than how and why this technology improved surgical outcomes as intended. We found that, as well as offering resources, robotic surgery took away resources from the theatre team, by physically reconfiguring the operating theatre and redistributing the surgical task load, essentially changing the context in which teamwork was performed. We constructed Context-Mechanism-Outcome configurations that explain how teamwork mechanisms were both constrained by the contextual changes, and triggered in the new context through the use of informal strategies. We conclude by reflecting on our application of realist evaluation to understand the potential impacts of robotic surgery on teamwork.

Background: Improving care for people living with dementia when they are admitted to hospital is a national priority. Interventions have been designed and implemented to support staff to improve how they provide care to patients with dementia. However, there is limited understanding of how these interventions work in practice and what the outcomes are for patients and their family carers. Objective: To develop, test, and refine a theory-driven explanation of what supports hospital staff to provide dementia-friendly care and with what outcomes for people living with dementia and their carers. Method: A two-phase study design employing realist methodology. Phase one was a realist review which combined evidence from stakeholder interviews and literature searches. Phase two used realist evaluation to analyse data collected from two NHS Hospital Trusts in the East of England to test the theory developed in phase one. Findings: Initial scoping in the realist review identified three candidate theories which structured the literature searches and analysis. Six related context-mechanism-outcome configurations were identified and collectively made the initial programme theory. The review found that single strategies, such as dementia awareness training, would not on their own change how staff provide care for patients with dementia. An important context was for staff to understand behaviour as a form of communication. Organisational endorsement for dementia care and clarity in staff roles was important for staff to recognise dementia care as a legitimate part of their work. The realist evaluation refined the programme theory. While the study sites had applied resources for patients with dementia differently, there were crosscutting themes which demonstrated how key mechanisms and contexts influenced staff actions and patient outcomes. When staff were allocated time to spend with patients and drew on their knowledge of the patient with dementia and dementia care skills, staff could provide care in ways that reassured patients and recognised their personhood. However, accepted organisational and social norms for care practices influenced whether staff considered providing skilled dementia care was an important contribution to the work on the ward. This impacted on how staff prioritised their work, which influenced whether they recognised and addressed patient needs such as pain or hunger, made attempts to reduce distress, and if patients and carers considered they were listened to. Organisational focuses, such as risk management, influenced how patient need was defined and how staffing resources were allocated. Staff commitment to continuing in dementia care was influenced by whether or not they valued dementia care as skilled work. Discussion: Single strategies, such as the use of dementia awareness training, will not on their own improve the outcomes for patients with dementia when they are admitted to hospitals. In addition, attention needs to be paid to the role of senior managers and their knowledge of dementia to support staff to provide care in ways that recognise the needs of the person. The way dementia care is valued within an organisation has implications for how resources are organised and how staff consider their role in providing dementia care. Evidence from observations demonstrated that when staff are supported to provide good dementia care, patients experienced positive outcomes in terms of their needs being addressed and reducing distress. Dementia care needs to be recognised as skilled work by the staff and the organisation.

Yes / Background Technology has the potential to provide support for self-management to people with congestive heart failure (CHF). This paper describes the results of a realist evaluation of the SMART Personalised Self-Management System (PSMS) for CHF. Methods The PSMS was used, at home, by seven people with CHF. Data describing system usage and usability as well as questionnaire and interview data were evaluated in terms of the context, mechanism and outcome hypotheses (CMOs) integral to realist evaluation. Results The CHF PSMS improved heart failure related knowledge in those with low levels of knowledge at baseline, through providing information and quizzes. Furthermore, participants perceived the self-regulatory aspects of the CHF PSMS as being useful in encouraging daily walking. The CMOs were revised to describe the context of use, and how this influences both the mechanisms and the outcomes. Conclusions Participants with CHF engaged with the PSMS despite some technological problems. Some positive effects on knowledge were observed as well as the potential to assist with changing physical activity behaviour. Knowledge of CHF and physical activity behaviour change are important self-management targets for CHF, and this study provides evidence to direct the further development of a technology to support these targets.

Yes / Objective: There is increasing interest in using artificial intelligence (AI) in pathology to improve accuracy and efficiency. Studies of clinicians' perceptions of AI have found only moderate acceptability, suggesting further research is needed regarding integration into clinical practice. This study aimed to explore stakeholders' theories concerning how and in what contexts AI is likely to become integrated into pathology. Materials and Methods: A literature review provided tentative theories that were revised through a realist interview study with 20 pathologists and 5 pathology trainees. Questions sought to elicit whether, and in what ways, the tentative theories fitted with interviewees' perceptions and experiences. Analysis focused on identifying the contextual factors that may support or constrain uptake of AI in pathology. Results: Interviews highlighted the importance of trust in AI, with interviewees emphasizing evaluation and the opportunity for pathologists to become familiar with AI as means for establishing trust. Interviewees expressed a desire to be involved in design and implementation of AI tools, to ensure such tools address pressing needs, but needs vary by subspecialty. Workflow integration is desired but whether AI tools should work automatically will vary according to the task and the context. Conclusions: It must not be assumed that AI tools that provide benefit in one subspecialty will provide benefit in others. Pathologists should be involved in the decision to introduce AI, with opportunity to assess strengths and weaknesses. Further research is needed concerning the evidence required to satisfy pathologists regarding the benefits of AI. / This work was undertaken as part of the National Pathology Imaging Co-operative. National Pathology Imaging Co-operative, NPIC (Project no. 104687) is supported by a £50m investment from the Data to Early Diagnosis and Precision Medicine strand of the government’s Industrial Strategy Challenge Fund, managed and delivered by UK Research and Innovation (UKRI).

Yes / Background: National audits aim to reduce variations in quality by stimulating quality improvement. However, varying provider engagement with audit data means that this is not being realised. Aim: The aim of the study was to develop and evaluate a quality dashboard (i.e. QualDash) to support clinical teams’ and managers’ use of national audit data. Design: The study was a realist evaluation and biography of artefacts study. Setting: The study involved five NHS acute trusts. Methods and results: In phase 1, we developed a theory of national audits through interviews. Data use was supported by data access, audit staff skilled to produce data visualisations, data timeliness and quality, and the importance of perceived metrics. Data were mainly used by clinical teams. Organisational-level staff questioned the legitimacy of national audits. In phase 2, QualDash was co-designed and the QualDash theory was developed. QualDash provides interactive customisable visualisations to enable the exploration of relationships between variables. Locating QualDash on site servers gave users control of data upload frequency. In phase 3, we developed an adoption strategy through focus groups. ‘Champions’, awareness-raising through e-bulletins and demonstrations, and quick reference tools were agreed. In phase 4, we tested the QualDash theory using a mixed-methods evaluation. Constraints on use were metric configurations that did not match users’ expectations, affecting champions’ willingness to promote QualDash, and limited computing resources. Easy customisability supported use. The greatest use was where data use was previously constrained. In these contexts, report preparation time was reduced and efforts to improve data quality were supported, although the interrupted time series analysis did not show improved data quality. Twenty-three questionnaires were returned, revealing positive perceptions of ease of use and usefulness. In phase 5, the feasibility of conducting a cluster randomised controlled trial of QualDash was assessed. Interviews were undertaken to understand how QualDash could be revised to support a region-wide Gold Command. Requirements included multiple real-time data sources and functionality to help to identify priorities. Conclusions: Audits seeking to widen engagement may find the following strategies beneficial: involving a range of professional groups in choosing metrics; real-time reporting; presenting ‘headline’ metrics important to organisational-level staff; using routinely collected clinical data to populate data fields; and dashboards that help staff to explore and report audit data. Those designing dashboards may find it beneficial to include the following: ‘at a glance’ visualisation of key metrics; visualisations configured in line with existing visualisations that teams use, with clear labelling; functionality that supports the creation of reports and presentations; the ability to explore relationships between variables and drill down to look at subgroups; and low requirements for computing resources. Organisations introducing a dashboard may find the following strategies beneficial: clinical champion to promote use; testing with real data by audit staff; establishing routines for integrating use into work practices; involving audit staff in adoption activities; and allowing customisation. Limitations: The COVID-19 pandemic stopped phase 4 data collection, limiting our ability to further test and refine the QualDash theory. Questionnaire results should be treated with caution because of the small, possibly biased, sample. Control sites for the interrupted time series analysis were not possible because of research and development delays. One intervention site did not submit data. Limited uptake meant that assessing the impact on more measures was not appropriate. Future work: The extent to which national audit dashboards are used and the strategies national audits use to encourage uptake, a realist review of the impact of dashboards, and rigorous evaluations of the impact of dashboards and the effectiveness of adoption strategies should be explored. Study registration: This study is registered as ISRCTN18289782. / This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 12. See the NIHR Journals Library website for further project information.

Yes / To capture stakeholders’ theories concerning how and in what contexts robot-assisted surgery becomes integrated into routine practice. A literature review provided tentative theories that were revised through a realist interview study. Literature-based theories were presented to the interviewees, who were asked to describe to what extent and in what ways those theories reflected their experience. Analysis focused on identifying mechanisms through which robot-assisted surgery becomes integrated into practice and contexts in which those mechanisms are triggered. Nine hospitals in England where robot-assisted surgery is used for colorectal operations. Forty-four theatre staff with experience of robot-assisted colorectal surgery, including surgeons, surgical trainees, theatre nurses, operating department practitioners and anaesthetists. Interviewees emphasised the importance of support from hospital management, team leaders and surgical colleagues. Training together as a team was seen as beneficial, increasing trust in each other’s knowledge and supporting team bonding, in turn leading to improved teamwork. When first introducing robot-assisted surgery, it is beneficial to have a handpicked dedicated robotic team who are able to quickly gain experience and confidence. A suitably sized operating theatre can reduce operation duration and the risk of de-sterilisation. Motivation among team members to persist with robot-assisted surgery can be achieved without involvement in the initial decision to purchase a robot, but training that enables team members to feel confident as they take on the new tasks is essential. We captured accounts of how robot-assisted surgery has been introduced into a range of hospitals. Using a realist approach, we were also able to capture perceptions of the factors that support and constrain the integration of robot-assisted surgery into routine practice. We have translated these into recommendations that can inform future implementations of robot-assisted surgery.

This thesis evaluates the legacy/impact of the London 2012 Games for a non-hosting sub-region Leicestershire, principally through evaluations of four legacy-programmes (i.e. the Workplace Challenge, Get Set, Sport Makers, and Games Makers programmes), drawing conclusions about lessons learned from the Leicestershire 2012 legacy experience. The selection of Leicestershire as a non-hosting sub-region reflects the fact that little is known about the legacy/ impact of the Games in such contexts. The nature of Olympic legacy evaluation is considered as a complex, wide-ranging, and multi-staged process. This thesis thus focuses on two main areas: i) conducting a systematic review of the literature (covering the period 1996-2011) to explore and establish an understanding of the concepts of Olympic legacy , and evidenced legacy/impact of hosting the Olympics in previous Games; ii) assessing the extent to which the London 2012 Games had impacted on the changed legacy outcomes for Leicestershire through realist evaluations. Together with realist evaluations, analytic logic models and the assessment of additionality approaches are adopted in this study, focusing on the four legacy-programmes evaluations, wherein quantitative and/or qualitative methodology are utilised in order to identify the causal mechanisms that produced the anticipated/unanticipated effects in their specific contexts. This study is an empirical example of the application of the realist evaluation and assessment of additionality. It also produces an evidence base for policy analysis in order to inform stakeholders thinking regarding sustaining the legacy of the Games and any future major sporting events by identifying lessons learned for non-hosting contexts.

This thesis has sought to examine the conceptualisation(s) of the field of Olympic education identified in the English language literature, and to evaluate the planning of Olympic education in practice, specifically in relation to the preparation of Olympic education programmes and systems for the Tokyo 2020 Olympics and Paralympics. When Pierre de Coubertin introduced the modern Olympic Games, one of the ideas for the revival of the Games was to educate young people through sport. Despite Coubertin s educational philosophy, the Olympic Games have long failed to represent ideals of fair play, equal opportunity, and international harmony but being replaced by bribery, corruption, commercialism, drug use and gender discrimination instead. The IOC has strengthened the roles and mission of the Olympic bodies in particular relation to the promotion of Olympic values and Olympism through the implementation of Olympic education. As a policy aim for the Olympic Movement, the development of Olympic education programmes has become a key goal for the IOC and thus host cities/nations. Providing a concept of Olympic and Paralympic education programmes in preparation for staging the Olympic Games is a compulsory requirement for host cities and nations. However, in spite of the IOC s recent explicit and intended commitment to the development of Olympic education policies in practice, explanation of Olympic education as a concept and a set of practices is imprecise and relatively underdeveloped in the Olympic related area. In addition, there is a lack of understanding of how universal values and concepts of Olympic education are perceived and communicated in culturally diverse contexts. The thesis is divided into two related parts, which seeks to provide two fundamental contributions to knowledge in this field. Part One is focused on a meta-narrative review of the English language literature on Olympic education. The methodology of a meta-narrative review is an extension of the systematic review process and facilitates the identification of the contribution of research traditions to the phenomena under review, in this case the conceptualisation and operationalisation of Olympic education. Through the process of meta-narrative review, six research traditions were identified: educational philosophy; critical sociology; curriculum development; education psychology; development of evaluation measures; and policy analysis and evaluation. II The results of the review identified how Olympic education has been conceptualised with various unfolding storylines in different research traditions, and this analysis subsequently provided the basis for the second key element of the study in the form of templates against which to evaluate the Olympic education programmes and systems associated with Tokyo 2020. Part Two employs a case study approach and is focused on the analysis of six cases using a realist evaluation methodology, employing analytic logic models and analysis of Context-Mechanism-Outcome (CMO) configurations. This facilitates the development of explicit and/or implicit causal claims about changes brought about by Japanese Olympic education programmes. The research has also contributed to developing a critical review of Olympic education programmes in a culturally specific, non-western context. Provision of Olympic education, within the context of national legislation requiring its introduction into the school curriculum developed by various stakeholders, represents a unique and culturally specific context for its study. Not only its education system, but also the cultural and historical values embedded within Japanese Olympic education programmes derive from the Japanese understanding of Olympism and universal Olympic values based on the Japanese values such as harmony, in particular applied in the effort in the recovery from national disasters, moral values learned from Judo and physical education, and Japanese ways of expressing hospitality. Thus, this case study of Tokyo 2020 acts as an exemplar in the diffusing of ways of developing and delivering the benefits of Olympic education programmes in culturally specific context.

Yes / Background: In the United Kingdom, stroke is the most significant cause of adult disability. Stroke survivors are frequently left with physical and psychological changes that can profoundly affect their functional ability, independence, and social participation. Research suggests that long-term, intense, task- and context-specific rehabilitation that is goal-oriented and environmentally enriched improves function, independence, and quality of life after a stroke. It is recommended that rehabilitation should continue until maximum recovery has been achieved. However, the increasing demand on services and financial constraints means that needs cannot be met through traditional face-to-face delivery of rehabilitation. Using a participatory design methodology, we developed an information communication technology–enhanced Personalized Self-Managed rehabilitation System (PSMrS) for stroke survivors with integrated insole sensor technology within an “intelligent shoe.”. The intervention model was based around a rehabilitation paradigm underpinned by theories of motor relearning and neuroplastic adaptation, motivational feedback, self-efficacy, and knowledge transfer. Objective: To understand the conditions under which this technology-based rehabilitation solution would most likely have an impact on the motor behavior of the user, what would work for whom, in what context, and how. We were interested in what aspects of the system would work best to facilitate the motor behavior change associated with self-managed rehabilitation and which user characteristics and circumstances of use could promote improved functional outcomes. Methods: We used a Realist Evaluation (RE) framework to evaluate the final prototype PSMrS with the assumption that the intervention consists of a series of configurations that include the Context of use, the underlying Mechanisms of change and the potential Outcomes or impacts (CMOs). We developed the CMOs from literature reviews and engagement with clinicians, users, and caregivers during a series of focus groups and home visits. These CMOs were then tested in five in-depth case studies with stroke survivors and their caregivers. Results: While two new propositions emerged, the second importantly related to the self-management aspects of the system. The study revealed that the system should also encourage independent use and the setting of personalized goals or activities. Conclusions: Information communication technology that purports to support the self-management of stroke rehabilitation should give significant consideration to the need for motivational feedback that provides quantitative, reliable, accurate, context-specific, and culturally sensitive information about the achievement of personalized goal-based activities.