This thesis is concerned with an initial assessment of the implementation of Studio Schools, a novel and highly flexible model of secondary education, in England. Responding to the methodological challenges towards evaluating a 'standard' national social programme that is encouraged to be adapted to context by local schools, the thesis also reports the development, operationalisation, and testing of a new approach towards the concept of implementation fidelity for evaluation science. The thesis commences by presenting the modern-historic foundations and challenges of the current English secondary education system that gave rise to Studio Schools, and describing the nature and objectives of the schools. This is followed by a discussion of the general challenges involved in the development and evaluation of complex social interventions and the specific challenges presented by the case of Studio Schools. The remainder of the thesis reports the development, use, and assessment of methods to overcome these challenges - with particular focus on evaluating implementation as part of process evaluations - as well as the current state of implementation in the schools. Delphi-inspired consensus methods were used in order to develop an explicit programme theory for Studio Schools where none previously existed, involving stakeholders in the theory specification process. The process demonstrated that stakeholders without a background in programme evaluation can agree to a specific and explicit theory of change after a programme was designed but prior to its evaluation. Next, a novel conceptual approach towards defining and measuring implementation fidelity was developed to translate a standard programme theory into flexible implementation measures. This approach focuses on the functions - or targeted change mechanisms - of a programme alongside its form of a given set of activities. Implementation measures were developed in the form of quantitative, paper-based questionnaires that were used to rate form- and function- focused fidelity of implementation of project-based learning (PBL) and personal coaching in schools on ordinal Likert scales. These measures were piloted and refined, and subsequently tested for their psychometric properties through the use of factor analysis in addition to established methods for determining the reliability of instruments in terms of internal consistency and inter-rater agreement. Findings show that it is feasible to monitor programme functions alongside form in process evaluations, and that the validity and reliability of measures based on this approach can be established using common psychometric methods. The measures developed earlier in the thesis were used by the doctoral candidate as well as teachers and students to rate the current state of implementation practices of PBL and coaching in Studio Schools was monitored over a period of four months in four participating schools. Ratings were based on observations made in-vivo or based on video- and audio recordings made during repeated visits to the schools. Quantitative implementation scores were calculated per rater group for PBL and coaching by aggregating ratings given to individual sessions, and were compared within and between schools. Spearman's correlation coefficients were calculated to assess correlation between form- and function-focused fidelity scores. The results of this study imply that implementation in Studio Schools likely varies substantially between individual schools and can be improved in all of them, but also suggest that the model could be evaluated for its effectiveness as long as implementation and process are carefully monitored. The additions of this thesis to the evaluation literature are considered, as well as its strengths and limitations and implications for practice and research.
| Identifer | oai:union.ndltd.org:bl.uk/oai:ethos.bl.uk:724952 |
| Date | January 2016 |
| Creators | Urk, Felix van |
| Contributors | Montgomery, Paul ; Bonell, Christopher |
| Publisher | University of Oxford |
| Source Sets | Ethos UK |
| Detected Language | English |
| Type | Electronic Thesis or Dissertation |
| Source | https://ora.ox.ac.uk/objects/uuid:5c73b308-efbf-48aa-91b1-f8c06b7eb885 |
Page generated in 0.0019 seconds