University of South Florida
Design and Validation of an Evaluation Checklist for Organizational Readiness for Evaluation Capacity Development
Abstract
dc:descriptionEvaluation capacity development (ECD) has been acknowledged as a system of processes to help organizations achieve sustainable evaluation practice. Examining the existing evaluation capacity of an organization before starting an ECD process is necessary and will increase the possibilities of success, determined by the establishment or strengthening of an evaluation system into the organization. In response to this need, this study involved the designing of the Organizational Readiness for Evaluation Capacity Development (ORECD) checklist and its initial validation, using a mixed method research design. The study was conducted in four phases, including: (a) the design of the ORECD checklist based on a review of the literature; (b) a review of the ORECD checklist by five experts to obtain face and content validity evidences, with emphasis on relevance and clarity of the items and how well the items fit the corresponding component; (c) a pretesting about the appropriateness of the wording of the items and format of the ORECD checklist by a sample of doctoral graduate students with formal training in evaluation and professional evaluators; and (d) a field study with 32 nonprofit organizations to determine the utility and benefits of using the ORECD checklist and potential improvements to the instrument. This phase generated information about the psychometric properties as well as consequential validity evidence. Findings indicated that the ORECD checklist has great potential to determine the readiness of an organization to develop evaluation capacity, as demonstrated by the feedback received from various groups of participants, establishing face, content, and consequential validity. Results from the psychometric analysis showed correlations that, for the most part, suggested that the components are measuring aspects of the same construct. In addition, the alpha for most of the components supported the reliability of the ORECD checklist. The two components with alphas close to but below .70 required modifications in order to improve their reliability. Also, it was necessary to modify or reword some of the items. Ongoing efforts should provide information about how the changes made to the ORECD checklist are working and additional validity evidences as the one that can be obtained through factor analysis. This will allow the exploration of the underlying structure of the ORECD checklist and its components. It is expected that the ORECD checklist can be a contribution to the body of literature about ECD helping to address organizational readiness in order to support and sustain the development of evaluation capacity within organizations.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy (Ph.D.)
- Level thesis:degree_level
- Dissertation
- Discipline thesis:degree_discipline
- Curriculum & Instruction
- Grantor
- University of South Florida
- Year dc:date
- 2014
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Walker-Egea, Connie F.
- Contributors dc:contributor
-
- Liliana Rodríguez-Campos, Ph.D.
- John Ferron, Ph.D.
- Robert Dedrick, Ph.D.
- Roger Boothroyd, Ph.D.
Subjects
dc:subject × 6Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.usf.edu/etd/5399