Program evaluation is becoming increasingly important in pediatric rehabilitation settings that adhere to FCS philosophy. This philosophy recognizes that each family is unique, that parents know their children best, and that optimal child functioning occurs within a family context. However, researchers know little about the specific evaluation activities occurring in these settings or the extent to which evaluators uphold FCS philosophy in their activities. The primary goal of this study is to examine the strengths, limitations, and consequences of current evaluation practice, including its compatibility with FCS philosophy. As a secondary goal, the study aims to understand the promise and prospects of alternative evaluation approaches that, in theory, are compatible with FCS philosophy. To address these goals, this study uses a mixed-methods approach and includes three phases. Phase 1 involves a survey of staff members involved with program evaluation at 15 Canadian pediatric rehabilitation centres. It determines the level of program evaluation occurring in these settings, verifies on the motivation for evaluation, and describes the degree to which evaluation activities are consistent with the FCS philosophy. Phase 2 involves interviews with staff members and explores the values, factors, and conditions that support and inhibit the evaluation of family-centred programs in pediatric rehabilitation settings as well as the benefits and limitations of using mainstream practices for evaluating these programs. Phase 3 then uses focus groups with staff members and interviews with parents to explore how the evaluation of family-centred programs can be improved as well as to identify the compatibility and practicality of using alternative evaluation approaches within these settings. Overall, the findings show that the amount of evaluation activities occurring within these centres is variable; that the majority of individuals working in program evaluation do not have formal training in it; and that the centres have limited resources for evaluation. The findings also demonstrate that participatory evaluation approaches are more compatible with FCS philosophy but that it might be difficult to implement such approaches given the limited resources and diverse characteristics of rehabilitation settings. In light of these circumstances, the study notes ways for improving program evaluation activities.

This study sought to better understand program evaluation capacity and competency amongst public health nurses. Program evaluation plays a vital role in public health and is an identified core competency for public health practice (Canadian Public Health Agency). In Part One, I conducted a scoping review to systematically map the current literature on this topic and to identify important areas for future research. Twenty-three articles were selected based on pre-established exclusion and inclusion criteria and the assistance of a secondary reviewer. The articles highlighted the value of program evaluation in public health and its importance as a nursing skill amidst the evolving health care sector. Themes identified included: a broader lack of public health competencies (including program evaluation) among all public health professionals; the complexities and challenges of evaluating public health interventions; and the uncertainty of what constitutes adequate evaluation competency in public health. Furthermore, my review noted inconsistent terminology to describe a public health nurse and the need for further exploration around the specific evaluation capacity of public health nurses. In Part Two, I explored the stated or expected evaluation competencies for public health nurses through a document review of relevant Canadian public health nursing core competencies, guidelines, and standards for practice. The identification of 52 stated evaluation competencies, demonstrates the assumption that public health nurses have competency and or capacity related to program evaluation and contrasts with the themes identified in my scoping review. Furthermore, the documents I reviewed included no specific reference to the Canadian Evaluation Society (CES), however some of the included content did align with the CES Program Evaluation Standards. This study demonstrates a misalignment between the discourse in the literature reviewed related to evaluation competency amongst public health nurses and the stated or assumed evaluation competencies put forth in leading public health nursing documentation. In the absence of any standardized evaluation training and preparation for public health nurses, further exploration is needed around what these broad evaluation competencies mean in practice and how they can be objectively assessed, exhibited, and better integrated into public health nursing education and evaluation capacity building activities. These questions warrant further investigation to ensure public health interventions are properly evaluated and that public health nurses have the competencies required for effective public health practice.

This thesis aimed to gain an in-depth understanding of effective program evaluation frameworks, particularly in the agriculture sector. Specifically, the analysis was focused on assessing monitoring and evaluation of public and private sector projects to assist the British Columbia Investment Agriculture Foundation (IAF), the client for this thesis, in identifying smart practices as a way to support constant improvement in their organization. This research involved conducting a literature review of the most recent and relevant literature on program evaluation, particularly works that related to nonprofit organizations in the agriculture sector, interviewing IAF staff, developing a jurisdictional scan of program evaluation frameworks in the Netherlands and New Zealand, and conducting a review of existing IAF evaluation documents to identify and discuss key themes for an effective program evaluation framework and provide examples of smart evaluation practices that may be adapted by IAF. The recommendations include integrating formative and summative evaluation practices, developing targeted programs with well-defined key performance indicators (where possible), and capitalizing on data visualization software for monitoring and reporting on project goals in real-time. / Graduate

Background In 2010, a pilot program was implemented by the Medication Assessment Consultants (MAC) to provide comprehensive medication management (CMM) services to patients in the community of Saskatoon, Saskatchewan. Clinical pharmacists working as consultants, independent of any retail pharmacy or health system organization, delivered the program. The goal of the MAC CMM service was to optimize therapeutic outcomes for individual patients through improved medication use, and to reduce the risk of adverse drug reactions (ADRs). Purpose The purpose of this study was to evaluate the independent consultant pharmacist model for delivering comprehensive medication management (CMM) services employed by the Medication Assessment Consultants (MAC) pilot program. Methods A program evaluation was performed on MAC, consisting of a document review of program materials (e.g., MAC Policy and Procedure Manual, electronic patient records) along with stakeholder interviews. The document review consisted of the collection of existing MAC documents, followed by an analysis of the information contained within each document. Patients, physicians and MAC staff were interviewed using a semi-structured interview approach. Patient and physician interviews were conducted by phone and by an interviewer external to the study. Interviews were continued until saturation was reached. The MAC staff was interviewed in person by the investigator. All interviews were recorded and transcribed verbatim. Thematic analysis was used to identify common themes by having three individuals independently review each group of stakeholder interview transcripts. Results During the 17-month pilot, 53 patients were referred to MAC, mostly from family physicians (79.2%). Patients were elderly (mean 71 years) and mostly female (67.9%). On average, patients were taking 13.3 medications and had 9.2 medical conditions. For the 42 patients for whom an assessment was completed, an average of 5.1 drug therapy problems (DTPs) per patient was identified. The document review revealed that MAC did not achieve all of its internal program objectives (e.g., to generate a consistent flow of patient referrals; to promote the service to physicians and patients; and to improve medication-related short-term outcomes for patients). All three interview groups reported a high level of satisfaction and support for the program. Interviewees described various ways in which they felt that they benefitted from the program, including medication regimen optimization (patients), support in dealing with complex medication regimens (physicians), and a strong sense of personal and professional satisfaction and fulfilment (MAC staff). The evaluation of the MAC program resulted in the identification of several program strengths (e.g., a strong, well-defined patient care process; an accessible service location; MAC pharmacist mentorship and support program), along with opportunities for improvement (e.g., expanded promotional activities; administrative support for the program; implementation of a formal satisfaction survey to obtain regular feedback from key stakeholders). Conclusion The results of this study suggest the independent consultant pharmacist model for the provision of CMM services has potential to be utilized as a new service delivery model (in addition to community pharmacies and primary health care teams) to provide CMM services in the primary health care system (PHCS). The study findings have identified several strengths and opportunities for improvement, which may be useful for future attempts at implementing the CMM service model.

Leading innovation is a difficult process because it is replete with tensions and paradoxes. Innovations require leaders to switch leadership styles depending on the context and the phase of innovation. This study used two leadership theoretical frameworks, transformational and transactional leaderships, to understand the leadership behaviors used to promote and manage the process of innovation. The purpose of this study was to explore leadership behaviors exhibited by evaluation team leaders during the process of innovation. The focus of the study was on leadership behaviors and study participants are individuals who identify as evaluators who led a team of two or more evaluators. This study used Critical Incident Technique (CIT) to better understand the leadership behaviors exhibited during the process of innovation. Through semi-structured interviews, participants described a specific innovation that he/she led during the past 24 months, the actions they took to lead and support their team, outcomes and their perspectives about the process. Contradictory behaviors were exhibited at all three key stages of innovation—insight, prototype, and adoption. Leaders described both transformational and transactional leadership behaviors at all the major innovation phases. Leaders were both people and task oriented in their leadership style.

Program evaluation has carved out its value in the International Development (ID) sector as the way that International Non-Governmental Organization (INGOs) report program performance, and more importantly learn on how to improve for future programs. Its rise to prominence has been an evolution between the traditional, outcomes and participatory evaluation paradigms.   Although program evaluation is regarded as a function of management, Project Management (PM) as a discipline has been scarcely part of ID’s growth and virtually non-existent when it comes to program evaluation. In fact, although ID is a project-based sector, it largely developed its own PM practices, independent from the PM discipline. The developments of PM in ID can also be viewed in paradigm shifts from the conventional, people-centered right down to critical approaches. The only exception to this has been the creation of PM4NGOs in 2011 – a PM association geared to tailor PM knowledge for national and international Non-Governmental Organizations (NGOs).   In this backdrop, the research employs a critical literature review and data collection process to investigate if PM has contributed to program evaluation and whether program evaluation practices seen in the sector today could inform PM4NGO’s guide: PMD Pro.   The study’s findings yield that PM has fallen short in contributing to program evaluation, despite the creation of PM4NGOs. Rather, what is evidenced in the PMD Pro guide is an adherence to traditional paradigms and an inability to capture where the sector is today: in an outcomes evaluation paradigm within a people-centered PM in ID era. This study ultimately provides a series of recommendations to update PM4NGO’s contribution to program evaluation and synchronize its relevance with program evaluation practices in the sector today

African Americans have been disproportionately affected by HIV since the beginning of the epidemic and the disparities have worsened over time (CDC, 2013a). African Americans comprise about 12% of the U.S. population but represented about 44% of all new HIV infections in 2010 (CDC, 2014a). Young people (age 13–24) accounted for 26% of all new HIV infections in 2010, despite persons in this age range comprising just 17% of the population (CDC, 2014c). Young African Americans (age 13-24) are affected in particular. In 2010, they comprised 57% of infections in this age range (CDC, 2014c). Substance use is a major factor in the sexual risk taking of young people. Substance use has been associated with risky behaviors such as unprotected sex and having multiple partners, which may put one at risk for contracting sexually transmitted infections (STIs) (Kaiser Family Foundation, 2002). This may help to explain why STIs are more prevalent among 18 to 24 year olds than any other age group. Prevalence estimates suggest that young people (age 15–24) acquire half of all new STIs (Satterwhite et al., 2008). Program evaluation is a social science activity that entails “collecting, analyzing, interpreting, and communicating information about the workings and effectiveness of social programs” (p. 2) (Rossi, Lipsey, & Freeman, 2004). This dissertation is an evaluation of the Raise 5 Project, which aimed to provide HIV and substance abuse prevention services to the African American students attending Virginia Commonwealth University (VCU). The project utilized four strategies: 1) evidence based interventions, 2) peer education and awareness, 3) HIV testing and counseling, and 4) a social marketing campaign. Two forms of evaluation, process and outcome, were utilized to assess the Raise 5 Project. Process evaluation assesses service utilization (i.e., program engagement) and program organization. This process evaluation sought to determine if the project’s four strategies operated as intended. Outcome evaluation assesses the intended outcomes of a program. This outcome evaluation examined if participants in the project’s evidence based interventions had reduced substance use and risky sexual behaviors.

The current study used a longitudinal design to evaluate a day treatment program for children with serious emotional disorder located in Richmond, Virginia (N = 101). Child and Adolescent Functional Assessment Scale (Hodges, 1984, 1999) results were analyzed from children who had attended the program for at least six months. Repeated Measures Analysis of Variance tests indicated that children attending YDT for nine months improved their overall functioning F(2, 217) = 26.23, p = .01 as well as their functioning at school F(3,279)= 10.822, p =.01, home F(3, 244) = 6.120, p = .01, and in the community F(3, 238) = 4.158, p =.01. In addition, children improved their thinking F(2, 232) = 5.595, p = .01, behavior toward others F(3, 279)= 24.230, p = .01 and decreased the amount of self-harm F(1, 121) = 7.546, p =.01. Children began YDT with an average total CAFAS score of score of 85.90; however, after nine months of treatment, children dropped 20.40 points to an average CAFAS score of 65.50 points. Results indicate that even the most functionally impaired children in the sample demonstrated some level of improvement. Implications for social work micro and macro practice are included.

In recent years, regional accreditation and the regional accreditation associations for higher education have experienced continuing criticism and become the objects of increased scrutiny. Higher education institutions look to the accreditation process as one of the principal means of justifying their actions and activities relative to their performance and results. Since the complaints and criticism directed toward higher education have not diminished but continued, and even increased, the regional accreditation associations find themselves to be the focus of similar complaints and criticisms. In the United States, we have no national system of accreditation. We rely on the actions and activities of one of the six regional accreditation associations. Each of these associations has its own separate and, in some cases, unique accreditation processes. Each publishes its own standards, its own handbooks, its own policies and practices, its own newsletters, and its own rules and regulations for accreditation. This situation can lead to inconsistencies, and could be a source of much confusion and misunderstanding when discussing the implications of an institution being accredited within a particular region. Logic models are diagrams or visual schematics that convey relationships between program processes and outcomes. This study uses logic modeling and logic model theory as the framework for an examination of the components of accreditation within two regional accreditation associations. A systematic and detailed methodology was developed in order to construct a logic model from existing handbooks and documents. As a result, two regional logic models were constructed, as well as a combined model based upon common elements. Implications of this study include the possible construction of a national accreditation logic model if the methodology is applied in the additional regional associations. This conceptual approach could lead to more consistency in the design, communication, and application of accreditation processes. Better understanding of, and less confusion concerning, the myriad of activities and processes required in a successful regional accreditation could lead to better, more effective, and more meaningful accreditation activities and results. This, in turn, could generate true growth and improvement in the actions, activities, and results achieved by our higher education institutions.

Thesis advisor: Lauren Saenz / Stakeholder involvement is widely acknowledged to be an important aspect of program evaluation (Mertens, 2007; Greene, 2005a; Brandon, 1998). However, limited work has been done to empirically study evaluators’ practices of stakeholder involvement and ways in which stakeholder involvement is affected or guided by various factors. As evaluators interact with and place value on the input of stakeholders, social, cultural, and historical backgrounds will always be infused into the context (Mertens & Wilson, 2012; MacNeil, 2005). The field of evaluation has done little to critically examine how such contexts impact evaluators’ perceptions of stakeholders and their involvement. The present study attempts to fill these gaps, focusing specifically on the relationships among evaluator identities and characteristics, evaluation models, and stakeholder involvement. Using the frameworks of critical evaluation theory (Freeman & Vasconcelos, 2010) and a theory of capital (Bourdieu, 1986), the present study utilized a sequential explanatory mixed methods approach. A sample of 272 practicing program evaluators from the United States and Canada provided quantitative survey data, while a sample of nine evaluators provided focus group and interview data. Regression analyses and thematic content analyses were conducted. Findings from the quantitative strand included relationships between: (1) measures of individualism-collectivism and stakeholder involvement outcomes, (2) contextual evaluation variables and stakeholder involvement outcomes, (3) use of use, values or social justice branch evaluation models and stakeholder involvement outcomes, and (4) whether the evaluator identified as a person of color and the diversity of involved stakeholders. Findings from the qualitative strand demonstrated the role of dominant frameworks of evaluation serving to perpetuate systems of power. Participating evaluators revealed ways in which they feel and experience systems of power acting on them, including participation in, recognition of, and responses to oppression. The qualitative strand showed that evaluation models may be used to help recognize power dynamics, but that they are also used to reinforce existing power dynamics. Implications and recommended directions for future research are discussed. / Thesis (PhD) — Boston College, 2015. / Submitted to: Boston College. Lynch School of Education. / Discipline: Educational Research, Measurement and Evaluation.