1 volume

This paper investigates the relationship between student-led consulting teams and community organizations as one part of Eller Social Innovation (ESI). The research conducted to analyze the relationship between these entities is most similar to program evaluation. Program evaluation has become commonplace in the nonprofit sector and it is important method for these organizations to establish the value of all of their activities (Carman, 2007). Through data collected from interviews and surveys of community organizations, this article seeks to understand the efficacy of student-led consulting projects and determine the effect that they have on these organizations. The answer to this question will inform stakeholders in ESI about the potential value that further collaboration with community organizations could create for both parties.

Mediational analysis quantifies proposed causal mechanisms through which treatments act on outcomes. In the presence of clustered data, conventional multiple regression mediational methods break down, requiring the use of hierarchical linear modeling techniques. As an additional consideration, nonlinear relationships in multilevel mediation models require unique specifications that are ignored if modeled linearly. Improper specification of nonlinear relationships can lead to a consistently overestimated mediated effect. This has direct implications for inferences regarding intervention causality and efficacy. The current investigation proposes a nonlinear multilevel mediation model to account for nonlinear relationships in clustered data. A simulation study is proposed to compare the statistical performance of the proposed nonlinear multilevel mediation model with that of conventional methods. / text

Despite the wealth of information on suicide prevention issues and the widespread implementation of suicide prevention strategies, program evaluation efforts have been limited. Lack of sound program evaluation remains one of the most significant barriers to identification and implementation of effective intervention and prevention strategies. The purpose of this study was two-fold: to conduct a summative evaluation of the gatekeeper suicide prevention program implemented at the Toronto Transit Commission, and to concomitantly, appraise the efficacy and effectiveness of the Kirkpatrick evaluation model as an analytical framework to guide suicide prevention program evaluations. The study used a two-phase, sequential mixed-method approach of converging quantitative and qualitative methodologies. The quantitative study employed a repeated measures design and examined the immediate and long-term effects of the gatekeeper program on attitudes, knowledge, intervention abilities. The qualitative study consisted of semi-structured interviews and explored participants’ effective and utility reactions to the gatekeeper training program. The results of this study indicated safeTALK and suicideAWARE training programs increased participants’ knowledge of suicide and suicidal behaviour, enhanced positive attitudes toward the suicidal individual, suicide intervention, and improved intervention skills. The empirical findings from this study support the premise that the Kirkpatrick evaluation model could be adapted for use in gatekeeper program evaluations. The model provides a highly relevant, well-rounded, rigorous approach to suicide prevention program evaluations.

Despite the wealth of information on suicide prevention issues and the widespread implementation of suicide prevention strategies, program evaluation efforts have been limited. Lack of sound program evaluation remains one of the most significant barriers to identification and implementation of effective intervention and prevention strategies. The purpose of this study was two-fold: to conduct a summative evaluation of the gatekeeper suicide prevention program implemented at the Toronto Transit Commission, and to concomitantly, appraise the efficacy and effectiveness of the Kirkpatrick evaluation model as an analytical framework to guide suicide prevention program evaluations. The study used a two-phase, sequential mixed-method approach of converging quantitative and qualitative methodologies. The quantitative study employed a repeated measures design and examined the immediate and long-term effects of the gatekeeper program on attitudes, knowledge, intervention abilities. The qualitative study consisted of semi-structured interviews and explored participants’ effective and utility reactions to the gatekeeper training program. The results of this study indicated safeTALK and suicideAWARE training programs increased participants’ knowledge of suicide and suicidal behaviour, enhanced positive attitudes toward the suicidal individual, suicide intervention, and improved intervention skills. The empirical findings from this study support the premise that the Kirkpatrick evaluation model could be adapted for use in gatekeeper program evaluations. The model provides a highly relevant, well-rounded, rigorous approach to suicide prevention program evaluations.

This dissertation consists of three independent essays evaluating the impact of public health interventions in two countries, Indonesia and Australia. The first two essays concern the national pro-poor health card program in Indonesia, which fully subsidises health care purchases by its recipients. In quantifying the impact of the program, the combination of propensity score matching and difference-in-differences techniques is used to address bias due to non-random assignment. Covariates to match include both variables affecting demand and eligibility and variables measuring local health care supply. Using data from the single, nationally-representative longitudinal study of Indonesian households, the program is found to have limited impact. The second essay evaluates the program??s impact on supply variables. In the absence of a wage revision or additional staff, the program lowers the incentives for health workers to maintain their public position. This is particularly true for those workers providing outpatient care as patients can not be controlled by waiting lists. The analysis finds some evidence of reduced number of full-time doctors in areas where the distribution of health cards were most extensive. The final essay uses Australian data to investigate changes in women??s preferences for cervical screening following a screening promotion campaign and a vaccination program. Discrete choice experiments were used to elicit preferences, and results from experiments conducted prior and after the interventions were compared. Several additional comparison groups were created based on spatial variations and by variation induced by randomisation. The study finds that the interventions have minor impact on how women valued various screening attributes. However, there was a general reduction in the willingness to screen, which was unexpected given the awareness campaign. Through simulation, it is shown that an effective way to increase the screening rate is through encouraging a more active role of the provider.

This dissertation consists of three independent essays evaluating the impact of public health interventions in two countries, Indonesia and Australia. The first two essays concern the national pro-poor health card program in Indonesia, which fully subsidises health care purchases by its recipients. In quantifying the impact of the program, the combination of propensity score matching and difference-in-differences techniques is used to address bias due to non-random assignment. Covariates to match include both variables affecting demand and eligibility and variables measuring local health care supply. Using data from the single, nationally-representative longitudinal study of Indonesian households, the program is found to have limited impact. The second essay evaluates the program??s impact on supply variables. In the absence of a wage revision or additional staff, the program lowers the incentives for health workers to maintain their public position. This is particularly true for those workers providing outpatient care as patients can not be controlled by waiting lists. The analysis finds some evidence of reduced number of full-time doctors in areas where the distribution of health cards were most extensive. The final essay uses Australian data to investigate changes in women??s preferences for cervical screening following a screening promotion campaign and a vaccination program. Discrete choice experiments were used to elicit preferences, and results from experiments conducted prior and after the interventions were compared. Several additional comparison groups were created based on spatial variations and by variation induced by randomisation. The study finds that the interventions have minor impact on how women valued various screening attributes. However, there was a general reduction in the willingness to screen, which was unexpected given the awareness campaign. Through simulation, it is shown that an effective way to increase the screening rate is through encouraging a more active role of the provider.

As the growth of online programs in higher education accelerated across the nation, concerns have been addressed regarding the quality of online programs. The evaluation of programs for quality assurance has become an important component of successful online programs (Kindgren, Nilsson, and Wiklund, 2012, Martinez, Liu, Watson, and Bichelmeyer, 2006). Deggs and Weaver (2009) stated that student perception is "essential to improving the quality of instruction in higher education" (p. 41). In the online program evaluation literature, majority of evaluation studies commonly collected students' perceptions of their learning outcomes and their learning experiences through survey questionnaires. Few studies had been conducted on using students' reflections in e-portfolios for program evaluation in higher education. The purpose of this study was to explore how students' reflections in their e-portfolio can be used as a mechanism for program evaluation. This study employed a qualitative content analysis method. Seventy students' reflections in their e-portfolios of an online master's program were collected. Three types of reflections in e-portfolios have been reviewed and analyzed: (1) students' reflections on their previous approaches to the five tasks of instructional design including design, development, utilization, management, and evaluation; (2) the students' reflections on their current approaches to these five instructional design tasks; and (3) the perceptions on their learning experiences in this online master's program. Two overarching themes emerged from the students' reflections including students' perceptions on their learning outcomes and their learning experiences in the program. The results of the study indicated that students' reflections in their e-portfolios provide another dimension of qualitative information for program evaluation with respect to students' perceptions, and this qualitative data can be used in conjunction with other methods of data collection to conduct a comprehensive evaluation of online programs. / Ph. D.

This proposed study examines whether different levels of informative feedback can change students’ achievement goal orientation. A factorial repeated-measures MANOVA will be conducted to investigate changes in levels of trichotomous goals and reading comprehension scores of community college students over one academic semester. I hypothesize that both scores will be responsive to the level of informative performance feedback. Participants who get more information about their performance and a strategy to solve similar tasks will show significantly greater increases in mastery and performance-approach goals and decreases in performance-avoidance goals. In addition, effects of demographic variables on the goal-regulation outcomes (e.g., sex and ethnicity) will be examined. Implications for future research and educational applications are presented. This report also includes an evaluation plan which details the components of the trust building program, a model for the program, and the proposed method to measure the reported outcomes. / text

Program evaluation is important for improving the activities and services of nonprofit organizations. However, the emphasis on program evaluation has created new challenges and uncertainties for nonprofit organizations. These concerns of evaluation capacity, quality and use are of particular importance in nonprofit contexts where resources are already stretched, and where staff size and capacity are limited. As such, this research study focuses on a specific niche of the nonprofit sector, exploring how stakeholders experience program evaluation in a nonprofit organization, which (a) operates exclusively in one region, (b) is low-resourced, and (c) identifies as providing educational services to a specific community. Using a phenomenological approach to study program evaluation, the study is informed by Colaizzi’s phenomenological approach to explore stakeholders’ lived experience of evaluation in this context. The findings suggest that several components are shared among participants. Their experience is described by four themes (1) rushed and pressed for time; (2) some good will come: a sense of optimism; (3) perceived inadequacy of evaluation; and (4) dedicated, but alone and isolated. From the phenomenological analysis, three underlying contextual factors of this shared experience also emerged, including dependence on the granting system, a suboptimal structure, and individuals’ backgrounds and training. In all, this study offers contributions to evaluation practice, theory and methodological development by studying program evaluation in a rarely examined context, and by contributing empirical data about program evaluation using a methodology that has rarely been used in this domain.