Abstract not available.

The question of evaluation capacity in governmental organizations has been studied empirically in only a limited fashion in recent years. The actual dimensions of evaluation capacity, or what evaluation capacity might look like in an organization, have not previously been identified systematically in the literature. This study sought to identify the key dimensions of evaluation capacity in Canadian federal government organizations as well as the manageable steps required to move from low to high capacity for each of these. The data collection approach was articulated around three phases. The first phase sought to identify the key dimensions of evaluation capacity in Canadian federal government organizations and characterize minimal and exemplary performance on each dimension through the development of a framework. This was achieved through a literature review and semi-structured key informant interviews. The second phase focused on refining and clarifying these dimensions by using key informant interviews. The third and final phase sought to validate the revised framework. This was achieved through its practical application in four federal government organizations and semi-structured interviews with those involved in applying the framework. The study concluded that evaluation capacity in Canadian federal government departments and agencies can be described through six main dimensions, each one broken down into further sub-dimensions. The capacity of these organizations on each of these dimensions and sub-dimensions can be assessed using four levels: low, developing, intermediate, and exemplary. The study found that organizations vary in terms of their capacity from one dimension to the next, and indeed, from one sub-dimension to the next. This study makes an important contribution to the body of knowledge on evaluation capacity; although much has been published in the evaluation literature on evaluation capacity building, the actual characteristics and attributes of evaluation capacity itself have not before been defined and described based on empirical data. Continuing research may focus on expanding the scope of the framework, while the main practical implication of this study is the use of the framework as an instrument of evaluation capacity for government organizations.

This exploratory research compares Q-matrices related to two multiple choice tests in mathematics. These Q-matrices are elaborated in three different conditions. In the first one, only the items are available. In the second one, the items are presented as set together with a factor analysis. In the third condition, the item difficulty and discrimination parameters are provided to the experts as well as the distractors' analysis results. The cognitive model RUM (Reparametrized Unified Model) is used to compare the cognitive structure of the different matrices related to the same test. The form of the matrices, the agreement between the experts and the parameters estimated with the RUM are compared. The results tend to show that condition 2 cognitive structure is stronger for both tests. The difference with conditions 1 or 3 is, however, not very large. The third condition leads the experts to pay attention to non relevant aspects of the answering process and leads to the worst quality matrices.

Fairness is a perennial ideal in educational assessment. For most of the 20th century, objective tests were viewed as a means of bypassing teachers' inconsistencies in grading, and fairness was defined in technical terms for the development and administration of tests. As the 21 st century opened, the limitations of standardized tests for supporting student learning had become apparent, and interest in the pedagogical potential of classroom assessment was renewed. Although teachers' assessment decisions can have profound, long-term effects for students, most of the guidance for fair assessment still focuses on standardized testing. This study was motivated by the need for a better understanding of fairness in the classroom environment where the primary purpose of assessment is ideally to support learning. I approached my research from the perspective of critical pragmatism, which assumes that diverse forms of knowledge are valuable, including ethical and practical knowledge. I drew on the practical wisdom (phronesis) of experienced teachers to understand fairness specifically in the assessment of writing in English Language Arts in Grades 7 to 12 in Ontario. Practical wisdom is a responsive network that combines theoretical and technical knowledge, moral beliefs and ethical knowledge, experience, personal characteristics, and an understanding of the particulars of practice. I began my research by immersing myself in the literature, investigating the educational context, tailoring my research tools, and purposefully selecting teachers to participate based on their education and experience. Six participants responded in writing to vignettes that illustrated fairness dilemmas, and we discussed their recommendations for fair assessment during follow-up interviews. Insight that emerged from this process is that some strategies contribute to the fairness of classroom assessment, such as developing a constructive learning environment, sharing expectations and criteria, and offering students relevant opportunities to learn and demonstrate their learning. At the crux, however, fairness rests in a teacher's ability to understand students and their learning, and to reflect about the ongoing interactions and decisions involved in the use of assessment for learning.

Le desir de comprendre l'origine de la disparite des pratiques d'evaluation des apprentissages en salle de classe et des incoherences qui souvent les caracterisent est le point de depart de cette recherche. Force fut de constater que l'absence d'un modele de qualite des resultats d'evaluation coherent avec la nature et le contexte des apprentissages est en grande partie responsable du laxisme observe. Qu'ils tirent leur origine du paradigme traditionnel de la psychometrie ou de celui plus moderne de l'edumetrie, les elements concourant a des resultats justes ont ete regroupes en quatre composantes: la pertinence de la tache d'evaluation par rapport aux habiletes visees, sa coherence avec l'activite pedagogique, sa transparence a l'egard de l'eleve et l'absence de desavantage circonstanciel. Tous ces elements, retenus pour leur apport a l'obtention de resultats justes, c'est-a-dire pour situer correctement un eleve dans son developpement, forment le concept de justesse des resultats. Ce dernier a ete confronte a la realite de la salle de classe, en collaboration avec cinq enseignants d'ecoles elementaires de l'Ontario. De l'avis des participants, les quatre composantes du concept de justesse sont toutes pertinentes a l'obtention de resultats justes. Cependant, cette conception de l'evaluation fait ressortir comme etant problematiques certaines situations d'evaluation en salle de classe, comme l'evaluation de taches effectuees en equipe et celle d'habiletes multiples enseignees de maniere integree. La discussion tente d'etablir si ces faits sont dus au manque de realisme du concept de justesse ou s'ils font plutot ressortir des situations d'evaluation en salle de classe menant a des resultats d'evaluation de pietre qualite. L'experience menee ici ne suffit pas a etablir la viabilite du critere de justesse comme modele de qualite des resultats d'evaluation des apprentissages en salle de classe. Toutefois, cette experience suggere fortement que cette organisation simplifiee de la realite est compatible avec la nature contemporaine de l'apprentissage et rend bien compte de la complexite de son evaluation. De ce point de vue, elle s'avere superieure aux criteres de validite et de fidelite presentement utilises pour guider les enseignants dans leurs interventions d'evaluation en salle de classe.

The Public Service has undergone widespread reorganizations which have had a considerable impact on human resources and, in order to achieve a continuity of services, the employees have been required to take training courses. Consequently, there has been a marked increase in popularity for courses of short duration and these courses are becoming more profitable from the economic point of view. Are they, however, in terms of the pedagogy? The evaluation of the courses given by Training and Development Canada relies on the Kirkpatrick's training program evaluation model. This model proposes the evaluation of training at four levels: (1) reaction, (2) learning, (3) behavior and (4) results. Kirkpatrick claims the existence of a positive relationship between these levels. The development of measuring tools has allowed for the measurement of the first three levels of the model for four selected courses. By using a sample of 77 federal public servants, the assumption of a positive relationship has been established, with the help of correlation coefficients, for the Orientation to Financial Management course. However, this assumption has not been established for the courses Negotiating Skills, Effective Presentations and Writing Memos and Letters.

L'examen de la fidélité et de l'efficacité des méthodes repose sur des données réelles, les résultats à un test de mathématiques de l'American College Testing Program (ACTP). Pour étudier l'efficacité des méthodes, nous avons déterminé quels items avaient un fonctionnement différentiel non négligeable. Pour ce faire, nous nous sommes servis d'un échantillon de 40 000 sujets qui constitue, à toutes fins utiles, la population d'origine. Les hommes y forment le groupe de référence; les femmes, le groupe focalisé. Ont été identifiés comme ayant un fonctionnement différentiel non négligeable tous les items dont la différence de difficulté standardisée est égale ou supérieure à 0,05 en valeur absolue, après arrondissement au centième le plus proche. Les items devaient également présenter un delta de Mantel-Haenszel ou une différence de difficulté standardisée modifiée qui les rangent parmi les items avec les indices les plus grands. De plus, ils devaient présenter des différences de difficulté standardisée qui sont plus grandes que celles observées à l'intérieur du groupe de référence et du groupe focalisé. Á partir des analyses effectuées, il appert que la méthode de standardisation modifiée offre des indices dont la stabilité est un peu moins grande que celle des deltas de Mantel-Haenszel. Malgré cela, sa capacité de détecter des items qui ont un fonctionnement différentiel se compare avantageusement à la méthode du delta de Mantel-Haenszel. Quelle que soit la méthode à l'étude, la stabilité des indices et leur capacité de détecter des items qui ont un fonctionnement différentiel non négligeable varient en fonction de la taille des échantillons. Règle générale, stabilité et efficacité tendent à augmenter avec l'accroissement du nombre de sujets dans les échantillons. Toutes les méthodes se comportent de façon analogue lorsqu'on se base sur des centiles pour déterminer quels items ont un fonctionnement différentiel dans les échantillons de taille réduite. Il en résulte peu de différence entre les méthodes pour des échantillons de même grandeur. Lorsqu'on se base sur des critères fixes, la méthode du khi carré de Mantel-Haenszel et la méthode de régression logistique affichent des tendances à l'opposé de la méthode du delta de Mantel-Haenszel et de la méthode de standardisation modifiée. Cette divergence résulte de la nature des indices fournis et se traduit par des différences importantes dans la détection des items. (Abstract shortened by UMI.)

Concept mapping is a conceptualization process that can be used by individuals and groups to develop conceptual frameworks which can guide evaluations and planning (Trochim, 1989). In research, these processes display individual and group representations of concepts about particular domains, illustrating potential relationships among them (Miles, 1994). Cognitive mapping processes involve the acquisition, store, access and utilization of spacial knowledge (Golledge, 1986). Empirical research using concept mapping technology has proliferated within the past fifteen years. Investigation of this research has revealed the existence of a wide variation of domains of inquiry and applications of concept mapping. Using non-traditional meta-analytic research techniques employed in prior reviews by Cousins and associates (Cousins, 1996; Cousins & Earl, 1992; Cousins & Leithwood, 1986; Ross, in press) and others (e.g., Leithwood & Montgomery, 1982), the empirical research studies are explored with relevance to evaluation theory and practice. Emphasis on concept mapping process variations and use in evaluation is ordered. This study provides researchers and evaluators with valuable empirical basis from which to make choices regarding selection and applications of concept mapping.

No description available.

No description available.