Abstract not available.

The question of evaluation capacity in governmental organizations has been studied empirically in only a limited fashion in recent years. The actual dimensions of evaluation capacity, or what evaluation capacity might look like in an organization, have not previously been identified systematically in the literature. This study sought to identify the key dimensions of evaluation capacity in Canadian federal government organizations as well as the manageable steps required to move from low to high capacity for each of these. The data collection approach was articulated around three phases. The first phase sought to identify the key dimensions of evaluation capacity in Canadian federal government organizations and characterize minimal and exemplary performance on each dimension through the development of a framework. This was achieved through a literature review and semi-structured key informant interviews. The second phase focused on refining and clarifying these dimensions by using key informant interviews. The third and final phase sought to validate the revised framework. This was achieved through its practical application in four federal government organizations and semi-structured interviews with those involved in applying the framework. The study concluded that evaluation capacity in Canadian federal government departments and agencies can be described through six main dimensions, each one broken down into further sub-dimensions. The capacity of these organizations on each of these dimensions and sub-dimensions can be assessed using four levels: low, developing, intermediate, and exemplary. The study found that organizations vary in terms of their capacity from one dimension to the next, and indeed, from one sub-dimension to the next. This study makes an important contribution to the body of knowledge on evaluation capacity; although much has been published in the evaluation literature on evaluation capacity building, the actual characteristics and attributes of evaluation capacity itself have not before been defined and described based on empirical data. Continuing research may focus on expanding the scope of the framework, while the main practical implication of this study is the use of the framework as an instrument of evaluation capacity for government organizations.

This exploratory research compares Q-matrices related to two multiple choice tests in mathematics. These Q-matrices are elaborated in three different conditions. In the first one, only the items are available. In the second one, the items are presented as set together with a factor analysis. In the third condition, the item difficulty and discrimination parameters are provided to the experts as well as the distractors' analysis results. The cognitive model RUM (Reparametrized Unified Model) is used to compare the cognitive structure of the different matrices related to the same test. The form of the matrices, the agreement between the experts and the parameters estimated with the RUM are compared. The results tend to show that condition 2 cognitive structure is stronger for both tests. The difference with conditions 1 or 3 is, however, not very large. The third condition leads the experts to pay attention to non relevant aspects of the answering process and leads to the worst quality matrices.

Fairness is a perennial ideal in educational assessment. For most of the 20th century, objective tests were viewed as a means of bypassing teachers' inconsistencies in grading, and fairness was defined in technical terms for the development and administration of tests. As the 21 st century opened, the limitations of standardized tests for supporting student learning had become apparent, and interest in the pedagogical potential of classroom assessment was renewed. Although teachers' assessment decisions can have profound, long-term effects for students, most of the guidance for fair assessment still focuses on standardized testing. This study was motivated by the need for a better understanding of fairness in the classroom environment where the primary purpose of assessment is ideally to support learning. I approached my research from the perspective of critical pragmatism, which assumes that diverse forms of knowledge are valuable, including ethical and practical knowledge. I drew on the practical wisdom (phronesis) of experienced teachers to understand fairness specifically in the assessment of writing in English Language Arts in Grades 7 to 12 in Ontario. Practical wisdom is a responsive network that combines theoretical and technical knowledge, moral beliefs and ethical knowledge, experience, personal characteristics, and an understanding of the particulars of practice. I began my research by immersing myself in the literature, investigating the educational context, tailoring my research tools, and purposefully selecting teachers to participate based on their education and experience. Six participants responded in writing to vignettes that illustrated fairness dilemmas, and we discussed their recommendations for fair assessment during follow-up interviews. Insight that emerged from this process is that some strategies contribute to the fairness of classroom assessment, such as developing a constructive learning environment, sharing expectations and criteria, and offering students relevant opportunities to learn and demonstrate their learning. At the crux, however, fairness rests in a teacher's ability to understand students and their learning, and to reflect about the ongoing interactions and decisions involved in the use of assessment for learning.

It was hypothesized that on an achievement test, items measuring complex cognitive objectives would exhibit a higher mean discrimination index — based on the whole test as criterion — than would an equal number of items measuring less complex cognitive objectives; and that the mean discrimination index of these items would in turn be higher than that of the same number of still less complex items. The proviso was made that the difficulty indices of the items be similarly distributed within the several categories of items, hereafter called "relevance-categories," since discriminating power is related to difficulty. The categories selected were, from simplest to most complex, the Knowledge, Comprehension, and Application categories of Bloom's "Taxonomy of Educational Objectives." An achievement test was constructed, consisting of items in all three categories, and covering the content of two units of the British Columbia university-programme grade nine science course. A try-out of this test, on 200 students in two schools, permitted negatively discriminating items to be rejected and, in addition, provided difficulty indices for the remaining items. It was possible to match forty Knowledge items and forty Comprehension items very closely for difficulty; however, the mean difficulty of the Application items was so high that they could not be used in a test of the hypothesis without reducing numbers too drastically in all categories. Two "equivalent forms," matched for content, relevance-category, and difficulty were constructed from these eighty items and administered to 530 students in three schools. The reliability coefficient of the total test, estimated by correlating the sub-test scores and applying the Spearman-Brown formula, was .84; those of the Knowledge and Comprehension categories were similarly found to be .69 and .77, respectively. Revised difficulty indices, based on the new and larger sample, were calculated. Their distribution within the two relevance categories were found to be very similar, though not as closely matched as on the basis of the try-out test. For each item, the point-biserial coefficient of correlation between item and total score was computed — this being the selected index of discrimination — and Fisher's z-transformation was applied to produce measures with more nearly an equal-unit scale, in the hope that the parametric t-test could be used. However, the shapes of the resulting distributions were such that they could not be claimed to be samples from a normal population or populations. Accordingly, the t-test was rejected in favour of the non-parametric Mann-Whitney test of "no difference in median discrimination indices." The respective medians were .27 and .30, in terms of Fisher's z-values, but the difference proved to be non-significant at the pre-selected l%-level of significance. It was concluded that this experiment provided no grounds for accepting the hypothesis of the study. However, the actual probability of obtaining, in random sampling from a single population, a difference as large as that observed was only about .10; in addition, the results consistently favoured the Comprehension items, whose discrimination indices exceeded those of the Knowledge items at the extremes as well as at the mean. It was therefore suggested that if adequate testing time could be obtained, the use of larger numbers of items in all categories might increase test-reliability and possibly produce a significant result. Suggestions were advanced, based upon observations from the data, for refining the experiment and for further research. / Education, Faculty of / Graduate

The purpose of the study was to compare the Goodenough and Stanford-Binet scores of children referred to a Mental Hygiene Clinic. The sample was composed of 150 children between the ages of six and twelve. The relationship between the two sets of scores was examined, comparing the IQ's, MA's, and MA's with CA held constant. Differences associated with age and sex, individual differences between the two scores, and the range of variations were noted. The inter-scorer reliability of the Goodenough was also investigated. The Goodenough inter-scorer reliability was found to be very high. The correlation between the Goodenough and Stanford-Binet scores was slight for the IQ's, higher for the MA's, and still higher for the MA's when the CA was held constant. A large majority of the children scored higher on their Stanford-Binet than on their Goodenough tests, and there were many large differences between a child's score on the Binet and his score on the Goodenough. There were no differences found between the two instruments that were associated with age, but there were marked differences associated with sex. The boys' drawings were much better indicators of their intellectual abilities, as measured by the Stanford-Binet, than the girls'. It was concluded that the Goodenough Draw-a-Man Test could not be safely used as an independent test of an individual's intelligence, in a Mental Hygiene Clinic. / Arts, Faculty of / Psychology, Department of / Graduate

The purpose of this investigation was the determination of the nature of certain imperfections in magnesium oxide crystals. Optical absorption and photoconductivity spectra of specimens cleaved from a number of larger pieces were measured. The effects of vacuum heating, of non-stoichiometry and of ultraviolet and X-ray irradiation were investigated. The nature of the imperfections could not be inferred from the experimental results but an energy level diagram consistent with all the data has been deduced. A comparison of the present work with pertinent data from the literature is presented and a basic error in previous photoconductivity measurements is pointed out. A method for the determination of the sign of the charge carriers excited during photoconductivity measurements is described. / Science, Faculty of / Physics and Astronomy, Department of / Graduate

Le desir de comprendre l'origine de la disparite des pratiques d'evaluation des apprentissages en salle de classe et des incoherences qui souvent les caracterisent est le point de depart de cette recherche. Force fut de constater que l'absence d'un modele de qualite des resultats d'evaluation coherent avec la nature et le contexte des apprentissages est en grande partie responsable du laxisme observe. Qu'ils tirent leur origine du paradigme traditionnel de la psychometrie ou de celui plus moderne de l'edumetrie, les elements concourant a des resultats justes ont ete regroupes en quatre composantes: la pertinence de la tache d'evaluation par rapport aux habiletes visees, sa coherence avec l'activite pedagogique, sa transparence a l'egard de l'eleve et l'absence de desavantage circonstanciel. Tous ces elements, retenus pour leur apport a l'obtention de resultats justes, c'est-a-dire pour situer correctement un eleve dans son developpement, forment le concept de justesse des resultats. Ce dernier a ete confronte a la realite de la salle de classe, en collaboration avec cinq enseignants d'ecoles elementaires de l'Ontario. De l'avis des participants, les quatre composantes du concept de justesse sont toutes pertinentes a l'obtention de resultats justes. Cependant, cette conception de l'evaluation fait ressortir comme etant problematiques certaines situations d'evaluation en salle de classe, comme l'evaluation de taches effectuees en equipe et celle d'habiletes multiples enseignees de maniere integree. La discussion tente d'etablir si ces faits sont dus au manque de realisme du concept de justesse ou s'ils font plutot ressortir des situations d'evaluation en salle de classe menant a des resultats d'evaluation de pietre qualite. L'experience menee ici ne suffit pas a etablir la viabilite du critere de justesse comme modele de qualite des resultats d'evaluation des apprentissages en salle de classe. Toutefois, cette experience suggere fortement que cette organisation simplifiee de la realite est compatible avec la nature contemporaine de l'apprentissage et rend bien compte de la complexite de son evaluation. De ce point de vue, elle s'avere superieure aux criteres de validite et de fidelite presentement utilises pour guider les enseignants dans leurs interventions d'evaluation en salle de classe.

The Public Service has undergone widespread reorganizations which have had a considerable impact on human resources and, in order to achieve a continuity of services, the employees have been required to take training courses. Consequently, there has been a marked increase in popularity for courses of short duration and these courses are becoming more profitable from the economic point of view. Are they, however, in terms of the pedagogy? The evaluation of the courses given by Training and Development Canada relies on the Kirkpatrick's training program evaluation model. This model proposes the evaluation of training at four levels: (1) reaction, (2) learning, (3) behavior and (4) results. Kirkpatrick claims the existence of a positive relationship between these levels. The development of measuring tools has allowed for the measurement of the first three levels of the model for four selected courses. By using a sample of 77 federal public servants, the assumption of a positive relationship has been established, with the help of correlation coefficients, for the Orientation to Financial Management course. However, this assumption has not been established for the courses Negotiating Skills, Effective Presentations and Writing Memos and Letters.

L'examen de la fidélité et de l'efficacité des méthodes repose sur des données réelles, les résultats à un test de mathématiques de l'American College Testing Program (ACTP). Pour étudier l'efficacité des méthodes, nous avons déterminé quels items avaient un fonctionnement différentiel non négligeable. Pour ce faire, nous nous sommes servis d'un échantillon de 40 000 sujets qui constitue, à toutes fins utiles, la population d'origine. Les hommes y forment le groupe de référence; les femmes, le groupe focalisé. Ont été identifiés comme ayant un fonctionnement différentiel non négligeable tous les items dont la différence de difficulté standardisée est égale ou supérieure à 0,05 en valeur absolue, après arrondissement au centième le plus proche. Les items devaient également présenter un delta de Mantel-Haenszel ou une différence de difficulté standardisée modifiée qui les rangent parmi les items avec les indices les plus grands. De plus, ils devaient présenter des différences de difficulté standardisée qui sont plus grandes que celles observées à l'intérieur du groupe de référence et du groupe focalisé. Á partir des analyses effectuées, il appert que la méthode de standardisation modifiée offre des indices dont la stabilité est un peu moins grande que celle des deltas de Mantel-Haenszel. Malgré cela, sa capacité de détecter des items qui ont un fonctionnement différentiel se compare avantageusement à la méthode du delta de Mantel-Haenszel. Quelle que soit la méthode à l'étude, la stabilité des indices et leur capacité de détecter des items qui ont un fonctionnement différentiel non négligeable varient en fonction de la taille des échantillons. Règle générale, stabilité et efficacité tendent à augmenter avec l'accroissement du nombre de sujets dans les échantillons. Toutes les méthodes se comportent de façon analogue lorsqu'on se base sur des centiles pour déterminer quels items ont un fonctionnement différentiel dans les échantillons de taille réduite. Il en résulte peu de différence entre les méthodes pour des échantillons de même grandeur. Lorsqu'on se base sur des critères fixes, la méthode du khi carré de Mantel-Haenszel et la méthode de régression logistique affichent des tendances à l'opposé de la méthode du delta de Mantel-Haenszel et de la méthode de standardisation modifiée. Cette divergence résulte de la nature des indices fournis et se traduit par des différences importantes dans la détection des items. (Abstract shortened by UMI.)