This dissertation illustrates how to detect the rater centrality effect in a simulation study that approximates data collected in large scale performance assessment settings. It addresses three research questions that: (1) which of several centrality-detection indices are most sensitive to the difference between effect raters and non-effect raters; (2) how accurate (and inaccurate), in terms of Type I error rate and statistical power, each centrality-detection index is in flagging effect raters; and (3) how the features of the data collection design (i.e., the independent variables including the level of centrality strength, the double-scoring rate, and the number of raters and ratees) influence the accuracy of rater classifications by these centrality-detection indices. The results reveal that the measure-residual correlation, the expected-residual correlation, and the standardized deviation of assigned scores perform better than the point-measure correlation. The mean-square fit statistics, traditionally viewed as potential indicators of rater centrality, perform poorly in terms of differentiating central raters from normal raters. Along with the rater slope index, the mean-square fit statistics did not appear to be sensitive to the rater centrality effect. All of these indices provided reasonable protection against Type I errors when all responses were double scored, and that higher statistical power was achieved when responses were 100% double scored in comparison to only 10% being double scored. With a consideration on balancing both Type I error and statistical power, I recommend the measure-residual correlation and the expected-residual correlation for detecting the centrality effect. I suggest using the point-measure correlation only when responses are 100% double scored. The four parameters evaluated in the experimental simulations had different impact on the accuracy of rater classification. The results show that improving the classification accuracy for non-effect raters may come at a cost of reducing the classification accuracy for effect raters. Some simple guidelines for the expected impact of classification accuracy when a higher-order interaction exists summarized from the analyses offer a glimpse of the "pros" and "cons" in adjusting the magnitude of the parameters when we evaluate the impact of the four experimental parameters on the outcomes of rater classification. / Ph. D.

This study involves investigating the impacts of ignoring nested data structure in Rasch/1PL item response theory (IRT) model via a two-level and three-level hierarchical generalized linear model (HGLM). Currently, Rasch/IRT models are frequently used in educational and psychometric researches for data obtained from multistage cluster samplings, which are more likely to violate the assumption of independent observations of examinees required by Rasch/IRT models. The violation of the assumption of independent observation, however, is ignored in the current standard practices which apply the standard Rasch/IRT for the large scale testing data. A simulation study (Study Two) was conducted to address this issue of the effects of ignoring nested data structure in Rasch/IRT models under various conditions, following a simulation study (Study One) to compare the performances of three methods, such as Penalized Quasi-Likelihood (PQL), Laplace approximation, and Adaptive Gaussian Quadrature (AGQ), commonly used in HGLM in terms of accuracy and efficiency in estimating parameters. As expected, PQL tended to produce seriously biased item difficulty estimates and ability variance estimates whereas almost unbiased for Laplace or AGQ for both 2-level and 3-level analysis. As for the root mean squared errors (RMSE), three methods performed without substantive differences for item difficulty estimates and ability variance estimates in both 2-level and 3-level analysis, except for level-2 ability variance estimates in 3-level analysis. Generally, Laplace and AGQ performed similarly well in terms of bias and RMSE of parameter estimates; however, Laplace exhibited a much lower convergence rate than that of AGQ in 3-level analyses. The results from AGQ, which produced the most accurate and stable results among three computational methods, demonstrated that the theoretical standard errors (SE), i.e., asymptotic information-based SEs, were underestimated by at most 34% when 2-level analyses were used for the data generated from 3-level model, implying that the Type I error rate would be inflated when the nested data structures are ignored in Rasch/IRT models. The underestimated theoretical standard errors were substantively more severe as the true ability variance increased or the number of students within schools increased regardless of test length or the number of schools. / Ph. D.

The literature is clear that intentional or unintentional clustering of data elements typically results in the inflation of the estimated standard error of fixed parameter estimates. This study is unique in that it examines the impact of multilevel data structures on subject ability which are random effect predictions known as empirical Bayes estimates in the one-parameter IRT / Rasch model. The literature on the impact of complex survey design on latent trait models is mixed and there is no "best practice" established regarding how to handle this situation. A simulation study was conducted to address two questions related to ability estimation. First, what impacts does design based clustering have with respect to desirable statistical properties when estimating subject ability with the one-parameter IRT / Rasch model? Second, since empirical Bayes estimators have shrinkage properties, what impacts does clustering of first-stage sampling units have on measurement validity-does the first-stage sampling unit impact the ability estimate, and if so, is this desirable and equitable? Two models were fit to a factorial experimental design where the data were simulated over various conditions. The first model Rasch model formulated as a HGLM ignores the sample design (incorrect model) while the second incorporates a first-stage sampling unit (correct model). Study findings generally showed that the two models were comparable with respect to desirable statistical properties under a majority of the replicated conditions-more measurement error in ability estimation is found when the intra-class correlation is high and the item pool is small. In practice this is the exception rather than the norm. However, it was found that the empirical Bayes estimates were dependent upon the first-stage sampling unit raising the issue of equity and fairness in educational decision making. A real-world complex survey design with binary outcome data was also fit with both models. Analysis of the data supported the simulation design results which lead to the conclusion that modeling binary Rasch data may resort to a policy tradeoff between desirable statistical properties and measurement validity. / Ph. D.

No / Patient-reported outcome measurement has become accepted as an important component of comprehensive outcomes research. Researchers wishing to use a patient-reported measure must either develop their own questionnaire (called an instrument in the research literature) or choose from the myriad of instruments previously reported. This article summarizes how previously developed instruments are best assessed using a systematic process and we propose a system of quality assessment so that clinicians and researchers can determine whether there exists an appropriately developed and validated instrument that matches their particular needs. These quality assessment criteria may also be useful to guide new instrument development and refinement. We welcome debate over the appropriateness of these criteria as this will lead to the evolution of better quality assessment criteria and in turn better assessment of patient-reported outcomes.

Previous research into the accuracy of referrals for glaucoma has shown that a large number of referrals to the Hospital Eye Service are false positive. Research in areas of healthcare other than ophthalmology has shown that psychological distress can be caused by false positive referrals. The present study aimed to evaluate the quality of referrals to the HES for all ocular pathologies, and also to quantify the proportion of these referrals that were false positive. Any commonality between false positive referrals was investigated. The psychological effect of being referred to the HES was also evaluated using the Hospital Anxiety and Depression Scale (HADS) and State-Trait Anxiety Inventory (STAI). Both scales were validated in this population with Rasch analysis before use. A final aim was to develop an improvement to the present referral pathway in order to reduce numbers of false positive referrals. The accuracy of referrals to the HES appears to improve as clinicians become more experienced, and greater numbers of false positive referrals are generated by female clinicians. Optometrists refer patients with a wide range of ocular diseases and in most cases include both fundus observations and visual acuity measurements in their referrals. GPs mainly refer patients with anterior segment disorders, particularly lid lesions, based on direct observation and symptoms. Illegibility and missing clinical information reduce the quality of many optometric referrals. Patients referred to the HES experience raised levels of anxiety as measured by the STAI and raised levels of depression as measured by the HADS-Depression subscale. As a method of assessing psychological distress, the questionnaires HADS-T (all items), STAI-S (State subscale) and STAI-T (Trait subscale) show good discrimination between patients when administered to a population of new ophthalmic outpatients, despite all having a floor effect. Subsequently a referral refinement service was developed which reduced numbers of unnecessary referrals and reduced costs for the NHS.

According to cognitive psychologists, consistent practice (i.e., tasks with a high degree of similarity) of a skill leads to the development of automaticity, with the degree of automatization being increased if the practice also involves exact repetition. Practice is skill specific, meaning that practicing one skill does not automatize related skills. Once achieved, automaticity has been characterized as the ability to perform a primary task with little interference from performing a concurrent secondary task. A second key characteristic of skill automatization is that it involves a gradual decrease in executive working memory (EWM) demands. The primary purpose of this study is to examine whether such claims put forth by cognitive psychologists are transferable to the automatization of second language (L2) bottom-up listening skills. The secondary purpose of this study is to explore whether reading while listening promotes listening automaticity to a greater extent than listening only, as has been argued in the L2 literature (Chang, 2011; Chang & Millett, 2014; Chang and Millett, 2016; Chang et al., 2018). The third purpose of this study is to examine whether visuospatial working memory (VWM) is a predictor of listening automaticity growth and to what degree it is related to EWM.Although L2 listening is regarded as a cognitive skill, the automatization of L2 listening has not been researched as described by cognitive psychologists. That is, using dual tasks and a reduction in EWM involvement as indices of automaticity, and providing consistent practice with an exact repetition component. In addition, the existing research into the question surrounding what kind of practice is more effective in bringing about L2 listening automaticity –listening while reading or listening only¬– has methodological flaws that have led to an inconclusive answer. To fill these gaps in the literature, I designed a listening-while-reading and a listening-only intervention that blended elements claimed to be conducive to automaticity development in cognitive psychology and L2 research, namely consistent training and exact repetition of stimuli and time pressure, while taking into account the methodological issues of previous L2 research. Data were collected from 290 Japanese high school students, who were assigned to the listening-while-reading, the listening-only, or a control group. Participants in the listening-only intervention listened to texts composed of a narrow range of vocabulary and grammar structures, which warranted consistency, whose speech rates increased by 10 words per minute (wpm) over the sessions from 100 wpm to 180 wpm. The listening-while-reading intervention involved listening to the same texts while reading the transcripts. Exact repetition of the stimuli was provided in both interventions by playing the texts two consecutive times in each intervention session. After the second listening, comprehension was assessed using a Japanese summary of the texts and a set of five multiple-choice questions. Degree of automatization was measured at three time points (i.e., Pretest, Posttest 1, and Posttest 2) separated by two cycles of training, through two dual tasks at each time point. The dual tasks, which consisted of listening to a text delivered at 160 wpm while drawing a series of vertical lines on blank paper and were followed by the abovementioned comprehension measures, were used to construct measures of automaticity. Dual-task data were linked through a Rasch longitudinal design, keeping text difficulty invariant across the dual tasks, with Rasch person measures for comprehension for subsequent analyses computed by combining scores on summaries and multiple-choice questions, while giving double weight to the summaries. Regarding the memory constructs, EWM was assessed through a speaking and a listening span task, whose scores were combined into a global EWM Rasch measure that was used in the analyses to test if the participants’ EWM costs diminished over time. Similarly, a single VWM Rasch measure was calculated by combining the participants’ scores on the Mr. Peanut task (DeAvila, 1974), which assesses visual memory, and the Corsi block span task (Corsi, 1972), which assesses spatial memory, as additional predictors of L2 listening automaticity. To my knowledge, measures of VWM have not previously been employed as predictors of listening ability. Results from a repeated-measures ANOVA and a multi-group latent growth model indicated that both intervention groups automatized their listening skills to a greater extent than the control group with the listening-only group developing automaticity to the largest extent. Although a reduction of EWM demands was not observed, possibly because the participants were not complete beginners and might have started the study with some degree of automaticity, the fact that they could perform the dual tasks less effortfully over time evidenced automatization. Interestingly, VWM significantly predicted growth in listening automaticity, which was interpreted to mean that as listeners automatize their listening skills, they become able to see what they hear. This result was explained in terms of attentional capacity limits, where higher-level processes, such as visualizing input, cannot be achieved until the lower-level processes such as word recognition or grammatical parsing have been automatized. Follow-up qualitative interviews corroborated the quantitative findings and extended them by suggesting that a small number of training sessions might be sufficient to develop listening automaticity. Results of an additional correlation analysis further showed that VWM and EWM are not highly related as has been shown by some experimental evidence in the working memory literature. The finding that listening only was superior to listening while reading is important because it contradicts previous research and suggests that to automatize listening, one needs to practice in real-operating conditions. Practical and theoretical implications are discussed. First, listening to texts whose speech rates increase over the practice sessions promotes listening automaticity. Next, claims by cognitive psychologists regarding the development of skill automaticity are largely generalizable to the natural conditions of classroom-based research focusing on the acquisition of real-world skills such as L2 listening. Last, the multi-component versus the unitary view of working memory. / Applied Linguistics

The purpose of this study was to construct a holistic education school evaluation tool using Montessori Erdkinder principles, and begin the validation process of examining the proposed tool. This study addresses a vital need in the holistic education community for a school evaluation tool. The tool construction process included using Erdkinder literature to justify the development of each item through the use of an item matrix, ultimately leading to the development of the 23 item formative Montessori Erdkinder School Evaluation Survey. The validation process included a series of three Rasch Rating Scale Model analyses with data from a sample school. The validation process used item anchoring estimates from the earlier analyses in the later analyses and included determining the tool’s dimensionality, reliability, item fit, possible differential item functioning, and comparing the order of item difficulty levels to the holistic model of Maslow’s hierarchy of needs. Results of the study showed that six items had issues with fit and would need to be revised, and that the items in the cognitive and moral facet will need to be revised to better match Maslow’s model. This study provides the foundation for the development of a holistic education evaluation or accreditation system, and constructed a resource that could be directly implemented in schools.

Students who are the first in their families to attend college are less likely to earn a college degree as compared to their continuing-generation peers. In efforts to increase college graduation rates for first-generation college students, support programs designed to assist first-generation college students are increasing in numbers. These first- generation programs are relying on existing research to build effective curriculums. Even though an extensive body of literature exists in the fields of self-efficacy and first- generation college students, research investigating the self-efficacy of first-generation college students are extremely limited. The research is further limited when examining academic self-efficacy and generational status. The purpose of this study is to investigate if parental levels of education affect college students’ self-reported levels of academic self-efficacy. The following research questions guided this study: 1) Do survey response hierarchies differ between first-generation college students and their continuing- generation counterparts on a scale that measures academic self-efficacy?, 2) Do levels of item endorsability vary based upon parental levels of education? and 3) Do the results produced from the college student survey support the existing literature on first- generation college students and academic-self-efficacy? Quality control indicators were utilized to assess the soundness of the instrument and to ensure that the rating scale functioned appropriately. Variable maps were used to compare and contrast student responses and item hierarchies. Pairwise differential item functioning (DIF) was used to examine item endorsability based upon levels of parental education. Results encourage practitioners to be mindful of the importance of data-informed decision making.

Cette recherche s’inscrit dans le cadre de l’évaluation des compétences langagières en français chez des adultes immigrants en vue de leur placement dans des cours de français. Elle porte sur la dimensionnalité, de même que sur la difficulté objective et subjective de tâches discrètes ou intégrées de compréhension écrite, à différents niveaux de maîtrise. Elle propose des analyses de l’estimation de la maîtrise linguistique en fonction de l’appartenance des candidats à des groupes linguistiques distincts. Pour mener à bien la recherche, un test de six textes et de 30 items a été créé. Il a été administré à 118 immigrants. Ces immigrants suivaient les cours de français proposés par le Ministère de l’immigration et des communautés culturelles du Québec (MICC) dans les écoles de langues de l’Université de Montréal et de l’Université du Québec à Montréal. Après administration, ce test a été soumis à des analyses portant sur la dimensionnalité et la difficulté des tâches discrètes et intégrées ainsi que sur les interactions entre ces tâches et les différents groupes de candidats. Des études plus précises ont été faites sur l’interaction entre le type de tâche, l’appartenance à un groupe linguistique pour des candidats et des items de niveau similaire. Enfin, des analyses ont permis d’étudier la perception de la difficulté des tâches par les candidats. L’étude, même si elle porte sur un test en rodage, permet de distinguer la dimensionnalité de tâches discrètes de celle de tâches intégrées. Elle permet également de constater les différences de fonctionnement entre ces deux types de tâches. Enfin, elle permet de comprendre l’interprétation de la difficulté par les candidats et, par ricochet, leur vision du test. In fine, des propositions sont formulées quant à l’opportunité d’utiliser des tâches discrètes et intégrées dans un test de positionnement adaptatif en français langue seconde. / This research has been conducted within the assessment procedure of the language competence of adult immigrants, for placement purposes in French courses. It relates to the dimensionality as well as the objective and subjective difficulty of discrete or integrated reading tasks at different proficiency levels. Analyses of linguistic proficiency estimates are proposed in relation with candidates’ linguistic groups. In order to conduct this study successfully, a 6-text and 30-item test has been constructed and administered to 118 immigrants. These immigrants were enrolled in French courses offered by the Ministry of Immigration and Cultural Communities (MICC) in language schools at the Université de Montreal and the Université du Québec à Montréal. After the administration, analyses have been made on the dimensionality and difficulty of the discrete and integrated tasks and on the interactions between the tasks and different groups of candidates. More detailed analyses have been made on candidates and items at similar levels. Finally, we were able to study the candidates’ perceptions of task difficulty. Although the study is based on a provisional test, the dimensionality of discrete and integrated tasks has been distinguished. Differences in the way the 2 types of task work have been shown. Finally, the candidates’ interpretation of difficulty and therefore, their view of the test, are better understood. In fine, proposals are made in regard with the proper use of discrete and integrated tasks in an adaptive placement test in second language.

L’apport disproportionné aux taux de criminalité des membres des gangs de rue est, nul doute, une proposition empirique robuste. De nombreuses études ont conclu que l’association aux gangs de rue est un facteur de risque supplémentaire à celui que constitue déjà la fréquentation de pairs délinquants au nombre des meilleurs prédicteurs de la délinquance avec les antécédents criminels et les traits antisociaux de la personnalité. Pourtant, la contribution spécifique de l’association aux gangs de rue à l’explication de la délinquance est largement méconnue. Au nombre des variables les plus souvent citées pour l’expliquer figure néanmoins le concept de l’adhésion à la culture de gang qui n’a toutefois jamais été spécifiquement opérationnalisé. Le but de la thèse est d’étudier la contribution spécifique de l’adhésion d’un contrevenant à la culture des gangs de rue à l’explication de la délinquance. Plus précisément, elle a comme objectifs de définir la culture des gangs de rue, d’opérationnaliser l’adhésion à la culture des gangs de rue, d’examiner la fidélité de la mesure de l’adhésion à la culture de gang et d’étudier sa relation avec la nature, la variété et la fréquence des conduites délinquantes de contrevenants placés sous la responsabilité des centres jeunesse et des services correctionnels du Québec. Trois articles scientifiques, auxquels un chapitre régulier est joint, ont servi la démonstration de la thèse. D’abord, le premier article présente les démarches relatives au développement de la première Mesure de l’adhésion à la culture de gang, la MACg. Plus précisément, l’article présente la recension des écrits qui a permis de proposer une première définition de la culture de gang et d’opérationnaliser le concept. Il fait aussi état de la démarche de la validation de la pertinence de son contenu et des données préliminaires qui révèlent la très bonne cohérence interne de la MACg. Cette première étude est suivie de la présentation, dans le cadre d’un chapitre régulier, des résultats de l’examen de la cotation des principaux indicateurs de la culture de gang. Cette démarche constitue un complément nécessaire à l’examen de la validité apparente de la MACg. Les résultats révèlent des degrés de concordance très satisfaisants entre les observations de divers professionnels des centres jeunesse et des services correctionnels du Québec qui ont été invités à coter les indicateurs de la culture de gang à partir de deux histoires fictives d’un contrevenant mineur et d’un second d’âge adulte. Puis, le deuxième article présente les résultats d’un premier examen de la fidélité de la MACg à l’aide du modèle de Rasch de la Théorie de la réponse aux items. Ses résultats soutiennent l’unidimensionnalité de la MACg et sa capacité à distinguer des groupes d’items et de personnes le long d’un continuum de gravité d’adhésion à la culture de gang. Par contre, le fonctionnement différentiel et le mauvais ajustement de certains items sont observés, ainsi que l’inadéquation de la structure de réponses aux items (de type Likert) privilégiée lors de l’élaboration de la MACg. Une version réaménagée de cette dernière est donc proposée. Enfin, le troisième et dernier article présente les résultats de l’examen de la relation entre la délinquance et l’adhésion d’un contrevenant à la culture de gang telle que mesurée par la MACg. Les résultats soutiennent l’apport unique de l’adhésion d’un contrevenant à la culture de gang à la diversité et à la fréquence des conduites délinquantes auto-rapportées par des contrevenants placés sous la responsabilité des centres jeunesse et des services correctionnels du Québec. Le score à l’échelle originale et réaménagée de la MACg s’avère, d’ailleurs, un facteur explicatif plus puissant que l’âge, la précocité criminelle, les pairs délinquants et la psychopathie au nombre des meilleurs prédicteurs de la délinquance. L’étude met aussi en lumière l’étroite relation entre une forte adhésion à la culture de gang et la présence marquée de traits psychopathiques annonciatrice de problèmes particulièrement sérieux. Malgré ses limites, la thèse contribuera significativement aux développements des bases d’un nouveau modèle explicatif de l’influence de l’association aux gangs de rue sur les conduites des personnes. La MACg pourra aussi servir à l’évaluation des risques des hommes contrevenants placés sous la responsabilité du système de justice pénale et à l’amélioration de la qualité des interventions qui leur sont dédiées. / The disproportionate contribution of street gang members to a city's crime rate is, no doubt, a strong empirical proposition. Numerous studies have concluded that street gang association is an additional risk factor tied to the association with delinquent peers which constitutes among the best predictors of delinquency along with criminal history and antisocial personality traits. However, the specific role that street gang association plays in explaining delinquency is largely unknown. Nevertheless, among the variables most often cited to explain this is the concept of adherence to the gang culture which has however never been specifically operationalized. The aim of this thesis is to study the specific contribution of an offender's adherence to the street gang culture to an explanation of delinquency. More specifically, its objectives are to define the street gang culture, operationalize adherence to the street gang culture, examine the reliability of the measurement of adherence to the street gang culture, and study its relationship with the nature, variety and frequency of delinquent behaviour of offenders placed under the responsibility of youth centres and correctional services in Quebec. Three scientific papers, to which a regular chapter has been attached, were used to demonstrate the thesis. The first paper presents the steps taken toward the development of the first Measurement of Adherence to the Gang Culture, the MACg. More specifically, the paper reviews the literature that led us to propose an initial definition of the gang culture and operationalize the concept. It also describes the approach to validating the relevance of its content and the preliminary data that demonstrate the measurement's strong internal consistency. This first study is followed by an examination of scoring of the main indicators of the gang culture, presented as a regular chapter. This approach is a necessary complement to the examination of the face validity of the MACg. The results indicate very satisfying levels agreement between observations of the various Quebec youth centre and correctional services professionals who were invited to score indicators of the gang culture based on two fictional stories of a juvenile offender and an adult offender.   Then, the second paper presents results of an initial study of the reliability of the MACg using the Rasch model approach to item response theory. Its findings support the one-dimensional nature of the MACg and its ability to distinguish groups of items and persons along a severity continuum of adherence to the gang culture. However, the differential functioning and poor fit of certain items were observed, in addition to the inadequacy of the type of response structure (Likert scale) chosen during the development of the MACg. Therefore, an adjusted version of the measurement is proposed. The third and final paper presents the results of a study examining the relationship between delinquency and an offender's adherence to the gang culture as measured by the MACg. The results support the unique contribution of an offender's adherence to the gang culture to the diversity and frequency of self-reported delinquent behaviour by offenders placed under the responsibility of youth centres and correctional services in Quebec. Moreover, the score on the original and adjusted MACg scales turned out to be a more powerful explanatory factor than age, criminal precocity, delinquent peers and psychopathy when it came to the best predictors of delinquency. The study also highlights the close relationship between a strong adherence to the gang culture and the marked presence of psychopathic traits which were warning signs of particularly serious problems. Despite its limitations, this thesis will significantly contribute to developing the foundation of a new model to explain the influence of street gang association on human behaviour. The MACg can also aid the risk assessment of male offenders placed under the responsibility of the criminal justice system and improve the quality of the treatments available to them.