College admission tests are of great importance for admissions systems in general and for candidates in particular. The SweSAT (Högskoleprovet in Swedish) has been used for college admission in Sweden for more than 30 years, and today it is alongside with the upper-secondary school GPA the most widely used instrument for selection of college applicants. Because of the importance that is placed on the SweSAT, it is essential that the scores are reliable and that the interpretations and uses of the scores are valid. The main purpose of this thesis was therefore to examine some assumptions that are of importance for the validity of the interpretation and use of SweSAT scores. The argument-based approach to validation was used as the framework for the evaluation of these assumptions.The thesis consists of four papers and an extensive introduction with summaries of the papers. The first three papers examine assumptions that are relevant for the use of SweSAT scores for admission decisions, while the fourth paper examines an assumption that is relevant for the use of SweSAT scores for providing diagnostic information. The first paper is a review of predictive validity studies that have been performed on the SweSAT. The general conclusion from the review is that the predictive validity of SweSAT scores varies greatly among study programs, and that there are many problematic issues related to the methodology of the predictive validity studies. The second paper focuses on an assumption underlying the current SweSAT equating design, namely that the groups taking different forms of the test have equal abilities. The results show that this assumption is highly problematic, and consequently a more appropriate equating design should be applied when equating SweSAT scores. The third paper examines the effect of textual item revisions on item statistics and preequating outcomes, using data from the SweSAT data sufficiency subtest. The results show that most kinds of revisions have a significant effect on both p-values and point-biserial correlations, and as a consequence the preequating outcomes are affected negatively. The fourth paper examines whether there is added value in reporting subtest scores rather than just the total score to the test-takers. Using a method derived from classical test theory, the results show that all observed subscores are better predictors of the true subscores than is the observed total score, with the exception of the Swedish reading comprehension subtest. That is, the subscores contain information that the test-takers can use for remedial studies and hence there is added value in reporting the subscores. The general conclusion from the thesis as a whole is that the interpretations and use of SweSAT scores are based on several questionable assumptions, but also that the interpretations and uses are supported by a great deal of validity evidence.

This research addresses the lack of an existing psychometrically sound Aspergers Syndrome self-screening instrument for adults. Initial instrument development procedures were carried out by creating an item pool using existing Asperger measures. Items were rewritten following common item writing rules suggested by several researchers as reference. Five new domains were created and the items were re-categorized. Expert panel of nine judges specialized in Psychometrics, ASD, School Psychology, and Speech-Language Pathology were asked to rate the relevancy of items to their domains in order to obtain evidence of content validity. These experts were chosen because of their relatedness to Aspergers Syndrome and their expertise in instrument development. First, the quality of the judges ratings were examined to identify any aberrant judges. Ratings were then analyzed using the remaining six judges using three descriptive and three quantitative methods to examine the representativeness and relevancy of each item to their domain. A total of 55 items were identified as satisfactory by the judges. The second part of this study was to compare the content validity analytical methods. It was concluded that the percentage agreement, the content validity index (CVI), and the content validity coefficients (VIK) were the best methods to use in selecting the satisfactory items. This research aims to bring more attention to the importance of psychometric properties in measures for the Autism Spectrum Disorder field. It also hope to shed some light on which content validity analyses would best be used under certain circumstances. Limitations of study and future directions were also discussed.

This research addresses the lack of an existing psychometrically sound Aspergers Syndrome self-screening instrument for adults. Initial instrument development procedures were carried out by creating an item pool using existing Asperger measures. Items were rewritten following common item writing rules suggested by several researchers as reference. Five new domains were created and the items were re-categorized. Expert panel of nine judges specialized in Psychometrics, ASD, School Psychology, and Speech-Language Pathology were asked to rate the relevancy of items to their domains in order to obtain evidence of content validity. These experts were chosen because of their relatedness to Aspergers Syndrome and their expertise in instrument development. First, the quality of the judges ratings were examined to identify any aberrant judges. Ratings were then analyzed using the remaining six judges using three descriptive and three quantitative methods to examine the representativeness and relevancy of each item to their domain. A total of 55 items were identified as satisfactory by the judges. The second part of this study was to compare the content validity analytical methods. It was concluded that the percentage agreement, the content validity index (CVI), and the content validity coefficients (VIK) were the best methods to use in selecting the satisfactory items. This research aims to bring more attention to the importance of psychometric properties in measures for the Autism Spectrum Disorder field. It also hope to shed some light on which content validity analyses would best be used under certain circumstances. Limitations of study and future directions were also discussed.

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal

Executive functioning skills are paramount to our ability to purposefully and successfully mediate our actions within our day-to-day environment. Dysfunction of the executive system can result in a multitude of behavioral manifestations in all stages of life. Increasing evidence supports the use of rating scales to obtain a more comprehensive and ecologically valid understanding of an individual’s executive functioning. The current thesis involves two articles examining the use of behavioral rating scales in the assessment of executive functions. Study 1: In response to a recent proliferation of executive functions rating scales, this article reviews and discusses currently available scales for the assessment of executive functions across the lifespan. Study 2: This study derived an executive functions screener from the Behavioral Assessment System for Children (BASC-2-SRP-COL) for use in young adults and evaluated it against a well-known executive function rating scale (the Behavior Rating Inventory of Executive Function-Adult Version). / Graduate / 0622 / 0632 / eduggan@uvic.ca

Although much educational research has investigated the relative effectiveness of different educational interventions and policies, little is known about the absolute net benefits of K-12 schooling independent of growth due to chronological age and out-of-school experience. The nearly universal policy of age tracking in schools makes this a difficult topic to investigate. However, a quasi-experimental regression discontinuity design can be used to separate observed test score differences between grades into independent age and schooling components, yielding an estimate of the net effects of school exposure at each grade level. In this study, a multilevel version of this design was applied to scores on 22 common ability and achievement tests from two major standardized test batteries. The ability battery contained 9 measures of Verbal, Quantitative, and Figural reasoning. The achievement battery contained 13 measures in the areas of Language, Mathematics, Reading, Social Studies, Science, and Sources of Information. The analysis was based on sample of over 20,000 students selected from a longitudinal database collected by a large U.S. parochial school system. The theory of fluid (Gf) and crystallized (Gc) intelligence predicts that these tests will show systematically different levels of sensitivity to schooling. Indeed, the achievement (Gc) tests were found to be three times more sensitive to schooling than they were to aging (one-year effect sizes of .41 versus .15), whereas the ability (Gf) tests were equally influenced by age (.18) and schooling (.19). Nonetheless, the schooling effect on most Gf tests was substantial, especially when the compounding over a typical school career is considered. This replicates the results of previous investigations of age and schooling using regression discontinuity methods and once again contradicts common interpretations of fluid ability. Different measures of a construct often exhibited varying levels of school sensitivity. Those tests that were less sensitive to schooling generally required reading, reasoning, transfer, synthesis, or translation; posed a wider range of questions; and/or presented problems in an unfamiliar format. Quantitative reasoning tests showed more sensitivity to schooling than figural reasoning tests, while verbal reasoning tests occupied a middle ground between the two. Schooling had the most impact on basic arithmetic skills and mathematical concepts, and a significantly weaker impact on the solution of math word problems. School-related gains on isolated language skills were much larger than gains on solving grammar problems in context. The weakest schooling impact overall was on reading comprehension where effects were no larger than those on verbal ability measures. An interesting dichotomy was found between spelling and paper folding (a measure of figural and spatial reasoning). Spelling skills showed robust schooling effects but a consistently negative age slope, a puzzling result which indicates that younger students in each group outperformed older students. Paper folding showed the opposite pattern, a large age effect and a small but consistently negative schooling effect. Results serve to rebut skepticism about both the impact of schooling on test scores and the validity of distinctions between ability and achievement. It is argued that the regression discontinuity design has great potential in the measurement of school effectiveness, while also offering a source of validity evidence for test developers and test users. Implications for theories of cognitive ability and future research on schooling effects are discussed.

No description available.

Personality, as a frequently used predictor of job performance, has often been criticized for its low criterion-related validity when compared to cognitive tests and some other predictors. The present study investigated incremental validity of narrow-trait personality to distinguish predictive from non-predictive facets of Conscientiousness and Extraversion dimensions. In addition, some intermediate mechanisms that may link the two personality dimensions with the criterion, such as different types of person-environment (P-E) fit and job involvement variables, were tested as well. The institution's job performance scale, NEO-PI-3 personality scale, person-organization (P-O) fit, needs-supplies (N-S) fit, demands-abilities (D-A) fit, job involvement questionnaire (JIQ), and demographic measures were administered in an online survey to 295 professional and civil service employees of a midsize Midwestern university. The sample was predominantly female and Caucasian with a mean age of 45.8 years and a median length of current employment of 5.1 years. Both personality dimensions were positively related to overall job performance. Conscientiousness was a stronger predictor of task performance, whereas Extraversion was related more consistently to contextual performance. In stepwise multiple regression analyses containing facets of personality dimensions as predictors of overall job performance, Competence emerged as the only facet of Conscientiousness, and Warmth and Assertiveness as the only facets of Extraversion that accounted for a meaningful amount of variance in the criterion. The use of narrow-trait personality to predict overall job performance enhances criterion-related validity of the construct and renders it a more efficient predictor of job performance than global-trait personality. Among the potential mediators, P-O and D-A fit partially mediated the personality-performance relationship providing evidence for the importance of perceptions of congruence in values and the ability to meet demands of the job. Current results are considered in light of limitations. Implications for theory, research, and practice, as well as future research directions are discussed.

No description available.

Recent stressful environments within military and non-military domains are producing a new challenge for the lab-based study of stress on task performance, one that requires knowledge of underlying cognitive-motivational and goal orientation factors. Results of recent stress on task performance research traditionally employ metaphorical explanations (i.e., resource theory) in order to rapidly apply stimulus-response outcomes to the real world counterparts. This dissertation provides an alternative perspective about these metaphorical, or black box, interpretations and reveals how they may be confounded with respect to the intended real world counterpart. To examine how voluntary human control can influence traditional stress/no-stress research findings, traditional as well as exploratory paradigms were presented. Both noise and time pressure conditions produced significant differences between experimental and control groups on visual discrimination. However, when analogous cash payment-contingency conditions were employed, the traditional stress/no-stress findings were not evident. In addition, a second experiment revealed that this trend of differences (and non-differences) held consistently over 30 minutes of interrupted task performance time. This study indicates the importance of developing more diagnostic measures that include assessments of how the differences between participants' and the generalized operators' goals and motivations may alter results in stressful task environments.