A validity study can be used to investigate the effectiveness of an exam and reveal both its strengths and weaknesses. This study concerns an investigation of the writing portfolio Level Achievement Test (LAT) at the English Language Center (ELC) of Brigham Young University (BYU). The writing portfolios of 251 students at five proficiency levels were rated by 11 raters. Writing portfolios consisted of two coursework essays, a self-reflection assignment, and a 30-minute timed essay. Quantitative methods included an analysis with Many-Facet Rasch Model (MFRM) software, called FACETS, which looked for anomalies in levels, classes, examinees, raters, writing criteria, and the rating scale categories. Qualitative methods involved a rater survey, rater Think Aloud Protocols (TAPs), and rater interviews. Results indicated that the exam has a high degree of validity based on the MFRM analysis. The survey and TAPs revealed that although raters follow a similar pattern for rating portfolios, they differed both in the time they took to rate portfolios and in the degree to which they favored the rating criteria. This may explain some of the discrepancies in the MFRM rater analysis. Conclusions from the MFRM analysis, surveys, TAPs, and interviews were all used to make recommendations to improve the rating process of the LAT, as well as to strengthen the relationship between LAT rating and classroom teaching and grading.

No description available.

Mémoire numérisé par la Direction des bibliothèques de l'Université de Montréal. / Ce mémoire présente le développement d'une stratégie de testing informatisée et interactive, centrée sur le diagnostic pédagogique dans le contexte de la classe. La nouvelle stratégie d'administration d'un test a été intégrée à un système de tests adaptatifs existant, conçu pour répondre à un besoin d'outils pour revaluation formative qui respectent les contraintes organisationnelles du milieu scolaire québécois. Le système développé a pour but de renseigner rapidement renseignant et l'élève de l'état des apprentissages de celui-ci sans délai, afin que renseignant puisse entreprendre des interventions correctives efficaces auprès des élèves. Le système de tests réalisé administre un test en deux phases. La première phase fournit un diagnostic structurel ainsi qu'une hypothèse du diagnostic causal reflétant les apprentissages d'un élève. La seconde phase du test permet d'établir un niveau de cohérence d'un élève présentant une difficulté cognitive préalablement identifiée dans la première phase et de communiquer ce diagnostic causal sous la forme d'un message qualitatif pour renseignant. Pour répondre aux objectifs de la première phase du test, nous avons modifié la programmation de la stratégie "stradaptative" existante pour la rendre plus flexible et indépendante des banques d'items. Nous avons procédé à la verification de l'algorithme de sélection des items "stradaptatif" par une étude de vérification simulant différents profils d'élèves. Pour répondre aux objectifs de la seconde phase, nous avons convenu d'utiliser la catégorie de leurres la plus fréquemment choisie par l'élève lors de la première phase du test comme élément d'information pour la seconde phase. Nous avons ajouté au système de tests diagnostiques existant, une stratégie de selection adaptative des items selon un modèle Rasch nécessaire pour estimer le niveau de difficulté cognitive de l'élève face à une catégorie de leurres donnée. n IV Nous avons établi, à partir des résultats des études de simulations, des intervalles de confiance pour les thêta moyens estimés correspondant à des niveaux de difficulté cognitive et des profils d'élèves différents. A partir de ces intervalles, nous avons défini trois niveaux de messages diagnostiques exprimant le niveau de cohérence de l'élève à choisir une catégorie de leurres donnée : la difficulté n'est pas cohérente, la difficulté présente un certain degré de cohérence, la difficulté est cohérente. Le diagnostic structurel et le diagnostic causal sont accessibles immédiatement après la passation du test par l'entremise de la fiche diagnostique individuelle de l'élève. Cette recherche a permis d'intégrer dans un même système de tests, la stratégie "stradaptative" dont la programmation a été rendue indépendante du contenu des banques et une stratégie complémentaire de sélection des items utilisant le modèle de Rasch qui permet de déterminer un niveau de cohérence de l'élève à choisir une certaine catégorie de leurres.

Data-driven learning is a sub-discipline of corpus linguistics that makes use of the analyses and tools of corpus linguistics in foreign and second language classroom (Johns, 1991; Johns & King, 1991). With this approach, learners become researchers rather than passive recipients of language rules (Johns, 1991). This study was an investigation of the impact of this approach as a form of written corrective feedback for in-service teachers of English participating in an online writing course at a teacher training institute in Japan. Data-driven learning is commonly utilized in conventional, face-to-face classrooms, or computer lab settings in which there is close direction from the instructor on how to interpret the output of a corpus query. The purpose of this study was to investigate how data-driven learning can be implemented in a blended online environment by providing training to develop the participants’ corpus competence (Charles, 2011; Flowerdew, 2010), which is defined as the ability to interpret data obtained from querying a corpus. This competence has been associated with becoming familiar with corpus methods, which include interpreting concordances, and in turn can aid in accurately repairing writing errors. This training, while initially presented in a face-to-face session at the beginning of the course, was sustained with support from resources on the course’s Moodle website and my comments in Microsoft Word documents. In addition, I applied a fine-grained approach to the analysis of the to examine the quality of participants’ interpretation of concordances. The mixed method triangulation convergence design (Creswell & Plano Clark, 2007, 2011) used in this study was based on data from four sources to examine the effectiveness of data-driven learning in an online environment as well as to observe how the participants interpreted concordances. One data set involved an analysis of the participants’ responses in drafts of their own writing to concordance-based feedback. The participants were given a prefabricated concordance, which was a concordance I generated. That concordance was attached to an error in the participants’ document and the participants used the information provided by the concordance to repair their writing error. The resulting data set, which contains the concordance, along with before and after comparisons of the writers’ repairs, shows how the participants’ interpretations of concordances aided the repairs. With the evidence of several trials over the course of four writing assignments, it was possible to see how the participants used the supplied concordance to repair their writing errors and in turn revealed their degree of corpus competence. A second data set obtained from think-aloud protocols from select participants was utilized to reveal how they interpreted the concordance during an error-repair task. This data revealed what kind of thought processes or noticing that occurred in this task. A third piece of evidence was derived from data obtained from the Moodle website via log files and other resources such as online documents and training quizzes. The purpose was to document which resources the participants accessed relating to data-driven learning training to investigate if those resources aided in their development of corpus competence. The fourth piece of evidence was a quiz developed online to compare the participants with a standard set of items. The quiz was used to investigate which participants successfully or unsuccessfully interpreted the concordances. This instrument, which was analyzed with the Rasch model, allowed for further comparison between the participants’ skill of interpreting concordances. These four data sources were triangulated and in the final analysis cross-referenced to examine how data-driven learning can be successfully applied in a blended online learning environment and how the training of corpus competence aided the learners in interpreting the concordances. / Teaching & Learning

No description available.

No description available.

Syftet med denna uppsats var att undersöka sociometerteorins sociologiska relevans, förenlighet med flerdimensionella och hierarkiska modeller av självbild samt mätinstrumentet Revised Janis and Field Scales mätegenskaper. En enkät administrerades till 225 studenter vid Karlstads universitet, varav 140 respondenter var kvinnor och 85 var män. Resultatet visar att principiell komponentanalys i huvudsak replikerar valideringsstudien från 1984 medan Raschanalys visar att mätinstrumentet har stora och allvarliga problem och behöver revideras innan användning eller kasseras. Vidare är den modell över självbilden som mätinstrumentet är avsett att fånga föråldrad och inaktuell. Resultatet visar även att sociometerteorin kan relateras till symbolisk interaktionism och därmed har en sociologisk relevans men att förenligheten med modeller över självbilden är begränsad till huruvida teorin kan anses beskriva mekanismen bakom den sociala självbilden. Diskussionen avhandlar självkänslans självuppfyllande profetia, att självhjälpsindustrin, media och allmänhet etiketterar företeelser som och tillskriver begreppet betydelse och relevans varpå det beforskas vidare trots att forskningsfältet verkar befinna sig i ett degenerativt tillstånd. Vidare diskuteras att självkänsla och självbild ej är utbytbara begrepp eftersom det senare begreppet har prediktiv validitet (dvs. har visats förutsäga andra utfall). Slutligen ges ett förslag att antingen rikta fokus mot att studera självmedkänsla, socialt stöd, socialt kapital, självbild och subjektivt välmående istället för det för närvarande fruktlösa självkänslabegreppet, eller att utveckla begreppet teoretiskt innan fler kvantitativa studier utförs. Ytterligare slutsatser och implikationer av uppsatsen diskuteras. / The purpose of this thesis was to investigate the sociometer theory’s sociological relevance, compatibility with multidimensional and hierarchical models of the self-concept and the measurement instrument Revised Janis and Field Scales measurement properties. A questionnaire was administered to 225 students at Karlstad University, of whom 140 respondents were female and 85 were male. The results shows that the principal components analysis in general replicates the validation study from 1984 while Rasch analysis demonstrates that the measurement instrument have major and severe problems and needs to be revised before use or discarded. Furthermore, the model of self-concept that the measurement instrument is intended to capture is outdated and obsolete. The results also shows that the sociometer theory can be related to symbolic interactionism and thus have a sociological relevance but that the compatibility with models of the self-concept is limited to whether the theory can be regarded as describing the mechanism behind the social self-concept. The discussion treats a self-fulfilling prophecy of self-esteem, that the self-help industry, media and public label phenomena as and ascribes the concept meaning and relevance whereon it is further researched despite that the research field seems to be in a degenerative state. Furthermore, self-esteem and self-concept are not interchangeable concepts since the latter concept has predictive validity (i.e. has been shown to predict other outcomes). Finally, a proposition is made to either direct focus at researching self-compassion, social support, social capital, self-concept and subjective well-being instead of the currently fruitless self-esteem concept, or to develop the concept theoretically before conducting more quantitative studies. Further conclusions and implications of the thesis are discussed.

This study investigated the difficulty order of 38 grammar structures obtained from an analysis of multiple-choice items using a Rasch analysis. The order was compared with the order predicted by processability theory and the order in which the structures appear in junior and senior high school textbooks in Japan. Because processability theory is based on natural speech data, a sentence repetition test was also conducted in order to compare the result with the order obtained from the multiple-choice tests and the order predicted by processability theory. The participants were 872 Japanese university students, whose TOEIC scores ranged from 200 to 875. The difficulty order of the 38 structures was displayed according to their Rasch difficulty estimates: The most difficult structure was subjunctive and the easiest one was present perfect with since in the sentence. The order was not in accord with the order predicted by processability theory, and the difficulty order derived from the sentence repetition test was not accounted for by processability theory either. In other words, the results suggest that processability theory only accounts for natural speech data, and not elicited data. Although the order derived from the repetition test differed from the order derived from the written tests, they correlated strongly when the repetition test used ungrammatical sentences. This study tentatively concluded that the students could have used their implicit knowledge when answering the written tests, but it is also possible that students used their explicit knowledge when correcting ungrammatical sentences in the repetition test. The difficulty order of grammatical structures derived from this study was not in accord with the order in which the structures appear in junior and senior high school textbooks in Japan. Their correlation was extremely low, which suggests that there is no empirical basis for textbook makers'/writers' policy regarding the ordering of grammar items. This study also demonstrated the difficulty of writing items testing the knowledge of the same grammar point that show similar Rasch difficulty estimates. Even though the vocabulary and the sentence positions were carefully controlled and the two items looked parallel to teachers, they often displayed very different difficulty estimates. A questionnaire was administered concerning such items, and the students' responses suggested that they seemed to look at the items differently than teachers and what they notice and how they interpret what they notice strongly influences item difficulty. Teachers or test-writers should be aware that it is difficult to write items that produce similar difficulty estimates and their own intuition or experience might not be the best guide for writing effective grammar test items. It is recommended to pilot test items to get statistical information about item functioning and qualitative data from students using a think-aloud protocol, interviews, or a questionnaire. / CITE/Language Arts

The primary purposes of this study were to identify motivational typologies of growth and stability and identify people who have crossed a boundary in terms of levels of hope and time perspective. This study draws upon two fields, philosophy and psychology. The philosophical framework traces its roots back to American pragmatism and Alfred North Whitehead’s Process Philosophy. The second set of theories proposed for investigation came from the relatively recent empirical endeavor known as positive psychology. Specifically, I tested the construct validity and predictive utility of hope and time perspective as predictors of academic time management and academic outcomes in a Japanese sample. The participants were 467 students attending one of the largest private universities in Japan. Several instruments were used to measure the relationship between hope and time perspective as independent variables and self-reported academic outcomes. The instruments were the Hope Disposition Survey, the Zimbardo Time Perspective Inventory, self-reported TOEIC and GPA scores, and the Vocabulary Size Test. The research design was a quantitative and qualitative mixed-methods research plan. Two relatively recent constructs from the area of positive psychology research known as hope theory (a goal-oriented construct) and the Zimbardo Time Perspective Inventory were utilized for empirical investigation. The use of a mixed-method research design allowed this study to add to our knowledge of the roles of hope and time management in goal directed behavior. The analytical tools included the Rasch model, confirmatory factor analyses (CFA), structural equation modeling (SEM), and latent growth curve modeling (LGM). The qualitative analysis was a phenomenological investigation (similar to a case study) into the relationship between affect, cognition, and motivation utilizing a Process Philosophy framework. Results of the Rasch and CFA indicated that hope and time perspective were viable constructs for this sample. The hope SEM results indicated that hope had a positive relationship with academic outcomes as hypothesized. The time perspective SEM indicated that future time perspective had a positive relationship and that present-hedonism had a negative relationship with academic behavior as hypothesized. LGM results indicated that study time management had a non-linear relationship with the academic calendar. Both sets of results must be considered with caution due to a design flaw in the data collection instruments and high levels of attrition for the LGMs. Finally, the interview results indicated that students in the sample were extrinsically motivated by situational variables such as professor signals of how to, how much, when to, and what to study and that transitions from secondary to tertiary level studies were difficult for students with low levels of hope. The results were interpreted to suggest that levels of student engagement in the sample were at a less than desirable level when compared to OECD or North American university expectations. However, results were considered to be generally supportive of hope and time perspective theory. / Applied Linguistics

Many researchers have highlighted the need for students to have a willingness to communicate (WTC) in second and foreign language classrooms. WTC is important because it is believed that WTC leads to eventual communication both inside and outside the classroom. Previous research into WTC has centered mainly on the use of structural models and trait, self-reported measurements of WTC, but recent research has shown that WTC is also heavily dependent on the situation. However, very few studies recognize this and have, thus, not employed situational measurements of WTC. After a thorough review of WTC literature, 13 pertinent constructs were modified to reflect the situation in the foreign language classroom. These were related to constructs of the classroom, beliefs about communication; self-determined motivation; self-perceived competence; communication anxiety and willingness to communicate. A preliminary study employing exploratory factor analysis and Rasch analysis, followed by a main study conducted with and confirmatory factor analysis and Rasch analysis were undertaken with first- and second-year Japanese university students. The factor analysis was used to investigate the underlying structures of the factors and the Rasch analysis was used to determine fit, category functioning and dimensionality. Nine reliable and unidimensional factors were brought forward from the main study, which were Classroom Efficacy Factor and Classroom Affective Factor as the two classroom factors; Intrinsic Motivation for Communication, Introjected Regulation for Communication and External Regulation for Communication as the three self-determined motivation constructs, and finally, Self-Perceived Competence, Communicative Anxiety and Willingness to Communicate split in two subconstructs of pair/work and whole class activities. The second half the study was the formulation of a structural equation model using the above constructs to predict situational WTC. The model also included an often under-utilized resource, the teacher, who assessed the learners’ actual communication to identify if WTC leads to language use. All the fit indices in the final model (N = 376) were good, and the model included three additional paths. The model indicated that classroom constructs led to motivation and self-perceived competence, which predicted confidence. Motivation led directly to WTC and indirectly to WTC through confidence. Lastly, WTC predicted actual communication. The constructs in this study can be applied in other studies of situational WTC. This study helps to both expand our understanding of constructs affecting situational WTC and actual communication, and provides more validity to the construct of situational WTC. It also reaffirms the importance of what happens in the classroom, which is main arena for communication in the EFL setting. / Language Arts