Data derived from universal screening procedures are increasingly utilized by schools to identify and provide additional supports to students at-risk of behavioral and emotional concerns. As screening has the potential to be resource intensive, effort has been placed on the development of efficient screening procedures, namely brief behavior rating scales. This study utilized classical test theory and generalizability theory to examine the extent to which differences among students, raters, occasions, and screening measures affect the meaningfulness of data derived from universal screening procedures. Teacher pairs from three middle school classrooms completed two brief behavior rating scales during fall and spring screening administrations for all students in their respective classrooms. Correlation coefficients examining interrater reliability, test-retest reliability, and concurrent validity were generally strong. Generalizability analyses indicated that the majority of variance in teacher ratings were attributable to student differences across all score comparisons, but differences between teacher ratings for particular students accounted for relatively large percentages of error variance among student behavior ratings. Although decision studies showed that increasing the number of screening occasions resulted in more generalizable data, the impact of increasing the number of raters resulted in more efficient screening procedures.

No description available.

The ability to make sense of a wide array of stimuli presupposes the human tendency to organize information in a meaningful way. Efforts to assess the degree to which students organize information meaningfully have been hampered by several factors including the idiosyncratic way in which individuals represent their knowledge either with words or visually. Concept maps have been used as tools by researchers and educators alike to assist students in understanding the conceptual interrelationships within a subject domain. One concept-map assessment in particular known as the construct-a-map task has shown great promise in facilitating reliable and valid inferences from student concept-map ratings. With all of its promise, however, the construct-a-map task is burdened with several rating difficulties. One challenge in particular is that no published rubric has been developed that accounts for the degree to which individual propositions are important to an understanding of the overall topic or theme of the map. This study represents an attempt to examine the psychometric properties of two construct-a-map tasks designed to overcome in part this rating difficulty. The reliability of the concept-map ratings was calculated using a person-by-rater-by-occasion fully crossed design. This design made it possible to use generalizability theory to identify and estimate the variance in the ratings contributed by the three factors mentioned, the interaction effects, and unexplained error. The criterion validity of the concept-map ratings was examined by computing Pearson correlations between concept-map and essay ratings and concept-map and interview transcript ratings. The generalizability coefficients for student mean ratings were moderate to very high: .73 and .94 for the first concept-mapping task and .74 and .87 for the second concept-mapping task. A relatively large percentage of the rating variability was contributed by the object of measurement. Both tasks correlated highly with essay and interview ratings: .62 to .81.

The purpose of this study was to refine a rating procedure used to assess intermediate elementary school students' ability to orally retell what they had read from two expository passages. Oral retellings from 28 fourth grade students were tape-recorded and rated on two different occasions by each of 4 raters. A four-facet (passage, day of test administration, rater, and rating occasion) generalizability study was conducted using a partially nested design. The six largest sources of variability identified in the G-study included (a) students, (b) the student-by-day interaction, (c) the interaction of passage with rater (nested within student and day), (d) the student-by-day-by-occasion interaction, (e) the passage-by-raters (nested within students and day)-by-occasion interaction, and (f) the residual. A D-study was conducted to predict the values of the error variances and generalizability indices for both relative and absolute decisions. The results show how the error variance and the generalizability coefficients vary as a function of the number of passages, days of test administration, raters, and rating occasions. The results of the D study indicate that adding an extra reading day would produce a greater increase in reliability than asking the students to read more passages, or using more raters or more rating occasions. To achieve the greatest gain in generalizability, teachers should have students read at least two passages on at least two separate days and have their retelling rated by at least two raters and then compute a mean rating for each student averaged across the various passages, testing days, and raters.

Intuition is an important aspect of counseling, several revered counselors have either attested to the powers of their intuition or have had such powers attributed to them by their contemporaries. Moreover, many counselors believe that their intuition is more influential in their work with clients than are evidence-based practices (EBPs). However, the academy criticizes intuition for its susceptibility to cognitive errors and its poor performance when compared to statistical methods. In addition, the exact nature of intuition’s role in counseling is largely unknown. Therefore, its contribution to client outcomes is equally a mystery, making it difficult for counselors to justify their reliance on its powers. Until this study, counselor intuition has been regarded as a, more or less, phantom construct in need of evidence to even suggest that it does in fact exist. Therefore, the purpose of this study was to develop the Counselor Intuition Scale (CIS). The construction of the CIS began by adapting the methodology of instruments already in existence and whose purpose was to measure interpersonal and emotional sensitivity. The construction of the CIS began by creating a series of 39 video segments (lasting approximately two minutes each) depicting a client discussing a presenting problem. The video segments were then reviewed by two rounds of counseling experts (N = 45) whose intuitive responses to the clients featured in the CIS were used to create the criterion responses of the instrument. The expert responses were analyzed using Q-Methodology, the results of which suggested that the counseling experts approached the clients from a unidimensional perspective, which the researcher named "counselor intuition." The expert ratings were also analyzed using generalizability theory to assess the consistency of expert responses, the results of which iv suggested that interrater reliability was excellent, ranging from .88 to .85. Lastly, the experts identified 263 criterion responses that can be used for the future development of the instrument. The implications of the study’s findings, as well as the recommendations for future research are discussed

No description available.

Natural disasters strike anywhere, disrupting local communication and transportation infrastructure, making the process of assessing specific local damage difficult, dangerous, and slow. The goal of Building Damage Assessment (BDA) is to quickly and accurately estimate the location, cause, and severity of the damage to maximize the efficiency of rescuers and saved lives. In current machine learning BDA solutions, attention operators are the most recent innovations adopted by researchers to increase generalizability and overall performances of Convolutional Neural Networks for the BDA task. However, the latter, nowadays exploit attention operators tailored to the specific task and specific neural network architecture, leading them to be hard to apply to other scenarios. In our research, we want to contribute to the BDA literature while also addressing this limitation. We propose Global Voxel Transformer Operators (GVTOs): flexible attention-operators originally proposed for Augmented Microscopy that can replace up-sampling, down-sampling, and size-preserving convolutions within either a U-Net or a general CNN architecture without any limitation. Dissimilar to local operators, like convolutions, GVTOs can aggregate global information and have input-specific weights during inference time, improving generalizability performance, as already proved by recent literature. We applied GVTOs on a state-of-the-art BDA model and named it GVT-BDNet. We trained and evaluated our proposal neural network on the xBD dataset; the largest and most complete dataset for BDA. We compared GVT-BDNet performance with the baseline architecture (BDNet) and observed that the former improves damaged buildings segmentation by a factor of 0.11. Moreover, GVT-BDNet achieves state-of-the-art performance on a 10% split of the xBD training dataset and on the xBD test dataset with an overall F1- score of 0.80 and 0.79, respectively. To evaluate the architecture consistency, we have also tested BDNet’s and GVT-BDNet’s generalizability performance on another segmentation task: Tree & Shadow segmentation. Results showed that both models achieved overall good performances, scoring an F1-score of 0.79 and 0.785, respectively. / Naturkatastrofer sker överallt, stör lokal kommunikations- och transportinfrastruktur, vilket gör bedömningsprocessen av specifika lokala skador svår, farlig och långsam. Målet med Building Damage Assessment (BDA) är att snabbt och precist uppskatta platsen, orsaken och allvarligheten av skadorna för att maximera effektiviteten av räddare och räddade liv. Nuvarande BDA-lösningar använder Convolutional Neural Network (CNN) och ad-hoc Attention Operators för att förbättra generaliseringsprestanda. Nyligen föreslagna attention operators är dock specifikt skräddarsydda för uppgiften och kan sakna flexibilitet för andra scenarier eller neural nätverksarkitektur. I vår forskning bidrar vi till BDA -litteraturen genom att föreslå Global Voxel Transformer Operators (GVTO): flexibla attention operators som kan appliceras på en CNN -arkitektur utan att vara bundna till en viss uppgift. Nyare litteratur visar dessutom att de kan öka utvinningen av global information och därmed generaliseringsprestanda. Vi tillämpade GVTO på en toppmodern CNN-modell för BDA. GVTO: er förbättrade skadessegmenteringsprestandan med en faktor av 0,11. Dessutom förbättrade de den senaste tekniken för xBD-testdatauppsättningen och nådde toppmodern prestanda på en 10% delning av xBD-träningsdatauppsättningen. Vi har också utvärderat generaliserbarheten av det föreslagna neurala nätverket på en annan segmenteringsuppgift (Tree Shadow segmentering), vilket uppnådde över lag bra prestationer.

(EN) Title: The process of pilot standardization of the Czech version of the questionnaire SERVQUAL for the sports services indurty. Objectives: This paper focus is to initiate the standardization process of the Czech version of the SERVQUAL questionnaire for fitness and recreational sport and prepare the ground for its future use in practice. Methods: First was SERVQUAL questionnaire translated into Czech language and distributed to two fitness centers during six months period. Then using Generalizability theory and faktor analysis the reliability and internal structure of this instrument was examined. Results: Provides information about use of this instrument in assesing service quality of fitness centers in Czech republic . Proposes recommendations for further modification and use the questionnaire in the Czech environment. Key words: Service quality, Measurement of service quality, Generalizability theory, Factor analysis. This research was supported by the Grant agency of Charles University, project no. 267811 Measuring sport services quality in fitness industry

In measuring second language learners’ writing proficiency, test takers’ performance on a particular assessment task is evaluated by raters using a set of criteria to generate writing scores. The scores are used by teachers, students, and parents to make inferences about their performance levels in real-life writing situations. To examine the accuracy of this inference, it is imperative that we investigate the sources of measurement error involved in the writing score. It is also important to ensure rater consistency, both within a single rater and between raters, to provide evidence that the scores are valid indicators of tested constructs. This mixed methods research addressed the validity of integrated listening-to-write (L-W) scores. More specifically, it examined the generalizability of L-W scores and raters’ decision-making processes and scoring challenges. A total of 198 high school English learners in Taiwan completed up to two L-W tasks, each of which required them to listen to an academic lecture and respond to a related writing prompt in English. Nine raters who had experience teaching English evaluated each student’s written materials using a holistic scale. This study employed a univariate two-facet random effects generalizability study (p × t × r) to investigate the effects of tasks and raters on the score variance. Subsequent decision studies (p × T × R) estimated standard error of measurement and generalizability coefficients. Post-rating stimulated recall interview data were analyzed qualitatively to explore raters’ alignment of rating scale descriptors, decision-making behaviors, and scoring challenges. The results indicated that the majority of score variance was explained by test takers’ ability difference in academic writing proficiency. The raters were similar in their stringency and did not contribute much to score variance. Due to a relatively large magnitude of person-by-task interaction effect, increasing the number of tasks, rather than raters, resulted in a much lower degree of error and higher degree of score generalizability. The ideal assessment procedure to achieve an acceptable level of score generalizability would be to administer two L-W tasks scored by two raters. When evaluating written materials for L-W tasks, nine raters primarily focused on the content of the essays and paid less attention to language-related features. The raters did not equally consider all aspects of essay features described in the holistic rubric. The most prominent scoring challenges included 1) assigning a holistic score while balancing students’ listening comprehension skills and writing proficiency and 2) assessing the degree of students’ successful reproduction of lecture content. The findings of this study have practical and theoretical implications for integrated writing assessments for high school EFL learners.

Persistent lack of mathematics achievement and disparity in achievement has led to the publication of research findings related to equitable teaching practices. Although the publication of such research provides insights about approaches for potentially increasing equity in mathematics education, teachers must be able to apply what has been learned from these studies to their classroom teaching practices. Despite the widespread expectation that teachers use research-supported teaching strategies to meet the needs of their diverse classrooms, the research to practice gap persists. Little research is currently available to guide mathematics teacher educators in how to prepare future teachers to apply research to teaching practices. Inspired by advancements in social work and other health-related fields, this study departed from the standard approach of preparing teachers to utilize specific, research- based teaching strategies to preparing teachers to engage in the meta-process of applying research to practice. This meta-process has been defined by the health-related disciplines as the process of evidence-based practice (EBP). This process is explicated in a conceptual framework that is composed of the following five steps. The practitioner (1) formulates an answerable practice question, (2) searches for the best research evidence, (3) critically appraises the evidence, (4) selects the best intervention for a specific practice context, and (5) evaluates the outcome of the intervention. The purpose of this study was to examine the process of preparing preservice elementary teachers of mathematics to engage in the five-step process of EBP. Because this process, which can be conceptualized as a routine of practice, has not been identified for the field of mathematics education previously, it was examined using a design-based research (DBR) methodological approach. There were two objectives to the study: (1) to create an empirically tested teaching intervention that mathematics teacher educators can use to prepare preservice teachers to apply research to teaching practice and (2) to create a system of assessment that supports the teaching of this intervention. The study involved five iterations of the DBR process that permited the intervention to be evaluated and revised after each iteration. Although each iteration is discussed, this study focuses primarily on the process used in the fifth iteration of the DBR process. This iteration took place in the context of a mathematics methods course in a clinically-rich, undergraduate residency program for initial preparation of elementary school teachers. The twelve participants were simultaneously enrolled in the methods course and embedded in co-teaching assignments at an elementary school. The intervention to prepare teachers to engage in EBP included two workshops that were co-facilitated by an education librarian and a mathematics teacher educator and a semester-long Education Research Project. The project required participants to identify a problem of practice related to teaching or learning mathematics, find relevant research to address that problem, create an intervention to apply the research findings to classroom instruction, implement that intervention, and collect data to evaluate the effectiveness of the designed intervention. Instruments used to collect data included: (1) a self-report Information Literacy Questionnaire, (2) a self-report Familiarity with the Process of Evidence-Based Practice in Education Scale, (3) the Education Research Project report, and (4) a standardized performance assessment. The standardized performance assessment was used to assess beginning proficiency with the process of EBP. Generalizeability theory was used to evaluate the reliability of the system created for the standardized performance assessment. The system that included three raters, two tasks, and two scoring occasions was found to be fairly reliable (absolute generalizability coefficient = .81). Results from this study revealed that participants were more successful at creating implementation plans and linking those plans to research than they were at modifying their plans to meet the needs of specific students or evaluating their research implementation. This study contributes to both research and mathematics education communities' understandings about the potential of EBP as a high-leverage routine of practice and the use of generalizability theory in the creation of a reliable assessment to evaluate this routine of practice. This study documents the complexity of the process of linking research to practice and provides an empirically tested conceptual framework for preparing preservice teachers to engage in this complex practice.