This study explores the differential validity of SJT responding formats (i.e., selecting a response to an SJT item). It was hypothesized that the SJT on which respondents identified the best and worst options would be related to cognitive ability because this type of SJT has a high cognitive load and thus acts like a knowledge test. It was also hypothesized that the SJT on which respondents rated the effectiveness of each option on a Likert scale would be related to personality because it taps into test taker?s behavioral tendencies. Results show that the best-and-worst SJT was not related to measures of education (a proxy for cognitive ability) or measures of personality. The SJT on which respondents rated the effectiveness of each option on a Likert scale was related to measures of personality but not education. Finally, because the Likert SJT has a greater number of responses, it was rescored as following the best-and-worst SJT keying. This converted SJT was related to neither measures of education nor measures of personality. The choose best-and-worst SJT significantly predicted performance, while the Likert SJT and a Likert SJT converted to a best-and-worst SJT did not predict performance.

This study explores the differential validity of SJT responding formats (i.e., selecting a response to an SJT item). It was hypothesized that the SJT on which respondents identified the best and worst options would be related to cognitive ability because this type of SJT has a high cognitive load and thus acts like a knowledge test. It was also hypothesized that the SJT on which respondents rated the effectiveness of each option on a Likert scale would be related to personality because it taps into test taker?s behavioral tendencies. Results show that the best-and-worst SJT was not related to measures of education (a proxy for cognitive ability) or measures of personality. The SJT on which respondents rated the effectiveness of each option on a Likert scale was related to measures of personality but not education. Finally, because the Likert SJT has a greater number of responses, it was rescored as following the best-and-worst SJT keying. This converted SJT was related to neither measures of education nor measures of personality. The choose best-and-worst SJT significantly predicted performance, while the Likert SJT and a Likert SJT converted to a best-and-worst SJT did not predict performance.

Doctor of Philosophy / Department of Psychology / Ronald G. Downey / Entrepreneurship represents an important path to job creation, product development and organizational competitive advantage. Therefore, the identification and retention of entrepreneurial talent is of primary importance. The Entrepreneurial Orientation Profile Inventory (EOPI) was developed to evaluate the Proactiveness, Innovativeness and Risk-Taking dimensions of Entrepreneurial Orientation using a situational judgment test (SJT) testing methodology. The current research outlines the initial development of the testing items and provides a preliminary review of the process used to develop a scoring key and evaluate the psychometric properties of the measure among two independent samples. Study 1 focused on developing a key to score and evaluate data in subsequent samples. In Study 1, 49 adult workers provided ratings regarding the most and least effective response to 12 business-related scenarios designed to measure the Proactiveness, Innovativeness and Risk-Taking dimensions of Entrepreneurial Orientation. Interrater consistency analyses were conducted to determine the correct rank order of the response options within the most and least effective response conditions. In the most effective condition, raters reached consensus on the correct ranking of the response options for 7 of the 12 items. In the least effective condition, raters reached consensus on the correct ranking of the response options for 9 of the 12 items. The highest ranked response option was identified as the “correct” response and used as a scoring key in Study 2. This finding suggests individuals are generally better at identifying a single best ineffective solution to a business-related problem, but less effective at identifying a single best effective solution to a business-related problem. Thus, when using an SJT format to evaluate business-related problems, asking respondents to identify the least effective responses is likely to provide better identification of a “correct” response. Items for which the adult sample reached agreement were retained for further examination in Study 2. Study 2 was conducted to evaluate the impact of three response option instruction and scoring methodologies (i.e., “most effective”, least effective” or a combined “most and least effective”) on the reliability and validity of the EOPI measure. Using a sample of 188 undergraduate students, the construct and criterion validity of the EOPI measure as a unidimensional composite and at the item level was evaluated. Across the three conditions, the results of the construct and criterion validity analyses generally failed to support the EOPI instrument as an effective method to evaluate Entrepreneurial Orientation at the composite level. The modest correlation coefficients among the criteria variables suggest a potentially broader measurement issue with currently available measures of Entrepreneurship in general. Within the “least effective” response instruction condition, minor significant results were found at the item level. A review of these items provides insight into how modifications of EOPI items may facilitate future item development. Further, the current research also suggests that biographical data may provide insight into the measurement of Entrepreneurial Orientation. A biodata-based unidimensional composite of Entrepreneurial Behavior was found to be both marginally reliable and significantly related to an alternative measure of Entrepreneurial Orientation. The development of additional biodata items that correlate with the current items is likely to improve the psychometric properties of the Entrepreneurial Orientation composite and provide insight into the role of previous experience as a valid and reliable indicator of Entrepreneurial Orientation and Entrepreneurship behaviors.

No description available.

No description available.

As a low fidelity work simulation, Situational Judgment Tests (SJTs) are an affordable and practical way of empirically linking training and on-the-job performance, thereby providing a viable means of evaluating training effectiveness. An issue, when utilizing SJTs, is deciding the appropriate manner in which the SJT should be scored. Traditional SJT scoring methodologies, while successfully utilized for selection and prediction, pose specific challenges when applied to a SJT designed to evaluate the effectiveness of a training program. This study discusses the shortcomings of traditional SJT scoring methodologies when used in the evaluation context. To overcome these challenges, an innovative scoring methodology, the Hybrid methodology, is presented. This study provides the detailed description of the Hybrid scoring key creation, and compares the Hybrid scoring key with two traditional scoring keys (Subject Matter Expert (SME) and Respondent-based scoring keys). Responses from a military training program are utilized to illustrate the distinctive effects of using the three different scoring approaches. The superiority of the hybrid scoring key, due to increased confidence in the key's accuracy, and findings regarding training evaluation are discussed. Future research directions and practical applications of the research are also discussed. / Master of Science

Mental toughness (MT) has been shown to predict outcomes across a variety of high-stress contexts such as athletics, the military, and the workplace. Despite this, researchers have struggled to reach consensus regarding how best to conceptualize and measure MT. Specifically, MT assessments have focused on measuring general MT rather than domain-specific MT. The current study proposes a measurement model of MT grounded in social-cognitive theory and introduced an assessment of MT within a situational judgment test framework to assess MT in the workplace. Participants completed a battery consisting of the new measure as well as measures intended to establish construct validity. Factor analyses suggested a three-factor solution fit the data best. Furthermore, cross-structure analyses indicated that the new assessment avoided common-method bias in responding, as evident by weak correlations with measures of other constructs. / Master of Science / Mental toughness (MT) has been shown to be a resource that buffers against the negative effects of distress and predicts outcomes across a variety of settings, including the workplace. However, widely used self-report MT questionnaires have numerous issues, such as a lack of context. The current study addressed a number of these issues by creating a measure of MT wherein respondents were given a workplace situation and asked the likelihood that they would respond in a variety of manners, thereby assessing MT as relevant to the workplace context. Three factors of MT were most prominent – task persistence, emotional control, and utilization of feedback. The measure introduced in the current study had small associations with existing self-report measures of MT, personality, and distress, suggesting that the new method of measuring MT avoided some issues inherent to self-report responding. This research laid promising groundwork for the future assessment of MT in the workplace.

This paper investigated the suggestion that situational judgment is a multidimensional evaluation methodology that assesses task and/or contextual job knowledge, and in any given situational judgment test (SJT), there may be items that better tap contextual knowledge while other items may better tap task knowledge. 233 undergraduate students completed questionnaires containing a situational judgment test, personality questionnaire, and cognitive ability test. Results supported the hypothesis that suggested personality significantly predicts contextual knowledge over and above cognitive ability, but did not support the prediction that cognitive ability significantly predicts task knowledge above and beyond personality. Preliminary results suggest that the lack of support for H2 may be due to the SJT utilized in this study, which appears to have tapped primarily contextual knowledge domains. Implications and directions for future research are suggested. / Master of Science

Situational judgment tests are analogous to earlier forms of "high fidelity" simulations such that an ostensible paradox emerges in the consistent finding of criterion-referenced validity but almost complete lack of construct validity evidence. The present study evaluates the extent to which SJT's can demonstrate convergent and discriminant validity by analyzing a SJT from a multitrait-multimethod perspective. A series of hierarchically nested confirmatory factor models were tested. Results indicate that the SJT demonstrates convergent and discriminant validity but also contains non-trivial amounts of construct-irrelevant method variance. Wide variability in the content and validation methods of SJT's are discussed as the reason previous attempts to find construct validity have failed. / Master of Science

The focus of the current dissertation was on the construction and validation of a situational judgment test (SJT) assessing lying/dishonesty in the workplace. The scale was designed to have two dimensions corresponding to two needs based on Socioanalytic Theory: (1) the need to get along and (2) the need to get ahead. Three studies were undertaken in order to create items, pilot test items, and assess both the construct and criterion-related validity of the scale. The result of Study 1 was a pilot-tested scale with six SJT items corresponding to each of the two SJT dimension (12 items total). The dimensions of the new SJT are referred to in the paper as LAl (lying to get along) and LAh (lying to get ahead). Study 2 examined convergent and discriminant validity as well as the nomological network of other individual difference variables expected to be related to LAl and/or LAh. The results of Study 2 largely supported the construct validity of the overall scale but many of the personality constructs based on the Five Factor Model (FFM) were not related to LAl or LAh. In Study 3, LAl and LAh were unrelated to several aspects of supervisor rated performance as well turnover intentions and objective career outcomes (e.g., promotions). The primary contribution of the dissertation was the creation of a construct valid SJT measuring lying in organizations whose dimensions could be assessed reliably using coefficient alpha as opposed to test-retest reliability. Future research directions and limitations of the study are discussed in the final section of the dissertation.