BACKGROUND:Available measures of patient-reported outcomes for complementary and alternative medicine (CAM) inadequately capture the range of patient-reported treatment effects. The Self-Assessment of Change questionnaire was developed to measure multi-dimensional shifts in well-being for CAM users. With content derived from patient narratives, items were subsequently focused through interviews on a new cohort of participants. Here we present the development of the final version in which the content and format is refined through cognitive interviews.METHODS:We conducted cognitive interviews across five iterations of questionnaire refinement with a culturally diverse sample of 28 CAM users. In each iteration, participant critiques were used to revise the questionnaire, which was then re-tested in subsequent rounds of cognitive interviews. Following all five iterations, transcripts of cognitive interviews were systematically coded and analyzed to examine participants' understanding of the format and content of the final questionnaire. Based on this data, we established summary descriptions and selected exemplar quotations for each word pair on the final questionnaire.RESULTS:The final version of the Self-Assessment of Change questionnaire (SAC) includes 16 word pairs, nine of which remained unchanged from the original draft. Participants consistently said that these stable word pairs represented opposite ends of the same domain of experience and the meanings of these terms were stable across the participant pool. Five pairs underwent revision and two word pairs were added. Four word pairs were eliminated for redundancy or because participants did not agree on the meaning of the terms. Cognitive interviews indicate that participants understood the format of the questionnaire and considered each word pair to represent opposite poles of a shared domain of experience.CONCLUSIONS:We have placed lay language and direct experience at the center of questionnaire revision and refinement. In so doing, we provide an innovative model for the development of truly patient-centered outcome measures. Although this instrument was designed and tested in a CAM-specific population, it may be useful in assessing multi-dimensional shifts in well-being across a broader patient population.

The purpose of this study was to propose and evaluate a procedure for revising an existing self-administered survey that is in need of item revision and/or scale reduction while maximizing validity and reliability. The procedure was demonstrated using the Client Evaluation of Self and Treatment (CEST; Joe, Broome, Rowan-Szal, & Simpson, 2002), a self-administered survey used in drug and alcohol treatment agencies. The procedure included confirmatory and exploratory factor analyses of a large dataset of completed CEST surveys, a readability analysis, and cognitive interviewing of two different groups of respondents to determine what problems they might have with CEST items. The cognitive interviewing revealed a number of issues that led to confusion among respondents, including items with two distinct concepts embedded, items containing absolutes and vague qualifiers, misinterpreted items, and terms and phrases respondents had difficulty understanding. The CEST was also judged to be long and potentially burdensome to respondents. Based on the results of this evaluation, a new survey—the Brief Assessment of Self in Context (BASIC)—also intended for use by substance abuse treatment providers, was constructed. First, factor analyses of the CEST and advice from an expert panel were used to determine which scales to retain. Next, quantitative analyses and cognitive interviewing helped determine which CEST items to retain and which to revise. Readability, sound item writing principles, and response format and scale requirements were also used to determine which items to include in the initial draft of the BASIC and guided item construction when needed. After the panel of experts provided feedback on the first revision, a final draft was prepared. Another round of cognitive interviewing was followed by administration of the final draft of the survey to a representative sample. The results indicated that the BASIC’s items are clear, unambiguous, and easy to interact with and understand, and that the instrument is an improvement over the CEST. In brief, the procedure demonstrated in this study produced a psychometrically sound instrument composed of items that are easy for respondents to access. / text