Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. There have been few studies of the evaluator effect in usability testing (UT), task-based testing with end-users. UEM comparison studies focus on counting usability problems detected, but we also need to assess the content of usability problem descriptions (UPDs) to more fully measure evaluation effectiveness. The goals of this research were to develop UPD guidelines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content. Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines. There was support for existence of an evaluator effect, even when watching pre-recorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practitioners received higher grades averaged across the 6 guidelines than students did, suggesting that the guidelines may be useful for grading reports. The grades for the guidelines were not correlated with thoroughness, suggesting that the guideline grades complement measures of problem detection. A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness. / Ph. D.

Heuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field. / A Avaliação Heurística (AH) é um método popular de inspeção de usabilidade. Entretanto, seus resultados são dependentes da experiência dos avaliadores. Este estudo explorou e descreveu a diferença na qualidade de resultados (relatórios) de AH colaborativa conduzida por grupos de avaliadores de composição distinta, considerando diferentes quantidades de avaliadores experientes em cada grupo. Vinte e sete (27) avaliadores contribuíram voluntariamente com este estudo, nove (9) experientes e 18 novatos. Assim, foram organizados sete (7) grupos de AH, de acordo com quatro (4) níveis diferentes do fator presença de avaliador experiente, variando de nenhum experiente até três (3) avaliadores experientes no mesmo grupo. Cada grupo de avaliadores concordou em entregar seus relatórios de AH para este estudo. A partir de tais relatórios, foi conduzida uma análise comparativa baseada em métodos específicos da área, e também baseado em uma análise de agrupamento com base em medidas de similaridade. Como resultado, descreveu-se as medidas F (F-measure) referentes ao relatório de cada grupo respeitando critérios estritos e relaxados de comparação. Além disto, foram descritos os dendrogramas resultados das análises de agrupamento. Os resultados mostraram que a qualidade de relatórios de AH colaborativas conduzidas por avaliadores experientes e novatos juntos pode ser mais similar à qualidade de relatórios de AH tradicional conduzida por múltiplos avaliadores experientes (Grupo Benchmark) do que à qualidade de relatórios de AH colaborativa conduzida por grupos formados apenas por avaliadores novatos (Grupo Baseline). Finalmente, discutiu-se resultados adicionais e implicações para pesquisas futuras na área.

Heuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field. / A Avaliação Heurística (AH) é um método popular de inspeção de usabilidade. Entretanto, seus resultados são dependentes da experiência dos avaliadores. Este estudo explorou e descreveu a diferença na qualidade de resultados (relatórios) de AH colaborativa conduzida por grupos de avaliadores de composição distinta, considerando diferentes quantidades de avaliadores experientes em cada grupo. Vinte e sete (27) avaliadores contribuíram voluntariamente com este estudo, nove (9) experientes e 18 novatos. Assim, foram organizados sete (7) grupos de AH, de acordo com quatro (4) níveis diferentes do fator presença de avaliador experiente, variando de nenhum experiente até três (3) avaliadores experientes no mesmo grupo. Cada grupo de avaliadores concordou em entregar seus relatórios de AH para este estudo. A partir de tais relatórios, foi conduzida uma análise comparativa baseada em métodos específicos da área, e também baseado em uma análise de agrupamento com base em medidas de similaridade. Como resultado, descreveu-se as medidas F (F-measure) referentes ao relatório de cada grupo respeitando critérios estritos e relaxados de comparação. Além disto, foram descritos os dendrogramas resultados das análises de agrupamento. Os resultados mostraram que a qualidade de relatórios de AH colaborativas conduzidas por avaliadores experientes e novatos juntos pode ser mais similar à qualidade de relatórios de AH tradicional conduzida por múltiplos avaliadores experientes (Grupo Benchmark) do que à qualidade de relatórios de AH colaborativa conduzida por grupos formados apenas por avaliadores novatos (Grupo Baseline). Finalmente, discutiu-se resultados adicionais e implicações para pesquisas futuras na área.

Heuristic Evaluation (HE) is a popular usability inspection method. Yet little is known about the effect the evaluators have on the outcome of HE. One potentially important feature of evaluators is their end-user status, that is, whether or not they are end-users for whom the interface is designed. I completed a detailed review of the HE literature, combined sources, developed an explicit method for conducting an HE and trained HE novices from different work domains using it. Using these methods I conducted a preliminary randomized crossover study (n=6) of the effect of end-user status during the inspection and merging stages of HE. I estimate a larger study of approximately 148 end-users would be needed to test hypotheses regarding end-user status. I demonstrated a novel measure of the effect of end-user status for the merging stage of HE, which I called the measure of matching similarity (MMS).

Heuristic Evaluation (HE) is a popular usability inspection method. Yet little is known about the effect the evaluators have on the outcome of HE. One potentially important feature of evaluators is their end-user status, that is, whether or not they are end-users for whom the interface is designed. I completed a detailed review of the HE literature, combined sources, developed an explicit method for conducting an HE and trained HE novices from different work domains using it. Using these methods I conducted a preliminary randomized crossover study (n=6) of the effect of end-user status during the inspection and merging stages of HE. I estimate a larger study of approximately 148 end-users would be needed to test hypotheses regarding end-user status. I demonstrated a novel measure of the effect of end-user status for the merging stage of HE, which I called the measure of matching similarity (MMS).