It is shown that a limitation of the various collation methods for paired comparison data currently available is their lack of validity when used in cases where the experiment is incomplete and particularly when the judgements are not replicated. Presented in this thesis is a reasonably thorough background to the method of paired comparisons and an overview of the existing methods for collating paired comparison data into a final ranking. As a result of the extensive review of existing collation methods, the thesis progresses logically to a new collation method that utilises all the available information from a set of pairwise preferences. The performance of the new collation method is extensively tested against existing methods by way of a simulation exercise which highlights the performance of the collation methods under different scenarios in terms of experiment size, experiment completeness and judgement consistency, as well as by considering the number of direct comparisons and the strength of competition. The new collation method and the existing collation method of Allen (1992) are applied to a set of real world data and the outcomes of the two methods are compared. The usefulness of paired comparisons in understanding the way judges use information to construct their own criteria when instructed to make preference decisions at a broad level is also considered and a real world application of this approach is performed. The main findings of this thesis are: FnThe new methodology generally provides an improved performance when there are more than 10 objects to be ranked; FnReplication of each pairwise judgement certainly improves the accuracy of the overall ranking, regardless of the level of judgement inconsistency; FnIn the case of non-replication, the accuracy of the final ranking greatly improves as judgement consistency improves. In other words, if it is not possible to replicate individual pairwise judgements then high judgement consistency is important for a reasonable result; In the case of replication, the accuracy of the returned ranking improves with judgement consistency only in the case of the new method. For the existing methods, the accuracy actually decreases marginally with the improvement of judgement consistency, particularly if there is a low level of experiment completeness; In terms of experiment completeness, for non-replicated experiments, there is an increase in the accuracy of the returned ranking as the proportion of possible pairwise preferences completed increases, but not to the same extent as an increase in judgement consistency. That is, judgement consistency is actually more important than experiment completeness. This suggests that control over the design of the experiment (the extent of completeness and which pairwise preferences are completed) is less important than judgement consistency and replication ¡V certainly a finding not found reported in the literature; The new method outperforms the existing methods when there is perfect or very high judgement consistency.
| Identifer | oai:union.ndltd.org:ADTP/195243 |
| Date | January 2007 |
| Creators | Lenton, Richard, n/a |
| Publisher | Griffith University. School of Australian Environmental Studies |
| Source Sets | Australiasian Digital Theses Program |
| Language | English |
| Detected Language | English |
| Rights | http://www.gu.edu.au/disclaimer.html), Copyright Richard Lenton |
Page generated in 0.0014 seconds