Back to search

University of Victoria (Canada)

A comparison of independent samples t-test, approximate randomization test and bootstrap randomization test to departures from population normality : a Monte Carlo study

Abstract

dc:description.abstract

A Monte Carlo study was undertaken to compare the two independent sample t-test (parametric) with the approximate randomization test and the bootstrap randomization test using data sampled from the Micceri distributions, a set of non-normal distributions. The three tests were compared with respect to Type I errors for varying sample sizes. Following the method of Sawilowsky and Blair ( 1992), Monte Carlo methods were used to sample the eight Micceri distributions with sample sizes of (5, 15), (7, 7), (10, 10), (10, 30), (20, 20), (20, 60), (60, 60). Observations were sampled independently with replacement from each of the eight distributions. Replicating Sawilowsky and Blair (1992), the t-test showed marked non-robustness when matched with many of Micceri's distributions and the departures in Type I error from the five percent pre-set value of the study were greater than those found in previous studies. These results suggest that the probability estimates associated with the t-test are unreliable. The present study extends these' results to the alternative nonparametric methods of the bootstrap and approximate randomization tests as follows: when the distribution is extremely asymmetric, the bootstrap method outperforms both the t-test and approximate randomization test. Also, for non-symmetric distributions, pooled sample sizes of at least 30 to 40 would appear necessary to overcome the tendency of the t-test to underestimate the preset value when compared to the bootstrap test. For symmetric distributions (skew of an absolute value less than one), the t-test meets the criterion (i.e. it is within 10% of the preset value) for pooled sample sizes of 20 or more. The t-test lacks robustness for small (less than 20) unequal sample sizes from non-symmetric distributions and also is not robust for the very small and equal samples (7, 7) even for symmetric distributions. The approximate randomization test is consistently conservative for small sample sizes or unequal sample sizes. Whereas for the large sample sizes (60,60), this test produces a Type I error 4.6% of the time versus the 5% present value. The approximate randomization test never over estimates the preset value under the criterion. Prior to further research, I recommend that anyone using small sample sizes (less than 20 subjects in either group) consider using both parametric and nonparametric tests. A frequency plot of the sample data should be examined for breaches of the symmetry assumption of the normal distribution. If such a breach does occur then nonparametric tests should be applied.

Degree

thesis:*
Level thesis:degree_level
Master of Arts M.A.
Department thesis:degree_department
Department of Psychology
Year dc:date.issued
1997

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Miles, Stephen Allen

Rights

dc:rights
Statement dc:rights
  • Available to the World Wide Web

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1828/18985

Chain of custody

source
Harvested from
University of Victoria (Canada)
Base URL
dspace.library.uvic.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Miles, Stephen Allen. A comparison of independent samples t-test, approximate randomization test and bootstrap randomization test to departures from population normality : a Monte Carlo study. Master of Arts M.A. thesis, 1997. https://hdl.handle.net/1828/18985