The objective of this dissertation is to understand the circumstances under which non-experimental methods yield unbiased estimates of the effect of magnet school attendance on student achievement. This dissertation has two main analyses. In the first analysis, non-experimental estimates (via multiple regression with observed covariates, analysis of covariance with student fixed effects, and propensity score matching) of the effect of attending one academically selective magnet school on 5th and 6th grade math and reading achievement are compared to experimental estimates found using lottery status as an instrumental variable (IV) for magnet school attendance. This analysis finds that multiple regression and propensity score matching yield estimates with sizeable positive bias that would likely lead a policymaker conclude the magnet school is more effective than it really is; in some cases, this bias represents over half a school years worth of learning. Student fixed-effects modeling performs the best of the non-experimental methods and in reading yields estimates of the magnet effect on 5th and 6th grade achievement that are not meaningfully different from the experimental IV estimates. The second analysis tests how well the experimental and non-experimental methods perform under various forms and rates of sample attrition. To investigate this issue I create a variety of samples with different forms and rates of artificial attrition among lottery winners, lottery losers, and non-participants and then run the experimental and non-experimental estimators on these simulated samples. The second analysis finds that the experimental IV estimates are less biased than the non-experimental estimates in almost all scenarios. The one exception is the student fixed-effects estimator, which performed as well or better than the experimental IV estimator as attrition rates exceeded 40%. Collectively, the findings raise caution against using multiple regression and propensity score matching to evaluate the causal impact of school choice programs, even under situations where attrition in the experimental sample is severe. Student fixed-effects modeling shows promise, particularly in reading, but only under high rates of sample attrition can one expect it to perform better than an analysis using randomly assigned comparison groups.
| Identifer | oai:union.ndltd.org:VANDERBILT/oai:VANDERBILTETD:etd-10052009-210839 |
| Date | 15 October 2009 |
| Creators | Stuit, David Alan |
| Contributors | Mark Berends, Thomas M. Smith, R. Dale Ballou, Matthew G. Springer |
| Publisher | VANDERBILT |
| Source Sets | Vanderbilt University Theses |
| Language | English |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | http://etd.library.vanderbilt.edu/available/etd-10052009-210839/ |
| Rights | unrestricted, I hereby certify that, if appropriate, I have obtained and attached hereto a written permission statement from the owner(s) of each third party copyrighted matter to be included in my thesis, dissertation, or project report, allowing distribution as specified below. I certify that the version I submitted is the same as that approved by my advisory committee. I hereby grant to Vanderbilt University or its agents the non-exclusive license to archive and make accessible, under the conditions specified below, my thesis, dissertation, or project report in whole or in part in all forms of media, now or hereafter known. I retain all other ownership rights to the copyright of the thesis, dissertation or project report. I also retain the right to use in future works (such as articles or books) all or part of this thesis, dissertation, or project report. |
Page generated in 0.002 seconds