The assessment of the performance of learners by means of benchmark experiments is established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to benchmark studies in a supervised learning situation based on artificial and real-world data. / Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

The purpose of this research is to examine the bootstrap and jackknife as methods for estimating the variance of the AUC from a study using a complex sampling design and to determine which characteristics of the sampling design effects this estimation. Data from a one-stage cluster sampling design of 10 clusters was examined. Factors included three true AUCs (.60, .75, and .90), three prevalence levels (50/50, 70/30, 90/10) (non-disease/disease), and finally three number of clusters sampled (2, 5, or 7). A simulated sample was constructed for each of the 27 combinations of AUC, prevalence and number of clusters. Estimates of the AUC obtained from both the bootstrap and jackknife methods provide unbiased estimates for the AUC. In general it was found that bootstrap variance estimation methods provided smaller variance estimates. For both the bootstrap and jackknife variance estimates, the rarer the disease in the population the higher the variance estimate. As the true area increased the variance estimate decreased for both the bootstrap and jackknife methods. For both the bootstrap and jackknife variance estimates, as number of clusters sampled increased the variance decreased, however the trend for the jackknife may be effected by outliers. The National Health and Nutrition Examination Survey (NHANES) conducted by the CDC is a complex survey which implements the use of the one-stage cluster sampling design. A subset of the 2001-2002 NHANES data was created looking only at adult women. A separate logistic regression analysis was conducted to determine if exposure to certain furans in the environment have an effect on abnormal levels of four hormones (FSH, LH, TSH, and T4) in women. Bootstrap and jackknife variance estimation techniques were applied to estimate the AUC and variances for the four logistic regressions. The AUC estimates provided by both the bootstrap and jackknife methods were similar, with the exception of LH. Unlike in the simulated study, the jackknife variance estimation method provided consistently smaller variance estimates than bootstrap. AUC estimates for all four hormones suggested that exposure to furans effects abnormal levels of hormones more than expected by chance. The bootstrap variance estimation technique provided better variance estimates for AUC when sampling many clusters. When only sampling a few clusters or as in the NHANES study where the entire population was treated as a single cluster, the jackknife variance estimation method provides smaller variance estimates for the AUC.

In this diploma thesis we explain the main principles and properties of bootstrap methods, that can be used to conduct the statistical inference in linear and nonlinear financial time series. We will introduce basic ideas of bootstrap methods for the case when observations can be considered as independent random variables, and afterwards we will describe more advanced methods, that can be successfully used when we are dealing with time series. Thesis deals with both parametric bootstrap methods, that we can use when the underlying parametric model of observations is available, as well as with nonparametric bootstrap methods that are used when more general nonparametric model of time series data is considered. The main objective is to compare particular bootstrap methods and show the usage of these methods on real world data. There is also a basic time series theory included in the work. 1

The research deals with bootstrap interval estimation of wildlife population sizes from multiple surveys in the Hluhluwe-Umfolosi Park. The jackknife procedure was also used to provide the standard errors for the survey estimates. The main wildlife speciese studied in the research were the White and Black Rhino. The survey estimates for the wildlife species were obtained using line transect sampling and mark-recapture methods respectively. The bootstrap and jackknife procedures were applied separately to each of the datasets. Bootstrap estimates for each of the time point were obtained and the confidence intervals of the bootstrap estimates were constructed using percentile and standard methods. The coverage probability was assessed using the Monte Carlo simulations. Only the nonparametric bootstrap was applied in this research and the results were compared to the jackknife results. The lengths of the confidence intervals were used to assess the confidence intervals with a shorter confidence interval being more exact. The estimates used for both the bootstrap and jackknife methodology were based on a simple state space model. The discrete state space model used was proposed by Fatti et al (2002). State space models provide a natural framework for estimating and predicting animal population abundance given partial or inexact information. The model takes into account the (unknown) birth rate in the population and all known losses (mortalities and relocations) and gains (introductions) in the population between successive surveys as well as the errors in the survey estimates.

Ringo M.H. Ho. / Thesis (M.Phil.)--Chinese University of Hong Kong, 1997. / Includes bibliographical references (leaves 75-81). / ACKNOWLEDGMENT --- p.2 / ABSTRACT --- p.3 / TABLE OF CONTENTS --- p.5 / LIST OF TABLES --- p.8 / Chapter CHAPTER 1 --- PREVIOUS STUDIES ON USING PROCRUSTES ROTATION TO ASSESS FACTORIAL INVARIANCE --- p.10 / Factorial invariance problem --- p.10 / Procrustes rotation with congruent coefficient as a way to test factorial invariance --- p.11 / Quantifying the Procrustes fit --- p.14 / Outline of the present study --- p.15 / Chapter CHAPTER 2 --- A CRITICAL EVALUATION OF THE PERMUTATION METHOD --- p.18 / Introduction --- p.18 / Method --- p.19 / Results and Discussions --- p.21 / Chapter CHAPTER 3 --- BOOTSTRAP TESTING PROCEDURE FOR A FULLY SPECIFIED TARGET --- p.24 / Introduction --- p.24 / A brief introduction to the bootstrap procedure --- p.24 / The bootstrap testing procedure for a fully specified target --- p.26 / Method --- p.28 / Results and Discussions --- p.28 / Chapter CHAPTER 4 --- BOOTSTRAP TESTING FOR A PARTIALLY SPECIFIED TARGET --- p.33 / Introduction --- p.33 / The bootstrap testing procedure for a partially specified target --- p.36 / Method --- p.38 / Quantifying the fit - congruence coefficients for the partial target rotation --- p.39 / Results and Discussions --- p.40 / Chapter CHAPTER 5 --- FURTHER EXTENSIONS OF THE BOOTSTRAP METHOD --- p.45 / Introduction --- p.45 / First extension - when correlation matrix is used --- p.45 / The modified bootstrap procedure --- p.45 / Method --- p.48 / Results and Discussions --- p.48 / Second extension - when raw data of the target sample is not available --- p.49 / The conditional bootstrap procedure for a fully specified target --- p.49 / Method --- p.50 / Results and Discussions --- p.51 / Chapter CHAPTER 6 --- THREE REAL EXAMPLES --- p.54 / Example 1 - Testing factorial invariance of CPAI between two random split samples --- p.54 / Results --- p.55 / Example 2 - Testing factorial invariance of CPAI between Chinese males and females --- p.56 / Results --- p.57 / Example 3 - Cross-cultural comparison of the Big Five Model between U. S. and Chinese samples --- p.58 / Results --- p.59 / Chapter CHAPTER 7 --- CONCLUSIONS --- p.62 / Practical remarks on the bootstrap procedure --- p.62 / A note on the transformation on the sample for constructing correct resampling space --- p.64 / Remarks on utilizing the congruence coefficients --- p.65 / How good are the congruence coefficients in detecting discrepancy between two factor structures? --- p.68 / Rule of thumb for factor congruence coefficient in checking factor replicability --- p.68 / Sample size requirement --- p.69 / Limitations of the present study --- p.70 / Direction of future studies --- p.71 / Concluding remarks --- p.73 / REFERENCES --- p.75 / NOTES --- p.82 / APPENDIX1 --- p.83 / TABLES 1 TO TABLES17 --- p.84

Cheung, Yu Hin Ray. / Thesis (M.Phil.)--Chinese University of Hong Kong, 2008. / Includes bibliographical references (leaves 47-49). / Abstracts in English and Chinese. / Chapter CHAPTER ONE: --- INTRODUCTION --- p.1 / Chapter CHAPTER TWO: --- A UNIFIED BOOTSTRAP PROCEDURE --- p.7 / Chapter CHAPTER THREE: --- A SIMULATION STUDY --- p.10 / Chapter CHAPTER FOUR: --- RESULTS --- p.18 / Chapter CHAPTER FIVE: --- DISCUSSION --- p.33 / Chapter CHAPTER SIX: --- CONCLUSION --- p.37 / APPENDICES --- p.38 / REFERENCES --- p.46

"This paper describes our work in exploring trading strategies for the leveraged exchange-traded funds, Direxion Daily Financial Bull 3X (FAS) and Direxion Daily Financial Bear 3X (FAZ) over the first three quarters of 2009. Using minute-by-minute stock data we are able to verify the accuracy of these ETFs in regards to their target of the Russell 1000 Financial Index (RIFIN). We are then able to quantify the returns and risks involved with trading strategies that seek to exploit the ETFs objectives, specifically momentum trades, tracking-error discrepancy trades, and a combination of the two strategies we term “discount-and-up.” Bootstrap simulation techniques are employed to measure values at risk and conditional tail expectations over 30 day time horizons for each strategy. Lastly, we demonstrate the dangers of traditional buy-and-hold investing with regards to leveraged ETFs."

The focus of this thesis is extremal dependence among spatial observations. In particular, this research extends the notion of the extremogram to the spatial process setting. Proposed by Davis and Mikosch (2009), the extremogram measures extremal dependence for a stationary time series. The versatility and flexibility of the concept made it well suited for many time series applications including from finance and environmental science. After defining the spatial extremogram, we investigate the asymptotic properties of the empirical estimator of the spatial extremogram. To this end, two sampling scenarios are considered: 1) observations are taken on the lattice and 2) observations are taken on a continuous region in a continuous space, in which the locations are points of a homogeneous Poisson point process. For both cases, we establish the central limit theorem for the empirical spatial extremogram under general mixing and dependence conditions. A high level overview is as follows. When observations are observed on a lattice, the asymptotic results generalize those obtained in Davis and Mikosch (2009). For non-lattice cases, we define a kernel estimator of the empirical spatial extremogram and establish the central limit theorem provided the bandwidth of the kernel gets smaller and the sampling region grows at proper speeds. We illustrate the performance of the empirical spatial extremogram using simulation examples, and then demonstrate the practical use of our results with a data set of rainfall in Florida and ground-level ozone data in the eastern United States. The second part of the thesis is devoted to bootstrapping and variance estimation with a view towards constructing asymptotically correct confidence intervals. Even though the empirical spatial extremogram is asymptotically normal, the limiting variance is intractable. We consider three approaches: for lattice data, we use the circular bootstrap adapted to spatial observations, jackknife variance estimation, and subsampling variance estimation. For data sampled according to a Poisson process, we use subsampling methods to estimate the variance of the empirical spatial extremogram. We establish the (conditional) asymptotic normality for the circular block bootstrap estimator for the spatial extremogram and show L2 consistency of the variance estimated by jackknife and subsampling. Then, we propose a portmanteau style test to check the existence of extremal dependences at multiple lags. The validity of confidence intervals produced from these approaches and a portmanteau style test are demonstrated through simulation examples. Finally, we illustrate this methodology to two data sets. The first is the amount of rainfall over a grid of locations in northern Florida. The second is ground-level ozone in the eastern United States, which are recorded on an irregularly spaced set of stations.

Orientador: Luzia Aparecida Trinca / Resumo: Em problemas de regressão, na busca por um modelo parcimonioso, o pesquisador pode se deparar com adversidades, por exemplo, a existência de colinearidade entre as regressoras, dificultando a seleção de variáveis. Dessa forma, com a implementação de ferramentas inspiradas nas propostas de Murray et al. (2013), Muller & Welsh (2010) e Jiang et al. (2009) no pacote mplot (Tarr et al., 2018) no software R, pode-se, gráfica e interativamente, estudar em detalhes a estabilidade e a importância de inclusão de covariáveis para a construção de modelos. Neste trabalho, medidas de estabilidade e probabilidade de inclusão de variáveis foram obtidas pelo método bootstrap. Medidas resumo de qualidade do ajuste são baseadas no critério de informação generalizado, que incorpora, como casos particulares, os critérios de informação de Akaike e o Bayesiano, e reflete a perda (associada ao ajuste de um modelo simplificado) mais uma penalização à complexidade do modelo. Ao aplicar a teoria de seleção de variáveis, utilizando as ferramentas gráficas no ajuste de um modelo de regressão linear Normal e regressão Binomial, foi possível reconhecer seu potencial e utilidade no processo de formulação de modelos, no qual a incorporação de conhecimento do especialista da área pode ser feita de maneira natural, já que o processo não é automático. Isso é mais um diferencial em relação aos métodos usuais de seleção de variáveis que também foram aplicados aos mesmos conjuntos de dados para efeito de discussã... (Resumo completo, clicar acesso eletrônico abaixo) / Abstract: In regression analysis, the search of a parsimonious model can be difficult due to collinearities among variables and other problems. Murray et al. (2013), Muller & Welsh (2010) e Jiang et al. (2009) proposed tools for model stability and variable inclusion plots that were re ned and implemented in the mplot package of Tarr et al. (2018), which allows interactive graphs and summaries of information relevant to model building. Stability measures and the probability of variable inclusion are obtained through bootstrapping. Goodness of fit measures are based on the generalized information criterion, which includes as particular cases the Akaike and Bayesian information criteria, given by a measure of loss of the fit and a penalization due to model complexity. Applying the method to t a Normal linear regression and a Binomial regression revealed its great potential and usefulness for model building, allowing expertise knowledge to be incorporated since the selection model is not automated. This is a further contrast to the usual selection methods which were also applied to the same datasets in order to discuss the differences. / Mestre

.