Cross validation (CV) is widely used for model assessment and comparison. In this thesis, we first review and compare three v-fold CV strategies: best single CV, repeated and averaged CV and double CV. The mean squared errors of the CV strategies in estimating the best predictive performance are illustrated by using simulated and real data examples. The results show that repeated and averaged CV is a good strategy and outperforms the other two CV strategies for finite samples in terms of the mean squared error in estimating prediction accuracy and the probability of choosing an optimal model. In practice, when we need to compare many models, conducting repeated and averaged CV strategy is not computational feasible. We develop an efficient sequential methodology for model comparison based on CV. It also takes into account the randomness in CV. The number of models is reduced via an adaptive, multiplicity-adjusted sequential algorithm, where poor performers are quickly eliminated. By exploiting matching of individual observations, it is sometimes even possible to establish the statistically significant inferiority of some models with just one execution of CV. This adaptive and computationally efficient methodology is demonstrated on a large cheminformatics data set from PubChem. Cross validated mean squared error (CVMSE) is widely used to estimate the prediction mean squared error (MSE) of statistical methods. For linear models, we show how CVMSE depends on the number of folds, v, used in cross validation, the number of observations, and the number of model parameters. We establish that the bias of CVMSE in estimating the true MSE decreases with v and increases with model complexity. In particular, the bias may be very substantial for models with many parameters relative to the number of observations, even if v is large. These results are used to correct CVMSE for its bias. We compare our proposed bias correction with that of Burman (1989), through simulated and real examples. We also illustrate that our method of correcting for the bias of CVMSE may change the results of model selection.

Cross validation (CV) is widely used for model assessment and comparison. In this thesis, we first review and compare three v-fold CV strategies: best single CV, repeated and averaged CV and double CV. The mean squared errors of the CV strategies in estimating the best predictive performance are illustrated by using simulated and real data examples. The results show that repeated and averaged CV is a good strategy and outperforms the other two CV strategies for finite samples in terms of the mean squared error in estimating prediction accuracy and the probability of choosing an optimal model. In practice, when we need to compare many models, conducting repeated and averaged CV strategy is not computational feasible. We develop an efficient sequential methodology for model comparison based on CV. It also takes into account the randomness in CV. The number of models is reduced via an adaptive, multiplicity-adjusted sequential algorithm, where poor performers are quickly eliminated. By exploiting matching of individual observations, it is sometimes even possible to establish the statistically significant inferiority of some models with just one execution of CV. This adaptive and computationally efficient methodology is demonstrated on a large cheminformatics data set from PubChem. Cross validated mean squared error (CVMSE) is widely used to estimate the prediction mean squared error (MSE) of statistical methods. For linear models, we show how CVMSE depends on the number of folds, v, used in cross validation, the number of observations, and the number of model parameters. We establish that the bias of CVMSE in estimating the true MSE decreases with v and increases with model complexity. In particular, the bias may be very substantial for models with many parameters relative to the number of observations, even if v is large. These results are used to correct CVMSE for its bias. We compare our proposed bias correction with that of Burman (1989), through simulated and real examples. We also illustrate that our method of correcting for the bias of CVMSE may change the results of model selection.

Cross validation (CV) is widely used for model assessment and comparison. In this thesis, we first review and compare three v-fold CV strategies: best single CV, repeated and averaged CV and double CV. The mean squared errors of the CV strategies in estimating the best predictive performance are illustrated by using simulated and real data examples. The results show that repeated and averaged CV is a good strategy and outperforms the other two CV strategies for finite samples in terms of the mean squared error in estimating prediction accuracy and the probability of choosing an optimal model. In practice, when we need to compare many models, conducting repeated and averaged CV strategy is not computational feasible. We develop an efficient sequential methodology for model comparison based on CV. It also takes into account the randomness in CV. The number of models is reduced via an adaptive, multiplicity-adjusted sequential algorithm, where poor performers are quickly eliminated. By exploiting matching of individual observations, it is sometimes even possible to establish the statistically significant inferiority of some models with just one execution of CV. This adaptive and computationally efficient methodology is demonstrated on a large cheminformatics data set from PubChem. Cross validated mean squared error (CVMSE) is widely used to estimate the prediction mean squared error (MSE) of statistical methods. For linear models, we show how CVMSE depends on the number of folds, v, used in cross validation, the number of observations, and the number of model parameters. We establish that the bias of CVMSE in estimating the true MSE decreases with v and increases with model complexity. In particular, the bias may be very substantial for models with many parameters relative to the number of observations, even if v is large. These results are used to correct CVMSE for its bias. We compare our proposed bias correction with that of Burman (1989), through simulated and real examples. We also illustrate that our method of correcting for the bias of CVMSE may change the results of model selection. / Science, Faculty of / Statistics, Department of / Graduate

Clustering is a technique used to partition unlabelled data into meaningful groups. This thesis will focus on the area of clustering called model-based clustering, where it is assumed that data arise from a finite number of subpopulations, each of which follows a known statistical distribution. The number of groups and shape of each group is unknown in advance, and thus one of the most challenging aspects of clustering is selecting these features. Cross-validation is a model selection technique which is often used in regression and classification, because it tends to choose models that predict well, and are not over-fit to the data. However, it has rarely been applied in a clustering framework. Herein, cross-validation is applied to select the number of groups and covariance structure within a family of Gaussian mixture models. Results are presented for both real and simulated data. / Ontario Graduate Scholarship Program

Locally weighted regression is a powerful tool that allows the estimation of different sets of coefficients for each location in the underlying data, challenging the assumption of stationary regression coefficients across a study region. The accuracy of LWR largely depends on how a researcher establishes the relationship across locations, which is often constructed using a weight matrix or function. This paper explores the different kernel functions used to assign weights to observations, including Gaussian, bi-square, and tri-cubic, and how the choice of weight variables and window size affects the accuracy of the estimates. We guide this choice through the cross-validation approach and show that the bi-square function outperforms the choice of other kernel functions. Our findings demonstrate that an optimal window size for LWR models depends on the cross-validation (CV) approach employed. In our empirical application, the full-sample CV guides the choice of a higher window-size case, and CV by proxy guides the choice of a lower window size. Since the CV by Proxy approach focuses on the predictive ability of the model in the vicinity of one specific point (usually a policy point/site), we note that guiding a model choice through this approach makes more intuitive sense when the aim of the researcher is to predict the outcome in one specific site (policy or target point). To identify the optimal weight variables, while we suggest exploring various combinations of weight variables, we argue that an efficient alternative is to merge all continuous variables in the dataset into a single weight variable. / M.A. / Locally weighted regression (LWR) is a statistical technique that establishes a relationship between dependent and explanatory variables, focusing primarily on data points in proximity to a specific point of interest/target point. This technique assigns varying degrees of importance to the observations that are in proximity to the target point, thereby allowing for the modeling of relationships that may exhibit spatial variability within the dataset. The accuracy of LWR largely depends on how researchers define relationships across different locations/studies, which is often done using a “weight setting”. We define weight setting as a combination of weight functions (determines how the observations around a point of interest are weighted before they enter the model), weight variables (determines proximity between the point of interest and all other observations), and window sizes (determines the number of observations that can be allowed in the local regression). To find which weight setting is an optimal one or which combination of weight functions, weight variables, and window sizes generates the lowest predictive error, researchers often employ a cross-validation (CV) approach. Cross-validation is a statistical method used to assess and validate the performance of a predictive model. It entails removing a host observation (a point of interest), predicting that point, and evaluating the accuracy of such predicted point by comparing it with its actual value. In our study, we employ two CV approaches. The first one is a full-sample CV approach, where we remove a host observation, and predict it using the full set of observations used in the given local regression. The second one is the CV by proxy approach, which uses a similar mechanism as full-sample CV to check the accuracy of the prediction, however, by focusing only on the vicinity points that share similar characteristics as a target point. We find that the bi-square function consistently outperforms the choice of Gaussian and tri-cubic weight functions, regardless of the CV approaches. However, the choice of an optimal window size in LWR models depends on the CV approach that we employ. While the full-sample CV method guides us toward the selection of a larger window size, the CV by proxy directs us toward a smaller window size. In the context of identifying the optimal weight variables, we recommend exploring various combinations of weight variables. However, we also propose an efficient alternative, which involves using all continuous variables within the dataset into a single-weight variable instead of striving to identify the best of thousands of different weight variable settings.

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / Crop models are important tools to improve the management and yield of agricultural systems. These improvements are helpful to meet the growing food and fuel demand without increase the crop areas. The conventional approach for calibrating/validating a crop model considers few to many experiments. However, few experiments could lead to higher uncertainties and a large number of experiments is too expensive. Traditionally, the classical procedure use to share an experimental dataset one part to calibrate and the other to validate the model. However, if only few experiments are available, split it could increase the uncertainties on simulation performance. On the other hand, to calibrate/validate the model using several experiments is too expensive and time consuming. Methods that can optimize these procedures, decreasing the processing time and costs, with a reliable performance are always welcome. The first chapter of this study was conducted to evaluate and compare a statistically robust method with the classical calibration/validation procedure. These two procedure, were applied to estimate the genetic coefficients of the CROPGRO-soybean model, using multiple experiments. The cross-validation leave-one-out method, was applied to 21 experiments, using the NA 5909 RG variety, across a southern state of Brazil. The cross-validation reduced the classical calibration/validation procedure average RMSE from 2.6, 4.6, 4.8, 7.3, 10.2, 677 and 551 to 1.1, 4.1, 4.1, 6.2, 6.3, 347 and 447 for emergence, R1, R3, R5, R7 (days), grains.m-2 and kg.ha-1, respectively. There was stability in the estimated ecotype and genetic coefficient among the 21 experiments. Considering the wide range of environment conditions, the CROPGRO-soybean model provided robust predictions of phenology, biomass and grain yield. Finally, to improve the calibration/validation procedure performance, the cross-validation method should be used whenever possible. For the second chapter of this study, the main objectives were to evaluate the calibration/validation uncertainties using different numbers of experiments and to find out the minimum number of experiments required for a reliable CROPGRO-Soybean simulation. This study also used 21 field experiments (BMX Potencia RR variety) sown in eight different locations of Southern Brazil between 2010 and 2014. The experiments were grouped in four classes (Individual sowings, season/year per location, experimental sites, and all data together). As the grouping level increase, the developmental stages RRMSE (%), decreased from 22.2% to 7.8% from individual sowings to all data together, respectively. The use of only one individual sowings experiment could lead to a RRMSE of 28.4, 48, and 36% for R1, LAI and yield, respectively. However, the largest decrease occurred from the individual sowings to the season/year per location. Then, is recommended, use at least the season/year per location (early, recommended and late sowing dates) class. It will allow understand the behavior of the variety, avoiding the high costs of several experiments and keeping a reliable performance of the model. / Modelos agrícolas são ferramentas importantes para aprimorar técnicas de manejo e consequentemente a eficiência dos sistemas agrícolas. Esse acréscimo na eficiência são úteis para atender a crescente demanda de alimentos e combustíveis, sem avançar a fronteira agrícola. A calibração e validação de um modelo agrícola, historicamente considerou conjuntos de dados que variam de poucos á muitos experimentos. Poucos experimentos podem aumentar as incertezas e muitos experimentos tem alto custo financeiro e demanda de tempo. Pelo método de partição em dois grupos, o conjunto de experimentos é dividido em duas partes, uma para calibrar e a outra validar o modelo. Se apenas um conjunto pequeno de experimentos está disponível, dividi-los pode prejudicar o desempenho do modelo. Assim, métodos que otimizem esses processos, diminuindo o tempo e o custo de experimentos necessários para a calibração e validação, são sempre bem vindos. O objetivo do primeiro capítulo desta tese, foi comparar o método tradicionalmente utilizado na calibração e validação de modelos com um método mais robusto (cross-validation). Ambos os métodos foram aplicados para estimar os coeficientes genéticos na calibração e validação do modelo CROPGRO-soybean, utilizando múltiplos experimentos. Um conjunto com os 3 experimentos mais detalhados foram utilizados para calibração utilizando o método de partição em dois grupos. Já o método cross-validation, foi aplicado utilizando 21 experimentos. A cultivar NA5909 RG foi selecionada por ser uma das mais cultivadas no sul do Brasil nos últimos 5 anos, conduzida em experimentos distribuídos em oitos locais do Estado do Rio Grande do Sul durante as safras de 2010/2011 ate 2013/2014. O método cross-validation reduziu os RMSEs encontrados no método tradicionalmente utilizado de 2.6, 4.6, 4.8, 7.3, 10.2, 677 e 551 para 1.1, 4.1, 4.1, 6.2, 6.3, 347 e 447 para emergência, R1, R3, R5, R7 (em dias), grãos.m-2 e kg.ha-1, respectivamente. Foi observado estabilidade na maioria das estimativas de coeficientes genéticos, o que sugere a possibilidade de utilizar um menor número de experimentos no processo. Considerando a ampla faixa de condições ambientais, o modelo apresentou desempenho satisfatório na previsão fenológica, de biomassa e produtividade. Para otimizar os processos de calibração e validação, indica-se que o método cross-validation seja utilizado sempre que possível. No segundo capítulo, o principal objetivo foi avaliar o desempenho do uso de diferentes números de experimentos, e estimar o número mínimo necessário para garantir desempenho satisfatório do modelo CROPGRO-soybean. Esse estudo também utilizou 21 experimentos, com a cultivar BMX Potência RR. Os experimentos foram organizados em quatro grupos: Grupo 1 (semeaduras individuais), grupo 2 (ano agrícola por local), grupo 3 (local experimental) e grupo 4 (todos os experimentos juntos). Conforme o número de experimentos aumentou, a variabilidade dos coeficientes e os erros relativos (RRMSE) diminuíram. O primeiro grupo apresentou os maiores erros relativos, com até 28.4, 48 e 36% de erros nas simulações de R1, IAF e produtividade, respectivamente. O maior decréscimo nos erros relativos, ocorreu quando avançamos do grupo 1 para o grupo 2. Em alguns casos os erros foram reduzidos em mais que duas vezes. Assim, considerando o elevado custo financeiro e a demanda de tempo que os grupos 3 e 4 apresentam, recomenda-se a escolha de pelo menos o grupo 2, com 3 experimentos no mesmo ano agrícola. Essa estratégia vai permitir um melhor entendimento sobre o desempenho da cultivar, além de calibrar e validar o modelo CROPGRO-soybean, evitando os altos custos de vários experimentos, garantindo o desempenho satisfatório do modelo.

The assessment of the performance of learners by means of benchmark experiments is established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to benchmark studies in a supervised learning situation based on artificial and real-world data. / Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

Master of Science / Department of Computing and Information Sciences / Doina Caragea / Social media affects our daily lives. It is one of the first sources for finding breaking news. In particular, Twitter is one of the popular social media platforms, with around 330 million monthly users. From local events such as Fake Patty's Day to across the world happenings - Twitter gets there first. During a disaster, tweets can be used to post warnings, status of available medical and food supply, emergency personnel, and updates. Users were practically tweeting about the Hurricane Sandy, despite lack of network during the storm. Analysis of these tweets can help monitor the disaster, plan and manage the crisis, and aid in research. In this research, we use the publicly available tweets posted during several disasters and identify the relevant tweets. As the languages in the datasets are different, Bing translation API has been used to detect and translate the tweets. The translations are then, used as training datasets for supervised machine learning algorithms. Supervised learning is the process of learning from a labeled training dataset. This learned classifier can then be used to predict the correct output for any valid input. When trained to more observations, the algorithm improves its predictive performance.

Indiana University-Purdue University Indianapolis (IUPUI) / Social harm involves incidents resulting in physical, financial, and emotional hardships such as crime, drug overdoses and abuses, traffic accidents, and suicides. These incidents require various law-enforcement and emergency responding agencies to coordinate together for mitigating their impact on the society. With the advent of advanced networking and computing technologies together with data analytics, law-enforcement agencies and people in the community can work together to proactively reduce social harm. With the aim of effectively mitigating social harm events in communities, this thesis introduces a distributed web application, Community Data Analytic for Social Harm (CDASH). CDASH helps in collecting social harm data from heterogenous sources, analyzing the data for predicting social harm risks in the form of geographic hotspots and conveying the risks to law-enforcement agencies. Since various stakeholders including the police, community organizations and citizens can interact with CDASH, a need for a trust framework arises, to avoid fraudulent or mislabeled incidents from misleading CDASH. The enhanced system, called Trusted-CDASH (T-CDASH), superimposes a trust estimation framework on top of CDASH. This thesis discusses the importance and necessity of associating a degree of trust with each social harm incident reported to T-CDASH. It also describes the trust framework with different trust models that can be incorporated for assigning trust while examining their impact on prediction accuracy of future social harm events. The trust models are empirically validated by running simulations on historical social harm data of Indianapolis metro area.

No description available.