DAMASCENO, Filipe Francisco Rocha. Uma Melhoria no Algoritmo K-médias utilizando o Estimador de James-Stein. 63 f. 2015. Dissertação (Mestrado em Ciência da Computação)-Universidade Federal do Ceará, Fortaleza, 2015. / Submitted by Jairo Viana (jairo@ufc.br) on 2017-01-10T20:15:30Z No. of bitstreams: 1 2015_dis_ffrdamasceno.pdf: 966516 bytes, checksum: b1c373b2bf5b4b9a4e77e52d1f6a643c (MD5) / Approved for entry into archive by Jairo Viana (jairo@ufc.br) on 2017-01-10T20:15:44Z (GMT) No. of bitstreams: 1 2015_dis_ffrdamasceno.pdf: 966516 bytes, checksum: b1c373b2bf5b4b9a4e77e52d1f6a643c (MD5) / Made available in DSpace on 2017-01-10T20:15:44Z (GMT). No. of bitstreams: 1 2015_dis_ffrdamasceno.pdf: 966516 bytes, checksum: b1c373b2bf5b4b9a4e77e52d1f6a643c (MD5) Previous issue date: 2016 / The clustering task constitutes one of the main machine learning problems. Among many proposed methods, k-means stands out by its simplicity and high applicability. It is notorious that k-means performance is directly related to the centroid estimation from data, which is usually obtained from the maximum likelihood estimation (MLE). In previous studies it was proposed an estimator called James-Stein (JS) estimator, being, in average, capable of overcoming MLE for vectors of data with more than 2 dimensions. Also in previous studies it was proposed a variation of k-means applying JS estimator, obtaining improvements due to its better precision when compared to MLE. In this study we propose a variation of the k-means algorithm using the JS estimator. / A tarefa de agrupamento constitui um dos principais problemas de aprendizado de máquina. Dentre os diversos métodos propostos destaca-se o k-médias por sua simplicidade e grande aplicabilidade. É notório que o desempenho do k-médias está relacionado à estimativa dos centroides a partir dos dados e esta, usualmente, é obtida a partir da estimativa de máxima verossimilhança (EMV). Em trabalhos anteriores foi proposto um estimador denominado estimador de James-Stein (JS), sendo este capaz de, em média, superar o EMV para vetores de dados com dimensão maior que 2. Também em trabalhos anteriores foi proposta uma alteração do k-médias aplicando o estimador JS, obtendo melhoras devido à sua maior precisão em relação ao EMV. Neste trabalho propõe-se uma nova variante do algoritmo k-médias utilizando o estimador JS.

Charnigo and Srinivasan originally developed compound estimators to nonparametrically estimate mean response functions and their derivatives simultaneously when there is one response variable and one covariate. The compound estimator maintains self consistency and almost optimal convergence rate. This dissertation studies, in part, compound estimation with multiple responses and/or covariates. An empirical comparison of compound estimation, local regression and spline smoothing is included, and near optimal convergence rates are established in the presence of multiple covariates. James and Stein proposed an estimator of the mean vector of a p dimensional multivariate normal distribution, which produces a smaller risk than the maximum likelihood estimator if p is at least 3. In this dissertation, we also extend their idea to a nonparametric regression setting. More specifically, we present Steinized local regression estimators of p mean response functions and their derivatives. We consider different covariance structures for the error terms, and whether or not a known upper bound for the estimation bias is assumed. We also apply Steinization to compound estimation, considering the application of Steinization to both pointwise estimators (for example, as obtained through local regression) and weight functions. Finally, the new methodology introduced in this dissertation will be demonstrated on numerical data illustrating the outcomes of a laboratory experiment in which radiation induces nanoparticles to scatter evanescent waves. The patterns of scattering, as represented by derivatives of multiple mean response functions, may be used to classify nanoparticles on their sizes and structures.

Thesis (Ph.D.)--Brown University, 2005. / Vita. Thesis advisor: Nancy Armstrong. Includes bibliographical references (leaves 179-190). Also available online.

No description available.

Made available in DSpace on 2014-07-29T14:52:09Z (GMT). No. of bitstreams: 1 tese cristiane.pdf: 805152 bytes, checksum: 17b10ac06483864b2875c174a70c8625 (MD5) Previous issue date: 2008-06-13 / This research had the objective of evaluating the effects of different statistical approaches for the selection and ranking of genotypes, in the context of maize varieties trials. For that, data from real trials designed in lattice were used, in the Goiás State, Brazil, in the growing seasons of 2002/2003, 2003/2004, 2004/2005 and 2005/2006, as well as data from simulated experiments, aiming to cover situations related to that reality. The study also intended to quantify the effects of the genotype by environment interactions (GxE) from the real trials, aiming for the environmental stratification for the maize cultivation in the State, pointing out the cultivar evaluation and recommendation. Considering those objectives, the study is divided in three scientific articles. In the first one (Chapter 3), the effects of approaches of fixed model (FF), mixed model with random effect of blocks (AF), mixed model with random effect of treatments (FA), random model (AA), and James-Stein s estimator (JS) were evaluated on the selection and ranking of genotypes tested on the maize varieties trials, coordinated by the Agência Goiana de Desenvolvimento Rural e Fundiário (AgenciaRural Goiás). The experiments, in number of 47, were installed in lattice design, with three replications, during the four cited harvest years. In the second article (Chapter 4), the same approaches were evaluated, in terms of accuracy, mean predictive deviation and precision of their estimates/predictions, considering the simulated trials, also in lattice. Forty-eight cases were considered, corresponding to the combinations of different experimental sizes (15, 54, 105, and 450 treatments), genotypic determination coefficients h2' (6%, 15%, 25%, 48%, 63% and 82%), and two probability distributions for the generation of genotypic effects (normal and uniform). One thousand trials were simulated for each case, reaching the total of 48,000 experiments. The third and last article (Chapter 5) refers to the study of the GxE interaction, emphasizing the already mentioned environmental stratification, where the winner genotypes approach in association with the AMMI analysis (additive main effects and multiplicative interaction model) was adopted. Among the results and conclusions achieved through this study, it is possible to point out: i) the adoption of statistical approaches with shrinkage effect on the genotypic means results in the selection of a lower number of genotypes, especially in those trials whose mean of the check cultivars (baseline to the genotypic selection) is higher than the experimental grand mean; this fact reduces the number of genotypes with low yield potential in the next cycles of the selection program; ii) the use of models with fixed effects of treatments leads to a higher percentage of selected genotypes, mainly in the experiments whose check varieties mean overcomes the experimental grand mean; iii) among the shrinkage statistic approaches evaluated, the AA model must be preferred for the selection of genotypes, due to its capacity for better predicting the parametric genotypic effects (higher accuracy and lower mean predictive deviation), no matter if these effects are normally or uniformly distributed; iv) on the other hand, the FF model shows the worst relative performance, except for the situations where the variability among the genetic treatments is high (h2 ®1,0); v) considering low values for h2 (6%), the FA model shows efficiency similar to the AA model; vi) two established environmental strata showed to be consistent throughout the years, even when the tested genotypes were altered from one harvest season to the other: Ipameri, Inhumas and Senador Canêdo (stable to four years), and Porangatu and Orizona (stable along three years); vii) considering the obtained clustering, it is possible to reduce, at least 16%, the number of test locations currently used, and/or substitute the redundant locations by test places which better represent the recommended target region, aiming to increase the evaluation efficiency of the GxE interaction, in the scope of the genetic plant breeding program; viii) the ALBandeirante variety presents high yield potential and adaptability to the maize cultivation conditions in the Goiás State. / This research had the objective of evaluating the effects of different statistical approaches for the selection and ranking of genotypes, in the context of maize varieties trials. For that, data from real trials designed in lattice were used, in the Goiás State, Brazil, in the growing seasons of 2002/2003, 2003/2004, 2004/2005 and 2005/2006, as well as data from simulated experiments, aiming to cover situations related to that reality. The study also intended to quantify the effects of the genotype by environment interactions (GxE) from the real trials, aiming for the environmental stratification for the maize cultivation in the State, pointing out the cultivar evaluation and recommendation. Considering those objectives, the study is divided in three scientific articles. In the first one (Chapter 3), the effects of approaches of fixed model (FF), mixed model with random effect of blocks (AF), mixed model with random effect of treatments (FA), random model (AA), and James-Stein s estimator (JS) were evaluated on the selection and ranking of genotypes tested on the maize varieties trials, coordinated by the Agência Goiana de Desenvolvimento Rural e Fundiário (AgenciaRural Goiás). The experiments, in number of 47, were installed in lattice design, with three replications, during the four cited harvest years. In the second article (Chapter 4), the same approaches were evaluated, in terms of accuracy, mean predictive deviation and precision of their estimates/predictions, considering the simulated trials, also in lattice. Forty-eight cases were considered, corresponding to the combinations of different experimental sizes (15, 54, 105, and 450 treatments), genotypic determination coefficients h2' (6%, 15%, 25%, 48%, 63% and 82%), and two probability distributions for the generation of genotypic effects (normal and uniform). One thousand trials were simulated for each case, reaching the total of 48,000 experiments. The third and last article (Chapter 5) refers to the study of the GxE interaction, emphasizing the already mentioned environmental stratification, where the winner genotypes approach in association with the AMMI analysis (additive main effects and multiplicative interaction model) was adopted. Among the results and conclusions achieved through this study, it is possible to point out: i) the adoption of statistical approaches with shrinkage effect on the genotypic means results in the selection of a lower number of genotypes, especially in those trials whose mean of the check cultivars (baseline to the genotypic selection) is higher than the experimental grand mean; this fact reduces the number of genotypes with low yield potential in the next cycles of the selection program; ii) the use of models with fixed effects of treatments leads to a higher percentage of selected genotypes, mainly in the experiments whose check varieties mean overcomes the experimental grand mean; iii) among the shrinkage statistic approaches evaluated, the AA model must be preferred for the selection of genotypes, due to its capacity for better predicting the parametric genotypic effects (higher accuracy and lower mean predictive deviation), no matter if these effects are normally or uniformly distributed; iv) on the other hand, the FF model shows the worst relative performance, except for the situations where the variability among the genetic treatments is high (h2 ®1,0); v) considering low values for h2 (6%), the FA model shows efficiency similar to the AA model; vi) two established environmental strata showed to be consistent throughout the years, even when the tested genotypes were altered from one harvest season to the other: Ipameri, Inhumas and Senador Canêdo (stable to four years), and Porangatu and Orizona (stable along three years); vii) considering the obtained clustering, it is possible to reduce, at least 16%, the number of test locations currently used, and/or substitute the redundant locations by test places which better represent the recommended target region, aiming to increase the evaluation efficiency of the GxE interaction, in the scope of the genetic plant breeding program; viii) the ALBandeirante variety presents high yield potential and adaptability to the maize cultivation conditions in the Goiás State. / O presente trabalho teve como objetivo avaliar diferentes abordagens estatísticas em relação à seleção e ordenação de genótipos, no contexto de ensaios varietais de milho. Para isso, utilizaram-se dados reais de ensaios delineados em látice, conduzidos no Estado de Goiás, nas safras 2002/2003, 2003/2004, 2004/2005 e 2005/2006, bem como dados de experimentos simulados, nos quais se buscaram cobrir situações similares a essa realidade. O estudo propôs-se, ainda, a quantificar os efeitos da interação de genótipos com ambientes (GxE), a partir dos ensaios reais, visando-se à estratificação ambiental para a cultura do milho no Estado, com ênfase na avaliação e recomendação de cultivares. A partir desses objetivos, o trabalho apresenta-se estruturado na forma de três artigos científicos. No primeiro deles (Capítulo 3), avaliaram-se os efeitos das abordagens de modelo fixo (FF), modelo misto com efeito aleatório de blocos (AF), modelo misto com efeito aleatório de tratamentos (FA), modelo aleatório (AA) e do estimador de James-Stein (JS), na seleção e ordenação de genótipos testados na rede dos ensaios de variedades de milho, coordenada pela Agência Goiana de Desenvolvimento Rural e Fundiário (AgenciaRural Goiás). Os experimentos, em número de 47, foram instalados em látice, com três repetições, tendo sido conduzidos durante os quatro anos agrícolas citados. No segundo artigo (Capítulo 4), as mesmas abordagens foram avaliadas em termos de acurácia, desvio preditivo médio e precisão de suas estimativas/predições, considerando-se os experimentos simulados, também em látice. Foram considerados 48 casos, correspondentes às combinações de diferentes tamanhos experimentais (15, 54, 105 e 450 tratamentos), coeficientes de determinação genotípica h2' (6%, 15%, 25%, 48%, 63% e 82%) e duas distribuições de probabilidade para a geração dos efeitos genotípicos (normal e uniforme). Foram gerados 1.000 ensaios para cada caso, totalizando 48.000 experimentos. O terceiro e último artigo (Capítulo 5) refere-se ao estudo da interação GxE, com ênfase na referida estratificação ambiental, para o qual se adotou a abordagem de genótipos vencedores, associada à análise AMMI (modelo de efeitos principais aditivos e interação multiplicativa). Entre os resultados e conclusões obtidos, destacam-se: i) a adoção de abordagens estatísticas que promovem shrinkage das médias genotípicas resultam na seleção de menor número de genótipos, especialmente quando à média das cultivares testemunhas (referência para a seleção genotípica), que é superior à média experimental, reduzindo o número de genótipos pouco produtivos nos ciclos seguintes do programa de seleção; ii) o uso de modelos com efeitos fixos de tratamentos leva a um maior percentual de seleção de genótipos, sobretudo nos experimentos cuja média das testemunhas supera a média experimental; iii) entre as abordagens estatísticas shrinkage avaliadas, o modelo AA deve ser preferido para a seleção de genótipos, em razão de sua melhor capacidade de predição dos efeitos genotípicos paramétricos (maior acurácia e menor desvio preditivo médio), independentemente de esses efeitos terem distribuição normal ou uniforme; iv) contrariamente, o modelo FF demonstra o pior desempenho relativo, excetuando-se as situações em que a variabilidade entre os tratamentos genéticos é elevada (h2 ®1,0); v) sob baixos valores de h2 (6%), o modelo FA apresenta eficiência similar ao modelo AA; vi) dois estratos ambientais estabelecidos mostraram-se consistentes, ao longo dos anos, mesmo alterando-se os genótipos testados de uma safra agrícola para a outra: Ipameri, Inhumas e Senador Canêdo, (estável em quatro anos), e Porangatu e Orizona (estável em três anos); vii) com os agrupamentos obtidos é possível reduzir, pelo menos 16%, o número de locais de teste atualmente utilizados e, ou, efetuar a substituição de locais redundantes por outros pontos de teste que melhor representem a região alvo da recomendação, de modo a aumentar a eficiência da avaliação da interação GxE, no âmbito do programa de melhoramento; viii) a variedade ALBandeirante apresenta alto potencial produtivo e grande adaptabilidade às condições de cultivo do milho no Estado de Goiás. / O presente trabalho teve como objetivo avaliar diferentes abordagens estatísticas em relação à seleção e ordenação de genótipos, no contexto de ensaios varietais de milho. Para isso, utilizaram-se dados reais de ensaios delineados em látice, conduzidos no Estado de Goiás, nas safras 2002/2003, 2003/2004, 2004/2005 e 2005/2006, bem como dados de experimentos simulados, nos quais se buscaram cobrir situações similares a essa realidade. O estudo propôs-se, ainda, a quantificar os efeitos da interação de genótipos com ambientes (GxE), a partir dos ensaios reais, visando-se à estratificação ambiental para a cultura do milho no Estado, com ênfase na avaliação e recomendação de cultivares. A partir desses objetivos, o trabalho apresenta-se estruturado na forma de três artigos científicos. No primeiro deles (Capítulo 3), avaliaram-se os efeitos das abordagens de modelo fixo (FF), modelo misto com efeito aleatório de blocos (AF), modelo misto com efeito aleatório de tratamentos (FA), modelo aleatório (AA) e do estimador de James-Stein (JS), na seleção e ordenação de genótipos testados na rede dos ensaios de variedades de milho, coordenada pela Agência Goiana de Desenvolvimento Rural e Fundiário (AgenciaRural Goiás). Os experimentos, em número de 47, foram instalados em látice, com três repetições, tendo sido conduzidos durante os quatro anos agrícolas citados. No segundo artigo (Capítulo 4), as mesmas abordagens foram avaliadas em termos de acurácia, desvio preditivo médio e precisão de suas estimativas/predições, considerando-se os experimentos simulados, também em látice. Foram considerados 48 casos, correspondentes às combinações de diferentes tamanhos experimentais (15, 54, 105 e 450 tratamentos), coeficientes de determinação genotípica h2' (6%, 15%, 25%, 48%, 63% e 82%) e duas distribuições de probabilidade para a geração dos efeitos genotípicos (normal e uniforme). Foram gerados 1.000 ensaios para cada caso, totalizando 48.000 experimentos. O terceiro e último artigo (Capítulo 5) refere-se ao estudo da interação GxE, com ênfase na referida estratificação ambiental, para o qual se adotou a abordagem de genótipos vencedores, associada à análise AMMI (modelo de efeitos principais aditivos e interação multiplicativa). Entre os resultados e conclusões obtidos, destacam-se: i) a adoção de abordagens estatísticas que promovem shrinkage das médias genotípicas resultam na seleção de menor número de genótipos, especialmente quando à média das cultivares testemunhas (referência para a seleção genotípica), que é superior à média experimental, reduzindo o número de genótipos pouco produtivos nos ciclos seguintes do programa de seleção; ii) o uso de modelos com efeitos fixos de tratamentos leva a um maior percentual de seleção de genótipos, sobretudo nos experimentos cuja média das testemunhas supera a média experimental; iii) entre as abordagens estatísticas shrinkage avaliadas, o modelo AA deve ser preferido para a seleção de genótipos, em razão de sua melhor capacidade de predição dos efeitos genotípicos paramétricos (maior acurácia e menor desvio preditivo médio), independentemente de esses efeitos terem distribuição normal ou uniforme; iv) contrariamente, o modelo FF demonstra o pior desempenho relativo, excetuando-se as situações em que a variabilidade entre os tratamentos genéticos é elevada (h2 ®1,0); v) sob baixos valores de h2 (6%), o modelo FA apresenta eficiência similar ao modelo AA; vi) dois estratos ambientais estabelecidos mostraram-se consistentes, ao longo dos anos, mesmo alterando-se os genótipos testados de uma safra agrícola para a outra: Ipameri, Inhumas e Senador Canêdo, (estável em quatro anos), e Porangatu e Orizona (estável em três anos); vii) com os agrupamentos obtidos é possível reduzir, pelo menos 16%, o número de locais de teste atualmente utilizados e, ou, efetuar a substituição de locais redundantes por outros pontos de teste que melhor representem a região alvo da recomendação, de modo a aumentar a eficiência da avaliação da interação GxE, no âmbito do programa de melhoramento; viii) a variedade ALBandeirante apresenta alto potencial produtivo e grande adaptabilidade às condições de cultivo do milho no Estado de Goiás.

變數選取方法已經成為各領域在處理多維度資料的工具。Zhou與Hwang在2005年，為了改善James-Stein positive part估計量(JS+)只能在完全模型(full model)與原始模型(origin model)兩者去做挑選，建立了具有Minimax性質同時加上門檻值的估計量，即James-Stein with Threshoding positive part估計量(JSWT+)。由於JSWT+估計量具有門檻值，使得此估計量可以在完全模型與其線性子集下做變數選取。我們想進一步了解如果將JSWT+估計量應用於線性迴歸分析時，藉由JSWT+估計具有門檻值的性質去做變數選取的效果如何？本文目的即是利用JSWT+估計量具有門檻值的性質，建立JSWT+估計量應用於線性迴歸模型變數挑選的流程。建立模擬資料分析，以可同時做係數壓縮及變數選取的LASSO方法與我們所提出JSWT+變數選取的流程去比較係數路徑及變數選取時差異比較，最後將我們提出JSWT+變數選取的流程對實際資料攝護腺癌資料(Tibshirani，1996)做變數挑選。則當考慮解釋變數個數小於樣本個數情況下，JSWT+與LASSO在變數選取的比較結果顯示，JSWT+表現的比較好，且可直接得到估計量的理想參數。

This thesis consists of three papers that makes independent contributions to the fields of forecast evaluation and financial econometrics. As such, the papers, chapter 1-3, can be read independently of each other. In Chapter 1, “Inferring an agent’s loss function based on a term structure of forecasts”, we provide conditions for identification, estimation and inference of an agent’s loss function based on an observed term structure of point forecasts. The loss function specification is flexible as we allow the preferences to be both asymmetric and to vary non-linearly across the forecast horizon. In addition, we introduce a novel forecast rationality test based on the estimated loss function. We employ the approach to analyse the U.S. Government’s preferences over budget surplus forecast errors. Interestingly, we find that it is relatively more costly for the government to underestimate the budget surplus and that this asymmetry is stronger at long forecast horizons. In Chapter 2, “Monitoring Systemic Risk”, we define systemic risk as the conditional probability of a systemic banking crisis. This conditional probability is modelled in a fixed effect binary response panel-model framework that allows for cross-sectional dependence (e.g. due to contagion effects). In the empirical application we identify several risk factors and it is shown that the level of systemic risk contains a predictable component which varies through time. Furthermore, we illustrate how the forecasts of systemic risk map into dynamic policy thresholds in this framework. Finally, by conducting a pseudo out-of-sample exercise we find that the systemic risk estimates provided reliable early-warning signals ahead of the recent financial crisis for several economies. Finally, in Chapter 3, “Equity Premium Predictability”, we reassess the evidence of out-of- sample equity premium predictability. The empirical finance literature has identified several financial variables that appear to predict the equity premium in-sample. However, Welch & Goyal (2008) find that none of these variables have any predictive power out-of-sample. We show that the equity premium is predictable out-of-sample once you impose certain shrinkage restrictions on the model parameters. The approach is motivated by the observation that many of the proposed financial variables can be characterised as ’weak predictors’ and this suggest that a James-Stein type estimator will provide a substantial risk reduction. The out-of-sample explanatory power is small, but we show that it is, in fact, economically meaningful to an investor with time-invariant risk aversion. Using a shrinkage decomposition we also show that standard combination forecast techniques tends to ’overshrink’ the model parameters leading to suboptimal model forecasts.

Motivated by the classic works of Charles M. Stein, we focus on developing risk-estimation frameworks for denoising problems in both one-and two-dimensions. We assume a standard additive noise model, and formulate the denoising problem as one of estimating the underlying clean signal from noisy measurements by minimizing a risk corresponding to a chosen loss function. Our goal is to incorporate perceptually-motivated loss functions wherever applicable, as in the case of speech enhancement, with the squared error loss being considered for the other scenarios. Since the true risks are observed to depend on the unknown parameter of interest, we circumvent the roadblock by deriving finite-sample un-biased estimators of the corresponding risks based on Stein’s lemma. We establish the link with the multivariate parameter estimation problem addressed by Stein and our denoising problem, and derive estimators of the oracle risks. In all cases, optimum values of the parameters characterizing the denoising algorithm are determined by minimizing the Stein’s unbiased risk estimator (SURE). The key contribution of this thesis is the development of a risk-estimation approach for choosing the two critical parameters affecting the quality of nonparametric regression, namely, the order and bandwidth/smoothing parameters. This is a classic problem in statistics, and certain algorithms relying on derivation of suitable finite-sample risk estimators for minimization have been reported in the literature (note that all these works consider the mean squared error (MSE) objective). We show that a SURE-based formalism is well-suited to the regression parameter selection problem, and that the optimum solution guarantees near-minimum MSE (MMSE) performance. We develop algorithms for both glob-ally and locally choosing the two parameters, the latter referred to as spatially-adaptive regression. We observe that the parameters are so chosen as to tradeoff the squared bias and variance quantities that constitute the MSE. We also indicate the advantages accruing out of incorporating a regularization term in the cost function in addition to the data error term. In the more general case of kernel regression, which uses a weighted least-squares (LS) optimization, we consider the applications of image restoration from very few random measurements, in addition to denoising of uniformly sampled data. We show that local polynomial regression (LPR) becomes a special case of kernel regression, and extend our results for LPR on uniform data to non-uniformly sampled data also. The denoising algorithms are compared with other standard, performant methods available in the literature both in terms of estimation error and computational complexity. A major perspective provided in this thesis is that the problem of optimum parameter choice in nonparametric regression can be viewed as the selection of optimum parameters of a linear, shift-invariant filter. This interpretation is provided by deriving motivation out of the hallmark paper of Savitzky and Golay and Schafer’s recent article in IEEE Signal Processing Magazine. It is worth noting that Savitzky and Golay had shown in their original Analytical Chemistry journal article, that LS fitting of a fixed-order polynomial over a neighborhood of fixed size is equivalent to convolution with an impulse response that is fixed and can be pre-computed. They had provided tables of impulse response coefficients for computing the smoothed function and smoothed derivatives for different orders and neighborhood sizes, the resulting filters being referred to as Savitzky-Golay (S-G) filters. Thus, we provide the new perspective that the regression parameter choice is equivalent to optimizing for the filter impulse response length/3dB bandwidth, which are inversely related. We observe that the MMSE solution is such that the S-G filter chosen is of longer impulse response length (equivalently smaller cutoff frequency) at relatively flat portions of the noisy signal so as to smooth noise, and vice versa at locally fast-varying portions of the signal so as to capture the signal patterns. Also, we provide a generalized S-G filtering viewpoint in the case of kernel regression. Building on the S-G filtering perspective, we turn to the problem of dynamic feature computation in speech recognition. We observe that the methodology employed for computing dynamic features from the trajectories of static features is in fact derivative S-G filtering. With this perspective, we note that the filter coefficients can be pre-computed, and that the whole problem of delta feature computation becomes efficient. Indeed, we observe an advantage by a factor of 104 on making use of S-G filtering over actual LS polynomial fitting and evaluation. Thereafter, we study the properties of first-and second-order derivative S-G filters of certain orders and lengths experimentally. The derivative filters are bandpass due to the combined effects of LPR and derivative computation, which are lowpass and highpass operations, respectively. The first-and second-order S-G derivative filters are also observed to exhibit an approximately constant-Q property. We perform a TIMIT phoneme recognition experiment comparing the recognition accuracies obtained using S-G filters and the conventional approach followed in HTK, where Furui’s regression formula is made use of. The recognition accuracies for both cases are almost identical, with S-G filters of certain bandwidths and orders registering a marginal improvement. The accuracies are also observed to improve with longer filter lengths, for a particular order. In terms of computation latency, we note that S-G filtering achieves delta and delta-delta feature computation in parallel by linear filtering, whereas they need to be obtained sequentially in case of the standard regression formulas used in the literature. Finally, we turn to the problem of speech enhancement where we are interested in de-noising using perceptually-motivated loss functions such as Itakura-Saito (IS). We propose to perform enhancement in the discrete cosine transform domain using risk-minimization. The cost functions considered are non-quadratic, and derivation of the unbiased estimator of the risk corresponding to the IS distortion is achieved using an approximate Taylor-series analysis under high signal-to-noise ratio assumption. The exposition is general since we focus on an additive noise model with the noise density assumed to fall within the exponential class of density functions, which comprises most of the common densities. The denoising function is assumed to be pointwise linear (modified James-Stein (MJS) estimator), and parallels between Wiener filtering and the optimum MJS estimator are discussed.

Motivated by the classic works of Charles M. Stein, we focus on developing risk-estimation frameworks for denoising problems in both one-and two-dimensions. We assume a standard additive noise model, and formulate the denoising problem as one of estimating the underlying clean signal from noisy measurements by minimizing a risk corresponding to a chosen loss function. Our goal is to incorporate perceptually-motivated loss functions wherever applicable, as in the case of speech enhancement, with the squared error loss being considered for the other scenarios. Since the true risks are observed to depend on the unknown parameter of interest, we circumvent the roadblock by deriving finite-sample un-biased estimators of the corresponding risks based on Stein’s lemma. We establish the link with the multivariate parameter estimation problem addressed by Stein and our denoising problem, and derive estimators of the oracle risks. In all cases, optimum values of the parameters characterizing the denoising algorithm are determined by minimizing the Stein’s unbiased risk estimator (SURE). The key contribution of this thesis is the development of a risk-estimation approach for choosing the two critical parameters affecting the quality of nonparametric regression, namely, the order and bandwidth/smoothing parameters. This is a classic problem in statistics, and certain algorithms relying on derivation of suitable finite-sample risk estimators for minimization have been reported in the literature (note that all these works consider the mean squared error (MSE) objective). We show that a SURE-based formalism is well-suited to the regression parameter selection problem, and that the optimum solution guarantees near-minimum MSE (MMSE) performance. We develop algorithms for both glob-ally and locally choosing the two parameters, the latter referred to as spatially-adaptive regression. We observe that the parameters are so chosen as to tradeoff the squared bias and variance quantities that constitute the MSE. We also indicate the advantages accruing out of incorporating a regularization term in the cost function in addition to the data error term. In the more general case of kernel regression, which uses a weighted least-squares (LS) optimization, we consider the applications of image restoration from very few random measurements, in addition to denoising of uniformly sampled data. We show that local polynomial regression (LPR) becomes a special case of kernel regression, and extend our results for LPR on uniform data to non-uniformly sampled data also. The denoising algorithms are compared with other standard, performant methods available in the literature both in terms of estimation error and computational complexity. A major perspective provided in this thesis is that the problem of optimum parameter choice in nonparametric regression can be viewed as the selection of optimum parameters of a linear, shift-invariant filter. This interpretation is provided by deriving motivation out of the hallmark paper of Savitzky and Golay and Schafer’s recent article in IEEE Signal Processing Magazine. It is worth noting that Savitzky and Golay had shown in their original Analytical Chemistry journal article, that LS fitting of a fixed-order polynomial over a neighborhood of fixed size is equivalent to convolution with an impulse response that is fixed and can be pre-computed. They had provided tables of impulse response coefficients for computing the smoothed function and smoothed derivatives for different orders and neighborhood sizes, the resulting filters being referred to as Savitzky-Golay (S-G) filters. Thus, we provide the new perspective that the regression parameter choice is equivalent to optimizing for the filter impulse response length/3dB bandwidth, which are inversely related. We observe that the MMSE solution is such that the S-G filter chosen is of longer impulse response length (equivalently smaller cutoff frequency) at relatively flat portions of the noisy signal so as to smooth noise, and vice versa at locally fast-varying portions of the signal so as to capture the signal patterns. Also, we provide a generalized S-G filtering viewpoint in the case of kernel regression. Building on the S-G filtering perspective, we turn to the problem of dynamic feature computation in speech recognition. We observe that the methodology employed for computing dynamic features from the trajectories of static features is in fact derivative S-G filtering. With this perspective, we note that the filter coefficients can be pre-computed, and that the whole problem of delta feature computation becomes efficient. Indeed, we observe an advantage by a factor of 104 on making use of S-G filtering over actual LS polynomial fitting and evaluation. Thereafter, we study the properties of first-and second-order derivative S-G filters of certain orders and lengths experimentally. The derivative filters are bandpass due to the combined effects of LPR and derivative computation, which are lowpass and highpass operations, respectively. The first-and second-order S-G derivative filters are also observed to exhibit an approximately constant-Q property. We perform a TIMIT phoneme recognition experiment comparing the recognition accuracies obtained using S-G filters and the conventional approach followed in HTK, where Furui’s regression formula is made use of. The recognition accuracies for both cases are almost identical, with S-G filters of certain bandwidths and orders registering a marginal improvement. The accuracies are also observed to improve with longer filter lengths, for a particular order. In terms of computation latency, we note that S-G filtering achieves delta and delta-delta feature computation in parallel by linear filtering, whereas they need to be obtained sequentially in case of the standard regression formulas used in the literature. Finally, we turn to the problem of speech enhancement where we are interested in de-noising using perceptually-motivated loss functions such as Itakura-Saito (IS). We propose to perform enhancement in the discrete cosine transform domain using risk-minimization. The cost functions considered are non-quadratic, and derivation of the unbiased estimator of the risk corresponding to the IS distortion is achieved using an approximate Taylor-series analysis under high signal-to-noise ratio assumption. The exposition is general since we focus on an additive noise model with the noise density assumed to fall within the exponential class of density functions, which comprises most of the common densities. The denoising function is assumed to be pointwise linear (modified James-Stein (MJS) estimator), and parallels between Wiener filtering and the optimum MJS estimator are discussed.