In this thesis, we study the representation of speech signals and the estimation of information-theoretical measures from observations containing features of the speech signal. The main body of the thesis consists of four research papers. Paper A presents a compact representation of the speech signal that facilitates perfect reconstruction. The representation is constituted of models, model parameters, and signal coefficients. A difference compared to existing speech representations is that we seek a compact representation by adapting the models to maximally concentrate the energy of the signal coefficients according to a selected energy concentration criterion. The individual parts of the representation are closely related to speech signal properties such as spectral envelope, pitch, and voiced/unvoiced signal coefficients, bene cial for both speech coding and modi cation. From the information-theoretical measure of entropy, performance limits in coding and classi cation can be derived. Papers B and C discuss the estimation of di erential entropy. Paper B describes a method for estimation of the di erential entropies in the case when the set of vector observations (from the representation) lie on a lower-dimensional surface (manifold) in the embedding space. In contrast to the method presented in Paper B, Paper C introduces a method where the manifold structures are destroyed by constraining the resolution of the observation space. This facilitates the estimation of bounds on classi cation error rates even when the manifolds are of varying dimensionality within the embedding space. Finally, Paper D investigates the amount of shared information between spectral features of narrow-band (0.3-3.4 kHz) and high-band (3.4-8 kHz) speech. The results in Paper D indicate that the information shared between the high-band and the narrow-band is insufficient for high-quality wideband speech coding (0.3-8 kHz) without transmission of extra information describing the high-band. / QC 20100914

Modern medicine has become reliant on medical imaging. Multiple modalities, e. g. magnetic resonance imaging (MRI), computed tomography (CT), etc. , are used to provide as much information about the patient as possible. The problem of geometrically aligning the resulting images is called image registration. Mutual information, an information theoretic similarity measure, allows for automated intermodal image registration algorithms. <br /><br /> In applications such as cancer therapy, diagnosticians are more concerned with the alignment of images over a region of interest such as a cancerous lesion, than over an entire image set. Attempts to register only the regions of interest, defined manually by diagnosticians, fail due to inaccurate mutual information estimation over the region of overlap of these small regions. <br /><br /> This thesis examines the region of union as an alternative to the region of overlap. We demonstrate that the region of union improves the accuracy and reliability of mutual information estimation over small regions. <br /><br /> We also present two new mutual information based similarity measures which allow for localized image registration by combining local and global image information. The new similarity measures are based on convex combinations of the information contained in the regions of interest and the information contained in the global images. <br /><br /> Preliminary results indicate that the proposed similarity measures are capable of localizing image registration. Experiments using medical images from computer tomography and positron emission tomography demonstrate the initial success of these measures. <br /><br /> Finally, in other applications, auto-detection of regions of interest may prove useful and would allow for fully automated localized image registration. We examine methods to automatically detect potential regions of interest based on local activity level and present some encouraging results.

Nearest neighbour methods are a classical approach in nonparametric statistics. The k-nearest neighbour classifier can be traced back to the seminal work of Fix and Hodges (1951) and they also enjoy popularity in many other problems including density estimation and regression. In this thesis we study their use in three different situations, providing new theoretical results on the performance of commonly-used nearest neighbour methods and proposing new procedures that are shown to outperform these existing methods in certain settings. The first problem we discuss is that of entropy estimation. Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this chapter, we seek entropy estimators that are efficient and achieve the local asymptotic minimax lower bound with respect to squared error loss. To this end, we study weighted averages of the estimators originally proposed by Kozachenko and Leonenko (1987), based on the k-nearest neighbour distances of a sample. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness, while the original unweighted estimator is typically only efficient in up to three dimensions. A related topic of study is the estimation of the mutual information between two random vectors, and its application to testing for independence. We propose tests for the two different situations of the marginal distributions being known or unknown and analyse their performance. Finally, we study the classical k-nearest neighbour classifier of Fix and Hodges (1951) and provide a new asymptotic expansion for its excess risk. We also show that, in certain situations, a new modification of the classifier that allows k to vary with the location of the test point can provide improvements. This has applications to the field of semi-supervised learning, where, in addition to labelled training data, we also have access to a large sample of unlabelled data.

Modern medicine has become reliant on medical imaging. Multiple modalities, e. g. magnetic resonance imaging (MRI), computed tomography (CT), etc. , are used to provide as much information about the patient as possible. The problem of geometrically aligning the resulting images is called image registration. Mutual information, an information theoretic similarity measure, allows for automated intermodal image registration algorithms. <br /><br /> In applications such as cancer therapy, diagnosticians are more concerned with the alignment of images over a region of interest such as a cancerous lesion, than over an entire image set. Attempts to register only the regions of interest, defined manually by diagnosticians, fail due to inaccurate mutual information estimation over the region of overlap of these small regions. <br /><br /> This thesis examines the region of union as an alternative to the region of overlap. We demonstrate that the region of union improves the accuracy and reliability of mutual information estimation over small regions. <br /><br /> We also present two new mutual information based similarity measures which allow for localized image registration by combining local and global image information. The new similarity measures are based on convex combinations of the information contained in the regions of interest and the information contained in the global images. <br /><br /> Preliminary results indicate that the proposed similarity measures are capable of localizing image registration. Experiments using medical images from computer tomography and positron emission tomography demonstrate the initial success of these measures. <br /><br /> Finally, in other applications, auto-detection of regions of interest may prove useful and would allow for fully automated localized image registration. We examine methods to automatically detect potential regions of interest based on local activity level and present some encouraging results.

The maximum spacing (MSP) method, introduced by Cheng and Amin (1983) and independently by Ranneby (1984), is a general estimation method for continuous univariate distributions. The MSP method, which is closely related to the maximum likelihood (ML) method, can be derived from an approximation based on simple spacings of the Kullback-Leibler information. It is known to give consistent and asymptotically efficient estimates under general conditions and works also in situations where the ML method fails, e.g. for the three parameter Weibull model. In this thesis it is proved under general conditions that MSP estimates of parameters in the Euclidian metric are strongly consistent. The ideas behind the MSP method are extended and a class of estimation methods is introduced. These methods, called generalized MSP methods, are derived from approxima­tions based on sum-functions of rath order spacings of certain information mea­sures, i.e. the ^-divergences introduced by Csiszår (1963). It is shown under general conditions that generalized MSP methods give consistent estimates. In particular, it is proved that generalized MSP methods give L1 consistent esti­mates in any family of distributions with unimodal densities, without any further conditions on the distributions. Other properties such as distributional robust­ness are also discussed. Several limit theorems for sum-functions of rath order spacings are given, for ra fixed as well as for the case when ra is allowed to in­crease to infinity with the sample size. These results provide a strongly consistent nonparametric estimator of entropy, as well as a characterization of the uniform distribution. Further, it is shown that Cressie's (1976) goodness of fit test is strongly consistent against all continuous alternatives. / digitalisering@umu

We investigate the extraction of cryptographic keying material from nano-scale variations of digital circuit outputs by using nested polar codes and neural leakage estimators. A runtime-efficient algorithm is developed to simulate such a system. A certain family of digital circuit outputs are known to be a source of randomness that can be used as a unique identifier for each output. By generating secret keys from these unique outputs, one can apply cryptographic methods by using the secret keys as the seed. One is required to store extra helper data generated first time the outputs are measured, since there is noise in digital circuit outputs, to be able to reconstruct the same key from every measurement of the same digital circuit. The generation of the secret keys and helper data follow a nested polar code construction, and they are generated in this thesis to estimate the Shannon entropy of the secret key and secrecy leakage to a passive attacker using neural networks. The estimators used illustrate, for the first time, that the system generates secret keys of almost maximum entropy and negligible secrecy leakage for practical cryptographic systems if the digital circuit outputs can be preprocessed to obtain almost independent and identically distributed (i.i.d.) random outputs distributed according to a binary uniform distribution. The algorithm design is evaluated and improvements for lower runtime are suggested. Ideas for future research are presented.

The theme of this thesis is to analyze the impact of policy interventions such as financing human development (HD), tax reform and conditional cash transfer programmes, under the framework of growth and sustainable development. These policy instruments are evaluated through the application of both partial and general equilibrium models, and the last paper concentrates on developing regional social accounting matrices (SAMs) as a core database for spatial general equilibrium modelling. Essay 1: Trade-offs in Achieving Human Development Goals for Bangladesh investigates the benefits and costs associated with alternative investment financing options for achieving HD goals by applying the MAMS (Maquette for Millennium Development Goals Studies) model. We find that full achievement of these goals would have led to a GDP loss that would have been significantly larger in the domestic borrowing scenario than in the tax scenario. The tax-financing alternative is thus the better option for financing large development programs. In terms of public spending composition, we find that, under some circumstances, a trade-off arises between overall Millennium Development Goal (MDG) progress and poverty reduction. Essay 2: Welfare impact of broadening VAT by exempting Small-Scale food markets: The case of Bangladesh analyses the welfare impacts of different VAT reforms. A general and uniform VAT on all commodities is preferred as it is more efficient and less administratively costly. However, due to equity concerns, food is normally exempted from VAT. On the other hand, exemptions on food mean that an implicit subsidy is provided to high-income households. Hence, we analyze a broad-based VAT regime with a high threshold that excludes small-scale operators (where the low-income households buy their products most, including food) and the simulation result shows that welfare improves for the low-income households. Essay 3: Effect of Girls’ Secondary School Stipend on Completed Schooling and Age at Marriage: Evidence from Bangladesh estimates the effect of a conditional cash transfer programme on education and age at marriage. We apply both difference in differences (DiD) and regression discontinuity methods to evaluate the impact of the policy instrument. Our estimation results show that the girls in the treatment group who were exposed to the programme had a higher average number of completed years of schooling and also delayed their first marriage compared to the girls in the control group. We also show that the DiD approach might produce a biased result as it does not consider the convergence effect. Essay 4: Estimation of Multiregional Social Accounting Matrices using Transport Data proposes a methodology for estimating multiregional SAMs from a national SAM by applying the cross-entropy method. The methodology makes possible the construction of regional SAMs that are consistent with official regional accounts and minimize deviations from transport data.

Abstract Advances in miniaturisation have led to the development of new wireless measurement technologies such as wireless sensor networks (WSNs). A WSN consists of low cost nodes, which are battery-operated devices, capable of sensing the environment, transmitting and receiving, and computing. While a WSN has several advantages, including cost-effectiveness and easy installation, the nodes suffer from small memory, low computing power, small bandwidth and limited energy supply. In order to cope with restrictions on resources, data processing methods should be as efficient as possible. As a result, high quality approximates are preferred instead of accurate answers. The aim of this thesis was to propose an efficient entropy approximation method for resource-constrained environments. Specifically, the algorithm should use a small, constant amount of memory, and have certain accuracy and low computational demand. The performance of the proposed algorithm was evaluated experimentally with three case studies. The first study focused on the online monitoring of WSN communications performance in an industrial environment. The monitoring approach was based on the observation that entropy could be applied to assess the impact of interferences on time-delay variation of periodic tasks. The main purpose of the additional two cases, depth of anaesthesia (DOA) –monitoring and benchmarking with simulated data sets was to provide additional evidence on the general applicability of the proposed method. Moreover, in case of DOA-monitoring, an efficient entropy approximation could assist in the development of handheld devices or processing large amount of online data from different channels simultaneously. The initial results from the communication and DOA monitoring applications as well as from simulations were encouraging. Therefore, based on the case studies, the proposed method was able to meet the stated requirements. Since entropy is a widely used quantity, the method is also expected to have a variety of applications in measurement systems with similar requirements. / Tiivistelmä Mekaanisten- ja puolijohdekomponenttien pienentyminen on mahdollistanut uusien mittaustekniikoiden, kuten langattomien anturiverkkojen kehittämisen. Anturiverkot koostuvat halvoista, paristokäyttöisistä solmuista, jotka pystyvät mittaamaan ympäristöään sekä käsittelemään, lähettämään ja vastaanottamaan tietoja. Anturiverkkojen etuja ovat kustannustehokkuus ja helppo käyttöönotto, rajoitteina puolestaan vähäinen muisti- ja tiedonsiirtokapasiteetti, alhainen laskentateho ja rajoitettu energiavarasto. Näiden rajoitteiden vuoksi solmuissa käytettävien laskentamenetelmien tulee olla mahdollisimman tehokkaita. Tämän työn tavoitteena oli esittää tehokas entropian laskentamenetelmä resursseiltaan rajoitettuihin ympäristöihin. Algoritmin vaadittiin olevan riittävän tarkka, muistinkulutukseltaan pieni ja vakiosuuruinen sekä laskennallisesti tehokas. Työssä kehitetyn menetelmän suorituskykyä tutkittiin sovellusesimerkkien avulla. Ensimmäisessä tapauksessa perehdyttiin anturiverkon viestiyhteyksien reaaliaikaiseen valvontaan. Lähestymistavan taustalla oli aiempi tutkimus, jonka perusteella entropian avulla voidaan havainnoida häiriöiden vaikutusta viestien viiveiden vaihteluun. Muiden sovellusesimerkkien, anestesian syvyysindikaattorin ja simulaatiokokeiden, päätavoite oli tutkia menetelmän yleistettävyyttä. Erityisesti anestesian syvyyden seurannassa menetelmän arvioitiin voivan olla lisäksi hyödyksi langattomien, käsikäyttöisten syvyysmittareiden kehittämisessä ja suurten mittausmäärien reaaliaikaisessa käsittelyssä. Alustavat tulokset langattoman verkon yhteyksien ja anestesian syvyyden valvonnasta sekä simuloinneista olivat lupaavia. Sovellusesimerkkien perusteella esitetty algoritmi kykeni vastaamaan asetettuihin vaatimuksiin. Koska entropia on laajalti käytetty suure, menetelmä saattaa soveltua useisiin mittausympäristöihin, joissa on samankaltaisia vaatimuksia.

In this thesis, we study statistical inference for entropy, divergence, and related functionals of one or two probability distributions. Asymptotic properties of particular nonparametric estimators of such functionals are investigated. We consider estimation from both independent and dependent observations. The thesis consists of an introductory survey of the subject and some related theory and four papers (A-D). In Paper A, we consider a general class of entropy-type functionals which includes, for example, integer order Rényi entropy and certain Bregman divergences. We propose U-statistic estimators of these functionals based on the coincident or epsilon-close vector observations in the corresponding independent and identically distributed samples. We prove some asymptotic properties of the estimators such as consistency and asymptotic normality. Applications of the obtained results related to entropy maximizing distributions, stochastic databases, and image matching are discussed. In Paper B, we provide some important generalizations of the results for continuous distributions in Paper A. The consistency of the estimators is obtained under weaker density assumptions. Moreover, we introduce a class of functionals of quadratic order, including both entropy and divergence, and prove normal limit results for the corresponding estimators which are valid even for densities of low smoothness. The asymptotic properties of a divergence-based two-sample test are also derived. In Paper C, we consider estimation of the quadratic Rényi entropy and some related functionals for the marginal distribution of a stationary m-dependent sequence. We investigate asymptotic properties of the U-statistic estimators for these functionals introduced in Papers A and B when they are based on a sample from such a sequence. We prove consistency, asymptotic normality, and Poisson convergence under mild assumptions for the stationary m-dependent sequence. Applications of the results to time-series databases and entropy-based testing for dependent samples are discussed. In Paper D, we further develop the approach for estimation of quadratic functionals with m-dependent observations introduced in Paper C. We consider quadratic functionals for one or two distributions. The consistency and rate of convergence of the corresponding U-statistic estimators are obtained under weak conditions on the stationary m-dependent sequences. Additionally, we propose estimators based on incomplete U-statistics and show their consistency properties under more general assumptions.