The objective of this study is to propose an efficient sampling-based RBDO using a new classification method to reduce the computational cost. In addition, accuracy improvement strategies for the Kriging method are proposed to reduce the number of expensive computer experiments. Current research effort involves: (1) developing a new classification method that is more efficient than conventional surrogate modeling methods while maintaining required accuracy level; (2) developing a sequential adaptive sampling method that inserts samples near the limit state function; (3) improving the efficiency of the RBDO process by using a fixed hyper-spherical local window with an efficient uniform sampling method and identification of active/violated constraints; and (4) improving the accuracy of the Kriging method by introducing several strategies. In the sampling-based RBDO, only accurate classification information is needed instead of accurate response surface. On the other hand, in general, surrogates are constructed using all available DoE samples instead of focusing on the limit state function. Therefore, the computational cost of surrogates can be relatively expensive; and the accuracy of the limit state (or decision) function can be sacrificed in return for reducing the error on unnecessary regions away from the limit state function. On the contrary, the support vector machine (SVM), which is a classification method, only uses support vectors, which are located near the limit state function, to focus on the decision function. Therefore, the SVM is very efficient and ideally applicable to sampling-based RBDO, if the accuracy of SVM is improved by inserting virtual samples near the limit state function. The proposed sequential sampling method inserts new samples near the limit state function so that the number of DoE samples is minimized. In many engineering problems, expensive computer simulations are used and thus the total computational cost needs to be reduced by using less number of DoE samples. Several efficiency strategies such as: (1) launching RBDO at a deterministic optimum design, (2) hyper-spherical local windows with an efficient uniform sampling method, (3) filtering of constraints, (4) sample reuse, (5) improved virtual sample generation, are used for the proposed sampling-based RBDO using virtual SVM. The number of computer experiments is also reduced by implementing accuracy improvement strategies for the Kriging method. Since the Kriging method is used for generating virtual samples and generating response surface of the cost function, the number of computer experiments can be reduced by introducing: (1) accurate correlation parameter estimation, (2) penalized maximum likelihood estimation (PMLE) for small sample size, (3) correlation model selection by MLE, and (4) mean structure selection by cross-validation (CV) error.

Time series prediction techniques have been used in many real-world applications such as financial market prediction, electric utility load forecasting, weather and environmental state prediction, and reliability forecasting. The underlying system models and time series data generating processes are generally complex for these applications and the models for these systems are usually not known a priori. Accurate and unbiased estimation of time series data produced by these systems cannot always be achieved using well known linear techniques, and thus the estimation process requires more advanced time series prediction algorithms. One type of time series interpolation and prediction algorithm that has been proven to be effective for these various types of applications is Support Vector Regression (SVR) [1], which is based on the Support Vector Machine (SVM) developed by Vapnik et al. [2, 3]. The underlying motivation for using SVMs is the ability of this methodology to accurately forecast time series data when the underlying system processes are typically nonlinear, non-stationary and not defined a-priori. SVMs have also been proven to outperform other non-linear techniques including neural-network based non-linear prediction techniques such as multi-layer perceptrons. As with most time series prediction algorithms, there are typically challenges associated in applying a given heuristic to any general problem. One difficult challenge in using SVR to solve these types of problems is the selection of free parameters associated with the SVR algorithm. There is no given heuristic to select SVR free parameters and the user is left to adjust these parameters in an ad hoc manner. The focus of this dissertation is to present an alternative to the typical ad hoc approach of tuning SVR for time series prediction problems by using Particle Swarm Optimization (PSO) to assist in the SVR free parameter selection process. Developed by Kennedy and Eberhart [4-8], PSO is a technique that emulates the process living creatures (such as birds or insects) use to discover food resources at a given geographic location. PSO has been proven to be an effective technique for many different kinds of optimization problems [9-11]. The focus of this dissertation is to present an alternative to the typical ad hoc approach of tuning SVR for time series prediction problems by using Particle Swarm Optimization (PSO) to assist in the SVR free parameter selection process. Developed by Kennedy and Eberhart [4-8], PSO is a technique that emulates the process living creatures (such as birds or insects) use to discover food resources at a given geographic location. PSO has been proven to be an effective technique for many different kinds of optimization problems [9-11].

Les compartiments cellulaires, de par les frontières membranaires qui les définissent, permettent l'accomplissement de taches métaboliques diverses au sein de la cellule. Cette spécialisation en domaines intracellulaires induit donc une différentiation dans la fonction des protéines qui les composent. Le grand nombre de gènes orphelins produits ces dernières années par les projets de séquençage motive la mise au point de méthodes efficaces pour la prédiction ab-initio de la localisation cellulaire des protéines.<br /><br />Ainsi la majorité de ce travail de thèse s'intéresse au problème de la prédiction du compartiment cellulaire d'une protéine à partir de sa séquence primaire.<br /><br />Nous nous sommes attachés à proposer des alternatives descriptives aux méthodes existantes de prédiction de la localisation cellulaire en utilisant : (1) de nouveaux descripteurs issus de la séquence nucléique, (2) une approche par chaînes de Markov cachées (CMC) et arbres de décision. L'approche par CMC est justifiée biologiquement a posteriori car elle permet la modélisation de signaux d'adressage conjointement à la prise en compte de la composition globale. En outre, l'étape de classification hiérarchique par arbre améliore nettement les résultats de classification. Les résultats obtenues lors des comparaisons avec les méthodes existantes et utilisant des descripteurs fondés sur la composition globale possèdent des performances similaires.

Nous étudions le problème fondamental de la sélection de variables descriptives d'un signal, sélection dédiée à divers traitements comme la classification d'objets dans une image. Nous définissons dans un premier temps une loi de probabilités sur les variables descriptives du signal et utilisons un algorithme de descente de gradient, exact puis stochastique pour identifier la bonne distribution de probabilités sur ces variables. Nous donnons alors diverses applications à la classification d'objets (chiffres manuscrits, détection de visages, de spam, ...).<br /> Dans un second temps, nous implémentons un algorithme de diffusion réfléchie sur l'espace des probabilités puis de diffusion réfléchie avec sauts pour permettre plus facilement de faire évoluer l'espace des variables, ainsi que la probabilité apprise. Cette seconde approche nécessite un effort particulier au niveau des simulations stochastiques, qui sont alors étudiées le plus clairement possible.<br />Nous concluons par quelques expériences dans les mêmes domaines que précédemment.

Les SVM (Support Vector Machines) sont de nouvelles techniques d'apprentissage statistique proposées par V.Vapnik en 1995. Elles permettent d'aborder des problèmes très divers comme le classement, la régression, la fusion, etc... Depuis leur introduction dans le domaine de la Reconnaissance de Formes (RdF), plusieurs travaux ont pu montrer l'efficacité de ces techniques principalement en traitement d'image. L'idée essentielle des SVM consiste à projeter les données de l'espace d'entrée (appartenant à deux classes différentes) non-linéairement séparables dans un espace de plus grande dimension appelé espace de caractéristiques de façon à ce que les données deviennent linéairement séparables. Dans cet espace, la technique de construction de l'hyperplan optimal est utilisée pour calculer la fonction de classement séparant les deux classes. Dans ce travail de thèse, nous avons étudié les SVM comme techniques de classement pour la Vérification Automatique du Locuteur (VAL) en mode dépendant et indépendant du texte. Nous avons également étudié les SVM pour des tâches de fusion en réalisant des expériences concernant deux types de fusion, la fusion de méthodes et la fusion de modes. Dans le cadre du projet PICASSO, nous avons proposé un système de VAL en mode dépendant du texte utilisant les SVM dans une application de mots de passe publics. Dans ce système, une nouvelle modélisation basée sur la transcription phonétique des mots de passe a été proposée pour construire les vecteurs d'entrée pour notre classifieur SVM. En ce qui concerne notre étude des SVM en VAL en mode indépendant du texte, nous avons proposé des systèmes hybrides GMM-SVM. Dans ces systèmes, trois nouvelles représentations de données ont été proposées permettant de réunir l'efficacité des GMM en modélisation et les performances des SVM en décision. Ce travail entre dans le cadre de nos participations aux évaluations internationales NIST. Dans le cadre du projet BIOMET sur l'authentification biométrique mené par le GET (Groupe des Écoles de Télécommunications), nous avons étudié les SVM pour deux tâches de fusion. La première concerne la fusion de méthodes où nous avons fusionné les scores obtenus par les participants à la tâche ``One Speaker Detection'' aux évaluations NIST'2001. La seconde concerne la fusion de modes menée sur les scores obtenus sur les quatre différentes modalités de la base de données M2VTS. Les études que nous avons réalisées représentent une des premières tentatives d'appliquer les SVM dans le domaine de la VAL. Les résultats obtenus montrent que les SVM sont des techniques très efficaces et surtout très prometteuses que ce soit pour le classement ou la fusion.

<p>The subject of this thesis is the application of Support Vector Machines on two totally different applications, facial expressions recognition and remote sensing.</p><p>The basic idea of kernel algorithms is to transpose input data in a higher dimensional space, the feature space, in which linear operations on the data can be processed more easily. These operations in the feature space can be expressed in terms of input data thanks to the kernel functions. Support Vector Machines is a classifier using this kernel method by computing, in the feature space and on basis of examples of the different classes, hyperplanes that separate the classes. The hyperplanes in the feature space correspond to non linear surfaces in the input space.</p><p>Concerning facial expressions, the aim is to train and test a classifier able to recognise, on basis of some pictures of faces, which emotion (among these six ones: anger, disgust, fear, joy, sad, and surprise) that is expressed by the person in the picture. In this application, each picture has to be seen has a point in an N-dimensional space where N is the number of pixels in the image.</p><p>The second application is the detection of camouflage nets hidden in vegetation using a hyperspectral image taken by an aircraft. In this case the classification is computed for each pixel, represented by a vector whose elements are the different frequency bands of this pixel.</p>

In the first part of this paper we show a similarity between the principle of Structural Risk Minimization Principle (SRM) (Vapnik, 1982) and the idea of Sparse Approximation, as defined in (Chen, Donoho and Saunders, 1995) and Olshausen and Field (1996). Then we focus on two specific (approximate) implementations of SRM and Sparse Approximation, which have been used to solve the problem of function approximation. For SRM we consider the Support Vector Machine technique proposed by V. Vapnik and his team at AT&T Bell Labs, and for Sparse Approximation we consider a modification of the Basis Pursuit De-Noising algorithm proposed by Chen, Donoho and Saunders (1995). We show that, under certain conditions, these two techniques are equivalent: they give the same solution and they require the solution of the same quadratic programming problem.

The Support Vector Machine (SVM) is a new and very promising classification technique developed by Vapnik and his group at AT&T Bell Labs. This new learning algorithm can be seen as an alternative training technique for Polynomial, Radial Basis Function and Multi-Layer Perceptron classifiers. An interesting property of this approach is that it is an approximate implementation of the Structural Risk Minimization (SRM) induction principle. The derivation of Support Vector Machines, its relationship with SRM, and its geometrical insight, are discussed in this paper. Training a SVM is equivalent to solve a quadratic programming problem with linear and box constraints in a number of variables equal to the number of data points. When the number of data points exceeds few thousands the problem is very challenging, because the quadratic form is completely dense, so the memory needed to store the problem grows with the square of the number of data points. Therefore, training problems arising in some real applications with large data sets are impossible to load into memory, and cannot be solved using standard non-linear constrained optimization algorithms. We present a decomposition algorithm that can be used to train SVM's over large data sets. The main idea behind the decomposition is the iterative solution of sub-problems and the evaluation of, and also establish the stopping criteria for the algorithm. We present previous approaches, as well as results and important details of our implementation of the algorithm using a second-order variant of the Reduced Gradient Method as the solver of the sub-problems. As an application of SVM's, we present preliminary results we obtained applying SVM to the problem of detecting frontal human faces in real images.

When training Support Vector Machines (SVMs) over non-separable data sets, one sets the threshold $b$ using any dual cost coefficient that is strictly between the bounds of $0$ and $C$. We show that there exist SVM training problems with dual optimal solutions with all coefficients at bounds, but that all such problems are degenerate in the sense that the "optimal separating hyperplane" is given by ${f w} = {f 0}$, and the resulting (degenerate) SVM will classify all future points identically (to the class that supplies more training data). We also derive necessary and sufficient conditions on the input data for this to occur. Finally, we show that an SVM training problem can always be made degenerate by the addition of a single data point belonging to a certain unboundedspolyhedron, which we characterize in terms of its extreme points and rays.

In statistical data mining research, datasets often have nonlinearity and high-dimensionality. It has become difficult to analyze such datasets in a comprehensive manner using traditional statistical methodologies. Kernel-based data mining is one of the most effective statistical methodologies to investigate a variety of problems in areas including pattern recognition, machine learning, bioinformatics, chemometrics, and statistics. In particular, statistically-sophisticated procedures that emphasize the reliability of results and computational efficiency are required for the analysis of high-dimensional data. In this dissertation, first, a novel wrapper method called SVM-ICOMP-RFE based on hybridized support vector machine (SVM) and recursive feature elimination (RFE) with information-theoretic measure of complexity (ICOMP) is introduced and developed to classify high-dimensional data sets and to carry out subset selection of the variables in the original data space for finding the best for discriminating between groups. Recursive feature elimination (RFE) ranks variables based on the information-theoretic measure of complexity (ICOMP) criterion. Second, a dual variables functional support vector machine approach is proposed. The proposed approach uses both the first and second derivatives of the degradation profiles. The modified floating search algorithm for the repeated variable selection, with newly-added degradation path points, is presented to find a few good variables while reducing the computation time for on-line implementation. Third, a two-stage scheme for the classification of near infrared (NIR) spectral data is proposed. In the first stage, the proposed multi-scale vertical energy thresholding (MSVET) procedure is used to reduce the dimension of the high-dimensional spectral data. In the second stage, a few important wavelet coefficients are selected using the proposed SVM gradient-recursive feature elimination (RFE). Fourth, a novel methodology based on a human decision making process for discriminant analysis called PDCM is proposed. The proposed methodology consists of three basic steps emulating the thinking process: perception, decision, and cognition. In these steps two concepts known as support vector machines for classification and information complexity are integrated to evaluate learning models.