Stochastic and data-distributed optimization algorithms have received lots of attention from the machine learning community due to the tremendous demand from the large-scale learning and the big-data related optimization. A lot of stochastic and deterministic learning algorithms are proposed recently under various application scenarios. Nevertheless, many of these algorithms are based on heuristics and their optimality in terms of the generalization error is not sufficiently justified. In this talk, I will explain the concept of an optimal learning algorithm, and show that given a time budget and proper hypothesis space, only those achieving the lower bounds of the estimation error and the optimization error are optimal. Guided by this concept, we investigated the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We proposed a novel algorithm named Accelerated Nonsmooth Stochastic Gradient Descent, which exploits the structure of common nonsmooth loss functions to achieve optimal convergence rates for a class of problems including SVMs. It is the first stochastic algorithm that can achieve the optimal O(1/t) rate for minimizing nonsmooth loss functions. The fast rates are confirmed by empirical comparisons with state-of-the-art algorithms including the averaged SGD. The Alternating Direction Method of Multipliers (ADMM) is another flexible method to explore function structures. In the second part we proposed stochastic ADMM that can be applied to a general class of convex and nonsmooth functions, beyond the smooth and separable least squares loss used in lasso. We also demonstrate the rates of convergence for our algorithm under various structural assumptions of the stochastic function: O(1/sqrt{t}) for convex functions and O(log t/t) for strongly convex functions. A novel application named Graph-Guided SVM is proposed to demonstrate the usefulness of our algorithm. We also extend the scalability of stochastic algorithms to nonlinear kernel machines, where the problem is formulated as a constrained dual quadratic optimization. The simplex constraint can be handled by the classic Frank-Wolfe method. The proposed stochastic Frank-Wolfe methods achieve comparable or even better accuracies than state-of-the-art batch and online kernel SVM solvers, and are significantly faster. The last part investigates the problem of data-distributed learning. We formulate it as a consensus-constrained optimization problem and solve it with ADMM. It turns out that the underlying communication topology is a key factor in achieving a balance between a fast learning rate and computation resource consumption. We analyze the linear convergence behavior of consensus ADMM so as to characterize the interplay between the communication topology and the penalty parameters used in ADMM. We observe that given optimal parameters, the complete bipartite and the master-slave graphs exhibit the fastest convergence, followed by bi-regular graphs.

Cette thèse s'intègre dans le cadre du traitement automatique du langage naturel. La problématique du résumé automatique de documents arabes qui a été abordée, dans cette thèse, s'est cristallisée autour de deux points. Le premier point concerne les critères utilisés pour décider du contenu essentiel à extraire. Le deuxième point se focalise sur les moyens qui permettent d'exprimer le contenu essentiel extrait sous la forme d'un texte ciblant les besoins potentiels d'un utilisateur. Afin de montrer la faisabilité de notre approche, nous avons développé le système "L.A.E", basé sur une approche hybride qui combine une analyse symbolique avec un traitement numérique. Les résultats d'évaluation de ce système sont encourageants et prouvent la performance de l'approche hybride proposée. Ces résultats, ont montré, en premier lieu, l'applicabilité de l'approche dans le contexte de documents sans restriction quant à leur thème (Éducation, Sport, Science, Politique, Reportage, etc.), leur contenu et leur volume. Ils ont aussi montré l'importance de l'apprentissage dans la phase de classement et sélection des phrases forment l'extrait final.

Šis magistrinis darbas aprašo plačiausiai naudojamus dirbtinio intelekto metodus ir galimybes juos taikyti kredito rizikos, kuri yra viena svarbiausių sričių bankininkystėje ir finansuose, vertinime. Pagrindinė problema yra rizikos, atsirandančios kreditoriui išduodant kreditą tam tikram individui ar bendrovei, vertinimas, naudojant įvairius matematinius, statistinius ar kitus metodus. Ši rizika atsiranda tada, kai skolininkas negali laiku grąžinti skolos kreditoriui, kas reiškia papildomus nuostolius. Ji gali pasireikšti, priklausomai nuo skolininko tipo (individas, bendrovė ar užsienio vyriausybė) bei finansinio instrumento tipo ar su juo atliekamo veiksmo (skolos teikimas, finansinių derivatyvų tranzakcijos ir kt.), todėl finansinės institucijos jos įvertinimui bei valdymui naudoja įvairius metodus nuo vertinimo balais bei skirtingų faktorių, tokių kaip valdymo bei veiklos strategijos bei politika, įvertinimo iki klasifikavimo pagal įvairius kriterijus, naudojant modernius ir sudėtingus metodus, tiek matematinius, tiek dirbtinio intelekto. Ši sritis plačiai tiriama ir daug naujų metodų bei sprendimų pastoviai randama. Šio darbo tyrimas sukoncentruotas į atraminių vektorių mašinų (angl.Support Vector Machines, sutr. SVM) metodų, kuris yra viena populiariausių dirbtinio intelekto bei mašininio mokymo metodų ir kurio efektyvumas daugeliu atveju įrodytas. Šiuo tyrimo tikslas yra ištirti galimybes pritaikyti SVM metodą čia aprašomai problemai bei realizuoti sistemą, naudojančią... [toliau žr. visą tekstą] / This master work describes the most widely used artificial intelligence methods and the possibilities to apply them in credit risk evaluation which is one of the most important fields in banking and in finance. The main problem here is to evaluate the risk arising when a creditor gives a credit to a particular individual or an enterprise, using various mathematical, statistical or other methods and techniques. This risk arises when the debtor isn’t able to pay for the loan to the creditor in time which means additional loss. It can appear in many forms depending on the type of debtor (individ-ual, enterprise, government of an abroad country) and type of financial instrument or action that is done with it (giving of a loan, transactions of financial derivatives, etc.), this is the reason why fi-nancial institutions and for it’s evaluation and management use various different methodologies which comprise a lot of methods and techniques from credit scoring (evaluating by a particular formula, usually linear) and evaluating different factors, like management and business strategies or policies, to classification by various criterions by using modern and sophisticated methods, either algebraic, either artificial intelligence and machine learning. This field is widely researched and many new techniques are being found. The research here is concentrated mainly on Support Vector Machines (abbr. SVM) which is one of the most popular artificial intelligence and machine learning... [to full text]

Cette thèse s'inscrit dans le contexte de la vision embarquée pour la détection et la reconnaissance d'obstacles routiers, en vue d'application d'assistance à la conduite automobile.A l'issue d'une étude bibliographique, nous avons constaté que la problématique de détection d'obstacles routiers, notamment des piétons, à l'aide d'une caméra embarquée, ne peut être résolue convenablement sans recourir aux techniques de reconnaissance de catégories d'objets dans les images. Ainsi, une étude complète du processus de la reconnaissance est réalisée, couvrant les techniques de représentation,de classification et de fusion d'informations. Les contributions de cette thèse se déclinent principalement autour de ces trois axes.Notre première contribution concerne la conception d'un modèle d'apparence locale basée sur un ensemble de descripteurs locaux SURF (Speeded Up RobustFeatures) représentés dans un Vocabulaire Visuel Hiérarchique. Bien que ce modèle soit robuste aux larges variations d'apparences et de formes intra-classe, il nécessite d'être couplé à une technique de classification permettant de discriminer et de catégoriser précisément les objets routiers. Une deuxième contribution présentée dans la thèse porte sur la combinaison du Vocabulaire Visuel Hiérarchique avec un classifieur SVM.Notre troisième contribution concerne l'étude de l'apport d'un module de fusion multimodale permettant d'envisager la combinaison des images visibles et infrarouges.Cette étude met en évidence de façon expérimentale la complémentarité des caractéristiques locales et globales ainsi que la modalité visible et celle infrarouge.Pour réduire la complexité du système, une stratégie de classification à deux niveaux de décision a été proposée. Cette stratégie est basée sur la théorie des fonctions de croyance et permet d'accélérer grandement le temps de prise de décision.Une dernière contribution est une synthèse des précédentes : nous mettons à profit les résultats d'expérimentations et nous intégrons les éléments développés dans un système de détection et de suivi de piétons en infrarouge-lointain. Ce système a été validé sur différentes bases d'images et séquences routières en milieu urbain.

We consider the problem of learning the parameters of a structured output prediction model, that is, learning to predict elements of a complex interdependent output space that correspond to a given input. Unlike many of the existing approaches, we focus on the weakly supervised setting, where most (or all) of the training samples have only been partially annotated. Given such a weakly supervised dataset, our goal is to estimate accurate parameters of the model by minimizing the regularized empirical risk, where the risk is measured by a user-specified loss function. This task has previously been addressed by the well-known latent support vector machine (latent SVM) framework. We argue that, while latent SVM offers a computational efficient solution to loss-based weakly supervised learning, it suffers from the following three drawbacks: (i) the optimization problem corresponding to latent SVM is a difference-of-convex program, which is non-convex, and hence susceptible to bad local minimum solutions; (ii) the prediction rule of latent SVM only relies on the most likely value of the latent variables, and not the uncertainty in the latent variable values; and (iii) the loss function used to measure the risk is restricted to be independent of true (unknown) value of the latent variables. We address the the aforementioned drawbacks using three novel contributions. First, inspired by human learning, we design an automatic self-paced learning algorithm for latent SVM, which builds on the intuition that the learner should be presented in the training samples in a meaningful order that facilitates learning: starting frome easy samples and gradually moving to harder samples. Our algorithm simultaneously selects the easy samples and updates the parameters at each iteration by solving a biconvex optimization problem. Second, we propose a new family of LVMs called max-margin min-entropy (M3E) models, which includes latent SVM as a special case. Given an input, an M3E model predicts the output with the smallest corresponding Renyi entropy of generalized distribution, which relies not only on the probability of the output but also the uncertainty of the latent variable values. Third, we propose a novel learning framework for learning with general loss functions that may depend on the latent variables. Specifically, our framework simultaneously estimates two distributions: (i) a conditional distribution to model the uncertainty of the latent variables for a given input-output pair; and (ii) a delta distribution to predict the output and the latent variables for a given input. During learning, we encourage agreement between the two distributions by minimizing a loss-based dissimilarity coefficient. We demonstrate the efficacy of our contributions on standard machine learning applications using publicly available datasets.

Patients with macular degeneration (MD) often fixate with a preferred retinal locus (PRL). Eye movements made while fixating with the PRL (in MD patients) has been observed to be maladaptive compared to those made while fixating with the fovea (normal sighted individuals). For example, in MD patients, PRL eye movements negatively affect fixation stability and re-fixation precision; consequently creating difficulty in reading and limits to their execution of other everyday activities. Abnormal eye movements from the PRL affect research on the physiological adaptations to MD. Specifically, previous research on cortical reorganization using functional magnetic resonance imaging (fMRI), indicates a critical need to accurately determine a MD patient's point of gaze in order to better infer existence of cortical reorganization. Unfortunately, standard MR compatible hardware eye-tracking systems do not work well with these patients. Their reduction in fixation stability often overwhelms the tracking algorithms used by these systems. This research investigates the use of an existing magnetic resonance imaging (MRI) based technique called Predictive Eye Estimation Regression (PEER) to determine the point of gaze of MD patients and thus control for fixation instability. PEER makes use of the fluctuations in the MR signal caused by eye movements to identify position of gaze. Engineering adaptations such as temporal resolution and brain coverage were applied to tailor PEER to MD patients. Also participants were evaluated on different fixation protocols and the results compared to that of the micro-perimeter MP-1 to test the efficacy of PEER. The fixation stability results obtained from PEER were similar to that obtained from the eye tracking results of the micro-perimeter MP-1. However, PEER's point of gaze estimations was different from the MP-1's in the fixation tests. The difference in this result cannot be concluded to be specific to PEER. In order to resolve this issue, advancements to PEER by the inclusion of an eye tracker in the scanner to run concurrently with PEER could provide more evidence of PEER's reliability. In addition, increasing the diversity of AMD patients in terms of the different scotoma types will help provide a better estimate of PEER flexibility and robustness.

Thesis (MScEng)--Stellenbosch University, 2013. / Wind energy has proven to be a viable source of clean energy, and the worldwide demand is growing rapidly. Variable speed topologies, with synchronous generators and full-scale converters, are becoming more popular, and the e ective control of these systems is a current trend in wind energy research. The purpose of this study is the modelling, design, simulation and implementation of a small-scale, variable speed wind energy conversion system, incorporating the position sensorless direct torque control with space vector modulation, of a permanent magnet synchronous generator, including an RLC converter lter. Another aim is the development of a gain scheduling algorithm that facilitates the high level control of the system. Mathematical models of the combined lter-generator model, in the stationary and rotating reference frames, are presented and discussed, from which equivalent approximate transfer functions are derived for the design of the controller gains. The design of the controller gains, RLC lter components, gain scheduling concept and maximum power point tracking controller are presented. It is discovered that the RLC lter damping resistance has a signi cant e ect on the resonance frequency of the system. The system is simulated dynamically in both Simulink and the VHDL-AMS programming language. Additionally, the maximum power point tracking controller is simulated in the VHDL-AMS simulation, including a wind turbine simulator. The simulation results demonstrate good dynamic performance, as well as the variable speed operation of the system. The practical results of torque and speed controllers show satisfactory performance, and correlate well with simulated results. The detailed gain scheduling algorithm is presented and discussed. A nal test of the complete system yields satisfactory practical results, and con rms that the objectives of this thesis have been reached.

The continuity in the financial and other related losses due to cyber-attacks prove the substantial growth of malware and their lethal proliferation techniques. Every successful malware attack highlights the weaknesses in the defence mechanisms responsible for securing the targeted computer or a network. The recent cyber-attacks reveal the presence of sophistication and intelligence in malware behaviour having the ability to conceal their code and operate within the system autonomously. The conventional detection mechanisms not only possess the scarcity in malware detection capabilities, they consume a large amount of resources while scanning for malicious entities in the system. Many recent reports have highlighted this issue along with the challenges faced by the alternate solutions and studies conducted in the same area. There is an unprecedented need of a resilient and autonomous solution that takes proactive approach against modern malware with stealth behaviour. This thesis proposes a multi-aspect solution comprising of an intelligent malware detection framework and an energy efficient hosting model. The malware detection framework is a combination of conventional and novel malware detection techniques. The proposed framework incorporates comprehensive feature heuristics of files generated by a bespoke static feature extraction tool. These comprehensive heuristics are used to train the machine learning algorithms; Support Vector Machine, Decision Tree, and Boosting to differentiate between clean and malicious files. Both these techniques; feature heuristics and machine learning are combined to form a two-factor detection mechanism. This thesis also presents a cloud-based energy efficient and scalable hosting model, which combines multiple infrastructure components of Amazon Web Services to host the malware detection framework. This hosting model presents a client-server architecture, where client is a lightweight service running on the host machine and server is based on the cloud. The proposed framework and the hosting model were evaluated individually and combined by specifically designed experiments using separate repositories of clean and malicious files. The experiments were designed to evaluate the malware detection capabilities and energy efficiency while operating within a system. The proposed malware detection framework and the hosting model showed significant improvement in malware detection while consuming quite low CPU resources during the operation.

Les convertisseurs d'électronique de puissance sont de plus en plus exploités notamment dans les applications nécessitant la variation de vitesse de machines. L'utilisation de composants plus performants et plus puissants couplés à de nouvelles structures multiniveaux autorise l'accès à de nouveaux champs applicatifs, ou des fonctionnements à plus haut rendement. Ces convertisseurs statiques sont capables de gérer, par un pilotage adapté, les transferts d'énergie entre différentes sources et différents récepteurs selon la famille de convertisseur utilisée. Au sein de l'interface de pilotage, un schéma particulier permet de générer des signaux de commande pour les interrupteurs, il s'agit de la modulation et peut être vue par deux approches différentes : L'approche intersective issue d'une comparaison modulante-horteuse (appelée en anglais carrier based PWM) et l'approche vectorielle où les signaux de pilotage des trois bras de ponts sont considérés comme un vecteur global unique (appelée Modulation Vectorielle SVM). Le but de la MLI est de générer une valeur moyenne de la tension la plus proche possible du signal modulé. La commande usuelle par comparaison modulante-porteuse dans le cas des architectures multiniveaux nécessite autant de porteuses triangulaires qu'il y a de cellules à commander au sein d'un bras. Plus généralement, la stratégie de modulation de chacune des topologies multiniveaux est choisie en se basant sur des critères à optimiser liés à la qualité les formes d'ondes produites ou obtenues, suite à la conversion. Le choix de la variable de commande à implémenter dans le schéma MLI fait appel à l'expertise de l'expérimentateur et se réfère peu au modèle mathématique initial qui peut-être établit pour caractériser le fonctionnement de l'architecture d'électronique de puissance. En ce qui concerne les stratégies vectorielles SVM, une absence de modèle compatible avec les modèles, basés sur une comparaison modulante porteuse, d'onduleurs est constatée. Les types d'onduleurs triphasés à deux ou à N niveaux de tension admettent un modèle sous forme d'équations d'un système linéaire compatible qui s'écrit sous la forme V = f(a) dans le cas d'une MLI sinusoïdale et V = f(1) dans le cas d'une SVM, avec V les tensions de phase, a les rapports cycliques et f les instants de commutation. Dans cette configuration basique il est constaté que la matrice liant ces tensions aux rapports cycliques (ou aux instants de commutation) n'admet pas d'inverse, ce qui revient à dire qu'il n'est pas possible, avec les théories usuelles des fonctions linéaires, de résoudre ce système afin d'exprimer les rapports cycliques (ou les instants de commutation) en fonction des tensions de références. C'est ce qui explique qu'aujourd'hui un bon nombre d'implémentations pratiques de modulation se fait, suite à une analyse expérimentale des conséquences d'un choix de stratégie sur les variables d'intérêt. / The power electronic converters are increasingly exploited in particular in applications requiring variable speed machines. The use of more effcient and more powerful components coupled with new multilevel structures widens the fields of application and allows high efficiency functioning. These converters are able to manage, with a suitable control, the energy transfer between different sources and different receivers depending on the used converter family. In the control interface, a particular pattern is used to generate control signais for the switches, it is the modulation. Generally, the modulation strategy takes two forms : a Modulation based on comparaison modulating - caiTier (Carrier based Pulse Width Modulation, (CPWM)) or a Vector Modulation (SVM). The purpose of the PWM is to generate a signal which has a mean value as nearest as possible to the desired sinusoidal signal. The usual control by PWM, in the case of multi-level architectures, requires as many triangular carriers as there are cells to be controlled within an arm. The modulation strategy selection for each multilevel topology is based on optimizing criterias related to the quality of the produced waveforms after the conversion. The choice of the variable to implement in the PWM scheme requires expertise of the experimenter and refers little to the initial mathematical model that can be established to characterize the operation of the power electronics architecture. Concerning the vector strategies SVM, the lack of a compatible model with PWM inverters is observed. The three-phase inverters with two or N voltage levels can be modeled in the form of equations of a compatible linear system that is written as V= f(a) in the case of a sinusoïdal PWM and V= f(1) in the case of SVM, with V represents phase voltages, ais a duty cycle and fthe switching instants. In this basic configuration, it is found that the matrix linking these voltages duty cycles (or switching times) adrnits no inverse, which means that it is not possible with the usuallinear functions theories to solve this system in order to express the duty ratios (or the instants of switching) as a function of the reference voltages. This is the reason that today a number of practical implementations of modulation is done after experimental analysis of the consequences of strategy choices on the variables of interest. This study proposes the development of a generic formulation for the modeling of voltage inverters and especially multilevel inverters. The development of generic models for the implementation of modulation strategies is illustrated. The extension of the average model to the three-phase systems is performed to the usual structures of N levels such as the floating capacity and H bridge inverters. The idea is to generalize the model to the multi-level architectures, whether by the sinusoidal PWM modulation expressing the alpha as an output variable, or by the SVM expressing tau. This thesis aims to define a modeling approach and mathematically express the set of solutions in order to generate modulation strategies for various architectures of inverters studied. This will be done using a tool for solving linear systems. This resolution is based on finding degrees of freedom, to be identified at first, then express them in a second step by establishing the link with the criteria to optimize for given architectures. Two examples of application have been implemented on conventional two levels of voltage inverters and the thtree levels flying capacitor voltage inverter.

Submitted by Automa??o e Estat?stica (sst@bczm.ufrn.br) on 2016-07-11T17:32:14Z No. of bitstreams: 1 JhonnataBezerraDeCarvalho_DISSERT.pdf: 3154941 bytes, checksum: aab207a1be81d327327698a3f8ff9a2d (MD5) / Approved for entry into archive by Arlan Eloi Leite Silva (eloihistoriador@yahoo.com.br) on 2016-07-15T21:46:07Z (GMT) No. of bitstreams: 1 JhonnataBezerraDeCarvalho_DISSERT.pdf: 3154941 bytes, checksum: aab207a1be81d327327698a3f8ff9a2d (MD5) / Made available in DSpace on 2016-07-15T21:46:07Z (GMT). No. of bitstreams: 1 JhonnataBezerraDeCarvalho_DISSERT.pdf: 3154941 bytes, checksum: aab207a1be81d327327698a3f8ff9a2d (MD5) Previous issue date: 2016-02-03 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior (CAPES) / O classificador M?quina de Suporte Vetorial, que vem do termo em ingl?s \textit{Support Vector Machine}, ? utilizado em diversos problemas em v?rias ?reas do conhecimento. Basicamente o m?todo utilizado nesse classificador ? encontrar o hiperplano que maximiza a dist?ncia entre os grupos, para aumentar o poder de generaliza??o do classificador. Neste trabalho, s?o tratados alguns problemas de classifica??o bin?ria com dados obtidos atrav?s da eletroencefalografia (EEG) e eletromiografia (EMG), utilizando a M?quina de Suporte Vetorial com algumas t?cnicas complementares, destacadas a seguir como: An?lise de Componentes Principais para a identifica??o de regi?es ativas do c?rebro, o m?todo do periodograma que ? obtido atrav?s da An?lise de Fourier, para ajudar a discriminar os grupos e a suaviza??o por M?dias M?veis Simples para a redu??o dos ru?dos existentes nos dados. Foram desenvolvidas duas fun??es no $software$ \textbf{R}, para a realiza??o das tarefas de treinamento e classifica??o. Al?m disso, foram propostos 2 sistemas de pesos e uma medida sumarizadora para auxiliar na decis?o do grupo pertencente. A aplica??o dessas t?cnicas, pesos e a medida sumarizadora no classificador, mostraram resultados bastantes satisfat?rios, em que os melhores resultados encontrados foram, uma taxa m?dia de acerto de 95,31\% para dados de est?mulos visuais, 100\% de classifica??o correta para dados de epilepsia e taxas de acerto de 91,22\% e 96,89\% para dados de movimentos de objetos para dois indiv?duos. / The classifier support vector machine is used in several problems in various areas of knowledge. Basically the method used in this classier is to end the hyperplane that maximizes the distance between the groups, to increase the generalization of the classifier. In this work, we treated some problems of binary classification of data obtained by electroencephalography (EEG) and electromyography (EMG) using Support Vector Machine with some complementary techniques, such as: Principal Component Analysis to identify the active regions of the brain, the periodogram method which is obtained by Fourier analysis to help discriminate between groups and Simple Moving Average to eliminate some of the existing noise in the data. It was developed two functions in the software R, for the realization of training tasks and classification. Also, it was proposed two weights systems and a summarized measure to help on deciding in classification of groups. The application of these techniques, weights and the summarized measure in the classier, showed quite satisfactory results, where the best results were an average rate of 95.31% to visual stimuli data, 100% of correct classification for epilepsy data and rates of 91.22% and 96.89% to object motion data for two subjects.