When performing predictive data mining, the use of ensembles is claimed to virtually guarantee increased accuracy compared to the use of single models. Unfortunately, the problem of how to maximize ensemble accuracy is far from solved. In particular, the relationship between ensemble diversity and accuracy is not completely understood, making it hard to efficiently utilize diversity for ensemble creation. Furthermore, most high-accuracy predictive models are opaque, i.e. it is not possible for a human to follow and understand the logic behind a prediction. For some domains, this is unacceptable, since models need to be comprehensible. To obtain comprehensibility, accuracy is often sacrificed by using simpler but transparent models; a trade-off termed the accuracy vs. comprehensibility trade-off. With this trade-off in mind, several researchers have suggested rule extraction algorithms, where opaque models are transformed into comprehensible models, keeping an acceptable accuracy.In this thesis, two novel algorithms based on Genetic Programming are suggested. The first algorithm (GEMS) is used for ensemble creation, and the second (G-REX) is used for rule extraction from opaque models. The main property of GEMS is the ability to combine smaller ensembles and individual models in an almost arbitrary way. Moreover, GEMS can use base models of any kind and the optimization function is very flexible, easily permitting inclusion of, for instance, diversity measures. In the experimentation, GEMS obtained accuracies higher than both straightforward design choices and published results for Random Forests and AdaBoost. The key quality of G-REX is the inherent ability to explicitly control the accuracy vs. comprehensibility trade-off. Compared to the standard tree inducers C5.0 and CART, and some well-known rule extraction algorithms, rules extracted by G-REX are significantly more accurate and compact. Most importantly, G-REX is thoroughly evaluated and found to meet all relevant evaluation criteria for rule extraction algorithms, thus establishing G-REX as the algorithm to benchmark against. / <p>Avhandling framlagd 2007-06-01 vid Högskolan i Skövde.</p><p>Opponent: Rögnvaldsson, Thorsteinn, Professor, Sektionen för informationsvetenskap, Data- och Elektroteknik, Högskolan i Halmstad.</p>

對於神經網路系統將提出一個法則萃取的方式，並從神經網路中得到相關法則。在這裡我們所提到的方法是根據反函數的觀念而得到的。 / A rule-extraction method of the layered feed-forward neural networks is proposed here for identifying the rules suggested in the network. The method that we propose for the trained layered feed-forward neural network is based on the inversion of the functions computed by each layer of the network. The new rule-extraction method back-propagates regions from the output layer back to the input layer, and we hope that the method can be used further to deal with the predicament of ANN being a black box.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23397号 / 情博第766号 / 新制||情||131(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 阿久津 達也, 教授 山本 章博, 教授 鹿島 久嗣 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

神經網路已經被成功地應用於解決各種分類及函數近似的問題，尤其因為神經網路是個萬能的近似器(universal approximator)，所以對於函數近似的問題效果更為顯著。以往對於此類問題雖然多數以線性的分析工具為主，但是實際上多數問題本質上是非線性的，所以對於非線性分析工具的需求其實是很大的。自1986年起，神經網路本身的運作一直被視為一個黑箱作業，難以判斷網路學習結果的合理性，更無法有效地幫助使用者增進其知識，因此提供一套合理及有效的神經網路分析方法是重要。 本文提出一套分析神網路系統的方法；利用線性規劃的技巧萃取及分析網路中的規則(rule)，而不需要對任何資料集做分析；進而利用統計無母數方法－符號檢定－歸納出網路中的知識。以債券評價為例，驗證此方法的可行性，實證結果亦顯示此方法所萃取出來的規則是合理的，且由這些萃取出的規則中，所歸納出來有關債券評價的知識多數是合理的。 / Neural networks have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful for function approximation problems because they have been shown to be uni-versal approximators. In the past, for function approximation problems, they were mainly analyzed via tools of linear analyses. However, most of the function approxi-mation problems needed tools of nonlinear analyses in fact. Thus, there is the much demand for tools of nonlinear analyses. Since 1986, the neural network is considered a black box. It is hard to determine if the learning result of a neural network is rea-sonable, and the network can not effectively help users to develop the domain knowl-edge. Thus, it is important to supply a reasonable and effective analytic method of the neural network. Here, we propose an analytic method of the neural network. It can extract rules from the neural network and analyze them via the Linear Programming and does not depend on any data analysis. Then we can generalize domain knowledge from these rules via the sign test, a statistical non-parameter method. We take the bond-pricing as an instance to examine the feasibility of our proposed method. The result shows that these extracted rules are reasonable by our method and that these generalized domain knowledge from these rules is also reasonable.

Técnicas de Aprendizado de Máquina não-simbólicas, como Redes Neurais Artificiais, Máquinas de Vetores de Suporte e combinação de classificadores têm mostrado um bom desempenho quando utilizadas para análise de dados. A grande limitação dessas técnicas é a falta de compreensibilidade do conhecimento armazenado em suas estruturas internas. Esta Tese apresenta uma pesquisa realizada sobre métodos de extração de representações compreensíveis do conhecimento armazenado nas estruturas internas dessas técnicas não-simbólicas, aqui chamadas de caixa preta, durante seu processo de aprendizado. A principal contribuição desse trabalho é a proposta de um novo método pedagógico para extração de regras que expliquem o processo de classificação seguido por técnicas não-simbólicas. Esse novo método é baseado na otimização (maximização) da similaridade entre rankings de classificação produzidos por técnicas de Aprendizado de Máquina simbólicas e não simbólicas (de onde o conhecimento interno esta sendo extraído). Experimentos foram realizados com vários conjuntos de dados e os resultados obtidos sugerem um bom potencial para o método proposto / Non-symbolic Machine Learning techniques, like Artificial Neural Networks, Support Vector Machines and Ensembles of classifiers have shown a good performance when they are used in data analysis. The strong limitation regarding the use of these techniques is the lack of comprehensibility of the knowledge stored in their internal structure. This Thesis presents an investigation of methods capable of extracting comprehensible representations of the knowledge acquired by these non-symbolic techniques, here named black box, during their learning process. The main contribution of this work is the proposal of a new pedagogical method for rule extraction that explains the classification process followed by non-symbolic techniques. This new method is based on the optimization (maximization) of the similarity between classification rankings produced by symbolic and non-symbolic (from where the internal knowledge is being extracted) Machine Learning techniques. Experiments were performed for several datasets and the results obtained suggest a good potential of the proposed method

Made available in DSpace on 2015-03-05T13:53:43Z (GMT). No. of bitstreams: 0 Previous issue date: 9 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / Extração de Regras de Redes Neurais Artificiais Aplicadas ao Problema da Previsão da Estrutura Secundária de Proteínas apresenta o estudo feito sobre a extração de conhecimento de Redes Neurais na forma de regras difusas. Na aplicação desta técnica, foi utilizado o problema da classificação da estrutura secundária de proteínas, em alfa, beta e coil, a partir da estrutura primária. Serão apresentadas as implementações feitas para viabilizar esta tarefa. Dentre elas: a implementação de recursos adicionais ao software de extração de regras; a definição de uma metodologia de extração de regras; a implementação desta metodologia; e a análise das regras extraídas. Dentre os recursos implementados no processo de extração, sra visto que o foco principal foi o de embasar o conhecimento extraído sobre um suporte estático e disponibilizar medidas complementares para sua avaliação. Na definição da metodologia, será visto que cuidados devem ser tomados na preparação da base de dados e na definição da estrutura da rede / This work presents a study about knowledge extraction from Neural Networks in the form of fuzzy rules. In the application of this technique, it was investigated the problem of classification of the protein secondary structure (alpha, beta and coil) from its primary structure. The implementations that make possible this task will be presented. Amongst them: the implementation of new features in the rule extraction software; the definition of a methodology for the rule extraction process; the implementation of this methodology; and the analysis of the rules. Amongst the implemented features in the rules extraction process, it will be noticed that the main point was to provide a statistical support for the knowledge extracted and to make available additional resources to measure this information. In the definition of this methodology, it will be seen that some considerations must be observed in the database preparation and in definition of the network structure. Observations that had been followed and pr

In this thesis, we discuss different techniques to bridge the gap between two different approaches to artificial intelligence: the symbolic and the connectionist paradigm. Both approaches have quite contrasting advantages and disadvantages. Research in the area of neural-symbolic integration aims at bridging the gap between them. Starting from a human readable logic program, we construct connectionist systems, which behave equivalently. Afterwards, those systems can be trained, and later the refined knowledge be extracted.

A new method is developed to determine a set of informative and refined interface assertions satisfied by functions that are represented by feed-forward neural networks. Neural networks have often been criticized for their low degree of comprehensibility.It is difficult to have confidence in software components if they have no clear and valid interface description. Precise and understandable interface assertions for a neural network based software component are required for safety critical applications and for theintegration into larger software systems. The interface assertions we are considering are of the form &quote if the input x of the neural network is in a region (alpha symbol) of the input space then the output f(x) of the neural network will be in the region (beta symbol) of the output space &quote and vice versa. We are interested in computing refined interface assertions, which can be viewed as the computation of the strongest pre- and postconditions a feed-forward neural network fulfills. Unions ofpolyhedra (polyhedra are the generalization of convex polygons in higher dimensional spaces) are well suited for describing arbitrary regions of higher dimensional vector spaces. Additionally, polyhedra are closed under affine transformations. Given a feed-forward neural network, our method produces an annotated neural network, where each layer is annotated with a set of valid linear inequality predicates. The main challenges for the computation of these assertions is to compute the solution of a non-linear optimization problem and the projection of a polyhedron onto a lower-dimensional subspace.

“Accent” is the pattern of speech pronunciation by which one can identify a person’s linguistic, social or cultural background. It is an important source of inter-speaker variability and a particular problem for automated speech recognition. The aim of the study was to investigate a new computational approach to accent classification which did not require phonemic segmentation or the identification of phonemes as input, and which could therefore be used as a simple, effective accent classifier. Through a series of structured experiments this study investigated the effectiveness of Support Vector Machines (SVMs) for speech accent classification using time-based units rather than linguistically-informed ones, and compared it to the accuracy of other machine learning methods, as well as the ability of humans to classify speech according to accent. A corpus of read-speech was collected in two accents of English (Arabic and “Indian”) and used as the main datasource for the experiments. Mel-frequency cepstral coefficients were extracted from the speech samples and combined into larger units of 10 to 150ms duration, which then formed the input data for the various machine learning systems. Support Vector Machines were found to classify the samples with up to 97.5% accuracy with very high precision and recall, using samples of between 1 and 4 seconds of speech. This compared favourably with a human listener study where subjects were able to distinguish between the two accent groups with an average of 92.5% accuracy in approximately 8 seconds. Repeating the SVM experiments on a different corpus resulted in a best classification accuracy of 84.6%. Experiments using a decision tree learner and a rule-based classifier on the original corpus gave a best accuracy of 95% but results over the range of conditions were much more variable than those using the SVM. Rule extraction was performed in order to help explain the results and better inform the design of the system. The new approach was therefore shown to be effective for accent classification, and a plan for its role within various other larger speech-related contexts was developed.

“Accent” is the pattern of speech pronunciation by which one can identify a person’s linguistic, social or cultural background. It is an important source of inter-speaker variability and a particular problem for automated speech recognition. The aim of the study was to investigate a new computational approach to accent classification which did not require phonemic segmentation or the identification of phonemes as input, and which could therefore be used as a simple, effective accent classifier. Through a series of structured experiments this study investigated the effectiveness of Support Vector Machines (SVMs) for speech accent classification using time-based units rather than linguistically-informed ones, and compared it to the accuracy of other machine learning methods, as well as the ability of humans to classify speech according to accent. A corpus of read-speech was collected in two accents of English (Arabic and “Indian”) and used as the main datasource for the experiments. Mel-frequency cepstral coefficients were extracted from the speech samples and combined into larger units of 10 to 150ms duration, which then formed the input data for the various machine learning systems. Support Vector Machines were found to classify the samples with up to 97.5% accuracy with very high precision and recall, using samples of between 1 and 4 seconds of speech. This compared favourably with a human listener study where subjects were able to distinguish between the two accent groups with an average of 92.5% accuracy in approximately 8 seconds. Repeating the SVM experiments on a different corpus resulted in a best classification accuracy of 84.6%. Experiments using a decision tree learner and a rule-based classifier on the original corpus gave a best accuracy of 95% but results over the range of conditions were much more variable than those using the SVM. Rule extraction was performed in order to help explain the results and better inform the design of the system. The new approach was therefore shown to be effective for accent classification, and a plan for its role within various other larger speech-related contexts was developed.