The sum-product network (SPN) is a recently-proposed generative, probabilistic model that is guaranteed to compute any joint or any marginal probability in time linear in the size of the model. An SPN is represented as a directed, acyclic graph (DAG) of sum and product nodes, with univariate probability distributions at the leaves. It is important to learn the structure of this DAG since the set of distributions representable by an SPN is constrained by it. We present the first batch structure learning algorithm for SPNs and show its advantage over learning the parameters of an SPN with fixed architecture. We propose a search algorithm for learning the structure of an SPN and show that its ability to learn a DAG-structured SPN makes it better for some tasks than algorithms that only learn tree-structured SPNs. We adapt the structure search algorithm to learn the structure of an SPN in the online setting and show that two other methods for online SPN structure learning are slower or learn models with lower likelihood. We also propose to combine SPNs with an autoencoder to model image data; this application of SPN structure learning shows that both models benefit from being combined.We are also the first to propose a distinction between nonmonotone and monotone SPNs, or SPNs with negative edge-weights and those without, respectively. We prove several important properties of nonmonotone SPNs, propose algorithms for learning a special class of nonmonotone SPN called the twin SPNs, and show that allowing negative edge-weights can help twin SPNs model some distributions more compactly than monotone SPNs.

Thèse numérisée par la Direction des bibliothèques de l'Université de Montréal.

O método do Gradiente Espectral, introduzido por Barzilai e Borwein e analisado por Raydan, para minimização irrestrita, é um método simples cujo desempenho é comparável ao de métodos tradicionais como, por exemplo, gradientes conjugados. Desde a introdução do método, assim como da sua extensão para minimização em conjuntos convexos, foram introduzidas várias combinações de passos espectrais diferentes, assim como de buscas lineares não monótonas diferentes. Dos resultados numéricos apresentados em vários trabalhos não é possível inferir se existem diferenças significativas no desempenho dos diversos métodos. Além disso, também não fica clara a relevância das buscas não monótonas como uma ferramenta em si próprias ou se, na verdade, elas são úteis apenas para permitir que o método seja o mais parecido possível com o método original de Barzilai e Borwein. O objetivo deste trabalho é comparar os diversos métodos recentemente introduzidos como combinações de diferentes buscas lineares não monótonas e diferentes passos espectrais para encontrar a melhor combinação e, a partir daí, aferir o desempenho numérico do método. / The Spectral Gradient method, introduced by Barzilai and Borwein and analized by Raydan for unconstrained minimization, is a simple method whose performance is comparable to traditional methods, such as conjugate gradients. Since the introduction of method, as well as its extension to minimization of convex sets, there were introduced various combinations of different spectral steplengths, as well as different nonmonotone line searches. By the numerical results presented in many studies it is not possible to infer whether there are siginificant differences in the performance of various methods. It also is not sure the relevance of the nonmonotone line searches as a tool in themselves or whether, in fact, they are usefull only to allow the method to be as similar as possible with the original method of Barzilai e Borwein. The objective of this study is to compare the different methods recently introduced as different combinations of nonmonotone linear searches and different spectral steplengths to find the best combination and from there, evaluating the numerical performance of the method.

Orientadores: Márcia Aparecida Gomes Ruggiero, Sandra Augusta Santos / Tese (doutorado) - Universidade Estadual de Campinas, Instituto de Matemática, Estatística e Computação Científica / Made available in DSpace on 2018-08-21T10:21:10Z (GMT). No. of bitstreams: 1 Begiato_RodolfoGotardi_D.pdf: 3815627 bytes, checksum: 59584610cfd737a94e68dc5bf3735e25 (MD5) Previous issue date: 2012 / Resumo: O objetivo desta tese é tratar da resolução de sistemas não lineares de grande porte, em que as funções são continuamente diferenciáveis, por meio de uma abordagem híbrida que utiliza um método iterativo com duas fases. A primeira fase consiste de versões sem derivadas do método do ponto fixo empregando parâmetros espectrais para determinar o tamanho do passo da direção residual. A segunda fase é constituída pelo método de Newton inexato em uma abordagem matrix-free, em que é acoplado o método GMRES para resolver o sistema linear que determina a nova direção de busca. O método híbrido combina ordenadamente as duas fases de forma que a segunda é acionada somente em caso de falha na primeira e, em ambas, uma condição de decréscimo não-monótono deve ser verificada para aceitação de novos pontos. Desenvolvemos ainda um segundo método, em que uma terceira fase de busca direta é acionada em situações em que o excesso de buscas lineares faz com que o tamanho de passo na direção do método de Newton inexato torne-se demasiadamente pequeno. São estabelecidos os resultados de convergência dos métodos propostos. O desempenho computacional é avaliado em uma série de testes numéricos com problemas tradicionalmente encontrados na literatura. Tanto a análise teórica quanto a numérica evidenciam a viabilidade das abordagens apresentadas neste trabalho / Abstract: This thesis handles large-scale nonlinear systems for which all the involved functions are continuously differentiable. They are solved by means of a hybrid approach based on an iterative method with two phases. The first phase is defined by derivative-free versions of a fixed-point method that employs spectral parameters to define the steplength along the residual direction. The second phase consists of a matrix-free inexact Newton method that employs the GMRES to solve the linear system that computes the search direction. The proposed hybrid method neatly combines the two phases in such a way that the second is called only in case the first one fails. To accept new points in both phases, a nonmonotone decrease condition upon a merit function has to be verified. A second method is developed as well, with a third phase based on direct search, that should act whenever too many line searches have excessively decreased the steplenght along the inexact- Newton direction. Convergence results for the proposed methods are established. The computational performance is assessed in a set of numerical experiments with problems from the literature. Both the theoretical and the experimental analysis corroborate the feasibility of the proposed strategies / Doutorado / Matematica Aplicada / Doutor em Matemática Aplicada

Les phénomènes de contact entre les corps, déformables ou non, sont omniprésents dans la vie courante. Leurs modélisations requièrent des outils mathématiques faisant appel à des systèmes d'équations aux dérivées partielles incluant des conditions aux limites non triviales pour décrire le contact. Si les aspects physiques de la mécanique du contact sont connus depuis longtemps, la théorie mathématique qui lui est dédiée reste relativement récente laissant ainsi place à de nombreux problèmes à investiguer. Ce travail porte sur la modélisation, l'analyse et la simulation numérique de tels problèmes. Il se situe à mi-chemin entre la mécanique du contact et les aspects mathématiques inhérents au type de problème qui en découle. L'objectif est ici d'étudier certaines catégories de problèmes faisant intervenir des conditions originales de contact (avec et sans frottement) à la fois d'un point de vue mathématique et numérique, afin d'apporter une contribution à la théorie mathématique, puis de mettre en avant quelques méthodes numériques adaptées à leur résolution dans un cadre spécifique. / Contact phenomena between bodies, whether they are deformable or not, abound in everyday life. Their modellings require mathematical tools using systems of partial differential equations and involving complex boundary conditions, in order to describe the contact. While the physical aspects of such phenomena have been known for a long time, the mathematical theory remains relatively recent which leaves room for numerous problems. This work focuses on the modelling, the analysis and the numerical simulations of such problems. It is located halfway between contact mechanics and the mathematical aspects inherent to the mechanical questions involved. Our aim is to study several groups of problems that include original contact conditions (with or without friction), both from a mathematical and numerical point of view, in order to contribute to the theory, and also to highlight several numerical methods used to solve specific contact problems.

O método do Gradiente Espectral, introduzido por Barzilai e Borwein e analisado por Raydan, para minimização irrestrita, é um método simples cujo desempenho é comparável ao de métodos tradicionais como, por exemplo, gradientes conjugados. Desde a introdução do método, assim como da sua extensão para minimização em conjuntos convexos, foram introduzidas várias combinações de passos espectrais diferentes, assim como de buscas lineares não monótonas diferentes. Dos resultados numéricos apresentados em vários trabalhos não é possível inferir se existem diferenças significativas no desempenho dos diversos métodos. Além disso, também não fica clara a relevância das buscas não monótonas como uma ferramenta em si próprias ou se, na verdade, elas são úteis apenas para permitir que o método seja o mais parecido possível com o método original de Barzilai e Borwein. O objetivo deste trabalho é comparar os diversos métodos recentemente introduzidos como combinações de diferentes buscas lineares não monótonas e diferentes passos espectrais para encontrar a melhor combinação e, a partir daí, aferir o desempenho numérico do método. / The Spectral Gradient method, introduced by Barzilai and Borwein and analized by Raydan for unconstrained minimization, is a simple method whose performance is comparable to traditional methods, such as conjugate gradients. Since the introduction of method, as well as its extension to minimization of convex sets, there were introduced various combinations of different spectral steplengths, as well as different nonmonotone line searches. By the numerical results presented in many studies it is not possible to infer whether there are siginificant differences in the performance of various methods. It also is not sure the relevance of the nonmonotone line searches as a tool in themselves or whether, in fact, they are usefull only to allow the method to be as similar as possible with the original method of Barzilai e Borwein. The objective of this study is to compare the different methods recently introduced as different combinations of nonmonotone linear searches and different spectral steplengths to find the best combination and from there, evaluating the numerical performance of the method.

<p>The problem under consideration is an unconstrained optimization&nbsp;problem with the objective function in the form of mathematical ex-pectation. The expectation is with respect to the random variable that represents the uncertainty. Therefore, the objective &nbsp;function is in fact deterministic. However, nding the analytical form of that objective function can be very dicult or even impossible. This is the reason why the sample average approximation is often used. In order to obtain reasonable good approximation of the objective function, we have to use relatively large sample size. We assume that the sample is generated at the beginning of the optimization process and therefore we can consider this sample average objective function as the deterministic one. However, applying some deterministic method on that sample average function from the start can be very costly. The number of evaluations of the function under expectation is a common way of measuring the cost of an algorithm. Therefore, methods that vary the sample size throughout the optimization process are developed. Most of them are trying to determine the optimal dynamics of increasing the sample size.</p><p>The main goal of this thesis is to develop the clas of methods that&nbsp;can decrease the cost of an algorithm by decreasing the number of&nbsp;function evaluations. The idea is to decrease the sample size whenever&nbsp;it seems to be reasonable - roughly speaking, we do not want to impose&nbsp;a large precision, i.e. a large sample size when we are far away from the&nbsp;solution we search for. The detailed description of the new methods&nbsp;<br />is presented in Chapter 4 together with the convergence analysis. It&nbsp;is shown that the approximate solution is of the same quality as the&nbsp;one obtained by dealing with the full sample from the start.</p><p>Another important characteristic of the methods that are proposed&nbsp;here is the line search technique which is used for obtaining the sub-sequent iterates. The idea is to nd a suitable direction and to search&nbsp;along it until we obtain a sucient decrease in the &nbsp;function value. The&nbsp;sucient decrease is determined throughout the line search rule. In&nbsp;Chapter 4, that rule is supposed to be monotone, i.e. we are imposing&nbsp;strict decrease of the function value. In order to decrease the cost of&nbsp;the algorithm even more and to enlarge the set of suitable search directions, we use nonmonotone line search rules in Chapter 5. Within that chapter, these rules are modied to t the variable sample size framework. Moreover, the conditions for the global convergence and the R-linear rate are presented.&nbsp;</p><p>In Chapter 6, numerical results are presented. The test problems&nbsp;are various - some of them are academic and some of them are real&nbsp;world problems. The academic problems are here to give us more&nbsp;insight into the behavior of the algorithms. On the other hand, data&nbsp;that comes from the real world problems are here to test the real&nbsp;applicability of the proposed algorithms. In the rst part of that&nbsp;chapter, the focus is on the variable sample size techniques. Different&nbsp;implementations of the proposed algorithm are compared to each other&nbsp;and to the other sample schemes as well. The second part is mostly&nbsp;devoted to the comparison of the various line search rules combined&nbsp;with dierent search directions in the variable sample size framework.&nbsp;The overall numerical results show that using the variable sample size&nbsp;can improve the performance of the algorithms signicantly, especially&nbsp;when the nonmonotone line search rules are used.</p><p>The rst chapter of this thesis provides the background material&nbsp;for the subsequent chapters. In Chapter 2, basics of the nonlinear&nbsp;optimization are presented and the focus is on the line search, while&nbsp;Chapter 3 deals with the stochastic framework. These chapters are&nbsp;here to provide the review of the relevant known results, while the&nbsp;rest of the thesis represents the original contribution.&nbsp;</p> / <p>U okviru ove teze posmatra se problem optimizacije bez ograničenja pri čcemu je funkcija cilja u formi matematičkog očekivanja. Očekivanje se odnosi na slučajnu promenljivu koja predstavlja neizvesnost. Zbog toga je funkcija cilja, u stvari, deterministička veličina. Ipak, odredjivanje analitičkog oblika te funkcije cilja može biti vrlo komplikovano pa čak i nemoguće. Zbog toga se za aproksimaciju često koristi uzoračko očcekivanje. Da bi se postigla dobra aproksimacija, obično je neophodan obiman uzorak. Ako pretpostavimo da se uzorak realizuje pre početka procesa optimizacije, možemo posmatrati uzoračko očekivanje kao determinističku funkciju. Medjutim, primena nekog od determinističkih metoda direktno na tu funkciju&nbsp; moze biti veoma skupa jer evaluacija funkcije pod ocekivanjem često predstavlja veliki tro&scaron;ak i uobičajeno je da se ukupan tro&scaron;ak optimizacije meri po broju izračcunavanja funkcije pod očekivanjem. Zbog toga su razvijeni metodi sa promenljivom veličinom uzorka. Većcina njih je bazirana na odredjivanju optimalne dinamike uvećanja uzorka.</p><p>Glavni cilj ove teze je razvoj algoritma koji, kroz smanjenje broja izračcunavanja funkcije, smanjuje ukupne tro&scaron;skove optimizacije. Ideja je da se veličina uzorka smanji kad god je to moguće. Grubo rečeno, izbegava se koriscenje velike preciznosti&nbsp; (velikog uzorka) kada smo daleko od re&scaron;senja. U čcetvrtom poglavlju ove teze opisana je nova klasa metoda i predstavljena je analiza konvergencije. Dokazano je da je aproksimacija re&scaron;enja koju dobijamo bar toliko dobra koliko i za metod koji radi sa celim uzorkom sve vreme.</p><p>Jo&scaron; jedna bitna karakteristika metoda koji su ovde razmatrani je primena linijskog pretražzivanja u cilju odredjivanja naredne iteracije. Osnovna ideja je da se nadje odgovarajući pravac i da se duž njega vr&scaron;si pretraga za dužzinom koraka koja će dovoljno smanjiti vrednost funkcije. Dovoljno smanjenje je odredjeno pravilom linijskog pretraživanja. U čcetvrtom poglavlju to pravilo je monotono &scaron;to znači da zahtevamo striktno smanjenje vrednosti funkcije. U cilju jos većeg smanjenja tro&scaron;kova optimizacije kao i pro&scaron;irenja skupa pogodnih pravaca, u petom poglavlju koristimo nemonotona pravila linijskog pretraživanja koja su modifikovana zbog promenljive velicine uzorka. Takodje, razmatrani su uslovi za globalnu konvergenciju i R-linearnu brzinu konvergencije.</p><p>Numerički rezultati su predstavljeni u &scaron;estom poglavlju. Test problemi su razliciti - neki od njih su akademski, a neki su realni. Akademski problemi su tu da nam daju bolji uvid u pona&scaron;anje algoritama. Sa druge strane, podaci koji poticu od stvarnih problema služe kao pravi test za primenljivost pomenutih algoritama. U prvom delu tog poglavlja akcenat je na načinu ažuriranja veličine uzorka. Različite varijante metoda koji su ovde predloženi porede se medjusobno kao i sa drugim &scaron;emama za ažuriranje veličine uzorka. Drugi deo poglavlja pretežno je posvećen poredjenju različitih pravila linijskog pretraživanja sa različitim pravcima pretraživanja u okviru promenljive veličine uzorka. Uzimajuci sve postignute rezultate u obzir dolazi se do zaključcka da variranje veličine uzorka može značajno popraviti učinak algoritma, posebno ako se koriste nemonotone metode linijskog pretraživanja.</p><p>U prvom poglavlju ove teze opisana je motivacija kao i osnovni pojmovi potrebni za praćenje preostalih poglavlja. U drugom poglavlju je iznet pregled osnova nelinearne optimizacije sa akcentom na metode linijskog pretraživanja, dok su u trećem poglavlju predstavljene osnove stohastičke optimizacije. Pomenuta poglavlja su tu radi pregleda dosada&scaron;njih relevantnih rezultata dok je originalni doprinos ove teze predstavljen u poglavljima 4-6.</p>

<p>Algorithmic trading is an automated process of order execution on electronic stock markets. It can be applied to a broad range of financial instruments, and it is&nbsp; characterized by a signicant investors&#39; control over the execution of his/her orders, with the principal goal of finding the right balance between costs and risk of not (fully) executing an order. As the measurement of execution performance gives information whether best execution is achieved, a signicant number of diffeerent benchmarks is&nbsp; used in practice. The most frequently used are price benchmarks, where some of them are determined before trading (Pre-trade benchmarks), some during the trading&nbsp; day (In-traday benchmarks), and some are determined after the trade (Post-trade benchmarks). The two most dominant are VWAP and Arrival Price, which is along with other pre-trade price benchmarks known as the Implementation Shortfall (IS).</p><p>We introduce Negative Selection as a posteriori measure of the execution algorithm performance. It is based on the concept of Optimal Placement, which represents the ideal order that could be executed in a given time win-dow, where the notion of ideal means that it is an order with the best execution price considering&nbsp; market &nbsp;conditions&nbsp; during the time window. Negative Selection is dened as a difference between vectors of optimal and executed orders, with vectors dened as a quantity of shares at specied price positionsin the order book. It is equal to zero when the order is optimally executed; negative if the order is not (completely) filled, and positive if the order is executed but at an unfavorable price.</p><p>Negative Selection is based on the idea to offer a new, alternative performance measure, which will enable us to find the&nbsp; optimal trajectories and construct optimal execution of an order.</p><p>The first chapter of the thesis includes a list of notation and an overview of denitions and theorems that will be used further in the thesis. Chapters 2 and 3 follow with a&nbsp; theoretical overview of concepts related to market microstructure, basic information regarding benchmarks, and theoretical background of algorithmic trading. Original results are presented in chapters 4 and 5. Chapter 4 includes a construction of optimal placement, definition and properties of Negative Selection. The results regarding the properties of a Negative Selection are given in [35]. Chapter 5 contains the theoretical background for stochastic optimization, a model of the optimal execution formulated as a stochastic optimization problem with regard to Negative Selection, as well as original work on nonmonotone line search method [31], while numerical results are in the last, 6th chapter.</p> / <p>Algoritamsko trgovanje je automatizovani proces izvr&scaron;avanja naloga na elektronskim berzama. Može se primeniti na &scaron;irok spektar nansijskih instrumenata kojima se trguje na berzi i karakteri&scaron;e ga značajna kontrola investitora nad izvr&scaron;avanjem njegovih naloga, pri čemu se teži nalaženju pravog balansa izmedu tro&scaron;ka i rizika u vezi sa izvr&scaron;enjem naloga. S ozirom da se merenjem performasi izvr&scaron;enja naloga određuje da li je postignuto najbolje izvr&scaron;enje, u praksi postoji značajan broj različitih pokazatelja. Najče&scaron;će su to pokazatelji cena, neki od njih se određuju pre trgovanja (eng. Pre-trade), neki u toku trgovanja (eng. Intraday), a neki nakon trgovanja (eng. Post-trade). Dva najdominantnija pokazatelja cena su VWAP i Arrival Price koji je zajedno sa ostalim &quot;pre-trade&quot; pokazateljima cena poznat kao Implementation shortfall (IS).</p><p>Pojam negative selekcije se uvodi kao &quot;post-trade&quot; mera performansi algoritama izvr&scaron;enja, polazeći od pojma optimalnog naloga, koji predstavlja idealni nalog koji se&nbsp; mogao izvrsiti u datom vremenskom intervalu, pri ćemu se pod pojmom &quot;idealni&quot; podrazumeva nalog kojim se postiže najbolja cena u trži&scaron;nim uslovima koji su vladali&nbsp; u toku tog vremenskog intervala. Negativna selekcija se defini&scaron;e kao razlika vektora optimalnog i izvr&scaron;enog naloga, pri čemu su vektori naloga defisani kao količine akcija na odgovarajućim pozicijama cena knjige naloga. Ona je jednaka nuli kada je nalog optimalno izvr&scaron;en; negativna, ako nalog nije (u potpunosti) izvr&scaron;en, a pozitivna ako je nalog izvr&scaron;en, ali po nepovoljnoj ceni.</p><p>Uvođenje mere negativne selekcije zasnovano je na ideji da se ponudi nova, alternativna, mera performansi i da se u odnosu na nju nađe optimalna trajektorija i konstrui&scaron;e optimalno izvr&scaron;enje naloga.</p><p>U prvom poglavlju teze dati su lista notacija kao i pregled definicija i teorema&nbsp; neophodnih za izlaganje materije. Poglavlja 2 i 3 bave se teorijskim pregledom pojmova i literature u vezi sa mikrostrukturom trži&scaron;ta, pokazateljima trgovanja i algoritamskim trgovanjem. Originalni rezultati su predstavljeni u 4. i 5. poglavlju. Poglavlje 4 sadrži konstrukciju optimalnog naloga, definiciju i osobine negativne selekcije. Teorijski i praktični rezultati u vezi sa osobinama negativna selekcije dati su u [35]. Poglavlje 5 sadrži teorijske osnove stohastičke optimizacije, definiciju modela za optimalno izvr&scaron;enje, kao i originalni rad u vezi sa metodom nemonotonog linijskog pretraživanja [31], dok 6. poglavlje sadrži empirijske rezultate.</p>

Η παρούσα διατριβή ασχολείται με την μελέτη και την εκπαίδευση Τεχνητών Νευρωνικών Δικτύων (ΤΝΔ) με μεθόδους Βελτιστοποίησης και τις εφαρμογές αυτών. Η παρουσίαση των επιμέρους θεμάτων και αποτελεσμάτων της διατριβής αυτής οργανώνεται ως εξής : Στο κεφάλαιο 1 παρέχουμε τους βασικούς ορισμούς και περιγράφουμε τη δομή και τη λειτουργία των ΤΝΔ. Στη συνέχεια, παρουσιάζουμε μια συντομή ιστορική αναδρομή, αναφέρουμε μερικά από τα πλεονεκτήματα της χρήσης των ΤΝΔ και συνοψίζουμε τους κύριους τομείς όπου τα ΤΝΔ εφαρμόζονται. Τέλος, περιγράφουμε τις βασικές κατηγορίες μεθόδων εκπαίδευσης. Το κεφάλαιο 2 αφιερώνεται στη μαθηματική θεμελίωση της εκπαίδευσης ΤΝΔ. Περιγράφουμε τη γνωστή μέθοδο της οπισθοδρομικής διάδοσης του σφάλματος (Backpropagation) και δίνουμε αποδείξεις σύγκλισης για μια κλάση μεθόδων εκπαίδευσης που χρησιμοποιούν μονοδιάστατες ελαχιστοποιήσεις. Στο τέλος του κεφαλαίου παρουσιάζουμε κάποια θεωρητικά αποτελέσματα σχετικά με την ικανότητα των ΤΝΔ να προσεγγίζουν άγνωστες συναρτήσεις. Στο κεφάλαιο 3 προτείνουμε μια νέα κλάση μεθόδων εκπαίδευσης ΤΝΔ και αποδεικνύουμε ότι αυτές έχουν την ιδιότητα της ευρείας σύγκλισης , δηλαδή συγκλίνουν σε ένα ελάχιστο της αντικειμενικής συνάρτησης σχεδόν από οποιαδήποτε αρχική συνθήκη. Τα αποτελέσματα μας δείχνουν ότι η προτεινόμενη τεχνική μπορεί να βελτιώσει οποιαδήποτε μέθοδο της κλάσης της οπισθοδρομικής διάδοσης του σφάλματος. Στο επόμενο κεφάλαιο παρουσιάζουμε τη γνωστή μέθοδο Quick-Prop και μελετάμε τις ιδιότητες σύγκλισής της. Με βάση το θεωρητικό αποτέλεσμα που προκύπτει, κατασκευάζουμε μια νέα τροποποίηση της μεθόδου Quick-Prop, που έχει την ιδιότητα της ευρείας σύγκλισης και βελτιώνει σημαντικά την κλασίκη Quick-Prop μέθοδο. Στα επόμενα δύο κεφάλαια μελετάμε την εκπαίδευση ΤΝΔ με μεθόδους ολικής Βελτιστοποίησης. Πιο συγκεκριμένα, στο Κεφάλαιο 5 προτείνουμε και μελετάμε διεξοδικά μια νέα κλάση μεθόδων που είναι ικανές να εκπαιδεύσουν ΤΝΔ με περιορισμένα ακέραια βάρη. Στη συνέχεια, επεκτείνουμε τις μεθόδους αυτές έτσι ώστε να υλοποιούνται σε παράλληλους υπολογιστές και να εκπαιδεύουν ΤΝΔ με χρήση συναρτήσεων κατωφλιών. Το κεφάλαιο 6 πραγματεύεται την εφαρμογή γνωστών μεθόδων όπως οι Γενετικοί Αλγόριθμοι, η μέθοδος της προσομοιωμένης ανόπτησης ( Simulated Annealing ) και η μέθοδος βελτιστοποίησης με σμήνος σωματιδίων (Particle Swarm Optimization) στην εκπαίδευση ΤΝΔ. Επίσης, παρουσιάζουμε νέους μετασχηματισμούς της αντικειμενικής συνάρτησης με σκοπό την σταδιακή εξάλειψη των τοπικών ελαχίστων της. Στο κεφάλαιο 7 κάνουμε μια σύντομη ανασκόπηση της στοχαστικής μεθόδου της πιο απότομης κλίσης (stochastic gradient descent) για την εκπαίδευση ΤΝΔ ανά πρότυπο εισόδου και προτείνουμε μια νέα τέτοια μέθοδο . Η νέα μέθοδος συγκρίνεται με άλλες γνωστές μεθόδους και τα πειράματά μας δείχνουν ότι υπερτερεί. Η παρουσίαση του ερευνητικού έργου για αυτή τη διατριβή ολοκληρώνεται με το Κεφάλαιο 8, όπου προτείνουμε και μελετάμε εκτενώς μη μονότονες μεθόδους εκπαίδευσης ΤΝΔ. Η τεχνική που προτείνουμε μπορεί να εφαρμοστεί σε κάθε μέθοδο της κλάσης της οπισθοδρομικής διάδοσης του σφάλματος με αποτέλεσμα η τροποποιημένη μέθοδος να έχει την ικανότητα , πολλές φορές, να αποφεύγει τοπικά ελάχιστα της αντικειμενικής συνάρτησης. Η παρουσίαση της διατριβής ολοκληρώνεται με το κεφάλαιο 9 και δύο Παραρτήματα. Το Κεφάλαιο 9 περιέχει τα γενικά συμπεράσματα της διατριβής. Στο παράρτημα Α παρουσιάζουμε συνοπτικά μερικά από τα προβλήματα εκπαίδευσης που εξετάσαμε στα προηγούμενα κεφάλαια και τέλος στο Παράρτημα Β δίνουμε την απόδειξη της μεθόδου της οπισθοδρομικής διάδοσης του σφάλματος. / -