Many sequential decision making problems under uncertainty arising in engineering, science and economics are often modelled as Markov Decision Processes (MDPs). In the setting of MDPs, the goal is to and a state dependent optimal sequence of actions that minimizes a certain long-term performance criterion. The standard dynamic programming approach to solve an MDP for the optimal decisions requires a complete model of the MDP and is computationally feasible only for small state-action MDPs. Reinforcement learning (RL) methods, on the other hand, are model-free simulation based approaches for solving MDPs. In many real world applications, one is often faced with MDPs that have large state-action spaces whose model is unknown, however, whose outcomes can be simulated. In order to solve such (large) MDPs, one either resorts to the technique of function approximation in conjunction with RL methods or develops application specific RL methods. A solution based on RL methods with function approximation comes with the associated problem of choosing the right features for approximation and a solution based on application specific RL methods primarily relies on utilizing the problem structure. In this thesis, we investigate the problem of choosing the right features for RL methods based on function approximation as well as develop novel RL algorithms that adaptively obtain best features for approximation. Subsequently, we also develop problem specie RL methods for applications arising in the areas of wireless sensor networks and road traffic control. In the first part of the thesis, we consider the problem of finding the best features for value function approximation in reinforcement learning for the long-run discounted cost objective. We quantify the error in the approximation for any given feature and the approximation parameter by the mean square Bellman error (MSBE) objective and develop an online algorithm to optimize MSBE. Subsequently, we propose the first online actor-critic scheme with adaptive bases to find a locally optimal (control) policy for an MDP under the weighted discounted cost objective. The actor performs gradient search in the space of policy parameters using simultaneous perturbation stochastic approximation (SPSA) gradient estimates. This gradient computation however requires estimates of the value function of the policy. The value function is approximated using a linear architecture and its estimate is obtained from the critic. The error in approximation of the value function, however, results in sub-optimal policies. Thus, we obtain the best features by performing a gradient descent on the Grassmannian of features to minimize a MSBE objective. We provide a proof of convergence of our control algorithm to a locally optimal policy and show numerical results illustrating the performance of our algorithm. In our next work, we develop an online actor-critic control algorithm with adaptive feature tuning for MDPs under the long-run average cost objective. In this setting, a gradient search in the policy parameters is performed using policy gradient estimates to improve the performance of the actor. The computation of the aforementioned gradient however requires estimates of the differential value function of the policy. In order to obtain good estimates of the differential value function, the critic adaptively tunes the features to obtain the best representation of the value function using gradient search in the Grassmannian of features. We prove that our actor-critic algorithm converges to a locally optimal policy. Experiments on two different MDP settings show performance improvements resulting from our feature adaptation scheme. In the second part of the thesis, we develop problem specific RL solution methods for the two aforementioned applications. In both the applications, the size of the state-action space in the formulated MDPs is large. However, by utilizing the problem structure we develop scalable RL algorithms. In the wireless sensor networks application, we develop RL algorithms to find optimal energy management policies (EMPs) for energy harvesting (EH) sensor nodes. First, we consider the case of a single EH sensor node and formulate the problem of finding an optimal EMP in the discounted cost MDP setting. We then propose two RL algorithms to maximize network performance. Through simulations, our algorithms are seen to outperform the algorithms in the literature. Our RL algorithms for the single EH sensor node do not scale when there are multiple sensor nodes. In our second work, we consider the problem of finding optimal energy sharing policies that maximize the network performance of a system comprising of multiple sensor nodes and a single energy harvesting (EH) source. We develop efficient energy sharing algorithms, namely Q-learning algorithm with exploration mechanisms based on the -greedy method as well as upper confidence bound (UCB). We extend these algorithms by incorporating state and action space aggregation to tackle state-action space explosion in the MDP. We also develop a cross entropy based method that incorporates policy parameterization in order to find near optimal energy sharing policies. Through numerical experiments, we show that our algorithms yield energy sharing policies that outperform the heuristic greedy method. In the context of road traffic control, optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users. This problem is hard to solve when simultaneously considering all the junctions in the road network. So, we propose a decentralized multi-agent reinforcement learning (MARL) algorithm for solving this problem by considering each junction in the road network as a separate agent (controller) to obtain dynamic TSC policies. We propose two approaches to minimize the average delay. In the first approach, each agent decides the signal duration of its phases in a round-robin (RR) manner using the multi-agent Q-learning algorithm. We show through simulations over VISSIM (microscopic traffic simulator) that our round-robin MARL algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm over two real road networks. In the second approach, instead of optimizing green light duration, each agent optimizes the order of the phase sequence. We then employ our MARL algorithms by suitably changing the state-action space and cost structure of the MDP. We show through simulations over VISSIM that our non-round robin MARL algorithms perform significantly better than the FST, SAT and the round-robin MARL algorithms based on the first approach. However, on the other hand, our round-robin MARL algorithms are more practically viable as they conform with the psychology of road users.

No description available.

Ce mémoire de maîtrise porte sur les algorithmes de « data mining » et de « machine learning », constitutifs d’un domaine que l’on appelle plus récemment la « science des données ». Pour essayer d’éclairer la portée et la spécificité des enjeux que leur usage soulève dans nos sociétés, il est proposé d’interroger le rapport qu’ils entretiennent avec les fondements et les limites des outils plus traditionnels de la statistique sociale/mathématique, bien documentés en sociologie, à l'égard notamment du « langage des variables » et du raisonnement expérimental « toutes choses égales par ailleurs ». En inscrivant l’approche au croisement de la sociologie de la connaissance et de la quantification, le cadre conceptuel s’inspire de l’épistémologie comparative de Gilles-Gaston Granger, de la « méta-épistémologie historique » de Ian Hacking et de la sociohistoire de la statistique sociale d’Alain Desrosières. Par l’idée de mises en forme algorithmique de la vie sociale, les algorithmes de calcul sont envisagés comme modes d’investigation, partiellement ou complètement automatisés, procédant à des mises en forme et en ordre plurielles et différenciées du social et de ses propriétés. À partir de données de Statistique Canada servant à étayer plus concrètement les formes de connaissances produites et les visées d’objets qu’elles délimitent en termes de possibilités et de contraintes d’expérience, la présente étude de cas entreprend d’examiner le clivage des méthodes « classiques » et « contemporaines » à l’intérieur du cadre supervisé de l’apprentissage. Pour ce faire, trois techniques/familles d’algorithmes sont comparées sous l’angle de leurs opérations d’analyse: 1) les méthodes de régression logistique, 2) les arbres de décision et 3) les forêts aléatoires. L’objectif de cette analyse sociologique théorique comme empirique est d’examiner comment ces approches opèrent certains modes de classification et facilitent ou défavorisent des représentations du monde et de l’individu. Le travail conduit plus généralement à ouvrir quelques pistes de réflexion quant aux rapports de compatibilité et d’incompatibilité des formes de raisonnement du style statistique et probabiliste avec certains états du développement de la sociologie. / This master's thesis focuses on data mining and machine learning algorithms, constituting a field more recently called “data science”. To try to shed light on the specificity of the issues they raise in our societies, it is proposed to question the relationship they maintain with the foundations and the limits of the more “classic” tools of mathematical statistics in sociology, with regard in particular to the “language of variables” and to the experimental reasoning “all other things being equal” (cetaris paribus). By placing the approach at the intersection of the sociology of knowledge and quantification, the conceptual framework is inspired by the comparative epistemology of Gilles-Gaston Granger (1920-2016), the historical meta-epistemology of Ian Hacking (1936-) and the sociohistory of social statistics by Alain Desrosières (1940-2013). Through the idea of “mises en forme algorithmique de la vie sociale”, computational algorithms are considered as partially or completely automated types of investigation, carrying out plural and differentiated of shaping and ordering of the social and its properties. Using data from Statistics Canada used to more concretely support the forms of knowledge produced as well as the possibilities and experience constraints that they define, this case study sets out to examine the divide between “classical” and more “contemporary” methods of analysis within the framework of “supervised” learning. To do this, three algorithm techniques (or families of algorithms) are compared from the angle of their knowledge operations: 1) logistic regressions, 2) decision trees and 3) random forests. The objective of this theoretical as well as empirical work is to examine how these approaches operate certain modes of classification, facilitate or disadvantage representations of the world and can also be performative in social activities. The research work more generally leads to opening up some avenues of reflection as to the compatibility and incompatibility relationships of the forms of reasoning of the statistical and probabilistic style with certain states of development in society and in sociology.

Le trouble du spectre de l’autisme (TSA) est un trouble neurodéveloppemental caractérisé par des déficits importants de la communication sociale et des interactions sociales ainsi que par la présence de comportements ou d'intérêts restreints et répétitifs. Les données empiriques suggèrent que les interventions découlant de l’analyse appliquée du comportement (AAC) sont les plus efficaces pour intervenir auprès des personnes ayant un TSA. Néanmoins, certaines lacunes en lien avec les interventions découlant de l’analyse du comportement existent. Notamment, le manque d’accessibilité aux services, le manque de connaissances quant aux facteurs sous-jacents à l’efficacité des interventions et les perceptions divergentes de l’AAC freinent son adoption à plus grande échelle. Cette thèse comprend trois études qui mettent à profit la technologie pour mieux comprendre ou améliorer ces enjeux entourant l’AAC. Dans le cadre ma première étude, les effets d’une formation interactive en ligne qui vise à enseigner aux parents des stratégies découlant de l’AAC pour réduire les comportements problématiques de leur enfant ont été évalués à l’aide d’un devis randomisé contrôlé avec liste d’attente. Les résultats de cette étude soutiennent le potentiel et l’efficacité de la formation pour augmenter la fréquence d’utilisation de stratégies d’intervention découlant de l’AAC par les parents ainsi que pour réduire l’occurrence et la sévérité des comportements problématiques de leur enfant. En revanche, aucune différence significative n’a été observée pour la mesure des pratiques parentales. Certains enjeux éthiques et pratiques entourant la dissémination de la formation en ligne complètement auto-guidées sont discutés. La deuxième étude de ma thèse doctorale visait donc à montrer comment utiliser des algorithmes d’apprentissage automatique pour identifier les personnes qui sont plus enclines à observer des améliorations suivant une intervention. Plus spécifiquement, l’utilisation de quatre algorithmes d’apprentissage automatique pour prédire les participants ayant pris part à la première étude de cette thèse qui étaient les plus propices à rapporter une diminution des comportements problématiques de leur enfant est démontrée. Cette étude soutient que des algorithmes d’apprentissage automatique peuvent être utilisés avec de petits échantillons pour soutenir la prise de décision des cliniciens et des chercheurs. La troisième étude cette thèse visait à quantifier l’information sur l’AAC publiée dans quatre sous-forums d’un forum internet, une ressource en ligne souvent utilisée par les familles pour identifier des interventions à utiliser après de leur enfant. Pour atteindre cet objectif, une procédure de forage de données a été réalisée. Les analyses de cette étude appuient que les parents qui fréquentent le forum sont exposés à une proportion importante de messages présentant une désapprobation de l’AAC pour intervenir auprès des personnes ayant un TSA ou bien une description inexacte des principes, méthodes, procédures ou interventions qui en découlent. Ensemble, les études effectuées dans le cadre de ma thèse doctorale mettent en évidence les bienfaits de la technologie pour l’intervention psychosociale, tant au niveau de l’évaluation que de l’intervention et du transfert de connaissances. Comme souligné dans les trois études de cette thèse, chacun des outils utilisés présente des limites et doit donc être utilisé pour soutenir les cliniciens et les chercheurs, et non pour remplacer leurs interventions et leur jugement clinique. Les études futures doivent continuer à s’intéresser à l’efficacité des outils technologiques, mais également aux facteurs sous-jacents qui favoriseront leur utilisation et aux considérations éthiques liées à leur emploi. / Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by significant deficits in social communication and social interactions and by the presence of restricted and repetitive behaviors or interests. Empirical evidence suggests that interventions based on applied behavior analysis (ABA) are the most effective for treating individuals with ASD. Nevertheless, interventions based on behavior analysis present some issues. In particular, intervention services are hard to access, knowledge about the underlying factors of the effectiveness of interventions is lacking and divergent perceptions about of ABA hamper the adoption of the science. This dissertation includes three studies in which technology is used to better understand or improve these issues regarding ABA. As part of my first study, the effects of a fully self-guided interactive web training (IWT) developed for teaching parents of children with ASD ABA-derived strategies to reduce their child's challenging behaviors were evaluated using a randomized waitlist trial. The results of this study support the effectiveness of the IWT for increasing the frequency of parents’ use of behavioral interventions as well as for reducing the frequency and severity of their child’s challenging behaviors. In contrast, no significant difference was observed for the measurement of parenting practices. Ethical and practical consideration regarding the dissemination of fully self-guided online trainings are discussed. The second study of my doctoral thesis aimed to show how to use machine learning algorithms to predict individuals who were most likely to improve following an intervention. Specifically, a demonstration of how to implement four machine learning algorithms to predict the participants from my first study who were the most likely to report a decrease in their child's iv challenging behaviors. This study argues that machine learning algorithms can be used with small samples to support clinicians’ and researchers’ decision making. The third study of my dissertation aimed to quantify the information about ABA published on four subforums of an internet forum; an online resource often used by families to identify potential interventions for their child. This goal was achieved through the use of a data mining procedure. The analyses showed that parents who visited the forum were exposed to a significant proportion of messages that disapproved of ABA for individuals with ASD or that inaccurately described its underlying principles, methods, procedures, or interventions. Together, the studies carried out as part of my doctoral dissertation highlight the benefits of technology to support assessments, interventions, and knowledge gains or transfer within psychosocial practices. As highlighted in the three studies of this dissertation, each of the tools used presents limitations and should therefore be used to support clinicians and researchers, and should not replace their interventions and clinical judgment. Future studies should continue to focus on the effectiveness of technological tools and on the underlying factors that will promote their use. Finally, researchers must reflect on the ethical considerations related to use of technology when working with humans.

Unternehmen stehen zunehmend vor der Herausforderung, dem drohenden Wissensverlust durch demografischen Wandel und Mitarbeiterabgang zu begegnen. In Zeiten voranschreitender Digitalisierung gilt es, große Datenmengen beherrschbar und nutzbar zu machen, mit dem Ziel, einerseits die Ressourceneffizienz innerhalb des Unternehmens zu erhöhen und anderseits den Kunden zusätzliche Dienstleistungen anbieten zu können. Vor dem Hintergrund, ein effizientes Qualitätsmanagement und eine vorausschauende Instandhaltung mit ein und demselben System zu realisieren, sind zunächst technologische Kennzahlen und die Prozessführung zu bestimmen. Im Bereich der intelligenten Instandhaltung ist es jedoch nicht immer möglich, Fehlerzustände von physischen Anlagen im Serienbetrieb als Datensatz abzufassen. Das bewusste Zulassen von Fehlern unter realen Produktionsbedingungen könnte zu fatalen Ausfällen bis hin zur Zerstörung der Anlage führen. Auch das gezielte Erzeugen von Fehlern unter stark kontrollierten Bedingungen kann zeitaufwendig, kostenintensiv oder sogar undurchführbar sein.

Companies are increasingly challenged by the impending loss of knowledge due to demographic change and employee loss. In times of advancing digitalization, it is important to make large datasets accessible and usable, aiming at increasing resource efficiency within the company on the one hand and being able to offer customers additional services on the other. Given the background of implementing efficient quality management and predictive maintenance with the same system, technological key figures and process control must first be determined. In the field of intelligent maintenance, however, it is not always possible to record error states of physical systems in series operation as a data set. Deliberately allowing faults to occur under real production conditions could lead to fatal failures or even the destruction of the system. The targeted generation of faults under highly controlled conditions can also be timeconsuming, cost-intensive, or even impractical.

[ES] Los altos costes energéticos, las constantes medidas regulatorias aplicadas por las administraciones para mantener bajos los costes sanitarios, así como los cambios en la normativa sanitaria que se han introducido en los últimos años, han tenido un impacto significativo en la industria farmacéutica y sanitaria. El paradigma Industria 4.0 engloba cambios en el modelo productivo tradicional de la industria farmacéutica con la inclusión de tecnologías que van más allá de la automatización tradicional. El objetivo principal es lograr medicamentos más rentables mediante la incorporación óptima de tecnologías como la analítica avanzada. El proceso de fabricación de las industrias farmacéuticas tiene diferentes etapas (mezclado, secado, compactado, recubrimiento, envasado, etc.) donde una de las etapas más costosas energéticamente es el proceso de secado. El objetivo durante este proceso es extraer el contenido de líquidos como el agua mediante la inyección de aire caliente y seco en el sistema. Este tiempo de secado normalmente está predeterminado y depende del volumen y el tipo de unidades de producto farmacéutico que se deben deshidratar. Por otro lado, la fase de precalentamiento puede variar dependiendo de varios parámetros como la experiencia del operador. Por lo tanto, es posible asumir que una optimización de este proceso a través de analítica avanzada es posible y puede tener un efecto significativo en la reducción de costes en todo el proceso de fabricación. Debido al alto coste de la maquinaria involucrada en el proceso de producción de medicamentos, es una práctica común en la industria farmacéutica tratar de maximizar la vida útil de estas máquinas que no están equipados con los últimos sensores. Así pues, es posible implementar un modelo de aprendizaje automático que utilice plataformas de analítica avanzada, como la computación en la nube, para analizar los posibles ahorros en el consumo de energía. Esta tesis está enfocada en mejorar el consumo de energía en el proceso de precalentamiento de un secador de lecho fluido, mediante la definición e implementación de una plataforma de computación en la nube IIOT (Industrial Internet of Things)-Cloud, para alojar y ejecutar un algoritmo de aprendizaje automático basado en el modelo Catboost, para predecir cuándo es el momento óptimo para detener el proceso y reducir su duración y, en consecuencia, su consumo energético. Los resultados experimentales muestran que es posible reducir el proceso de precalentamiento en un 45% de su duración en tiempo y, en consecuencia, reducir el consumo de energía hasta 2.8 MWh por año. / [CAT] Els elevats costos energètics, les constants mesures reguladores aplicades per les administracions per mantenir uns costos assistencials baixos, així com els canvis en la normativa sanitària que s'han introduït en els darrers anys, han tingut un impacte important en el sector farmacèutic i sanitari. El paradigma de la indústria 4.0 engloba els canvis en el model de producció tradicional de la indústria farmacèutica amb la inclusió de tecnologies que van més enllà de l'automatització tradicional. L'objectiu principal és aconseguir fàrmacs més rendibles mitjançant la incorporació òptima de tecnologies com l'analítica avançada. El procés de fabricació de les indústries farmacèutiques té diferents etapes (mescla, assecat, compactació, recobriment, envasat, etc.) on una de les etapes més costoses energèticament és el procés d'assecat. L'objectiu d'aquest procés és extreure el contingut de líquids com l'aigua injectant aire calent i sec al sistema. Aquest temps de procediment d'assecat normalment està predeterminat i depèn del volum i del tipus d'unitats de producte farmacèutic que cal deshidratar. D'altra banda, la fase de preescalfament pot variar en funció de diversos paràmetres com l'experiència de l'operador. Per tant, podem assumir que una optimització d'aquest procés mitjançant analítiques avançades és possible i pot tenir un efecte significatiu de reducció de costos en tot el procés de fabricació. A causa de l'elevat cost de la maquinària implicada en el procés de producció de fàrmacs, és una pràctica habitual a la indústria farmacèutica intentar maximitzar la vida útil d'aquestes màquines que no estan equipats amb els darrers sensors. Així, es pot implementar un model d'aprenentatge automàtic que utilitza plataformes de analítiques avançades com la computació en núvol, per analitzar l'estalvi potencial del consum d'energia. Aquesta tesis està enfocada a millorar el consum d'energia en el procés de preescalfament d'un assecador de llit fluid, mitjançant la definició i implementació d'una plataforma IIOT (Industrial Internet of Things)-Cloud computing, per allotjar i executar un algorisme d'aprenentatge automàtic basat en el modelatge Catboost, per predir quan és el moment òptim per aturar el procés i reduir-ne la durada, i en conseqüència el seu consum energètic. Els resultats de l'experiment mostren que és possible reduir el procés de preescalfament en un 45% de la seva durada en temps i, en conseqüència, reduir el consum d'energia fins a 2.8 MWh anuals. / [EN] High energy costs, the constant regulatory measures applied by administrations to maintain low healthcare costs, and the changes in healthcare regulations introduced in recent years have all significantly impacted the pharmaceutical and healthcare industry. The industry 4.0 paradigm encompasses changes in the traditional production model of the pharmaceutical industry with the inclusion of technologies beyond traditional automation. The primary goal is to achieve more cost-efficient drugs through the optimal incorporation of technologies such as advanced analytics. The manufacturing process of the pharmaceutical industry has different stages (mixing, drying, compacting, coating, packaging, etc..), and one of the most energy-expensive stages is the drying process. This process aims to extract the liquid content, such as water, by injecting warm and dry air into the system. This drying procedure time usually is predetermined and depends on the volume and the kind of units of a pharmaceutical product that must be dehydrated. On the other hand, the preheating phase can vary depending on various parameters, such as the operator's experience. It is, therefore, safe to assume that optimization of this process through advanced analytics is possible and can have a significant cost-reducing effect on the whole manufacturing process. Due to the high cost of the machinery involved in the drug production process, it is common practice in the pharmaceutical industry to try to maximize the useful life of these machines, which are not equipped with the latest sensors. Thus, a machine learning model using advanced analytics platforms, such as cloud computing, can be implemented to analyze potential energy consumption savings. This thesis is focused on improving the energy consumption in the preheating process of a fluid bed dryer by defining and implementing an IIOT (Industrial Internet of Things) Cloud computing platform. This architecture will host and run a machine learning algorithm based on Catboost modeling to predict when the optimum time is reached to stop the process, reduce its duration, and consequently its energy consumption. Experimental results show that it is possible to reduce the preheating process by 45% of its time duration, consequently reducing energy consumption by up to 2.8 MWh per year. / Barriga Rodríguez, R. (2023). Optimization of Fluid Bed Dryer Energy Consumption for Pharmaceutical Drug Processes through Machine Learning and Cloud Computing Technologies [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/195847

This study investigates the performance of Logistic Regression, k-Nearest Neighbors (KNN), and Random Forest algorithms in predicting customer churn within an e-commerce platform. The choice of the mentioned algorithms was due to the unique characteristics of the dataset and the unique perception and value provided by each algorithm. Iterative models ‘examinations, encompassing preprocessing techniques, feature engineering, and rigorous evaluations, were conducted. Logistic Regression showcased moderate predictive capabilities but lagged in accurately identifying potential churners due to its assumptions of linearity between log odds and predictors. KNN emerged as the most accurate classifier, achieving superior sensitivity and specificity (98.22% and 96.35%, respectively), outperforming other models. Random Forest, with sensitivity and specificity (91.75% and 95.83% respectively) excelled in specificity but slightly lagged in sensitivity. Feature importance analysis highlighted "Tenure" as the most impactful variable for churn prediction. Preprocessing techniques differed in performance across models, emphasizing the importance of tailored preprocessing. The study's findings underscore the significance of continuous model refinement and optimization in addressing complex business challenges like customer churn. The insights serve as a foundation for businesses to implement targeted retention strategies, mitigating customer attrition, and promote growth in e-commerce platforms.

In Chapter 2, we propose several two-timescale simulation-based actor-critic algorithms for solution of infinite horizon Markov Decision Processes (MDPs) with finite state-space under the average cost criterion. On the slower timescale, all the algorithms perform a gradient search over corresponding policy spaces using two different Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates. On the faster timescale, the differential cost function corresponding to a given stationary policy is updated and averaged for enhanced performance. A proof of convergence to a locally optimal policy is presented. Next, a memory efficient implementation using a feature-vector representation of the state-space and TD (0) learning along the faster timescale is discussed. A three-timescale simulation based algorithm for solution of infinite horizon discounted-cost MDPs via the Value Iteration approach is also proposed. An approximation of the Dynamic Programming operator T is applied to the value function iterates. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are presented using the proposed algorithms. Next, in Chapter 3, we develop three simulation-based algorithms for finite-horizon MDPs (FHMDPs). The first algorithm is developed for finite state and compact action spaces while the other two are for finite state and finite action spaces. Convergence analysis is briefly sketched. We then concentrate on methods to mitigate the curse of dimensionality that affects FH-MDPs severely, as there is one probability transition matrix per stage. Two parametrized actor-critic algorithms for FHMDPs with compact action sets are proposed, the ‘critic’ in both algorithms learning the policy gradient. We show w.p1convergence to a set with the necessary condition for constrained optima. Further, a third algorithm for stochastic control of stopping time processes is presented. Numerical experiments with the proposed finite-horizon algorithms are shown for a problem of flow control in communication networks. Towards stochastic optimization, in Chapter 4, we propose five algorithms which are variants of SPSA. The original one measurement SPSA uses an estimate of the gradient of objective function L containing an additional bias term not seen in two-measurement SPSA. We propose a one-measurement algorithm that eliminates this bias, and has asymptotic convergence properties making for easier comparison with the two-measurement SPSA. The algorithm, under certain conditions, outperforms both forms of SPSA with the only overhead being the storage of a single measurement. We also propose a similar algorithm that uses perturbations obtained from normalized Hadamard matrices. The convergence w.p.1 of both algorithms is established. We extend measurement reuse to design three second-order SPSA algorithms, sketch the convergence analysis and present simulation results on an illustrative minimization problem. We then propose several stochastic approximation implementations for related algorithms in flow-control of communication networks, beginning with a discrete-time implementation of Kelly’s primal flow-control algorithm. Convergence with probability1 is shown, even in the presence of communication delays and stochastic effects seen in link congestion indications. Two relevant enhancements are then pursued :a) an implementation of the primal algorithm using second-order information, and b) an implementation where edge-routers rectify misbehaving flows. Also, discrete-time implementations of Kelly’s dual algorithm and primal-dual algorithm are proposed. Simulation results a) verifying the proposed algorithms and, b) comparing stability properties with an algorithm in the literature are presented.

In this thesis, our focus is on low-complexity, high-performance detection algorithms for multi-antenna communication receivers. A key contribution in this thesis is the demonstration that efficient algorithms from metaheuristics and machine learning can be gainfully adapted for signal detection in multi- antenna communication receivers. We first investigate a popular metaheuristic known as the reactive tabu search (RTS), a combinatorial optimization technique, to decode the transmitted signals in large-dimensional communication systems. A basic version of the RTS algorithm is shown to achieve near-optimal performance for 4-QAM in large dimensions. We then propose a method to obtain a lower bound on the BER performance of the optimal detector. This lower bound is tight at moderate to high SNRs and is useful in situations where the performance of optimal detector is needed for comparison, but cannot be obtained due to very high computational complexity. To improve the performance of the basic RTS algorithm for higher-order modulations, we propose variants of the basic RTS algorithm using layering and multiple explorations. These variants are shown to achieve near-optimal performance in higher-order QAM as well. Next, we propose a new receiver called linear regression of minimum mean square error (MMSE) residual receiver (referred to as LRR receiver). The proposed LRR receiver improves the MMSE receiver by learning a linear regression model for the error of the MMSE receiver. The LRR receiver uses pilot data to estimate the channel, and then uses locally generated training data (not transmitted over the channel) to find the linear regression parameters. The LRR receiver is suitable for applications where the channel remains constant for a long period (slow-fading channels) and performs well. Finally, we propose a receiver that uses a committee of linear receivers, whose parameters are estimated from training data using a variant of the AdaBoost algorithm, a celebrated supervised classification algorithm in ma- chine learning. We call our receiver boosted MMSE (B-MMSE) receiver. We demonstrate that the performance and complexity of the proposed B-MMSE receiver are quite attractive for multi-antenna communication receivers.