Machine learning (ML) has experienced immense growth in the last decade; applications of ML are found in nearly every sphere of society. We gather a vast amount of data during our day-to-day operations, and the machines use this data to make intelligent decisions for us. Transport is an essential part of our life. Since early times it has been a critical requirement of human beings, and as we progressed, its need has risen tremendously. Nowadays, we cannot think of a life without any means of transportation. A vehicle is the primary means of transportation in the modern industrial world. Most of the world's population uses a vehicle for transportation needs. The internal combustion engine (ICE) has been the vehicle's primary power source since the early days. Approximately 99% of world transport uses internal combustion engine-based vehicles. As with every other device, Internal combustion technology has seen enormous improvement during the past half a century. These gradual improvements have made these ICEs better than ever before. But as we have progressed, we have also been exposed to the harms of combustion-based power sources like ICEs. Their effect on the environment has been damaging. The future way is to limit the use of hydrocarbon-based fuels for combustion. As it's nearly impossible to eradicate the use of these fuels instantly as most of the population is deeply dependent on them, the best way to proceed is to find alternate transportation sources like EVs. But as these technologies are developing, it's of great importance; we use our modern ML-based technologies to make the current ICE-based vehicles as fuel-efficient as possible so that this transition to electric-based vehicles becomes smooth and the effect of ICEs is minimized on the environment. The thesis presents a deep reinforcement learning (DRL) based technique, which can develop an optimal policy to effectively reduce vehicle fuel consumption by providing intelligent driving inputs to the vehicle. The study shows that it is possible to use (DRL) methods to improve fuel efficiency in a simulated vehicular environment. Moreover, the models seem to develop new types of operational vehicle policies that are very unconventional but result in improved fuel efficiency in simulated environments. Although these cannot be implemented in the current form in the actual environment, modified versions of these policies may be able to work in real-life scenarios.

Dynamic grasping is the action of, using only contact force, manipulating the position of a moving object in space. Doing so with a robot is a quite complex task in itself, but is one with wide-ranging applications. Today, the work of automating most processes in society undergoes rapidly, and for many of these processes, the grasping of objects has a natural place. This work has explored using deep reinforcement learning techniques for dynamic grasping, in a simulated environment. Deep Deterministic Policy Gradient was chosen and evaluated for the task, both by itself and in combination with the Hindsight Experience Replay buffer. The reinforcement learning agent observed the initial state of the target object and the robot in the environment, simulated using AGX Dynamics, and then determined with what speed to move to which position. The agent's chosen action was relayed to ABB's virtual controller, which controlled the robot in the simulation. This meant that the agent was tasked with, in advance, parametrizing a predefined set of instructions to the robot, in such a way that the moving target object would be grasped and picked up. Doing it in this matter, as opposed to having the agent continuously control the robot, was a necessary challenge making it possible to utilize the intelligence already created for the virtual controller. It also means that transferring the things learned by an agent in a simulated environment to a real-world environment becomes easier. The accuracy of the target policy for the simpler agent was 99.07%, while the accuracy of the agent with the more advanced replay buffer came up to 99.30%. These results show promise for the future, both as we expect further fine-tuning to raise them even more, and as they indicate that deep reinforcement learning methods can be highly applicable to the robotics systems of today.

Deep Reinforcement Learning (DRL) is a new research area, which integrates deep neural networks into reinforcement learning algorithms. It is revolutionizing the field of AI with high performance in the traditional challenges, such as natural language processing, computer vision etc. The current deep reinforcement learning algorithms enable an end to end learning that utilizes deep neural networks to produce effective actions in complex environments from high dimensional sensory observations, such as raw images. The applications of deep reinforcement learning algorithms are remarkable. For example, the performance of trained agent playing Atari video games is comparable, or even superior to a human player. Current studies mostly focus on training single agent and its interaction with dynamic environments. However, in order to cope with complex real-world scenarios, it is necessary to look into multiple interacting agents and their collaborations on certain tasks. This thesis studies the state-of-the-art deep reinforcement learning algorithms and techniques. Through the experiments conducted in several 2D and 3D game scenarios, we investigate how DRL models can be adapted to train multiple agents cooperating with one another, by communications and physical navigations, and achieving their individual goals on complex tasks. / Djup förstärkningsinlärning (DRL) är en ny forskningsdomän som integrerar djupa neurala nätverk i inlärningsalgoritmer. Det har revolutionerat AI-fältet och skapat höga förväntningar på att lösa de traditionella problemen inom AI-forskningen. I detta examensarbete genomförs en grundlig studie av state-of-the-art inom DRL-algoritmer och DRL-tekniker. Genom experiment med flera 2D- och 3D-spelscenarion så undersöks hur agenter kan samarbeta med varandra och nå sina mål genom kommunikation och fysisk navigering.

<p>This thesis targets the orchestration challenge of the Wearable Internet of Things (IoT) systems, for optimal configurations of the system in terms of energy efficiency, computing, and  data transmission activities. We have firstly investigated the reinforcement learning on the  simulated IoT environments to demonstrate its effectiveness, and afterwards studied the algorithm  on the real-world wearable motion data to show the practical promise. More specifically, firstly,  challenge arises in the complex massive-device orchestration, meaning that it is essential to  configure and manage the massive devices and the gateway/server. The complexity on the massive  wearable IoT devices, lies in the diverse energy budget, computing efficiency, etc. On the phone  or server side, it lies in how global diversity can be analyzed and how the system configuration  can be optimized. We therefore propose a new reinforcement learning architecture, called boosted  deep deterministic policy gradient, with enhanced actor-critic co-learning and multi-view state?transformation. The proposed actor-critic co-learning allows for enhanced dynamics abstraction  through the shared neural network component. Evaluated on a simulated massive-device task, the proposed deep reinforcement learning framework has achieved much more efficient system  configurations with enhanced computing capabilities and improved energy efficiency. Secondly, we have leveraged the real-world motion data to demonstrate the potential of leveraging  reinforcement learning to optimally configure the motion sensors. We used paradigms in  sequential data estimation to obtain estimated data for some sensors, allowing energy savings since  these sensors no longer need to be activated to collect data for estimation intervals. We then  introduced the Deep Deterministic Policy Gradient algorithm to learn to control the estimation  timing. This study will provide a real-world demonstration of maximizing energy efficiency wearable IoT applications while maintaining data accuracy. Overall, this thesis will greatly  advance the wearable IoT system orchestration for optimal system configurations.   </p>

This research focuses on addressing two pertinent problems in machine learning (ML) which are (a) the supervised classification of time series and (b) the need for large amounts of labeled images for training supervised classifiers. The novel contributions are two-fold. The first problem of time series classification is addressed by proposing to transform time series into domain-specific 2D features such as scalograms and recurrence plot (RP) images. The second problem which is the need for large amounts of labeled image data, is tackled by proposing a new way of using a reinforcement learning (RL) technique as a supervised classifier by using multimodal (joint representation) scalograms and RP images. The motivation for using such domain-specific features is that they provide additional information to the ML models by capturing domain-specific features (patterns) and also help in taking advantage of state-of-the-art image classifiers for learning the patterns from these textured images. Thus, this research proposes a multimodal fusion (MMF) - deep reinforcement learning (DRL) approach as an alternative technique to traditional supervised image classifiers for the classification of time series. The proposed MMF-DRL approach produces improved accuracy over state-of-the-art supervised learning models while needing fewer training data. Results show the merit of using multiple modalities and RL in achieving improved performance than training on a single modality. Moreover, the proposed approach yields the highest accuracy of 90.20% and 89.63% respectively for two physiological time series datasets with fewer training data in contrast to the state-of-the-art supervised learning model ChronoNet which gave 87.62% and 88.02% accuracy respectively for the two datasets with more training data.

No description available.

Large-Scale Digital Twins Notification System (LSDTNS) monitors a Digital Twin (DT) cluster for a predefined critical state, and once it detects such a state, it sends a Notification Event (NE) to a predefined recipient. Additionally, the time from producing the DT's Complex Event (CE) to sending an alarm has to be less than a predefined deadline. However, addressing scalability and multi-objectives, such as deployment cost, resource utilization, and meeting the deadline, on top of process scheduling, presents a complex challenge. Therefore, this thesis presents a complex methodology consisting of three contributions that address system scalability, multi-objectivity and scheduling of CE processes using Reinforcement Learning (RL). The first contribution proposes the IoT Notification System Architecture based on a micro-service-based notification methodology that allows for running and seamlessly switching between various CE reasoning algorithms. Our proposed IoT Notification System architecture addresses the scalability issue in state-of-the-art CE Recognition systems. The second contribution proposes a novel methodology for multi-objective optimization for cloud provisioning (MOOP). MOOP is the first work dealing with multi-optimization objectives for microservice notification applications, where the notification load is variable and depends on the results of previous microservices subtasks. MOOP provides a multi-objective mathematical cloud resource deployment model and demonstrates effectiveness through the case study. Finally, the thesis presents a Scheduler for large-scale Critical Notification applications based on a Deep Reinforcement Learning (SCN-DRL) scheduling approach for LSDTNS using RL. SCN-DRL is the first work dealing with multi-objective optimization for critical microservice notification applications using RL. During the performance evaluation, SCN-DRL demonstrates better performance than state-of-the-art heuristics. SCN-DRL shows steady performance when the notification workload increases from 10% to 90%. In addition, SCN-DRL, tested with three neural networks, shows that it is resilient to sudden container resources drop by 10%. Such resilience to resource container failures is an important attribute of a distributed system.

This dissertation investigates potential applications for Machine Learning (ML) and real-time fouling monitors in Reverse Osmosis (RO) desalination. The main objective was to develop a framework that minimizes the cost of membrane fouling by deploying AI-generated cleaning patterns and real-time fouling monitoring. Membrane manufacturers and researchers typically recommend cleaning (standard operating procedure – SOP) when normalized permeate flow, a performance metric tracking the decline of permeate flow/output from its initial baseline with respect to operating pressure, reaches 0.85-0.90 of baseline values. This study used estimates of production cost, internal profitability metrics, and permeate volume output to evaluate and compare the impact of time selection for cleaning intervention. The cleanings initiated when the normalized permeate flow reached 0.85 represented the control for cleaning intervention times. In deciding optimal times for cleaning intervention, a Deep Reinforcement Learning (RL) agent was trained to signal cleaning between 0.85-0.90 normalized with a cost-based reward system. A laboratory-scale RO flat membrane desalination system platform was developed as a model plant, and data from the platform and used to train the model and examine both simulated and actual control of when to trigger membrane cleaning, replacing the control operator's 0.85 cleaning threshold. Compared to SOP, the intelligent operator showed consistent savings in production costs at the expense of total permeate volume output. The simulated operation using the RL initiated yielded 9% less permeate water but reduced the cost per unit volume ($/m3) by 12.3%. When the RL agent was used to initiate cleaning on the laboratory-scale RO desalination system platform, the system produced 21% less permeate water but reduced production cost ($/m3) by 16.0%. These results are consistent with an RL agent that prioritizes production cost savings over product volume output. / Doctor of Philosophy / The decreasing supply of freshwater sources has made desalination technology an attractive solution. Desalination—or the removal of salt from water—provides an opportunity to produce more freshwater by treating saline sources and recycled water. One prominent form of desalination is Reverse Osmosis (RO), an energy intensive process in which freshwater is forced from a pressurized feed through a semipermeable membrane. A significant limiting cost factor for RO desalination is the maintenance and replacement of semipermeable RO membranes. Over time, unwanted particles accumulate on the membrane surface in a process known as membrane fouling. Significant levels of fouling can drive up costs, negatively affect product quality (permeate water), and decrease the useful lifetime of the membrane. As a result, operators employ various fouling control techniques, such as membrane cleaning, to mitigate its effects on production and minimize damage to the membrane. This dissertation investigates potential applications for Machine Learning (ML) and real-time fouling monitors in Reverse Osmosis (RO) desalination. The main objective was to develop a framework that minimizes the cost of membrane fouling by deploying AI-generated cleaning patterns and real-time fouling monitoring. Membrane manufacturers and researchers typically recommend cleaning (standard operating procedure – SOP) when normalized permeate flow, a performance metric tracking the decline of permeate flow/output from its initial baseline with respect to operating pressure, reaches 0.85-0.90 of baseline values. This study used estimates of production cost, internal profitability metrics, and permeate volume output to evaluate and compare the impact of time selection for cleaning intervention. The cleanings initiated when the normalized permeate flow reached 0.85 represented the control for cleaning intervention times. In deciding optimal times for cleaning intervention, a Deep Reinforcement Learning (RL) agent was trained to signal cleaning between 0.85-0.90 normalized with a cost-based reward system. A laboratory-scale RO flat membrane desalination system platform was developed as a model plant, and data from the platform and used to train the model and examine both simulated and actual control of when to trigger membrane cleaning, replacing the control operator's 0.85 cleaning threshold. Compared to SOP, the intelligent operator showed consistent savings in production costs at the expense of total permeate volume output. The simulated operation using the RL initiated yielded 9% less permeate water but reduced the cost per unit volume ($/m3) by 12.3%. When the RL agent was used to initiate cleaning on the laboratory-scale RO desalination system platform, the system produced 21% less permeate water but reduced production cost ($/m3) by 16.0%. These results are consistent with an RL agent that prioritizes production cost savings over product volume output.

Tractography is a tool that is often used to study structural brain connectivity from diffusion magnetic resonance imaging data. Despite its ability to visualize fibers in the white brain matter, it results in a high number of invalid streamlines. For the sake of research and clinical applications, it is of great interest to reduce this number and so improve the quality of tractography. Over the years, many solutions have been proposed, often with a need for ground truth data. As such data for tractography is very difficult to generate even with expertise, it is meaningful to instead use methods like reinforcement learning that does not have such a requirement. In 2021 a deep reinforcement learning tractography network was published: Track-To-Learn. There is however still room for improvement in the reward function of the framework and this is what we focused on in this thesis. Firstly we successfully reproduced some of the published results of Track-To-Learn and observed that almost 20 % of the streamlines were anatomically plausible. Continuously we modified the reward function by giving a reward boost to streamlines which started or terminated within a specified mask. This addition resulted in a small increase of plausible streamlines for a more realistic dataset. Lastly we attempted to include anatomical filtering in the reward function. The produced results were however not enough to draw any valid conclusions about the influence of the modification. Nonetheless, the work of this thesis showed that including further fiber specific anatomical constraints in the reward function of Track-To-Learn could possibly improve the quality of the generated tractograms and would be of interest in both research and clinical settings.

Operation of Autonomous Mobile Robots (AMRs) of all forms that include wheeled ground vehicles, quadrupeds and humanoids in dynamically changing GPS denied environments without a-priori maps, exclusively using onboard sensors, is an unsolved problem that has potential to transform the economy, and vastly improve humanity's capabilities with improvements to agriculture, manufacturing, disaster response, military and space exploration. Conventional AMR automation approaches are modularized into perception, motion planning and control which is computationally inefficient, and requires explicit feature extraction and engineering, that inhibits generalization, and deployment at scale. Few works have focused on real-world end-to-end approaches that directly map sensor inputs to control outputs due to the large amount of well curated training data required for supervised Deep Learning (DL) which is time consuming and labor intensive to collect and label, and sample inefficiency and challenges to bridging the simulation to reality gap using Deep Reinforcement Learning (DRL). This dissertation presents a novel method to efficiently train DRL with significantly fewer samples in a constrained racetrack environment at physical limits in simulation, transferred zero-shot to the real-world for robust end-to-end AMR navigation. The representation learned in a compact parameter space with 2 fully connected layers with 64 nodes each is demonstrated to exhibit emergent behavior for Out-of-Distribution (OOD) generalization to navigation in new environments that include unstructured terrain without maps, dynamic obstacle avoidance, and navigation to objects of interest with vision input that encompass low light scenarios with the addition of a night vision camera. The learned policy outperforms conventional navigation algorithms while consuming a fraction of the computation resources, enabling execution on a range of AMR forms with varying embedded computer payloads. / Doctor of Philosophy / Robots with wheels or legs to move around environments improve humanity's capabilities in many applications such as agriculture, manufacturing, and space exploration. Reliable, robust mobile robots have the potential to significantly improve the economy. A key component of mobility is navigation to either explore the surrounding environment, or travel to a goal position or object of interest by avoiding stationary, and dynamic obstacles. This is a complex problem that has no reliable solution, which is one of the main reasons robots are not present everywhere, assisting people in various tasks. Past and current approaches involve first mapping an environment, then planning a collision-free path, and finally executing motor signals to traverse along the path. This has several limitations due to the lack of detailed pre-made maps, and inability to operate in previously unseen, dynamic environments. Furthermore, these modular methods require high computation resources due to the large number of calculations required for each step that prevents high real-time speed, and functionality in small robots with limited weight capacity for onboard computers, that are beneficial for reconnaissance, and exploration tasks. This dissertation presents a novel Artificial Intelligence (AI) method for robot navigation that is more computationally efficient than current approaches, with better performance. The AI model is trained to race in simulation at multiple times real-time speed for cost-effective, accelerated training, and transferred to a physical mobile robot where it retains its training experience, and generalizes to navigation in new environments without maps, with exploratory behavior, and dynamic obstacle avoidance capabilities.