We implement reinforcement learning in the context of wireless communications in two very different settings. In the first setting, we study the use of reinforcement learning in an underwater acoustic communications network to adapt its transmission frequencies to avoid interference and potential malicious jammers. To that effect, we implement a reinforcement learning algorithm called contextual bandits. The harsh environment of an underwater channel provides a challenging problem. The channel may induce multipath and time delays which lead to time-varying, frequency-selective attenuation. These factors are also influenced by the distance between the transmitter and receiver, the subbands the interference is located within, and the power of the transmitter. We show that the agent is effectively able to avoid frequency bands that have degraded channel quality or that contain interference, both of which are dynamic or time-varying . In the second setting, we study the use of reinforcement learning to adapt the modulation and power scheme of a jammer seeking to disrupt a wireless communications system. To achieve this, we make use of a linear contextual bandit to learn to jam the victim system. Prior work has shown that with the use of linear bandits, improved convergence is achieved to jam a single-carrier system using time-domain jamming schemes. However, communications systems today typically employ orthogonal frequency division multiplexing (OFDM) to transmit data, particularly in 4G/5G networks. This work explores the use of linear Thompson Sampling (TS) to jam OFDM-modulated signals. The jammer may select from both time-domain and frequency-domain jamming schemes. We demonstrate that the linear TS algorithm is able to perform better than a traditional reinforcement learning algorithm, upper confidence bound-1 (UCB-1), in terms of maximizing the victim's symbol error rate. We also draw novel insights by observing the action states, to which the reinforcement learning algorithm converges. We then investigate the design and modification of the context vector in the hope of in- creasing overall performance of the bandit, such as decreased learning period and increased symbol error rate caused to the victim. This includes running experiments on particular features and examining how the bandit weights the importance of the features in the context vector. Lastly, we study how to jam an OFDM-modulated signal which employs forward error correction coding. We extend this to leverage reinforcement learning to jam a 5G-based system implementing some aspects of the 5G protocol. This model is then modified to introduce unreliable reward feedback in the form of ACK/NACK observations to the jammer to understand the effect of how imperfect observations of errors can affect the jammer's ability to learn. We gain insights into the convergence time of the jammer and its ability to jam the victim, as well as improvements to the algorithm, and insights into the vulnerabilities of wireless communications for reinforcement learning based jamming. / Master of Science / In this thesis we implement a class of reinforcement learning known as contextual bandits in two different applications of communications systems and jamming. In the first setting, we study the use of reinforcement learning in an underwater acoustic communications network to adapt its transmission frequencies to avoid interference and potential malicious jammers. We show that the agent is effectively able to avoid frequency bands that have degraded channel quality or that contain interference, both of which are dynamic or time-varying. In the second setting, we study the use of reinforcement learning to adapt the jamming type, such as using additive white Gaussian noise, and power scheme of a jammer seeking to disrupt a wireless communications system. To achieve this, we make use of a linear contextual bandit which implies that the contexts that the jammer is able to observe and the sampled probability of each arm has a linear relationship with the reward function. We demonstrate that the linear algorithm is able to outperform a traditional reinforcement learning algorithm in terms of maximizing the victim's symbol error rate. We extend this work by examining the impact of the context feature vector design, LTE/5G-based protocol specifics (such as error correction coding), and imperfect reward feedback information. We gain insights into the convergence time of the jammer and its ability to jam the victim, as well as improvements to the algorithm, and insights into the vulnerabilities of wireless communications for reinforcement learning based jamming.

Approximately, two-thirds of earth's surface is covered by water. There is a growing interest from the military and commercial communities in having, an efficient, secure and high bandwidth underwater wireless communication (UWC) system for tactical underwater applications such as oceanography studies and offshore oil exploration. The existing acoustic and radio frequency (RF) technologies are severely limited in bandwidth because of the strong frequency dependent attenuation of sound in seawater and the high conductivity of seawater at radio frequencies, respectively. Recently, underwater wireless optical communication (UWOC) has been proposed as the best alternative or complementary solution to meet this challenge. Taking advantage of the low absorption window of seawater in blue-green (400-550 nm) regime of the electromagnetic spectrum, UWOC is expected to establish secure, efficient and high data rate communication links over short and moderate distances (< 100 m) for versatile applications such as underwater oil pipe inspection, remotely operated vehicle (ROV) and sensor networks. UWOC uses the latest gallium nitrite (GaN) visible light-emitting diode (LED) and laser diode (LD) transmitters. Although some research on LED lased UWOC is being conducted, both the military and academic 5 research communities are favoring the use of laser beams, which potentially could enhance the available bandwidth by up to three orders of magnitude. However, the underwater wireless channel is optically very challenging and difficult to predict. The propagation of laser beams in seawater is significantly affected by the harsh marine environments and suffers from severe attenuation which is a combined effect of absorption and scattering, optical turbulence, and multipath effects at high transmission rates. These limitations distort the intensity and phase structure of the optical beam leading to a decrease in signal-to-noise ratio (SNR) which ultimately degrades the performance of UWOC links by increasing the probability of error. In this dissertation, we seek to experimentally demonstrate the feasibility of short range (≤ 20 m) UWOC systems over various underwater channel water types using different modulation schemes as well as to model and describe the statistical properties of turbulence-induced fading in underwater wireless optical channels using laser beam intensity fluctuations measurements.