Cal Poly
Design and Parametric Study of a MEMS-based Reservoir Computer for Reinforcement Learning
Abstract
dc:description.abstractSingle-node reservoir computing (RC) is a hardware-efficient approach to machine learning, leveraging the dynamics of physical systems. In this work, two reinforcement learning algorithms, Q-learning and Proximal Policy Optimization (PPO), are applied to a simulated micro-electro-mechanical system (MEMS)-based reservoir computer to solve both discrete and continuous control tasks. MEMS-based reservoirs are low-power, compact, and their natural frequencies (kHz to MHz) pair well with real-time control loops. To explore the relationship between reservoir dynamics and learning performance, a parametric study is conducted on two reservoir hyperparameters, reservoir size and neuron separation, using CartPole-v1 and MountainCar-v0. The RC successfully learns multiple tasks with both algorithms. Most notably, the system learned a successful gait for the 4-degree-of-freedom bipedal locomotion task, BipedalWalker-v3. In the parametric studies, increasing reservoir size led to increased robustness to random initialization while increasing typical performance. A strong sensitivity to reservoir initialization was observed, motivating reservoir initialization searches during tuning. Optimal neuron separation was shown to be task dependent; however, each task was solved with neuron separations near the fundamental period of the MEMS device. In contrast to ultra-fast optical RCs, MEMS-based reservoir computers can operate at frequencies compatible with real-time sensing and control, enabling integration into mechanical systems. Overall, this work demonstrates the successful application of reinforcement learning algorithms to MEMS-based reservoir computing on both discrete and continuous control tasks while highlighting the relationship between reservoir dynamics and reinforcement learning performance.
Degree
thesis:*- Name thesis:degree_name
- MS in Mechanical Engineering
- Discipline thesis:degree_discipline
- Mechanical Engineering
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Carr, Andrew P
- Contributors dc:contributor
-
- Mohammad Hasan
- Mechanical Engineering
- College of Engineering
Subjects
dc:subject × 12Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.calpoly.edu/theses/3304