Back to search

University of British Columbia

Learning temporal action chunking for motor control

Abstract

dc:description

Deep reinforcement learning has had significant success at learning motor control tasks. Typically, these policies are fully closed loop or ‘state indexed’, implying a control policy that is queried at every control time step with the current state in order to estimate the best current action corresponding to that state. However, this approach ignores the inherent predictability of many systems, wherein the future states and actions are often quite predictable and can thus be controlled in an open-loop or ‘time indexed’ fashion. Chunking of action sequences is a well-established mechanism in cognitive systems to enhance learning capabilities and learning efficiency. By modeling actions in time-indexed chunks, one reduces the computational and perceptual demands required for control. Learning this type of temporal action abstraction remains under-explored. We present a method that learns a chunk-based state-and-time-indexed policy from any existing state-indexed reinforcement learning policy, with minimal added complexity. We show that with a straightforward multi-layer neural network, the chunk-based policy can decrease the required control frequency significantly. In particular, we show a reduction from 60Hz to 10Hz for the control of a 3D humanoid capable of robust and realistic movement across varying terrain. We further propose an adaptive runtime algorithm that can leverage long action chunks while reverting to single-step actions as needed in order to achieve robust behavior.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Gou, Ruiyu
Contributors dc:contributor
  • Van de Panne, M. (Michiel), 1965-

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/89057

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Gou, Ruiyu. Learning temporal action chunking for motor control. master's thesis, University of British Columbia, 2024. http://hdl.handle.net/2429/89057