Back to search

University of British Columbia

Energy as reward : an ecological foundation for the structure of attentional fixation in computational agents during audiovisual speech communication

Abstract

dc:description

Speech communication is a multimodal process. In this process, the act of producing symbolic linguistic structure affects the patterning of various energy media in the surround, which can be sensed, and subsequently transformed, into neural processes which likewise influence the cognitive state of a perceiver. With respect to vision in particular, there is likewise widespread recognition that the distribution of perceivers’ attention – where they fixate- plays a fundamental role in this process, but there is still considerable uncertainty as to what information in the visible signal is beneficial to perceivers, and why fixation patterns are structured in a way which tends to be distributed across various physiological structures, including, but not limited to, both the eyes and the mouth. In this dissertation, I approach this problem from a computational perspective, specifically via the development of a Deep Reinforcement Learning (DRL) agent that learns to adjust its visual fixations in order to predict the spatiotemporal structure of the acoustic speech signal. The reward signal that is maximized by this agent is based upon an ecologically oriented optimization principle – the Maximum Power Principle (MPP; [1, 2]) – which asserts that self-organizing living systems behave in such a way as to maximize the throughput of available energies in the environment. In experiments with two Audiovisual Speech datasets, I demonstrate that the agent’s fixation behaviours, when structured optimally with respect to this criterion, support successful learning of cross-modal relations between visible and auditory speech. More generally, I also suggest that this explicit link between the algorithmic notion of reward – which has recently been proposed to be sufficient for explaining all manner of intelligent behavior [3] - with the physical notion of energy could serve to explain how generally intelligent behavior manifests across all kinds of different environments: a universal heuristic that may well apply to biological and computational organisms alike.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy - PhD
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Linguistics
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Fuhrman, Robert
Contributors dc:contributor
  • Babel, Molly

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/90496

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Fuhrman, Robert. Energy as reward : an ecological foundation for the structure of attentional fixation in computational agents during audiovisual speech communication. doctoral thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/90496