Back to search

University of Waterloo

Biological Plausibility in Modern Hopfield Networks

Abstract

dc:description.abstract

Modern Hopfield Networks (HNs) have the ability to store a large number of target memories (e.g. binary patterns) and then recall a memory in its entirety when prompted by a sub-set or perturbed version of it; in this sense, these networks demonstrate properties of a Content Addressable Memory (CAM). Associative memory operates in a similar manner, with memories being elicited by cues of partial or noisy information. Hence, these models provide a basis for modelling associative memory in humans. However, these modern HNs often use functions that rely on many-body synapses and are thus not accurate to neurobiology. More biologically realistic versions of modern HNs have been proposed, although these implementations often still utilize the softmax function. Computing the softmax for a single node in a layer requires the knowledge of all other nodes in that layer, resulting in non-local computations. The softmax function also causes each target memory stored in the network to correspond to the activation of a single hidden node. Such one- hot representations are not robust, and not observed in biological neural networks. In this thesis, we investigate more biologically plausible versions of a modern HN that uses the softmax function. Specifically, we introduce a more biological version of this network by using a local softmax function – the locLSE network. To ensure each target memory has a distributed hidden representation, we propose a network that applies population coding to the hidden layer via the Neural Engineering Framework (NEF) – the Distributed locLSE network. Both proposed networks can learn the connection weights using a local learning rule derived from gradient descent on the energy function. It is found that the proposed, more biologically accurate, versions of the original modern HN still demonstrate capabilities of a CAM at similar rates to the original. Lastly, the NEF is applied to the entire locLSE network by implementing it using a software tool called Nengo. The locLSE network in Nengo is found to behave as a CAM for small datasets of target memories, and provided there are enough neurons in the network.

Degree

thesis:*
Grantor dc:publisher
University of Waterloo
Year dc:date.issued
2022

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Snow, Mallory
Advisor dc:contributor.advisor
  • Orchard, Jeff

Subjects

dc:subject × 4

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/10012/18965

Chain of custody

source
Harvested from
University of Waterloo
Base URL
dspacemainprd01.lib.uwaterloo.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Snow, Mallory. Biological Plausibility in Modern Hopfield Networks. University of Waterloo, 2022. http://hdl.handle.net/10012/18965