This dissertation modifies error-correcting codes and Bloom filters to create high-capacity associative memories. These associative memories use principled statistical inference and are distributed as no single component contains enough information to complete the task by itself, yet the components can collectively solve the task by passing information to each other. These associative memories are also robust to hardware failure as their distributed nature ensures there is no single point of failure. This dissertation starts by simplifying a Bloom filter so that it tolerates hardware failure (albeit with reduced performance). An efficient associative memory is created by performing inference over the set of items stored in the Bloom filter. This architecture suggests a modification which forgets old patterns stored in the associative memory (known as a palimpsest memory). It is shown that overwriting old patterns in an independent manner reduces performance, but is still comparable to the well-known Hopfield network. The lost performance can be regained using integer storage which allows the superposition of the pattern representation, or ensuring bits are not overwritten independently using concepts from errorcorrecting codes. The final task performs recall in continuous time using components which are more similar to neurons than used in the rest of the dissertation. The resulting memory has the exciting ability to recall many patterns simultaneously. Statistical inference ensures gradual degradation of the performance as an associative memory is overloaded. Since many definitions of associative memory capacity rely on the existence of catastrophic failure a new definition of capacity is provided. In spite of some biologically unrealistic attributes, this work is relevant to the understanding of the brain as it provides high performance solutions to the associative memory task which is known to be relevant to the brain.