Associative Neural Memories: Theory and Implementation
by Mohamad H. Hassoun
(Oxford Press, 1993)
This edited volume brings together significant works on
associative neural memory theory (architecture, learning, analysis, and
design) and hardware implementation (VLSI and optoelectronic) by leading
international researchers. The purpose of this book is to integrate recent
fundamental and significant research results on associative neural memories
into a single volume, and present the material in a clear and organized
format which makes it accessible to researchers and students.
Associative neural memories are a class of artificial neural networks (connectionist nets) which have gained substantial attention relative to other neural net paradigms. Associative memories have been the subject of research since the early seventies. Recent interest in these memories has been spurred by the seminal work of Hopfield in the early eighties, who has shown how a simple discrete nonlinear dynamical system can exhibit associative recall of stored binary patterns through collective computing. Since then, a number of important contributions have appeared in conference proceedings and technical journals addressing various issues of associative neural memories, including multiple-layer architectures, recording/storage algorithms, capacity, retrieval dynamics, fault-tolerance, and hardware implementation. Currently, associative neural memories are among the most extensively studied and understood neural paradigms. They have been studied as possible models of biological associative phenomena, as models of cognition and categorical perception, as high-dimensional nonlinear dynamical systems, as collective computing nets, as error-correcting nets, and as fault- tolerant content addressable computer memories.
This book is organized into an introductory chapter and four parts: Biological and Psychological Connections, Artificial Associative Neural Memory Models, Analysis of Memory Dynamics and Capacity, and Implementation. The group of chapters in the first part deals with associative neural models which have close connections to biological and/or psychological aspects of memory. This group consists of three chapters by D. Alkon et al., P. Kanerva, and J. Anderson. The second part of this book consists of three chapters by Y.-F. Wang et al., A. Dembo, and B. Baird and F. Eeckman. These chapters present more complex extensions of the simple associative memory models covered in the introductory chapter, and study their recall capabilities.The analysis of artificial associative neural memory dynamics, capacity, and error-correction capabilities are addressed in part three, which comprises the seven chapters by S.-I. Amari and H.-F. Yanai, R. Paturi, F. Waugh et al., S. Hui et al., G. Pancha and S. Venkatesh, S. Yoshizawa et al., and P.-C. Chung and T. Krile. The last part of the book deals with hardware implementation of associative neural memories (some of these memories and/or their associated recording algorithms constitute variations and/or extensions to those discussed in earlier chapters). Here, three chapters by A. G. Andreou and K. A. Boahen, M. Verleysen et al., and T.-D. Chiueh and R. Goodman address electronic VLSI implementations. Two additional chapters, one by F. T. S. Yu and the other by K. Kyuma et al., present optoelectronic implementations.
Chapter 1 is an introduction to basic dynamic associative memory (DAM) architectures and their associated learning/recording algorithms. DAMs are treated as collective nonlinear dynamical systems in which information retrieval is accomplished through an evolution of the system's state in a high dimensional (binary) state space. This chapter reviews some basic supervised learning/recording algorithms and derives the necessary conditions for perfect storage and retrieval of memories in a simple DAM. The characteristics of high-performance DAMs are outlined and such general issues as stability, capacity, and retrieval dynamics are discussed. Also, references to other chapters of the book are made so as to point out architecture extensions, additional learning algorithms, formal analysis, and other associative neural memory issues which are briefly described in this introductory chapter.
Chapter 2 demonstrates the important contributions that neurobiology can make to the design of artificial neural networks in general and associative learning nets in particular. It describes some results of biochemical and biophysical experiments that elucidate the properties of the Hermissenda visual-vestibular network, followed by descriptions of two computer models, each representing a different level of aggregation of the essential features of learning and memory. A biologically-based computer model called Dystal is presented. The model demonstrates efficient associative learning and is applied to problems in face recognition and optical character recognition applications.
Chapter 3 describes and analyzes a class of associative memories known as "sparse distributed memory" and relates it to associative memory models of the cerebellum, digital random access memory, and other sparse memory models reported in the literature. The chapter presents a unified formulation of a broad class of two-layer feedforward associative memory architectures, and advances the concept of "pattern computing" as a new computing model, as contrasted to numeric computing and symbolic computing.
Chapter 4 focuses on the brain-state-in-a-box (BSB) associative memory as a low-order approximation to a broad range of human cognitive operations. The chapter presents the theory of the BSB model and informally describes some of its mathematical properties. Simulations are presented that show how BSB can model psychological response time.
Chapters 5-7 present more complex models of artificial associative neural memories as compared to those reviewed in chapter 1. Chapter 5 addresses bidirectional associative memory (BAM), originally proposed independently by B. Kosko and Okajima et al. in 1987, and proposes several alternative recording schemes for improved recall. In chapter 6, a class of high-density associative memory models with such desirable properties as high capacity, controllable basins of attraction, and fast convergence speed is proposed and analyzed. In chapter 7, the "projection algorithm"-based network and its extensions are proposed for the guaranteed associative memory storage of analog patterns, continuous sequences, and chaotic attractors in the same network, with no spurious attractors. In this chapter, the authors concentrate on mathematical analysis and engineering-oriented applications of the projection algorithm-based memory. The chapter also attempts to relate the emergent dynamical behavior of interconnected modules of the proposed network to those of cortical computations by taking the view that oscillatory and possibly chaotic network modules form the actual cortical substrate of diverse sensory, motor, and cognitive operations.
The next group of seven chapters, chapters 8-14, deals with the analysis of various aspects of associative memory, such as capacity, convergence dynamics, shaping basin of attractions, effects of non-monotonic activation functions on retrieval dynamics, and fault tolerance. Chapter 8 formulates and presents a unified approach to the analysis of various architectures of associative memories based on a statistical neurodynamical method which allows for the analysis of associative recall dynamics and storage capacity. The method is applied to cross-correlation, cascaded, cyclic, autocorrelation, and associative sequence generator associative memories. Chapter 9 presents a detailed, rigorous mathematical analysis of convergence in the synchronous and asynchronous updated Hopfield associative memory. Theorems are presented, along with their proofs, on the amount of error-correction and the rate of convergence as a function of the number of fundamental memories. The stability and dynamics of analog parallel updated associative memories are studied in chapter 10. The operation of these dynamic analog nets as associative memories is explained in terms of phase diagrams, relating memory loading to neuron activation function slopes, for correlation and generalized inverse recording. Chapter 11 examines the stability of the generalized BSB associative memory. It characterizes the stability and location of all fixed points of the BSB model for different weight matrices. In chapter 12, a family of algorithms is discussed for DAM recording in terms of memory capacity and algorithm complexity. The chapter also emphasizes recording schemes for controlling the basins of attraction of selected memories and/or controling the basins of attraction of selected memory vector/pattern components. Chapter 13 adopts a piecewise linear non- monotonic neuron activation function in a dynamic autocorrelation associative memory and investigates the existence and stability of equilibrium states. This chapter also gives theoretical estimates on the capacity of such memories. Chapter 14 analyzes the effects of the faults characteristic of optical and electronic implementation technologies on implemented associative memory retrieval characteristics.
Chapters 15-19 cover hardware implementations of associative neural memories. In chapter 15, some basic building blocks for VLSI implementation of neural circuitry are described. The basic principles of analog VLSI architectures are discussed in connection with precision limitations encountered with such technology. The chapter describes how to overcome some of these limitations by appropriate designs of artificial neurons and adaptation of basic associative learning algorithms. Chapter 16 describes a hybrid analog/digital CMOS chip implementation of a high- capacity exponential correlation associative memory (ECAM), along with simulations and experimental validation. The chapter also analyzes the storage capacity and error-correction characteristics of ECAMs, and the effect of hardware-limited exponentiation dynamic range on capacity. In chapter 17, a scalable, efficient, and fault-tolerant chip design of a novel bidirectional associative memory architecture, based on subthreshold current mode MOS transistors, is described and validated through simulations. The design techniques employed in this chapter allow for compact implementation which can potentially lead to associative memory chips with densities approaching that of static random access memory (RAM). An optical implementation of a dynamic single-layer associative memory based on a liquid crystal television (LQTV) spatial light modulator (SLM) is described in chapter 18. Robust associative retrieval is demonstrated in this optical memory for several recording algorithms. This chapter also describes how a high-dimensional set of memories can be handled by employing a space-time sharing architecture. Finally, chapter 19 considers the implementation of neural network architectures employing 3-D optical neurochips with on-chip analog memory capabilities based on integrated LED and variable sensitivity photodetector (VSPD) arrays. Experimental results are reported for a two-layer winner-takes-all-based associative memory for stamp classification and a two-layer perceptron net employing back error propagation learning.
In putting this book together, an effort was made to include those researchers who are in many cases the originators of significant ideas on associative neural memories. However, as in any book like this, it would be impossible to include all significant work on this topic. Therefore, my strategy was to invite contributions byleading researchers who are able to relate their ideas to others in the literature, and who, in some cases, present a unifying framework for the study of associative neural memories.
I would like to take this opportunity to thank those who have contributed in various ways to the completion of this book. This project would not have been successful without the enthusiasm and professional cooperation of the contributing authors and their high-quality chapter contributions. I would like to thank the National Science Foundation for support of my work on associative neural memories through a Presidential Young Investigator Award (Grant ECE-9057896). In particular, my thanks go to Dr. Paul Werbos of NSF for his support of my research ideas since 1988. Also, thanks go to Ford Motor Company, Sun Microsystems, Unisys Corporation, and Zenith Data Systems for their valuable support, which has contributed directly or indirectly to the success of this project. Special thanks to Donald C. Jackson of Oxford University Press for his interest and help in publishing this book. I also take this opportunity to thank my wife Amal for her understanding and support, and thank my daughter Lamees for her patience.
Mohamad H. Hassoun
Detroit, June 1992