Table Of Contentfile:///C|/New%20Text%20Document%20(2).txt
Table of Contents
Preface
Acknowledgments
Addresses of the PDP Research Group
I THE PDP PERSPECTIVE
1 The Appeal of Parallel Distributed Processing
J.L. McClelland, D.E. Rumelhart, and G.E. Hinton
2 A General Framework for Parallel Distributed Processing
D.E. Rumelhart, G.E. Hinton, and J.L. McClelland
3 Distributed Representations
G.E. Hinton, J.L. McClelland, and D.E. Rumelhart
4 PDP Models and General Issues in Cognitive Science
D.E. Rumelhart and J.L. McClelland
II BASIC MECHANISMS
5 Feature Discovery by Competitive Learning
D.E. Rumelhart and D. Zipser
6 Information Processing in Dynamical Systems: Foundations of Harmony Theory
P. Smolensky
file:///C|/New%20Text%20Document%20(2).txt (1 of 2)9/25/2008 5:03:17 PM
file:///C|/New%20Text%20Document%20(2).txt
7 Learning and Relearning in Holtzmann Machines
G.E. Hinton and T.J. Sejnowski
8 Learning Internal Representations by Error Propagation
D.E. Rumelhart, G.E. Hinton, and R.J. Williams
III FORMAL ANALYSIS
9 An Introduction to Linear Algebra in Parallel Distributed Processing
M.I. Jordan
10 The Logic of Activation Functions
R.J. Williams
11 An Analysis of the Delta Rule and the Learning of Statistical Associations
G.O. Stone
12 Resource Requirements of Standard and Programmable Nets
J.L. McClelland
13 P3: A Parallel Network Simulating System
D. Zipser and D.E. Rabin
References
Index
file:///C|/New%20Text%20Document%20(2).txt (2 of 2)9/25/2008 5:03:17 PM
(cid:0)
Preface
(cid:0)
One of the great joys of science lies in the moment of shared
discovery. One person's half-baked suggestion resonates in the mind of
another and suddenly takes on a definite shape. An insightful critique
of one way of thinking about a problem leads to another, better under-
standing. An incomprehensible simulation result suddenly makes sense
as two people try to understand it together.
This book grew out of many such moments. The seeds of the book
were sown in our joint work on the interactive activation model of word
perception. Since then, each of us has worked with the other and with
other collaborators. The results of these collaborations are reported in
several of the chapters of this book. The book also contains many
chapters by other colleagues whose explorations have become
intertwined with ours. Each chapter has its own by-line, but each also
reflects the influences of other members of the group. We hope the
result reflects some of the benefits of parallel distributed processing!
The idea of parallel distributed processing- the notion that intelli-
gence emerges from the interactions of large numbers of simple pro-
cessing units- has come and gone before. The idea began to seem
more and more attractive to us as the contrast between our convictions
about basic characteristics of human perception, memory, language, and
thought and the accepted formal tools for capturing mental processes
became more apparent. Symbol-processing machines, for all their Tur-
ing equivalence, had failed to provide useful frameworks for capturing
x PREFACE
the simple insights about the interactive nature of processing that had
lead to such models as the HEARSAY model of speech understanding.
More generally, they had failed to provide a framework for representing
knowledge in a way that allowed it to be accessedb y content and effec-
tively combined with other knowledge to produce useful automatic
syntheses that would allow intelligence to be productive. And they
made no contact with the real strengths and weaknesses of the
hardware in the brain. A Cray computer can perform on the order of
100 million double-precision multiplications in a second, but it does not
exhibit natural intelligence. How then are we to understand the capa-
bilities of human thought, given the time constants and noisiness
inherent in neural systems? It seemed obvious that to get any process-
ing done in real time, the slow, noisy hardware in the brain would have
to do massively parallel processing.
As our interest in parallel mechanisms developed, we began to study
the work of others who shared our convictions and to build on their
work. Particularly important in this regard was Hinton and J. A.
Anderson's (1981) Parallel Models of Associative Memory. Indeed, we
see our book as a descendant of their book on two accounts. First, the
material presented here represents further developments on the work
presented in Hinton and Anderson's book. Second, we owe a particular
intellectual debt to both Hinton and Anderson. Our interest in distrib-
uted, associative memories goes back to interactions with Jim Ander-
son, beginning as early as 1968. Our interest in these topics began in
earnest, however, during the period when we were developing the
interactive activation model of word perception, in 1979, shortly after
Geoffrey Hinton began a postdoctoral fellowship at UCSD. Geoffrey's
crisp explanations showed us the potential power and generality of
models created from connections among simple processing units, and
fit together nicely with our own developing conviction that various
aspects of perception, language processing, and motor control were best
thought of in terms of massively parallel processing (see McClelland,
1979, and Rumelhart, 1977, for our earliest steps in this direction).
The project culminating in this book formally began in December,
1981 when the two of us and Geoffrey Hinton decided to work together
exploring the implications of network models and to write a book out-
lining our conclusions. We expected the project to take about six
months. We began in January 1982 by bringing a number of our col-
leagues together to form a discussion group on these topics. During
the first six months we met twice weekly and laid the foundation for
most of the work presented in these volumes. Our first order of busi-
ness was to develop a name for the class of models we were investigat-
ing. It seemed to us that the phrase parallel distributedp rocessing( POP
.
PREFACE XI
for short) best captured what we had in mind . It emphasized the paral-
lel nature of the processing , the use of distributed representations and
distributed control , and the fact that these were general processing sys-
tems, not merely memories we were studying , as the phrase associative
memory suggests. Thus the POP research group was born. Hinton and
McClelland left after the first six months - Hinton to CMU and
McClelland to MIT and later to CMU . The POP research group , how -
ever, has continued regular meetings at UCSO up to the present time .
The group has varied from five or six of us at times to as many as 15
or more at other times , and there is now a parallel group of about 15 or
so psychologists and computer scientists at CMU .
Shortly after leaving UCSO in 1982, Hinton began working with
Terrence Sejnowski on the Boltzmann machine (Chapter 7) and decided
to dl"'JP from the role of organizer of the project to a contributor , so he
could spend more time working on the implications of the Boltzmann
machine . Thus , the primary responsibility for putting the book
together fell to the two of us. At first we expected to complete the
book within a year after we began our work . Soon, however , it became
clear that there was much work to be done and many directions to
explore . Thus, our work continued and expanded as we and our col-
leagues followed the implications of the POP approach in many dif-
ferent ways .
A good deal has happened since we began this project . Though much
of the initial groundwork was laid in early 1982, most of the material
described in these volumes did not take its present form until much
later .
The work has been interdisciplinary and represents what we consider
a true cognitive science approach. Although the two of us have been
trained as cognitive psychologists , the POP group as a whole includes
people from a wide range of backgrounds . It includes people trained in
physics, mathematics , neuroscience , molecular biology , and computer
sciences, as well as in psychology . We also envision an interdisciplinary
audience for our book. We are cognitive psychologists and we hope,
primarily , to present POP models to the community of cognitive
psychologists as alternatives to the models that have dominated cogni-
tive psychology for the past decade or so. We also, however , see our-
selves as studying architectures for computation and methods for artifi -
cial intelligence . Therefore , we hope that this book will be seen as
relevant to researchers in computer science and artificial intelligence .
Also, the POP approach provides a set of tools for developing models
of the neurophysiological basis of human information processing, and
so we hope portions of these books will seem relevant to neuroscien -
tists as well .
..
XII PREFACE
ORGANIZATION OF THE BOOK
Our book consists of six parts , three in each of the two volumes .
The overall structure is indicated in the accompanying table. Part I pro-
vides an overview . Chapter 1 presents the motivation for the approach
and describes much of the early work that lead to the developments
reported in later sections. Chapter 2 describes the POP framework in
more formal terms . Chapter 3 focuses on the idea of distributed
representation , and Chapter 4 provides a detailed discussion of several
general issues that the POP approach has raised and explains how these
issues are addressed in the various later chapters of the book.
The remaining parts of the book present different facets of our
explorations in parallel distributed processing. The chapters in Part II
address central theoretical problems in the development of models of
parallel distributed processing, focusing for the most part on fundamen -
tal problems in learning . The chapters in Part III describe various
mathematical and computational tools that have been important in
the development and analysis of POP models . Part IV considers
A CONDENSED TABLE OF CONTENTS
VOLUME I
I. THE POP PERSPECTIVE II. BASIC MECHANISMS III . FORMAL ANALYSES
1. The Appeal of POP 5. Competitive Learning 9. Linear Algebra
2. A Framework for PDP 6. Harmony Theory 10. Activation Functions
3. Distributed 7. Boltzmann Machines 11. The Delta Rule
Representations 8. Learning by 12. Resource Requirements
4. General Issues Error Propagation 13. Parallel Network
Simulator
VOLUME II
--
IV . PSYCHOLOGICAL V. BIOLOGICAL VI . CONCLUSION
PROCESSES MECHANISMS
14. Schemata and POP 20. Anatomy and 26. Reflections
15. Speech Perception Physiology Future Directions
16. Model of Reading 21. Computation in
17. Learning and Memory the Brain
18. Morphology Acquisition 22. Neural and
19. Sentence Processing Conceptual Levels
23. Place Recognition
24. Neural Plasticity
25. Amnesia
.. .
PREFACEXIII
applications and implications of PDP models to various aspects of
human cognition , including perception , memory , language, and higher -
level thought processes. Part V considers the relation between parallel
distributed processing models and the brain , reviews relevant aspects of
the anatomy and physiology , and describes several models that apply
POP models to aspects of the neurophysiology and neuropsychology of
information processing , learning , and memory . Part VI contains two
short pieces: a reflection on PDP models by Don Norman and a brief
discussion of our thoughts about promising future directions .
How to read this book? It i~ too long to read straight through. Nor
is it designed to be read this way. Chapter 1 is a good entry point for
readers unfamiliar with the POP approach, but beyond that the various
parts of the book may be approached in various orders, as one might
explore the different parts of a complex object or machine . The vari-
ous facets of the POP approach are interrelated , and each part informs
the others~ but there are few strict sequential dependencies . Though
we have tried to cross-reference ideas that come up in several places,
we hope that most chapters can be understood without reference to the
rest of the book. Where dependencies exist they are noted in the intro -
ductory sections at the beginning of each part of the book.
This book charts the explorations we and our colleagues have made
in the microstructure of cognition . There is a lot of terrain left to be
explored . We hope this book serves as a guide that helps others join us
in these ongoing explorations .
December1 985 JamesL . McClelland
PI1TSBURG, PHENNSYLVANIA
David E. Rumelhart
LA JOLLA, CALIFORNIA
(cid:0)
Acknowledgments
(cid:0)
As we have already said, nearly all the ideas in this book were born
out of interactions, and one of our most important acknowledgments is
to the environment that made these interactions possible. The Institute
for Cognitive Science at UCSD and the members of the Institute have
made up the core of this environment.
Don Norman, our colleague and friend, the Founder and Director of
the Institute, deserves special credit for making ICS an exciting and
stimulating place, for encouraging our explorations in parallel distrib-
uted processing, and for his central role in arranging much of the finan-
cial support this book has benefited from (of which more below). The
atmosphere depends as well on the faculty, visiting scholars, and gradu-
ate students in and around ICS. The members of the PDP Research
Group itself, of course, have played the most central role in helping to
shape the ideas found in this book. All those who contributed to the
actual contents of the book are listed on the cover page; they have all
contributed, as well, in many other ways. Several other participants in
the group who do not have actual contributions to the book also
deserve mention. Most prominent among these are Mike Mozer and
Yves Chauvin, two graduate students in the Cognitive Science Lab, and
Gary Cottrell, a recent addition to the group from the University of
Rochester.
Several other members of the intellectual community in and around
ICS have played very important roles in helping us to shape our
thoughts. These include Liz Bates, Michael Cole, Steve Draper, Don
.
XVI ACKNOWLEDGMENTS
Gentner , Ed Hutchins , Jim Hollan , Jean Mandler , George Mandler ,
Jeff Miller , Guy van Orden , and many others , including the participants
in Cognitive Science 200.
There are also several colleagues at other universities who have
helped us in our explorations . Indeed , the annual connectionist
workshops (the first of which resulted in the Hinton and Anderson
book) have been important opportunities to share our ideas and get
feedback on them from others in the field , and to learn from the con -
tributions of others . Jim Anderson , Dana Ballard, Jerry Feldman ,
Geoff Hinton and Terry Sejnowski all had a hand in organizing dif-
ferent ones of these meetings ; and we have learned a great deal from
discussions with them and other participants , particularly Andy Barto,
Scott Fahlman , Christof von der Malsburg , John Hopfield , Dave
Touretzky , and more recently Mark Fanty and Gene Charniak .
McClelland 's discussions at MIT (particularly with Jerry Fodor and
Molly Potter) helped in the clarification of several aspects of our think -
ing, and various colleagues at and around CMU - particularly John
Anderson , Mark Derthick , Dave Klahr , Brian MacWhinney , and Jeff
Sokolov - have contributed a great deal through discussions over the
last year and a half or so, as we have worked toward the completion of
the book . Others one or both of us have interacted with a great deal
include Bill Brewer , Neal Cohen , Al Collins , Billy Salter, Ed Smith , and
Walter Schneider . All of these people have contributed more or less
directly to the development of the ideas presented in this book.
An overlapping group of colleagues deserves credit for helping us
improve the book itself . Jim Anderson , Andy Barto, Larry Barsalou,
Chris Reisbeck , Walter Schneider , and Mark Seidenberg all read several
chapters of the book and sent useful comments and suggestions . Many
other people read and commented on individual chapters, and we are
sincerely grateful for their careful contributions, which we acknowledge
in the appropriate chapters.
This project owes a tremendous amount to the help of the excellent
staff of the Institute for Cognitive Science. Kathy Farrelly , in particu-
lar, has played an enormous role in all aspects of the production of the
book; her cheerful, thoughtful , and very careful assistance made the
production of the book run much more smoothly than we have had any
right to hope and allowed us to keep working on the content of some of
the chapters even as the final production was rolling forward on other
sections. Eileen Conway's assistance with graphics and formatting has
also been invaluable and we are very grateful to her as well. Mark Wal-
len kept the computers running , served as chief programming consul-
tant and debugger par excellence , and tamed troff , the phototypesetter .
Without him we would never have gotten all the formatting to come
out right . Karol Lightner worked very hard toward the end of the
