Table Of ContentPerspectives in Ne ura! Computing
Springer-Verlag Landan Ltd.
Alsointhisseries:
Gustavo DecoandDraganObradovic
AnInformation-TheoreticApproachto Neural Computing
0-387-94666-7
Achilleas Zapranisand Apostolos-Paul Refenes
PrinciplesofNeural Model Identification,Selection and Adequacy
1-85233-139-9
WalterJ.Freeman
Neurodynamics:AnExplorationinMesoscopicBrainDynamics
1-85233-616-1
H.Malmgren,MBorgaand1.Niklasson(Eds)
Artificial NeuralNetworksinMedicineand Biology
1-85233-289-1
Mark Girolami
Advances in IndependentComponentAnalysis
1-85233-263-8
Robert M.Frenchand Jacques P.Sougne (Eds)
ConnectionistModels ofLearning, Developmentand Evolution
1-85233-354-5
RobertoTagliaferriandMaria Marinaro (Eds)
Neural Nets WIRNVietri-01
1-85233-505-X
JimmyShadboltandJohn G.Taylor (Eds)
Neural Networks andthe FinancialMarkets
1-85233-531-9
RelatedTitle
RobertHecht-NielsenandThomasMcKenna (Eds)
ComputationalModels forNeuroscience:HumanCortical InformationProcessing
1-85233-593-9
Artur S. d'Avila Garcez, Krysia B. Broda and
DovM. Gabbay
Neural-Symbolic
Learning Systems
Foundations and Applications
, Springer
Artur S. d'Avila Garcez, MEng, MSc, DIC, PhD
Department of Computing, City University, Northampton Square,
London ECI V OHB
Krysia B. Broda, BSc, BA, MSc, PhD
Department of Computing, Imperial College, 180 Queen's Gate, London SW7 2BZ
Dov M. Gabbay, FRSC, FAvH, FRSA
Department of Computer Science, King's College, Strand, London WC2R 2LS
Series Editor
J.G. Taylor, BA, BSc, MA, PhD, FlnstP
Centre for Neural Networks, Department of Mathematics, King's College,
Strand, London WC2R 2LS, UK
British Library Cataloguing in Publication Data
d'Avila Garcez, Artur S.
Neural-symbolic leaming systems : foundations and
applications. -(Perspectives in neural computing)
l.Neural networks (Computer Science)
I.Title II.Broda, Krysia I1I.Gabbay, Dov M., 1945-
006.3'2
Library of Congress Cataloging-in-Publication Data
d'Avila Garcez, Artur S., 1970-
Neural-symbolic learning systems: foundations and applications / Artur S. d'A vila
Garcez, Krysia B. Broda, and Dov. M. Gabbay.
p. cm. --(Perspectives in neural computing)
Includes bibliographical references and index.
ISBN 978-1-85233-512-0 ISBN 978-1-4471-0211-3 (eBook)
DOI 10.1007/978-1-4471-0211-3
1. Neural networks (Computer science) 2. Artificial intelligence. 1. Broda, Krysia,
1949-II. Gabbay, Dov M., 1945-III. Title. IV. Series.
QA76.87 .D38 2002
006.3'2--dc21 2002070771
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms of licences issued by the
Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to
the publishers.
Perspectives in Neural Computing ISSN 1431-6854
http://www.springer.co.uk
@Springer-Verlag London 2002
Originally published by Springer-Verlag London Limited in 2002
The use of registered names, trademarks etc. in this publication does not imply, even in the absence of a
specific statement, that such names are exempt from the relevant laws and regulations and therefore free
for general use.
The publisher makes no representation, express or implied, with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for any errors or
omissions that may be made.
Typesetting: lan Kingston Editorial Services, Nottingham, UK
34/3830-543210 Printed on acid-free paper SPIN 10838895
ToSimoneGarcez- A.G.
ToParimalPal- K.B.
ToLydiaRivlin- D.G.
Preface
Computing Science and Artificial Intelligence are concerned with producing
devices that help or replace human beings in their daily activities. To be
successful,adequatemodellingoftheseactivitiesneedsto be carriedout.This
urgent need formodelshasgiven rise to newdisciplinesaswellas accelerating
the development ofold disciplines.Amongthese are Logic and Computation,
Neural Networks, Fuzzy Systems and Causal Probabilistic Networks.
Learning Systems play a central role in this new landscape, compris
ing collections of techniques developed within each of the above disciplines.
Notwithstandingthis, human beings do not use these techniques in isolation,
but in an integrated way. Therefore the study of how these techniques could
complement each other - and the understanding of the role of each of them
in a particular application - is bound to be the way forward towards the
development of more effective intelligent systems.
In this book,wepresent thekey theory and algorithms that form the core
of the integration of Artificial Neural Networks and Logic Programming. By
combining Neural Networks and Logic Programming into Neural-Symbolic
Learning Systems, our aim isto benefit from theadvantages presented by the
Connectionist and Symbolist paradigms of Artificial Intelligence. The book
starts with an intuitive presentation of Neural-Symbolic Learning Systems,
illustrated with a worked example.It then presents a concise introduction to
Logic Programming and Artificial Neural Networks from a neural-symbolic
integration perspective. Basic concepts from other disciplines are introduced
as the need arises, focusing on their relevance to neural-symbolic integration.
The book is then divided into three main parts, presenting recent advances
on the theoretical foundations of (i) Knowledge Representation and Learn
ing, (ii) Knowledge Extraction and (iii) Inconsistency Handling in neural
symbolic systems.
The book presents a balance between theory and practice, witheach part
containing the results of applications using real-world problems in diverse
areas such as: DNA Sequence Analysis, Power Systems Fault Diagnosis and
Software Requirements Specifications. It will be valuable to researchers and
graduate students in the areas of Engineering, Computer Science, Artificial
Intelligence, Machine Learning and Neurocomputing, who require a compre-
vii
viii Preface
hensive introduction to Neural-Symbolic Learning Systems, as wellas knowl
edgeofthe latest research issuesrelevant to the field.It willalsobe ofuse to
intelligent systems practitioners and to those interested in the applications
of hybrid systems ofArtificial Intelligence.
Acknowledgements
We would like to thank Professor John Taylor, editor of the series, the late
Professor Imad Torsun, Dr Gerson Zaverucha, Dr Luis Alfredo Carvalho, Dr
Ruy Milidiu, Professor Valmir Barbosa, Dr Alberto de Souza, Dr Odinaldo
Rodrigues, Dr Sanjay Modgiland Dr LuisLamb fortheir valuable comments
and suggestions. We also would like to thank Rosie Kemp and the entire
production team at Springer-Verlag fortheir effectiveassistance in all phases
ofthis project. Artur Garcez was partially funded by the Brazilian Research
Council CAPES and bythe British Research CouncilEPSRC, whilehewasa
PhD studentand thenaResearch Associate at ImperialCollege.Heisgrateful
to Professor JeffreyKramer, Professor BasharNuseibeh, Dr AlessandraRusso
and the Requirements Engineering group at Imperial College,Department of
Computing, for many useful discussions.
Artur wouldliketo thank hiswife,Simone, forher lovingcare and under
standing; his parents, Lucia and Cicero;his sister, Cristiane; and his friends
for their continuous support and encouragement. Krysia would liketo thank
Paul for his patience and encouragement. Dov would like to thank his wife,
Lydia, for many years ofsupport and understanding.
London, May 2002
Artur Garcez,Krysia Broda, Dov Gabbay
About the Authors
Artur Garcez is a Lecturer in the Department ofComputing at City Uni
versity, London, and a Research Consultant to Performance Systems and
Methods Ltd. He is known for his work on the theoretical foundations of
Neural-Symbolic Integration: knowledge insertion, revision and extraction;
and its applications to Machine Learning,Bioinformatics and Software Engi
neering. He holds an M.Eng. in Computing Engineering from the Pontifical
Catholic University of Rio de Janeiro, Brazil, an M.Sc. in Computing and
Systems Engineering from the Federal University of Rio de Janeiro, Brazil,
and a Ph.D. in Computing from Imperial College, London, UK.
Krysia Broda is a Senior Lecturer in the Department of Computing at
Imperial College,London. She holds a Ph.D. in Automated Reasoning from
Imperial College.Shehas wideexperience onLogicProgrammingand anum
ber of recent publications on Labelled Deductive Systems (LDS) for non
standard logicssuch asResource Logicsand Substructural and FuzzyLogics,
and on transformation and abduction methods in LDS. Her current research
interests include knowledgerepresentation in neural networks and knowledge
extractionfromtrained neural networks,and its applications to goal-directed
reactive robots.
Dov Gabbay is Augustus de Morgan Professor ofLogicat the Department
of Computer Science and the Department of Philosophy at King's College,
London. He received a B.Sc. in Mathematics and Physics, a M.Sc. in Logic,
and aPh.D.in Logicfromthe Hebrew University,Jerusalem.Hehas initiated
several research areas in Logic and Computation, having made significant
contributionsto the areas ofTemporal, Multimodal and Monmonotonic Log
ics and their applications; Logic and Language; and the theory of inference
systems via Labelled Deductive Systems. He is Fellowof the Royal Society
ofCanada, and the editor ofseveral international Logicjournals and several
multi-volume Logichandbooks.
ix
Table of Contents
1. Introduction and Overview 1
1.1 Why Integrate Neurons and Symbols?..................... 1
1.2 Strategies ofNeural-Symbolic Integration. ................. 3
1.3 Neural-Symbolic Learning Systems 5
1.4 A Simple Example. ..................................... 7
1.5 Howto Read this Book. ................................ 10
1.6 Summary......... ........ .. ...... .. .... .... ... ........ 12
2. Background.. ..... ... ... ........ ... ... ..... .......... ... . 13
2.1 General Preliminaries. .................................. 13
2.2 Inductive Learning 14
2.3 Neural Networks 15
2.3.1 Architectures ... ... ... ....... .... ...... ..... .. ... 16
2.3.2 Learning Strategy ................................ 19
2.3.3 Recurrent Networks 21
2.4 LogicProgramming..................................... 23
2.4.1 What isLogicProgramming? ...................... 23
2.4.2 Fixpoints and Definite Programs ................... 26
2.5 Nonmonotonic Reasoning. ............................... 29
2.5.1 Stable Models and Acceptable Programs. ........... 29
2.6 BeliefRevision ......................................... 34
2.6.1 Truth Maintenance Systems ....................... 37
2.6.2 Compromise Revision. ............................ 39
Part I. Knowledge Refinement in Neural Networks
3. Theory Refinement in Neural Networks. .. ... ..... ....... 43
3.1 Inserting Background Knowledge ......................... 44
3.2 MassivelyParallel Deduction............................. 56
3.3 Performing Inductive Learning ........................... 58
3.4 Adding Classical Negation. .............................. 59
xi
xii TableofContents
3.5 Adding Metalevel Priorities .............................. 64
3.6 Summary and Further Reading. .......................... 84
4. Experiments on Theory Refinement 87
4.1 DNA Sequence Analysis ................................. 87
4.2 Power Systems Fault Diagnosis. .......................... 97
4.3 Discussion... ......... ... ..... ... ......... .......... ... 106
4.4 Appendix 108
Part II. Knowledge Extraction from Neural Networks
5. Knowledge Extraction from Trained Networks '" ." 113
5.1 The Extraction Problem 114
5.2 The Case ofRegular Networks 120
5.2.1 Positive Networks 121
5.2.2 Regular Networks 127
5.3 The General Case Extraction 137
5.3.1 Regular Subnetworks 138
5.3.2 KnowledgeExtraction from Subnetworks 139
5.3.3 Assembling the Final Rule Set 151
5.4 Knowledge Representation Issues 153
5.5 Summary and Further Reading 155
6. Experiments on Knowledge Extraction 159
6.1 Implementation 159
6.2 The Monk's Problems 166
6.3 DNA Sequence Analysis 168
6.4 Power Systems Fault Diagnosis 173
6.5 Discussion 176
Part III. Knowledge Revision in Neural Networks
7. Handling Inconsistencies in Neural Networks 183
7.1 Theory Revision in Neural Networks 183
7.1.1 The Equivalence with Truth Maintenance Systems 184
7.1.2 Minimal Learning 186
7.2 SolvingInconsistencies in Neural Networks 192
7.2.1 Compromise Revision 194
7.2.2 Foundational Revision 195
7.2.3 Nonmonotonic Theory Revision 200
7.3 Summary of the Chapter 207
