Bayesian Programming

378 Pages·2013·6.84 MB·English

by  Pierre Bessiere

Bayesian
Programming
K13774_FM.indd 1 10/28/13 2:17 PM
Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series
SERIES EDITORS
Ralf Herbrich Thore Graepel
Amazon Development Center Microsoft Research Ltd.
Berlin, Germany Cambridge, UK
AIMS AND SCOPE
This series reflects the latest advances and applications in machine learning and pat-
tern recognition through the publication of a broad range of reference works, text-
books, and handbooks. The inclusion of concrete examples, applications, and meth-
ods is highly encouraged. The scope of the series includes, but is not limited to, titles
in the areas of machine learning, pattern recognition, computational intelligence,
robotics, computational/statistical learning theory, natural language processing,
computer vision, game AI, game theory, neural networks, computational neurosci-
ence, and other relevant topics, such as machine learning applied to bioinformatics
or cognitive science, which might be proposed by potential contributors.
PUBLISHED TITLES
MACHINE LEARNING: An Algorithmic Perspective
Stephen Marsland
HANDBOOK OF NATURAL LANGUAGE PROCESSING,
Second Edition
Nitin Indurkhya and Fred J. Damerau
UTILITY-BASED LEARNING FROM DATA
Craig Friedman and Sven Sandow
A FIRST COURSE IN MACHINE LEARNING
Simon Rogers and Mark Girolami
COST-SENSITIVE MACHINE LEARNING
Balaji Krishnapuram, Shipeng Yu, and Bharat Rao
ENSEMBLE METHODS: FOUNDATIONS AND ALGORITHMS
Zhi-Hua Zhou
MULTI-LABEL DIMENSIONALITY REDUCTION
Liang Sun, Shuiwang Ji, and Jieping Ye
BAYESIAN PROGRAMMING
Pierre Bessière, Emmanuel Mazer, Juan-Manuel Ahuactzin, and Kamel Mekhnacha
K13774_FM.indd 2 10/28/13 2:17 PM
Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series
Bayesian
Programming
Pierre Bessière
CNRS, Paris, France
Emmanuel Mazer
CNRS, Grenoble, France
Juan-Manuel Ahuactzin
ProbaYes, Puebla, Mexico
Kamel Mekhnacha
ProbaYes, Grenoble, France
K13774_FM.indd 3 10/28/13 2:17 PM
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20131023
International Standard Book Number-13: 978-1-4398-8033-3 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
To the late Edwin Thompson Jaynes
for his doubts about certitudes
and for his certitudes about probabilities
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
Contents
Foreword xv
Preface xvii
1 Introduction 1
1.1 Probability an alternative to logic . . . . . . . . . . . . . . 1
1.2 A need for a new computing paradigm . . . . . . . . . . . . 5
1.3 A need for a new modeling methodology . . . . . . . . . . . 5
1.4 A need for new inference algorithms . . . . . . . . . . . . . 8
1.5 A need for a new programming language and new hardware 10
1.6 A place for numerous controversies . . . . . . . . . . . . . . 11
1.7 Running real programs as exercises . . . . . . . . . . . . . . 12
I Bayesian Programming Principles 15
2 Basic Concepts 17
2.1 Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 The normalization postulate . . . . . . . . . . . . . . . . . . 19
2.4 Conditional probability . . . . . . . . . . . . . . . . . . . . 19
2.5 Variable conjunction . . . . . . . . . . . . . . . . . . . . . . 20
2.6 The conjunction postulate (Bayes theorem) . . . . . . . . . 20
2.7 Syllogisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.8 The marginalizationrule . . . . . . . . . . . . . . . . . . . . 22
2.9 Joint distribution and questions . . . . . . . . . . . . . . . . 23
2.10 Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.11 Parametric forms . . . . . . . . . . . . . . . . . . . . . . . . 26
2.12 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.13 Specification = Variables + Decomposition +
Parametric forms . . . . . . . . . . . . . . . . . . . . . . . . 29
2.14 Description = Specification + Identification . . . . . . . . . 29
2.15 Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.16 Bayesian program= Description + Question . . . . . . . . 31
2.17 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
vii
viii Contents
3 Incompleteness and Uncertainty 35
3.1 Observing a water treatment unit . . . . . . . . . . . . . . 35
3.1.1 The elementary water treatment unit . . . . . . . . 36
3.1.2 Experimentation and uncertainty . . . . . . . . . . . 38
3.2 Lessons, comments, and notes . . . . . . . . . . . . . . . . . 40
3.2.1 The effect of incompleteness . . . . . . . . . . . . . . 40
3.2.2 The effect of inaccuracy . . . . . . . . . . . . . . . . 41
3.2.3 Not taking into account the effect of ignored variables
may lead to wrong decisions . . . . . . . . . . . . . . 42
3.2.4 From incompleteness to uncertainty . . . . . . . . . . 43
4 Description = Specification + Identification 47
4.1 Pushing objects and following contours . . . . . . . . . . . 48
4.1.1 The Khepera robot . . . . . . . . . . . . . . . . . . . 48
4.1.2 Pushing objects . . . . . . . . . . . . . . . . . . . . . 49
4.1.3 Following contours . . . . . . . . . . . . . . . . . . . 53
4.2 Description of a water treatment unit . . . . . . . . . . . . 56
4.2.1 Specification . . . . . . . . . . . . . . . . . . . . . . 56
4.2.2 Identification . . . . . . . . . . . . . . . . . . . . . . 59
4.2.3 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 59
4.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3 Lessons, comments, and notes . . . . . . . . . . . . . . . . 60
4.3.1 Description = Specification + Identification . . . . . 60
4.3.2 Specification = Variables + Decomposition + Forms 61
4.3.3 Learning is a means to transform incompleteness into
uncertainty . . . . . . . . . . . . . . . . . . . . . . . 62
5 The Importance of Conditional Independence 65
5.1 Water treatment center Bayesianmodel . . . . . . . . . . . 65
5.2 Description of the water treatment center . . . . . . . . . . 66
5.2.1 Specification . . . . . . . . . . . . . . . . . . . . . . . 66
5.2.2 Identification . . . . . . . . . . . . . . . . . . . . . . 70
5.2.3 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 71
5.3 Lessons, comments, and notes . . . . . . . . . . . . . . . . . 71
5.3.1 Independence versus conditional independence . . . . 71
5.3.2 The importance of conditional independence . . . . . 73
6 Bayesian Program = Description + Question 75
6.1 Water treatment center Bayesianmodel (end) . . . . . . . . 76
6.2 Forward simulation of a single unit . . . . . . . . . . . . . . 76
6.2.1 Question . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.3 Forward simulation of the water treatment center . . . . . . 78
6.3.1 Question . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Contents ix
6.4 Control of the water treatment center . . . . . . . . . . . . 81
6.4.1 Question (1) . . . . . . . . . . . . . . . . . . . . . . . 81
6.4.2 Results (1) . . . . . . . . . . . . . . . . . . . . . . . . 81
6.4.3 Question (2) . . . . . . . . . . . . . . . . . . . . . . . 82
6.4.4 Results (2) . . . . . . . . . . . . . . . . . . . . . . . . 84
6.5 Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.5.1 Question . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.6 Lessons, comments, and notes . . . . . . . . . . . . . . . . . 87
6.6.1 BayesianProgram= Description + Question . . . . 87
6.6.2 The essence of Bayesianinference . . . . . . . . . . . 88
6.6.3 No inverse or direct problem . . . . . . . . . . . . . . 89
6.6.4 No ill-posed problem . . . . . . . . . . . . . . . . . . 89
II Bayesian Programming Cookbook 91
7 Information Fusion 93
7.1 “Naive” Bayes sensor fusion . . . . . . . . . . . . . . . . . . 94
7.1.1 Statement of the problem . . . . . . . . . . . . . . . 94
7.1.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 94
7.1.3 Instance and results . . . . . . . . . . . . . . . . . . . 96
7.2 Relaxing the conditional independence fundamental
hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.2.1 Statement of the problem . . . . . . . . . . . . . . . 102
7.2.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 103
7.2.3 Instance and results . . . . . . . . . . . . . . . . . . . 103
7.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.3.1 Statement of the problem . . . . . . . . . . . . . . . 105
7.3.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 106
7.3.3 Instance and results . . . . . . . . . . . . . . . . . . . 106
7.4 Ancillary clues . . . . . . . . . . . . . . . . . . . . . . . . . 108
7.4.1 Statement of the problem . . . . . . . . . . . . . . . 108
7.4.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 108
7.4.3 Instance and results . . . . . . . . . . . . . . . . . . . 110
7.5 Sensor fusion with false alarm . . . . . . . . . . . . . . . . . 113
7.5.1 Statement of the problem . . . . . . . . . . . . . . . 113
7.5.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 114
7.5.3 Instance and results . . . . . . . . . . . . . . . . . . . 114
7.6 Inverse programming . . . . . . . . . . . . . . . . . . . . . . 116
7.6.1 Statement of the problem . . . . . . . . . . . . . . . 116
7.6.2 Bayesianprogram . . . . . . . . . . . . . . . . . . . . 117
7.6.3 Instance and results . . . . . . . . . . . . . . . . . . . 118