Applied Stochastic Processes

205 Pages·2013·7.844 MB·\205

by  Ming Liao (Author)

Applied
Stochastic
Processes
K20401_FM.indd 1 6/19/13 12:20 PM
K20401_FM.indd 2 6/19/13 12:20 PM
Applied
Stochastic
Processes
Ming Liao
Auburn University
Alabama, USA
K20401_FM.indd 3 6/19/13 12:20 PM
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does
not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of MAT-
LAB® software or related products does not constitute endorsement or sponsorship by The MathWorks
of a particular pedagogical approach or particular use of the MATLAB® software.
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20130614
International Standard Book Number-13: 978-1-4665-8934-6 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copy-
right.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222
Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that pro-
vides licenses and registration for a variety of users. For organizations that have been granted a pho-
tocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
i i
i i
Contents
1 Probability and stochastic processes 1
1.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Random variables and their distributions . . . . . . . . 3
1.3 Mathematical expectation . . . . . . . . . . . . . . . . . 6
1.4 Joint distribution and independence . . . . . . . . . . . 8
1.5 Convergence of random variables . . . . . . . . . . . . . 11
1.6 Laplace transform and generating functions . . . . . . . 12
1.7 Examples of discrete distributions . . . . . . . . . . . . 14
1.8 Examples of continuous distributions . . . . . . . . . . 15
1.9 Stochastic processes . . . . . . . . . . . . . . . . . . . . 17
1.10 Stopping times . . . . . . . . . . . . . . . . . . . . . . . 19
1.11 Conditional expectation . . . . . . . . . . . . . . . . . . 21
2 Poisson processes 29
2.1 Introduction to Poisson processes . . . . . . . . . . . . 29
2.2 Arrival and inter-arrival times of Poisson processes . . . 33
2.3 Conditional distribution of arrival times . . . . . . . . . 36
2.4 Poisson processes with different types of events . . . . . 39
2.5 Compound Poisson processes . . . . . . . . . . . . . . . 41
2.6 Nonhomogeneous Poisson processes . . . . . . . . . . . 43
3 Renewal processes 47
3.1 An introduction to renewal processes . . . . . . . . . . 47
3.2 Renewal reward processes . . . . . . . . . . . . . . . . . 51
3.3 Queuing systems . . . . . . . . . . . . . . . . . . . . . . 55
3.4 Queue lengths, waiting times, and busy periods . . . . 60
3.5 Renewal equation . . . . . . . . . . . . . . . . . . . . . 66
3.6 Key renewal theorem . . . . . . . . . . . . . . . . . . . 71
3.7 Regenerative processes . . . . . . . . . . . . . . . . . . 77
3.8 Queue length distribution and PASTA . . . . . . . . . . 79
v
i i
i i
i i
i i
vi
4 Discrete time Markov chains 87
4.1 Markov property and transition probabilities . . . . . . 87
4.2 Examples of discrete time Markov chains . . . . . . . . 91
4.3 Multi-step transition and reaching probabilities . . . . . 94
4.4 Classes, recurrence, and transience . . . . . . . . . . . . 97
4.5 Periodicity, class property, and positive recurrence . . . 103
4.6 Expected hitting time and hitting probability . . . . . . 106
4.7 Stationary distribution . . . . . . . . . . . . . . . . . . 110
4.8 Limiting properties . . . . . . . . . . . . . . . . . . . . 117
5 Continuous time Markov chain 123
5.1 Markov property and transition probability . . . . . . . 123
5.2 Transition rates . . . . . . . . . . . . . . . . . . . . . . 125
5.3 Stationary distribution and limiting properties . . . . . 130
5.4 Birth and death processes . . . . . . . . . . . . . . . . . 138
5.5 Exponential queuing systems . . . . . . . . . . . . . . . 141
5.6 Time reversibility . . . . . . . . . . . . . . . . . . . . . 149
5.7 Hitting time and phase-type distributions . . . . . . . . 155
5.8 Queuing systems with time-varying rates . . . . . . . . 159
6 Brownian motion and beyond 165
6.1 Brownian motion . . . . . . . . . . . . . . . . . . . . . 165
6.2 Standard Brownian motion and its maximum . . . . . . 169
6.3 Conditional expectation and martingales . . . . . . . . 172
6.4 Brownian motion with drift . . . . . . . . . . . . . . . . 175
6.5 Stochastic integrals . . . . . . . . . . . . . . . . . . . . 178
6.6 Itˆo’s formula and stochastic differential equations . . . 182
6.7 A single stock market model . . . . . . . . . . . . . . . 186
Bibliography 195
Index 197
i i
i i
i i
i i
Preface
The purpose of these notes is to present a concise account of applied
stochastic processes as usually covered in a first-year graduate course,
emphasizing applications and practical computation, but also devel-
oping an essentially complete mathematical theory. The topics, after
a quick review of basic probability, include Poisson processes, renewal
processes, discrete time and continuous time Markov chains, Brownian
motion, and an introduction to stochastic differential equations. The
main applications are queues, but other examples are also considered,
including the mathematical model of a single stock market. We have
tried to present the materials in a quick to-the-point fashion, possibly
with some motivation and short examples, but without much digres-
sion. The mathematical theory is developed with strong emphasis on
probability intuition, often informally, to be easily accessible, but al-
ways based on sound reasoning. Most sections end with a few closely
related exercises. The solutions to all exercises are included in a sepa-
rate solutions manual.
The short bibliography contains the books and papers used in the
preparation of the notes. It is not intended to be comprehensive, and
so does not include many other good books on applied stochastic pro-
cesses.
Studentsofmystochasticprocessclassesenduredtheearlyversions
ofthenotes,andprovidedusefulinputs.Iwishtothanktheanonymous
reviewers for the helpful comments and my colleague Erkan Nane for
providing a list of errors. A lot of effort has been put into these notes,
and my wife’s support and understanding are always important for me
to complete such a project.
MATLAB and Simulink are registered trademarks of The Math-
Works, Inc. For product information, please contact:
The MathWorks, Inc.
3 Apple Hill Drive
Natick, MA 01760-2098 USA
Tel: [phone]
vii
i i
i i
i i
i i
viii
Fax: [phone]
E-mail: [email protected]
Web: www.mathworks.com
i i
i i
i i
i i
Chapter 1
Probability and stochastic processes
The first eight sections summarize basic concepts and facts in proba-
bility theory with little explanation and very few proofs. The reader
may consult a standard graduate text on probability, such as [2], for
more details. The stochastic processes, stopping times, and conditional
expectation are introduced in the last three sections.
1.1 Probability
Sample space and events: The set of all possible outcomes is called
the sample space and is denoted by Ω. A collection of subsets of Ω,
including Ω itself and the empty set ∅, are called events. This will be
defined more precisely in the note on the measure theory later, but a
reader who is not familiar with the measure theory may just assume
any subset of Ω is an event. We will assume the reader is familiar with
the basic set notation and operations such as the membership ∈, the
inclusion ⊂, the union ∪, the intersection ∩, and the complement Ec
of an event E.
Probability: The probability P(E) is a function defined for every
event E, which satisfies the following three basic axioms (or rules):
(a) 0 ≤ P(E) ≤ 1.
(b) P(Ω) = 1 and P(∅) = 0.
∑
(c) P(∪∞ E ) = ∞ P(E ) for any sequence of disjoint events
n=1 n n=1 n
E ;E ;E ;:::,
1 2 3
Property (c) is called the countable additivity of the probability as
a sequence of objects is called a countable set in mathematics.
The sample space Ω together with a probability P is called a prob-
1
i i
i i