Review of (Temporally Homogeneous) Markov Chains

Notations and Definitions

Markov Chain

A Markov chain is a stochastic process X = { X t , t ∈ T } X=\{X_t,t\in\mathcal{T}\} X={Xt,tT}. Suppose the state space S \mathcal{S} S is countable. The Markov chain has the following property:
p r ( X n + 1 = j ∣ X 0 = i 0 , ⋯   , X n − 1 = i n − 1 , X n = i ) = p r ( X n + 1 = j ∣ X n = i ) = p i j pr(X_{n+1}=j|X_{0}=i_0,\cdots,X_{n-1}=i_{n-1},X_n=i)=pr(X_{n+1}=j|X_n=i)=p_{ij} pr(Xn+1=jX0=i0,,Xn1=in1,Xn=i)=pr(Xn+1=jXn=i)=pijfor i , j , i n − 1 , ⋯   , i 0 ∈ S i,j,i_{n-1},\cdots,i_0\in\mathcal{S} i,j,in1,,i0S. This equation reveals two things:

  1. Given the present space, the future is conditionally independent of the past.
  2. The transition probabilities p i j p_{ij} pij do not depend on time n n n. Markov chain is therefore time-homogeneous.

Transition Matrix

Transition matrix is defined to be P = ( p i j ) S × S \mathsf{P}=(p_{ij})_{\mathcal{S}\times \mathcal{S}} P=(pij)S×S. The transition properties satisfy ∑ j ∈ S p i j = 1 \sum_{j\in\mathcal{S}}p_{ij}=1 jSpij=1, i.e., each row of P \mathsf{P} P adds up to 1.
Let p i j n p^n_{ij} pijn be the ( i , j ) (i,j) (i,j)-th element of matrix P n \mathsf{P}^n Pn. Then by the definition of matrix multiplication
p i j n = ∑ i 1 , … , i n − 1 ∈ S n − 1 p i , i 1 p i 1 , i 2 ⋯ p i n − 1 , j . p_{ij}^n=\sum_{i_{1}, \ldots, i_{n-1} \in S^{n-1}} p_{i, i_{1}} p_{i_{1}, i_{2}} \cdots p_{i_{n-1}, j}. pijn=i1,,in1Sn1pi,i1pi1,i2pin1,j. Note that p r ( X n + 2 = j ∣ X n = i ) = ∑ z p r ( X n + 2 = j ∣ X n + 1 = z , X n = i ) p r ( X n + 1 = z ∣ X n = i ) = ∑ z p r ( X n + 2 = j ∣ X n + 1 = z ) p r ( X n + 1 = z ∣ X n = i ) = ∑ z p z j p i z pr(X_{n+2}=j|X_n=i)=\sum_{z}pr(X_{n+2}=j|X_{n+1}=z,X_n=i)pr(X_{n+1}=z|X_n=i)=\sum_{z}pr(X_{n+2}=j|X_{n+1}=z)pr(X_{n+1}=z|X_n=i)=\sum_{z}p_{zj}p_{iz} pr(Xn+2=jXn=i)=zpr(Xn+2=jXn+1=z,Xn=i)pr(Xn+1=zXn=i)=zpr(Xn+2=jXn+1=z)pr(Xn+1=zXn=i)=zpzjpiz. Similarly, we find out that p r ( X n = j ∣ X 0 = i ) = p i j n pr(X_n=j|X_0=i)=p^n_{ij} pr(Xn=jX0=i)=pijn. Let α = ( p r ( X 0 = i ) ) i ∈ S \alpha=(pr(X_0=i))_{i\in\mathcal{S}} α=(pr(X0=i))iS, a row vector of sum equal to 1, be the initial distribution, we have
p r ( X n = j ) = ( α P n ) j . pr(X_n=j)=(\alpha\mathsf{P}^n)_j. pr(Xn=j)=(αPn)j.As a consequence,
p r ( X n = j ∣ X m = i ) = p i j n − m . pr(X_n=j|X_m=i)=p_{ij}^{n-m}. pr(Xn=jXm=i)=pijnm.In the following we also use the notation P i ( X n = j ) = p i j n . P_i(X_n=j)=p^n_{ij}. Pi(Xn=j)=pijn.

Chapman-Kolmogorov Equations

p i j m + n = ∑ z ∈ S p i z m p z j n , p^{m+n}_{ij}=\sum_{z\in\mathcal{S}}p^m_{iz}p^n_{zj}, pijm+n=zSpizmpzjn,i.e.,
P x ( X n + m = z ) = ∑ y P x ( X m = y ) P y ( X n = z ) . P_x(X_{n+m}=z)=\sum_{y}P_x(X_{m}=y)P_{y}(X_{n}=z). Px(Xn+m=z)=yPx(Xm=y)Py(Xn=z).For a general state space, we have
π k , n ( x , A ) = ∫ X π k , m ( x , d y ) π m , n ( y , A ) . \pi_{k,n}(x,A)=\int_{\mathcal{X}}\pi_{k,m}(x,dy)\pi_{m,n}(y,A). πk,n(x,A)=Xπk,m(x,dy)πm,n(y,A).for any k < m < n k<m<n k<m<n.

Hitting Time is Stopping Time

Hitting time, τ = min ⁡ { n ≥ 1 : X n ∈ A } \tau=\min\{n \geq 1: X_n\in A\} τ=min{n1:XnA}, is a stopping time because by def. { τ = n } = { X k ∉ A , k < n , X n ∈ A } \{\tau=n\}=\{X_k\notin A,k<n,X_n\in A\} {τ=n}={Xk/A,k<n,XnA}.

The σ \sigma σ-field associated to a stopping time

F τ = { A : A ∈ F ∞  and  A ∩ { τ ≤ n } ∈ F n  for each  n } \mathcal{F}_{\tau}=\left\{A: A \in \mathcal{F}^{\infty} \quad \text { and } A \cap\{\tau \leq n\} \in \mathcal{F}_{n} \quad \text { for each } n\right\} Fτ={A:AF and A{τn}Fn for each n}One can check that F τ \mathcal{F}_{\tau} Fτ is a σ \sigma σ-field and τ ∈ F τ \tau\in\mathcal{F}_{\tau} τFτ.

Strong Markov Property

P x ( X τ + 1 ∈ A 1 , ⋯   , X τ + n ∈ A n ∣ F τ ) = P X τ ( X 1 ∈ A 1 , ⋯   , X n ∈ A n ) P_x(X_{\tau+1}\in A_1,\cdots, X_{\tau+n}\in A_n|\mathcal{F}_{\tau})=P_{X_{\tau}}(X_1\in A_1,\cdots, X_n\in A_n) Px(Xτ+1A1,,Xτ+nAnFτ)=PXτ(X1A1,,XnAn)The strong Markov property says, loosely speaking, that a Markov chain degenerates, or starts anew, at a stopping time.

Recurrence and Transience

Define τ y \tau_y τy be the the first recurrence time to state y y y and τ y n = min ⁡ { n > τ y n − 1 , X n = y } \tau^n_y=\min\{n>\tau^{n-1}_y,X_n=y\} τyn=min{n>τyn1,Xn=y} be the n-th recurrence time to state y y y. Let ρ x y = P x ( τ y < ∞ ) \rho_{xy}=P_x(\tau_y<\infty) ρxy=Px(τy<). The state y y y is said to be recurrent if ρ y y = 1 \rho_{yy}=1 ρyy=1 and transient if ρ y y < 1 \rho_{yy}<1 ρyy<1.

IFF Condition for Recurrence

Let N y = ∑ i = 1 ∞ 1 X i = y N_y=\sum_{i=1}^\infty 1_{X_i=y} Ny=i=11Xi=y be the number of visits to y y y. Then
E x [ N y ] = ∑ k = 1 ∞ p r ( N y ≥ k ) = ∑ k = 1 ∞ P x ( τ y k < ∞ ) = ∑ k = 1 ∞ ρ x y ρ y y k − 1 = ρ x y 1 − ρ y y . E_x[N_y]=\sum_{k=1}^\infty pr(N_y\geq k)=\sum_{k=1}^\infty P_x(\tau_y^k<\infty)=\sum_{k=1}^\infty \rho_{xy}\rho_{yy}^{k-1}=\frac{\rho_{xy}}{1-\rho_{yy}}. Ex[Ny]=k=1pr(Nyk)=k=1Px(τyk<)=k=1ρxyρyyk1=1ρyyρxy.It is easy to check that
E y [ N y ] = ∑ k = 1 ∞ ρ y y k = ρ y y 1 − ρ y y . E_y[N_y]=\sum_{k=1}^\infty \rho_{yy}^k=\frac{\rho_{yy}}{1-\rho_{yy}}. Ey[Ny]=k=1ρyyk=1ρyyρyy.
If y y y is transient, we have E x N y < ∞ E_xN_y<\infty ExNy<. Hence, we have the following theorem:
y y y is transient IFF E y [ N y ] = ∞ E_y[N_y]=\infty Ey[Ny]=.

Accessible and Communicate

State x x x is said to be accessible from state y y y if ρ x y > 0 \rho_{xy}>0 ρxy>0. State x x x is said to communicate with state y y y if x x x and y y y is accessible to each other.

Closed and Irreducible

A set C C C is closed if x ∈ C x\in C xC and ρ x y > 0 \rho_{xy}>0 ρxy>0 implies y ∈ C y\in C yC. C C C is irreducible if x , y ∈ C x,y\in C x,yC implies ρ x y > 0 \rho_{xy}>0 ρxy>0 (All states communicate with each other). C C C is recurrent if all states are reccurent.

Irreducible implies Recurrent

Let C C C be a finite closed set. Then C C C contains a recurrent state. Moreover, if C C C is irreducible, then C C C must be recurrent.

Stationary Measures

A measure μ \mu μ is a stationary measure IFF ∑ x μ x p x y = μ y \sum_{x}\mu_xp_{xy}=\mu_y xμxpxy=μy, i.e., μ P = μ \mu\mathsf{P}=\mu μP=μ. If μ \mu μ is additionally a probability measure, we say μ \mu μ is a stationary distribution.

A Proposition

Let x x x be a recurrent state. Then
μ y = ∑ n = 0 ∞ P x ( X n = y , τ x > n ) \mu_y=\sum_{n=0}^\infty P_x(X_n=y,\tau_x>n) μy=n=0Px(Xn=y,τx>n)defines a stationary measure.

Uniqueness of Stationary Measure

If a Markov chain p p p is irreducible and recurrent then the stationary measure is unique up to constant multiples.

Proposition II

If there is a stationary distribution then all states y y y that have π y \pi_y πy > 0 are recurrent.

Proposition III

If p p p is irreducible and has a stationary distribution π \pi π, then π x = 1 E x τ x \pi_x=\frac{1}{E_x\tau_x} πx=Exτx1.

Positive Reccurent and Null Recurrent

If E x τ x < ∞ E_x\tau_x<\infty Exτx<, x x x is positive recurrent. Otherwise it is null recurrent.

Equivalence under Irreducible

If p p p is irreducible, then the following are equivalent:

  1. Some x x x is positive recurrent.
  2. There is a stationary distribution.
  3. All states are positive recurrent.

Proposition IV

If p is irreducible and has a stationary distribution π \pi π, then any other stationary measure is a multiple of π \pi π.

Decomposition Property

The state space has the unique representations S = ∪ i C i ∪ T \mathcal{S}=\cup_i C_i\cup T S=iCiT where T T T is transient and C i C_i Ci are closed irriducible recurrent.

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值