Sufficient Statistic (充分统计量)

Sufficient statistic - Wikipedia

Sufficient statistic - arizona

定义

统计量是一些随机样本$X_1,X_2,\cdots ,X_n$的函数
$$T=r(X_1,X_2,\cdots ,X_n).$$
样本$X$的分布$f_{\theta }(X)=f(X;\theta )$由位置参数$\theta$决定, 通常我们通过极大似然估计
$$\underset{\theta }{\max }P(X_1,X_2,\cdots ,X_n;\theta )=\prod _{i=1}^{n}P(X_i;\theta )=\prod _{i=1}^n{f}_{\theta }(X_i).$$
而充分统计量是指这样的统计量:
P ( { X i } ∣ T = t ; θ ) = P ( { X i } ∣ T = t ) , P(\{X_i\}|T=t;\theta) = P(\{X_i\}|T=t), P({Xi​}∣T=t;θ)=P({Xi​}∣T=t),
即在给定$T(X)=t$的情况下, { X i } \{X_i\} {Xi​}的条件联合分布与未知参数$\theta$无关.

Example: 考虑伯努利分布, 成功的概率为$p$, 失败的概率为$1-p$, 有$n$个独立同分布的样本$X_1,X_2,\cdots ,X_n$, 则:
P ( { X i } ; p ) = p ∑ i X i ( 1 − p ) n − ∑ i X i , P(\{X_i\};p) = p^{\sum_i X_i}(1-p)^{n-\sum_i X_i}, P({Xi​};p)=p∑i​Xi​(1−p)n−∑i​Xi​,
实际上(后面会讲到)$T={\sum }_i^n{X}_i$为其一充分统计量. 实际上,
P ( { X i } ∣ T = t ; p ) = P ( { X i } , T = t ; p ) P ( T = t ; p ) = I [ ∑ i n X i = t ] ⋅ p t ( 1 − p ) n − t C n t p t ( 1 − p ) n − t = I [ ∑ i n X i = t ] C n t . P(\{X_i\}|T=t;p) = \frac{P(\{X_i\}, T=t; p)}{P(T=t;p)} = \frac{\mathbb{I}[{\sum_{i}^nX_i=t]}\cdot p^t (1-p)^{n-t}}{C_n^t p^t (1-p)^{n-t}}=\frac{\mathbb{I}[\sum_i^n X_i = t]}{C_n^t}. P({Xi​}∣T=t;p)=P(T=t;p)P({Xi​},T=t;p)​=Cnt​pt(1−p)n−tI[∑in​Xi​=t]⋅pt(1−p)n−t​=Cnt​I[∑in​Xi​=t]​.
显然与位置参数$p$无关.

充分统计量特别的意义, 比如上面提到的极大似然估计, 由于
P ( { X i } ; θ ) = P ( { X i } , T ; θ ) = P ( { X i } ∣ T ; θ )   P ( T ; θ ) = P ( { X i } ∣ T )   P ( T ; θ ) , P(\{X_i\};\theta) = P(\{X_i\}, T;\theta) = P(\{X_i\}|T;\theta) \:P(T;\theta) = P(\{X_i\}|T) \:P(T;\theta), P({Xi​};θ)=P({Xi​},T;θ)=P({Xi​}∣T;θ)P(T;θ)=P({Xi​}∣T)P(T;θ),
由于 P ( { X i } ∣ T ) P(\{X_i\}|T) P({Xi​}∣T)与$\theta$无关, 所以最大化上式等价于
$$\underset{\theta }{\max }P(T;\theta )=P(r(X_1,X_2,\cdots ,X_n);\theta ).$$

特别地, 有时候标量$T$并不充分, 需要$T=(T_1,T_2,\cdots ,T_k)$ 整体作为充分统计量, 比如当正态分布地$\mu ,\sigma$均为未知参数的时候, $T=(\frac{1}{n}{\sum }_i{X}_i,\frac{1}{n-1}{\sum }_i(X_i-\bar{X}{)}^2)$. 性质和上面的别无二致, 所以下面也不特别说明了.

当置于贝叶斯框架下时, 可以发现:
P ( θ ∣ { X i } ) = P ( { X i } , θ ) P ( { X i } ) = P ( { X i } , T , θ ) P ( { X i } , T ) = P ( { X i } ∣ T , θ ) P ( T ∣ θ ) P ( { X i } , T ) = P ( { X i } ∣ T ) P ( T ∣ θ ) P ( { X i } , T ) = P ( θ ∣ T ) . P(\theta|\{X_i\}) = \frac{P(\{X_i\}, \theta)}{P(\{X_i\})} = \frac{P(\{X_i\}, T, \theta)}{P(\{X_i\}, T)} = \frac{P(\{X_i\}| T, \theta) P(T|\theta)}{P(\{X_i\}, T)} = \frac{P(\{X_i\}| T) P(T|\theta)}{P(\{X_i\}, T)} = P(\theta|T). P(θ∣{Xi​})=P({Xi​})P({Xi​},θ)​=P({Xi​},T)P({Xi​},T,θ)​=P({Xi​},T)P({Xi​}∣T,θ)P(T∣θ)​=P({Xi​},T)P({Xi​}∣T)P(T∣θ)​=P(θ∣T).
即给定 { X i } \{X_i\} {Xi​}或者$T$, $\theta$的条件(后验)分布是一致的.

特别地, 我们可以用互信息来定义充分统计量, $T$为充分统计量, 当且仅当
$$I(\theta ;X)=I(\theta ;T(X)).$$
注: 一般情况下$I(\theta ;X)\ge I(\theta ;T(X))$.

充分统计量的判定

用上面的标准来判断充分统计量是非常困难的一件事, 好在有Fisher-Neyman分离定理:

Factorization Theorem: { X i } \{X_i\} {Xi​}的联合密度函数为$f_{\theta }(X)$, 则$T$是关于$\theta$的充分统计量当且仅当存在非负函数$g,h$满足
$$f(X_1,X_2,\cdots ,X_n;\theta )=h(X_1,X_2,\cdots ,X_n)g(T;\theta ).$$
注: $T$可以是$T=(T_1,T_2,\cdots ,T_k)$.

proof:

$\Rightarrow$
p ( X 1 , X 2 , ⋯   , X n ; θ ) = p ( { X i } ∣ T ; θ ) = p ( { X i } ∣ T ; θ ) p ( T ; θ ) = p ( { X i } ∣ T ) p ( T ; θ ) p(X_1,X_2,\cdots, X_n;\theta) = p(\{X_i\}|T;\theta) = p(\{X_i\}|T;\theta)p(T;\theta) = p(\{X_i\}|T)p(T;\theta) p(X1​,X2​,⋯,Xn​;θ)=p({Xi​}∣T;θ)=p({Xi​}∣T;θ)p(T;θ)=p({Xi​}∣T)p(T;θ)
此时
g ( T ; θ ) = p ( T ; θ ) , h ( X 1 , X 2 , ⋯   , X n ) = p ( { X i } ∣ T ) . g(T;\theta) = p(T;\theta), \\ h(X_1, X_2,\cdots, X_n) = p(\{X_i\}|T). g(T;θ)=p(T;θ),h(X1​,X2​,⋯,Xn​)=p({Xi​}∣T).

$\Leftarrow$

为了符号简便, 令 X = { X 1 , X 2 , ⋯   , X n } X = \{X_1, X_2,\cdots, X_n\} X={X1​,X2​,⋯,Xn​}.
$$\begin{array}{ll}p(T=t;\theta )& ={\int }_{T(X)=t}p(X,T=t;\theta )\mathrm{d}X\\ & ={\int }_{T(X)=t}f(X;\theta )\mathrm{d}X\\ & ={\int }_{T(X)=t}h(X)g(T=t;\theta )\mathrm{d}X\\ & ={\int }_{T(X)=t}h(X)\mathrm{d}X\cdot g(T=t;\theta )\end{array}.$$
则
$$\begin{array}{ll}p(X|T=t;\theta )& =\frac{p(X,T=t;\theta )}{p(T=t;\theta )}\\ & =\frac{p(X;\theta )}{p(T=t;\theta )}\\ & =\frac{h(X)g(T=t;\theta )}{{\int }_{T(X)=t}h(X)\mathrm{d}X\cdot g(T=t;\theta )}\\ & =\frac{h(X)}{{\int }_{T(X)=t}h(X)}.\end{array}$$
与$\theta$无关.

注: 上述的证明存疑.

最小统计量

最小统计量S, 即

1. S是充分统计量;
2. 充分统计量$T$, 存在$f$, 使得$S=f(T)$.

注: 若$T$是充分统计量, 则任意的可逆函数$f$得到的$f(T)$也是充分统计量.

例子

$U[0,\theta ]$

均匀分布, 此时
p ( X 1 , X 2 , ⋯   , X n ; θ ) = 1 θ n I [ 0 ≤ min ⁡ { X i } ] ⋅ I [ max ⁡ { X i } ≤ θ ] , p(X_1, X_2,\cdots, X_n;\theta) = \frac{1}{\theta^n} \mathbb{I}[0\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta], p(X1​,X2​,⋯,Xn​;θ)=θn1​I[0≤min{Xi​}]⋅I[max{Xi​}≤θ],
故
T = max ⁡ { X i } ,   g ( T ; θ ) = I [ max ⁡ { X i } ⋅ 1 θ n ,   h ( X ) = I [ 0 ≤ min ⁡ { X i } ] . T = \max \{X_i\}, \: g(T;\theta) = \mathbb{I}[\max \{X_i\} \cdot \frac{1}{\theta^n}, \: h(X) = \mathbb{I}[0\le \min \{X_i\}]. T=max{Xi​},g(T;θ)=I[max{Xi​}⋅θn1​,h(X)=I[0≤min{Xi​}].

$U[\alpha ,\beta ]$

p ( X 1 , X 2 , ⋯   , X n ; α , β ) = 1 ( β − α ) n I [ α ≤ min ⁡ { X i } ] ⋅ I [ max ⁡ { X i } ≤ θ ] , p(X_1, X_2,\cdots, X_n;\alpha,\beta) = \frac{1}{(\beta - \alpha)^n} \mathbb{I}[\alpha\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta], p(X1​,X2​,⋯,Xn​;α,β)=(β−α)n1​I[α≤min{Xi​}]⋅I[max{Xi​}≤θ],

T = ( min ⁡ { X i } , max ⁡ { X i } ) , g ( T ; α , β ) = 1 ( β − α ) n I [ α ≤ min ⁡ { X i } ] ⋅ I [ max ⁡ { X i } ≤ θ ] , h ( X ) = 1. T = (\min \{X_i\}, \max \{X_i\}), \\ g(T;\alpha, \beta) = \frac{1}{(\beta - \alpha)^n} \mathbb{I}[\alpha\le \min \{X_i\}] \cdot \mathbb{I}[\max \{X_i\} \le \theta], \\ h(X) = 1. T=(min{Xi​},max{Xi​}),g(T;α,β)=(β−α)n1​I[α≤min{Xi​}]⋅I[max{Xi​}≤θ],h(X)=1.

Poisson

$$P(X;\lambda )=\frac{{\lambda }^X{e}^{-\lambda }}{X!}.$$

$$p(X_1,X_2,\cdots ,X_n;\lambda )=e^{-n\lambda }{\lambda }^{{\sum }_i{X}_i}\cdot \frac{1}{{\prod }_i{X}_i!}.$$

$$\begin{gathered} T=\sum _i{X}_i, \\ g(T;\theta )=e^{-n\lambda }\cdot {\lambda }^T, \\ h(X)=\frac{1}{{\prod }_i{X}_i!}. \end{gathered}$$

Normal

$$P(X;\mu ,\sigma )=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(X-\mu {)}^2}{2{\sigma }^2}).$$

$$p(X_1,X_2,\cdots ,X_n;\mu ,\sigma )=(2\pi {\sigma }^2{)}^{-\frac{n}{2}}\exp (-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(X_i-\bar{X}{)}^2)\exp (-\frac{n}{2{\sigma }^2})(\mu -\bar{X}{)}^2.$$
若$\sigma$已知:
$$\begin{gathered} T=\frac{1}{n}\sum X_i=\bar{X}, \\ g(T;\mu )=(2\pi {\sigma }^2{)}^{-\frac{n}{2}}\exp (-\frac{n}{2{\sigma }^2})(\mu -T{)}^2, \\ h(X)=\exp (-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(X_i-\bar{X}{)}^2). \end{gathered}$$

若$\sigma$未知:
$$\begin{gathered} T=(\bar{X},s^2),s^2=\frac{{\sum }_{i=1}^n(X_i-\bar{X}{)}^2}{n-1}, \\ g(T;\mu ,\sigma )=(2\pi {\sigma }^2{)}^{-\frac{n}{2}}\exp (-\frac{n-1}{2{\sigma }^2}{s}^2)\exp (-\frac{n}{2{\sigma }^2})(\mu -\bar{X}{)}^2, \\ h(X)=1. \end{gathered}$$

指数分布

$$p(X)=\frac{1}{\lambda }{e}^{-\frac{X}{\lambda }},X\ge 0.$$

$$p(X_1,X_2,\cdots ,X_n;\lambda )=\frac{1}{{\lambda }^n}{e}^{-\frac{{\sum }_{i=1}^n{X}_i}{\lambda }}.$$

$$\begin{gathered} T=\sum _{i=1}^n{X}_i, \\ g(T;\lambda )=\frac{1}{{\lambda }^n}{e}^{-\frac{T}{\lambda }}, \\ h(X)=1. \end{gathered}$$

Gamma

$$\Gamma (\alpha ,\beta )=\frac{1}{\Gamma (\alpha ){\beta }^{\alpha }}{X}^{\alpha -1}{e}^{-\frac{X}{\beta }}.$$

$$p(X_1,X_2,\cdots ,X_n;\alpha ,\beta )=\frac{1}{(\Gamma (\alpha ){\beta }^{\alpha }{)}^n}(\prod _i{X}_i{)}^{\alpha -1}{e}^{-\frac{{\sum }_i{X}_i}{\beta }}.$$

$$\begin{gathered} T=(\prod _i{X}_i,\sum _i{X}_i), \\ g(T;\theta )=\frac{1}{(\Gamma (\alpha ){\beta }^{\alpha }{)}^n}(\prod _i{X}_i{)}^{\alpha -1}{e}^{-\frac{{\sum }_i{X}_i}{\beta }}, \\ h(X)=1. \end{gathered}$$