2021-03-02

Moment Generating Function

Definition

For a random variable $X$, if $E[e^{tx}]$ exists $\forall t\in (-h,h)$ for some $h>0$, moment generating function $M(t)=E[e^{tx}]$

Properties

$X$ is a random variable with MGF $M_x(t)$ that exists $\forall t\in (-h,h)$ where $h>0$,

• $Y=aX+b$ where $a,b\in \mathbb{R}$ and $a\ne 0$. Then, $M_y(t)=e^{bt}{M}_x(at)\forall t\in (-\frac{h}{|a|},\frac{h}{|a|})$
• $M(0)=1$ ,and
• $M^{(k)}(0)=E[x^k]$ where $M^{(k)}(t)=\frac{d^k}{d{t}^k}M(t)$ for $k=1,2,...$
• Uniqueness: $M_x(t)=M_y(t)\forall t\in (-h,h)⟺X$ and $Y$ have the same distribution

Multivariate Random Function

Definition

Suppose $X$ and $Y$ are $r.v$ defined on the sample space $S$,

• Joint $CDF$ of $X$ and $Y$:
  $F(x,y)=P(\underset{intersectionof2events}{\underbrace{X\le x,Y\le y}})\forall (x,y)\in {\mathbb{R}}^2$
• Marginal $CDF$ of $X$ and $Y$: (holds for both discrete and continuous $r.v.$)
  $F_1(x)/F_x(x)=\underset{y\to \infty }{\lim }F(x,y)=P(X\le x),\forall x\in \mathbb{R}$
  $F_2(y)/F_y(y)=\underset{x\to \infty }{\lim }F(x,y)=P(Y\le y),\forall y\in \mathbb{R}$
• Marginal $PDF$ of $X$ and $Y$:
  • For discrete $r.v.$,
    • $f_x(x)=P(X=x)=\sum _{ally}f(x,y)\forall x\in \mathbb{R}$
    • $f_y(y)=P(Y=y)=\sum _{allx}f(x,y)\forall y\in \mathbb{R}$
  • For continuous $r.v.$,
    • $f_x(x)={\int }_{-\infty }^{\infty }f(x,y)dy\forall x\in \mathbb{R}$
    • $f_y(y)={\int }_{-\infty }^{\infty }f(x,y)dx\forall y\in \mathbb{R}$

Independence

Recall: Independent events are $P(A\cap B)=P(A)P(B)$

Theorem

$X$ ans $Y$ are $r.v.$

1. If joint CDF $F(x,y)$, marginal CDFs $F_1(x)$ and $F_2(y)$, then $X$ and $Y$ are independent $⟺F(x,y)=F_1(x)F_2(y)\forall (x,y)\in {\mathbb{R}}^2$
2. If joint $pdf/pmf$ $f(x,y)$ and marginal $pdfs/pmfs$ $f_1(x)$ and $f_2(y)$ and A 1 = { s : f 1 ( x ) > 0 } A_1=\{s:f_1(x)>0\} A1​={s:f1​(x)>0}, A 2 = { y : f 2 ( y ) > 0 } A_2=\{y:f_2(y)>0\} A2​={y:f2​(y)>0}, $X$ and $Y$ are independent $⟺f(x,y)=f_1(x)f_2(y)\forall (x,y)\in (A_1\times A_2)$
3. (factorization theorem of independence) If joint $pdf/pmf$ $f(x,y)$ and A 1 = { s : f 1 ( x ) > 0 } A_1=\{s:f_1(x)>0\} A1​={s:f1​(x)>0}, A 2 = { y : f 2 ( y ) > 0 } A_2=\{y:f_2(y)>0\} A2​={y:f2​(y)>0}, $X$ and $Y$ are independent $⟺\exists g(y)\ge 0$ such that $f(x,y)=g(x)h(y)\forall (x,y)\in (A_1\times A_2)$
4. If support $A$ is not rectangular, then $X$ and $Y$ must be dependent $i.e.$
   $\exists (x,y)s.t.x\in A_1,y\in A_2,but(x,y)\notin Ai.e.$
   $f_1(x)>0,f_2(y)>0,f(x,y)=0$
5. if $X$ and $Y$ are independent random variables, then if $g,h$ are functions, then $g(x)$ and $h(y)$ are independent. (the reverse is false)

Joint Expectations

Definition

• Suppose $X$ and $Y$ are bivariate discrete random variables and $h(x,y)$ is a real-valued function. Then, if $\sum _{(x,y)\in A}|h(x,y)|f(x,y)<\infty$, then $E[h(x,y)]=\sum _{(x,y)\in A}h(x,y)f(x,y)$. Otherwise, $E[h(x,y)]$ DNE.
• Suppose $X$ and $Y$ are bivariate continuous random variables and $h(x,y)$ is a real-valued function. Then, if ${\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }|h(x,y)|f(x,y)dxdy<\infty ,E[h(x,y)]={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }h(x,y)f(x,y)dxdy$. Otherwise, $E[h(x,y]$ DNE.

Properties

1. Linearity
2. If $X$ and $Y$ are independent, then $\forall functiong,h,E[g(X)h(Y)]=E[g(X)]E[h(Y)]$

Covariance

$Cov(X,Y)=E[(x-{\mu }_x)(y-{\mu }_y)],where{\mu }_x=E[X],{\mu }_y=E[Y]$
$Cov(X,Y)=0$, we say $X$ and $Y$ are uncorrelated

Properties

1. $Cov(X,Y)=E[XY]-E[X]E[Y]$
2. $X$ and $Y$ are independent $⟹Cov(X,Y)=0$
3. $Cov(X,X)=Var(X)$
4. $Var(aX+bY)=a^{2}Var(X)+b^{2}Var(Y)+2abCov(X,Y)$
5. $Var[{\sum }_{i=1}^n{a}_i{X}_i]={\sum }_{i=1}^n{a}_i^{2}Var(X_i)+{\sum }_{i\ne j}{a}_i{a}_{j}Cov(X_i,X_j)$
6. $Cov(X+Y,Z)=Cov(X,Y)+Cov(Y,Z)$

Correlation Coefficient

$$\rho (x,y)=\frac{Cov(X,Y)}{\sqrt{Var(X)}\sqrt{Var(Y)}}$$
$-1<\rho <1$ measures the strength of linear relationship between $X$ and $Y$

Conditional Distribution

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)},givenP(B)>0$$
Suppose $X$ and $Y$ are bivariate random variables with joint pmf $f(x,y)$, then

1. conditional pmf/pdf of $X$ given $Y=y$ is
   $$f_1(x\mid y)=\frac{f(x,y)}{f_2(y)},given{f}_2(y)>0$$
2. conditional pmf/pdf of $Y$ given $X=x$ is
   $$f_2(y\mid x)=\frac{f(x,y)}{f_1(x)},given{f}_1(x)>0$$

Properties

$X$ and $Y$ are random variables with marginal pdf/pmf $f_1(x)$ and $f_2(y)$ and marginal support $A_1$ and $A_2$

1. $X$ and $Y$ are independent $⟺f_1(x\mid y)=f_1(x)\forall x\in A_1\vee f_2(y\mid x)=f_2(y)\forall y\in A_2$
2. Product Rule: $f(x,y)=f_1(x\mid y)f_2(y)=f_2(y\mid x)f_1(x)$

Conditional Expectations

Function $g$, the conditional expectation of $g(Y)$ given $X=x$ is:
$$E[g(Y)\mid X=x]=\{\begin{array}{ll}\sum _{ally}g(y)f_x(y\mid x)& \text{if Y is discrete r.v.}\\ {\int }_{-\infty }^{\infty }g(y)f_2(y\mid x)dy& \text{if Y is continuous r.v.}\end{array},unless\sum _{ally}g(y)f_x(y\mid x)/{\int }_{-\infty }^{\infty }g(y)f_2(y\mid x)dy\text{ does not converge, in which case}E[g(Y)\mid X=x]DNE$$

• $g(Y)=Y$, $E[Y\mid X=x]$ is called the conditional mean
• $g(y)=(y-E[Y\mid X=x]{)}^2⟹\begin{array}{rl}Var[Y\mid X=x]& =E[(Y-E[Y\mid X=x]{)}^2\mid X=x]\\ & =E[Y^2\mid X=x]-(E[y\mid X=x]{)}^2\end{array}$ is called conditional variance

Properties

1. Independence: If $X$ and $Y$ are independent, $\forall g(\cdot )andh(\cdot ),E[g(x)\mid Y=y]=E[g(x)]$ and $E[h(y)\mid X=x]=E[h(y)]$
2. Substitution Rule: $E[h(X,Y)\mid X=x]=E[h(x,Y)\mid X=x]$
3. Double Expectation: $E[G(Y)]=E[E[g(Y)\mid X]]$
4. $Var[Y]=E[\underset{functionh(x)=Var[Y\mid X=x]appliedtor.v.X}{\underbrace{Var(Y\mid X)}}]+Var[E[Y|X]]$
   1. $E[Var(Y|X)]$ :
      • find $Var(Y|X)$
        • figure out the expression for $h(x)=Var(Y|X)$
        • substitute $X$ for big $x$ in that expression
      • calculate $E[h(X)]$
   2. $Var[E[Y|X]]$:
      • find $E[Y|X]$
      • calculate $Var[\stackrel{\text{h˜(x)}}{\overbrace{E[Y|X]}}]$

Joint Moment Generating Function(MGF)

$X$ and $Y$ are random variables. If $E[e^{t_{1}x+t_{2}y}]$ exists $\forall t_1\in (-h_1,h_1)$ and $t_2\in (-h_2,h_2)$ for $h_1,h_2>0$
Then, $M(t_1,t_2)=E[e^{t_{1}x+t_{2}y}]\forall t_1,t_2$such that $E[\cdot ]$ exists is called the joint MGF

Marginal joint MGF

Given $M(t_1,t_2)$

• $M_x(t_1)=M(t_1,0)=E[e^{t_{1}x+0y}]=E[e^{t_{1}x}]$
• $M_y(t_2)=M(0,t_2)=E[e^{0x+t_{2}y}]=E[e^{t_{2}y}]$

Properties

$X$ and $Y$ are random variables with joint MGF $M(t_1,t_2)$. Then,
$X$ and $Y$ are independent $⟺M(t_1,t_2)=M_x(t_1)M_y(t_2)$

Multinomial Distribution

$$\begin{gathered} (X_1,X_2,...,X_k)\sim Multinomial(n;p_1,p_2,...,p_k)\text{ is discrete random variable with joint pmf} \\ f(x_1,x_2,...,x_k)=\{\begin{array}{ll}\frac{n!}{x_1!x_2!...x_k!}{p}_1^{x_1}{p}_2^{x_2}...p_k^{x_k}& x_i=0,1,...,n,i=1,2,...,k,{\sum }_{i=1}^k{x}_1=n\\ 0& o.w.\end{array} \\ for0<p_i<1,\sum _{i=1}^k{p}_i=1 \end{gathered}$$
Concrete example: k boxes randomly pick one of the k boxes, probability of picking the $i^{th}$ box being $p_i$. Repeat picking n times independently. $X_i$ is number of times box $i$ was picked, $i=1,2,...,k$

Properties

$(X_1,X_2,...,X_k)\sim Multinomial(n;p_1,p_2,...,p_k)$

1. Joint MGF $M(t_1,t_2,...,t_k)=E[e^{t_1{x}_1+t_2{x}_2+...+t_k{x}_k}]=(p_1{e}^{t_1}+...+p_k{e}^{t_k}{)}^n\forall (t_1,t_2,...,t_k)\in {\mathbb{R}}^k$
2. $X_i\sim Binomial(n,p_i)fori=1,2,...,k$
3. $T=X_i+X_j(i\ne j)⟹T\sim Binomial(n,p_i+p_j)$
4. $$\begin{gathered} E[X_i]=n{p}_i \\ Var[X_i]=n{p}_i(1-p_i) \\ Cov(X_i,X_j)=-n{p}_i{p}_j(i\ne j) \end{gathered}$$
5. $X_i\mid X_j=x_j\sim Bin(n-x_j,\frac{p_i}{1-p_j}),i\ne j$
6. $X_i\mid X_i+X_j=t\sim Bin(t,\frac{p_i}{p_i+p_j}),i\ne j$

Bivariate Normal Distribution

x → = ( x 1 x 2 ) ∼ B V N ( μ → , Σ )   w h e r e   X 1   a n d   X 2  are continutous random variables with joint pdf  f ( x 1 , x 2 ) = 1 2 π ∣ Σ ∣ 1 / 2 e x p { − ( x − μ ) Σ − 1 ( x − μ ) T 2 } , w h e r e   x = ( x 1 x 2 ) , μ = ( μ 1 μ 2 ) , Σ = ( σ 1 2 ρ σ 1 σ 2 ρ σ 1 σ 2 σ 2 2 ) i s   p o s i t i v e   d e f i n i t e \overrightarrow{x}=\begin{pmatrix}x_1\\x_2\end{pmatrix}\sim BVN(\overrightarrow{\mu},\Sigma)\ where\space X_1\space and\space X_2 \text{ are continutous random variables with joint pdf }\\f(x_1,x_2)=\frac{1}{2\pi |\Sigma|^{1/2}}exp\{\frac{-(x-\mu)\Sigma^{-1}(x-\mu)^T}{2}\},\\ where\ x=\begin{pmatrix}x_1\\x_2\end{pmatrix},\mu=\begin{pmatrix}\mu_1\\ \mu_2\end{pmatrix},\Sigma=\begin{pmatrix}\sigma_1^2 & \rho\sigma_1\sigma_2\\ \rho\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix} is\ positive\ definite x =(x1​x2​​)∼BVN(μ ​,Σ) where X1​ and X2​ are continutous random variables with joint pdf f(x1​,x2​)=2π∣Σ∣1/21​exp{2−(x−μ)Σ−1(x−μ)T​},where x=(x1​x2​​),μ=(μ1​μ2​​),Σ=(σ12​ρσ1​σ2​​ρσ1​σ2​σ22​​)is positive definite

Properties

1. $X_1,X_2$ has joint MGF M ( t 1 , t 2 ) = E [ e t 1 x 1 + t 2 x 2 ] = e x p { t T μ + 1 2 t T Σ t } ∀ t = ( t 1 t 2 ) ∈ R 2 M(t_1,t_2)=E[e^{t_1x_1+t_2x_2}]=exp\{t^T\mu+\frac{1}{2}t^T\Sigma t\}\forall t=\begin{pmatrix}t_1\\t_2\end{pmatrix}\in\R^2 M(t1​,t2​)=E[et1​x1​+t2​x2​]=exp{tTμ+21​tTΣt}∀t=(t1​t2​​)∈R2
2. M x 1 ( t ) = M ( t , 0 ) = e x p { t 1 μ 1 + 1 2 t 1 2 σ 1 2 } → X 1 ∼ N ( μ , σ 1 2 ) M x 2 ( t ) = M ( 0 , t ) = e x p { t 2 μ 2 + 1 2 t 2 2 σ 2 2 } → X 2 ∼ N ( μ , σ 2 2 ) M_{x_1}(t)=M(t,0)=exp\{t_1\mu_1+\frac{1}{2}t_1^2\sigma_1^2\}\to X_1\sim N(\mu,\sigma_1^2)\\M_{x_2}(t)=M(0,t)=exp\{t_2\mu_2+\frac{1}{2}t_2^2\sigma_2^2\}\to X_2\sim N(\mu,\sigma_2^2) Mx1​​(t)=M(t,0)=exp{t1​μ1​+21​t12​σ12​}→X1​∼N(μ,σ12​)Mx2​​(t)=M(0,t)=exp{t2​μ2​+21​t22​σ22​}→X2​∼N(μ,σ22​)
3. $Cov(X_1,X_2)=\rho {\sigma }_1{\sigma }_2$
4. $\rho =0⟺X_{1}and{X}_2$ are independent and correlated
5. $c=\left(\begin{array}{c}{c}_1\\ c_2\end{array}\right)\ne 0,c^{T}X=c_1{x}_1+c_2{x}_2\sim N({\mu }^{T}c,c^T\Sigma c)$
6. Nonsingular matrix $A\in {\mathbb{R}}^{2\times 2},b\in {\mathbb{R}}^2,thenAX+b\sim BVN(A\mu ,A\Sigma A^T)$
7. $((x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\sim {\chi }^2$