Selected Review from Mathematical Statistics

Selected Review from Mathematical Statistics

The Major section of this brief review list is modified or extracted based on the online material posted by Penn State University Eberly College of Science and notes from Mathematical Statistics and Data Analysis textbook by John A. Rice, University of California, Berkeley.

Note:
This post is for personal review only! If you wanna forward this post for other usage, please ask the editor/author/organization listed above.

1. Discrete Random Variable

1. 1 Definition:

A discrete random variable is a random variable that can take on only a finite or at most a countably infinite number of values.

1. 2 Probability mass function (pmf)

For discrete random variable, if there is a function $p$ such that $p(x_i)=P(X=x_i)$ and ${\sum }_{i}p(x_i)=1$, then this function is called the probability mass function, or the frequency function of the random variable $X$.

1. 3 Cumulative distribution function (cdf)

Cumulative distribution function $cdf$ of a random variable, which is defined to be
$$F(x)=P(X\le x),-\infty <x<\infty$$The cumulative distribution function is non-decreasing and satisfies:
$$\underset{x\to -\infty }{\lim }F(x)=0and\underset{x\to \infty }{\lim }F(x)=1$$

1. 4 Independent Variable

In the case of two discrete random variables $X$ and $Y$, taking on possible values $x_1,x_2,\dots ,$ and $y_1,y_2,\dots$, $X$ and $Y$ are said to be independent if, for all $i$ and $j$:
$$P(X=x_{i}andY=y_j)=P(X=x_i)P(Y=y_j)$$The definition is extended to collections of more than two discrete random variablesin the obvious way; for example, $X$, $Y$, and $Z$ are said to be mutually independent if, for all $i$, $j$, and $k$:
$$P(X=x_i,Y=y_j,Z=z_k)=P(X=x_i)P(Y=y_j)P(Z=z_k)$$

1. 5 Discrete Random Variable Distribution

1. 5. 1 Bernoulli Distribution

A Bernoulli random variable takes on only two values: 0 and 1, with probabilities $ 1-p $ and $p$, respectively. Its frequency function is thus
$$\begin{array}{rl}p(1)=& p\\ p(0)=& 1-p\\ p(x)=& 0,ifx\ne 0andx\ne 1\end{array}$$An alternative and sometimes useful representation of this function is
$$p(x)=\{\begin{array}{lcl}{p}^x(1-p{)}^{1-x}& & ifx=0orx=1\\ 0& & otherwise\end{array}$$ If $A$ is an event, then the indicator random variable, $I_A$, takes on the value 1 if $A$ occurs and the value 0 if $A$ does not occur:
$$I_A(\omega )=\{\begin{array}{lcl}1,& & if\omega \in A\\ 0,& & otherwise\end{array}$$ $I_A$ is a Bernoulli random variable. In applications, Bernoulli random variables often occur as indicators. A Bernoulli random variable might take on the value 1 or 0 according to whether a guess was a success or a failure.

1. 5. 2 Binomial Distribution

Suppose that $n$ independent experiments, or trials, are performed, where $n$ is a fixed number, and that each experiment results in a “success” with probability $p$ and a “failure” with probability $1-p$. The total number of successes, $X$, is a binomial random variable with parameters $n$ and $p$.

The probability that $X=k$, or $p(k)$, can be found in the following way: Any particular sequence of k successes occurs with probability $p^k(1-p{)}^{n-k}$ , from the multiplication principle. The total number of such sequences is $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ , since there are $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ ways to assign $k$ successes to $n$ trials. $P(X=k)$ is thus the probability of any particular sequence times the number of such sequences:
$$P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$$A random variable with a binomial distribution can be expressed in terms of independent Bernoulli random variables.

Specifically, let $X_1,X_2,\dots ,X_n$ be independent Bernoulli random variables with $p(X_i=1)=p$. Then $Y=X_1+X_2+\cdots +X_n$ is a binomial random variable.
Besides, a Bernoulli random variable can also be considered as a special case of a binomial random variable with $n=1$.

1. 5. 3 Geometric Distribution

The geometric distribution is also constructed from independent Bernoulli trials, but from an infinite sequence. On each trial, a success occurs with probability $p$, and $X$ is the total number of trials up to and including the first success. So that $X=k$, there must be $k-1$ failures followed by a success. From the independence of the trials, this occurs with probability
$$p(k)=P(X=k)=(1-p{)}^{k-1}p,k=1,2,3,\dots$$Note that these probabilities sum to 1:
$$\sum _{k=1}^{\infty }(1-p{)}^{k-1}p=p\sum _{j=0}^{\infty }(1-p{)}^j=1$$

1. 5. 4 Negative Binomial Distribution

The negative binomial distribution arises as a generalization of the geometric distribution. Suppose that a sequence of independent trials, each with probability of success $p$, is performed until there are $r$ successes in all; let $X$ denote the total number of trials. To find $P(X=k)$, we can argue in the following way: Any particular such sequence has probability $p^r(1-p{)}^{k-r}$, from the independence assumption. The last trial is a success, and the remaining $r-1$ successes can be assigned to the remaining $k-1$ trials $\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)$ in ways. Thus,
$$P(X=k)=\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)p^r(1-p{)}^{k-r}$$A negative binomial random variable can be expressed as the sum of $r$ independent geometric random variables: the number of trials up to and including the first success plus the number of trials after the first success up to and including the second success, . . . plus the number of trials from the $(r-1)$st success up to and including the $r$th success.

1. 5. 5 Hypergeometric Distribution

Suppose that an urn contains $n$ balls, of which $r$ are black and $n-r$ are white. Let $X$ denote the number of black balls drawn when taking $m$ balls without replacement.
$$P(X=k)=\frac{\left(\genfrac{}{}{0ex}{}{r}{k}\right)\left(\genfrac{}{}{0ex}{}{n-r}{m-k}\right)}{\left(\genfrac{}{}{0ex}{}{n}{m}\right)}$$ $X$ is a hypergeometric random variable with parameters $r$, $n$, and $m$.

1. 5. 6 Poisson Distribution

The Poisson frequency function with parameter $\lambda (\lambda >0)$ is
$$P(X=k)=\frac{{\lambda }^k}{k!}{e}^{-\lambda },k=0,1,2,\dots$$ Since $e^{\lambda }={\sum }_{k=0}^{\infty }\frac{{\lambda }^k}{k!}$, it follows that the frequency function sums to 1. The shape varies as a function of $\lambda$.
The Poisson distribution can be derived as the limit of a binomial distribution as the number of trials, $n$, approaches infinity and the probability of success on each trial, $p$, approaches zero in such a way that $np=\lambda$. The binomial frequency function is
$$p(k)=\frac{n!}{k!(n-k)!}{p}^k(1-p{)}^{n-k}$$Setting $np=\lambda$, this expression becomes
$$\begin{array}{rl}p(k)=& \frac{n!}{k!(n-k)!}(\frac{\lambda }{n}{)}^k(1-\frac{\lambda }{n}{)}^{n-k}\\ =& \frac{{\lambda }^k}{k!}\frac{n!}{(n-k)!}\frac{1}{n^k}(1-\frac{\lambda }{n}{)}^n(1-\frac{\lambda }{n}{)}^{-k}\end{array}$$As $$\begin{array}{rl}n\to & \infty \\ \frac{\lambda }{n}\to & 0\\ \frac{n!}{(n-k)!k!}\to & 1\\ (1-\frac{\lambda }{n}{)}^n\to & e^{-\lambda }\end{array}$$and$$(1-\frac{\lambda }{n}{)}^{-k}\to 1$$We thus have $$p(x=k)\to \frac{{\lambda }^k{e}^{-\lambda }}{k!}$$
which is the Poisson frequency function.

The Poisson distribution often arises from a model called a Poisson process for the distribution of random events in a set $S$, which is typically one-, two-, or three dimensional, corresponding to time, a plane, or a volume of space. Basically, this model states that if $S_1,S_2,\dots ,S_n$ are disjoint subsets of $S$, then the numbers of events in these subsets, $N_1,N_2,\dots ,N_n$, are independent random variables that follow Poisson distributions with parameters $\lambda |S_1|$, $\lambda |S_2|$, $\dots$ , $\lambda |S_n|$, where $\lambda |S_i|$ denotes the measure of $S_i$ (length, area, or volume, for example). The crucial assumptions here are that events in disjoint subsets are independent of each other and that the Poisson parameter for a subset is proportional to the subset’s size.

Keep Updating~
For personal review usage only! If you wanna forward this post for other usage, please ask the editor/author/organization listed at the beginning.