数据挖掘（凑标题字数）

Chapter Two

Data dispersion characteristics

Center

Mean: $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$, $\mu =\frac{\sum x}{N}$
Weighted Mean: $\bar{x}=\frac{{\sum }_{i=1}^n{w}_i{x}_i}{{\sum }_{i=1}^n{w}_i}$

Median(for grouped data): $median=L_1+(\frac{n/2-(\sum freq)l}{fre{q}_{median}})width$

Mode: $mean-mode=3\times (mean-median)$
mean > median, positively skewed
mean < median, negatively skewed

Quartiles: $Q_1$(25th percentile), $Q_3$(75th percentile)
Inter-quartile range: $IQR=Q_3-Q_1$
Five number summary: min, $Q_1$, median, $Q_3$, max
Boxplot: ends of the box are the quartiles; median is marked; add whiskers, and plot outliers individually
Outlier: usually, a value higher/lower than 1.5 x IQR

Variance:
unbiased estimation: $s^2=\frac{1}{n-1}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2=\frac{1}{n-1}[{\sum }_{i=1}^n{x}_i^2-\frac{1}{n}({\sum }_{i=1}^n{x}_i{)}^2]$
biased estimation: ${\sigma }^2=\frac{1}{n}{\sum }_{i=1}^n(x_i-\mu {)}^2=\frac{1}{n}{\sum }_{i=1}^n{x}_i^2-{\mu }^2$

Pixel-Oriented Visualization Techniques

Similarity and Dissimilarity

	1	0	sum
1	q	r	q + r
0	s	t	s + t
sum	q + s	r + t	p

Distance measure for symmetric binary variables: $d(i,j)=\frac{r+s}{q+r+s+t}$
Distance measure for asymmetric binary variables: $d(i,j)=\frac{r+s}{q++r+s}$

Here, the asymmetric means the loss cost is different, for some data sets, like the FP is an absolute majority.

Jaccard coefficient (similarity measure for asymmetric binary variables): $si{m}_{Jaccard}(i,j)=\frac{q}{q+r+s}$

Minowski distance(L-h norm): $d(i,j)=\sqrt[h]{|x_{i1}-x_{j1}{|}^h+|x_{i2}-x_{j2}{|}^h+\cdots +|x_{ip}-x_{jp}{|}^h}$

Properties:

1. $d(i,j)>0$ if $i\ne j$ and $d(i,i)=0$ (Positive definiteness)
2. $d(i,j)=d(j,i)$ (Symmetry)
3. $d(i,j)⩽d(i,k)+d(k,j)$ (Triangle Inequality)

A distance that satifies these properties is a metric.

$h=1:$ Mabhattan distance $d(i,j)=|x_{i1}-x_{j1}|+|x_{i2}-x_{j2}|+\cdots +|x_{ip}-x_{jp}|$
$h=2:$ Euclidean distance $d(i,j)=\sqrt{|x_{i1}-x_{j1}{|}^2+|x_{i2}-x_{j2}{|}^2+\cdots +|x_{ip}-x_{jp}{|}^2}$
$h\to \infty :$ supernum distance $d(i,j)=li{m}_{h\to \infty }({\sum }_{f=1}^p|x_{if}-x_{jf}|{)}^{\frac{1}{h}}=ma{x}_f^p|x_{if}-x_{jf}|$

Ordinal Variables: $z_{if}=\frac{r_{if}-1}{M_f-1}$

$d(i,j)=\frac{{\sum }_{f=1}^p{\delta }_{ij}^{(f)}{d}_{ij}^{(f)}}{{\sum }_{f=1}^p{\delta }_{ij}^{(f)}}$

$cos(d_1,d_2)=\frac{d_1\cdot d_2}{||d_1||\cdot ||d_2||}$ to evaluate the similarity of sentences.

Chapter Three

Data Processing

Data cleaning, Data integration, Data reduction, Data transformation and data discretization.

${\chi }^2$(chi-square) test

${\chi }^2=\sum \frac{(Observerd-Expected{)}^2}{Excepted}$
The larger the Χ2 value, the more likely the variables are related.

Correlation coefficient(Pearson’s product moment coefficient)

$r_{A,B}=\frac{{\sum }_{i=1}^n(a-\bar{A})(b-\bar{B})}{(n-1){\sigma }_A{\sigma }_B}=\frac{{\sum }_{i=1}^n(a_i{b}_i)-n\bar{A}\bar{B}}{(n-1){\sigma }_A{\sigma }_B}$

$r_{A,B}>0$ means A and B are positively correlated.

Let ${a_k}^{\prime }=(a_k-mean(A))/std(A),{b_k}^{\prime }=(b_k-mean(B))/std(B)$,
then $correlation(A,B)=A^{\prime }\cdot B^{\prime }$

Covariance

$Cov(A,B)=E((A-\bar{A})(B-\bar{B}))=\frac{{\sum }_{i=1}^n(a_i-\bar{A})(b-\bar{B})}{n}$
$r_{A,B}=\frac{Cov(A,B)}{{\sigma }_A{\sigma }_B}$
$Cov(A,B)=E((A-\bar{A})(B-\bar{B}))=E(A\cdot B)-\bar{A}\bar{B}$

Data reduction

Unsupervised:

1. Latent Semantic Indexing (LSI): truncated SVD
2. Principal Component Analysis (PCA)
3. Independent Component Analysis (ICA)
4. Canonical Correlation Analysis (CCA)

Supervised:

1. Linear Discriminant Analysis (LDA)

Semi-supervised:

1. Semi-supervised Discriminant Analysis (SDA)

Linear:

1. Latent Semantic Indexing (LSI): truncated SVD
2. Principal Component Analysis (PCA)
3. Linear Discriminant Analysis (LDA)
4. Canonical Correlation Analysis (CCA)

Nonlinear:

1. Nonlinear feature reduction using kernels
2. Manifold learning

Dimensionality reduction (Feature reduction):

1. Feature extraction
2. Feature selection

Selection: choose a best subset of size d from the available p features.
Extraction: given p features (set X), extract d new features (set Z) by linear or non-linear combination of all the p features.

PCA

Given { x 1 , . . . , x n } ∈ R p \{x_1, ..., x_n\} \in \mathbb{R}^p {x1​,...,xn​}∈Rp, target: get the $a$ to maxmize the $var(z)$, here $z=ax$

$$\begin{array}{rl}var(z)& =E((z-\bar{z}{)}^2)\\ & =\frac{1}{n}\sum _{i=1}^n(a{x}_i-a\bar{x}{)}^2\\ & =\frac{1}{n}\sum _{i=1}^n{a}^T(x_i-\bar{x})(x_i-\bar{x}{)}^{T}a\\ & =a^{T}Sa\\ S& =\frac{1}{n}\sum _{i=1}^n(x_i-\bar{x})(x_i-\bar{x}{)}^T\end{array}$$

which means $ma{x}_a{a}^{T}Sa,s.t.a^{T}a=1$.
We use Lagrange method to solve the problem.

$$\begin{array}{r}L=a^{T}Sa-\lambda (a^{T}a-1)\\ \frac{\partial L}{\partial a}=2Sa-2\lambda a=0\end{array}$$

So, $\lambda$ and $a$ is the pair of eigenvalue and eigenvector of S. Then $var(z)=a^T\lambda a=\lambda$. So the lambda is chosen from large to small.

Next, $ma{x}_{a_2}{a}_2^{T}S{a}_2,s.t.a_2^T{a}_2=1,cov(z^{(2)},z^{(1)})=0$ if we want another PCA.
$cov(z^{(2)},z^{(1)})=a_2^{T}S{a}_1=\lambda a_2^T{a}_1$, so(I don’t know) $S{a}_2=\lambda a_2$, and the $\lambda$ is the second largest eigenvalue.
Dimension reduction: $\chi \in {\mathbb{R}}^{p\times n}\to A^T\chi \in {\mathbb{R}}^{d\times n}$
Original data(Reconstruction): $A^T\chi \in {\mathbb{R}}^{d\times n}\to \bar{X}=A(A^{T}X)\in {\mathbb{R}}^{p\times n}$

Main theoretical result:
The matrix A consisting of the first d eigenvectors of the covariance matrix S solves the following optimization problem
$mi{n}_{A\in {\mathbb{R}}^{p\times d}}||\chi -A{A}^{T}X|{|}_F^2,s.t.A^{T}A=I_d$

LDA(Linear Discriminant Analysis)

Find a transformation a, such that the a^TX_1 and a^TX_2 are maximally separated & each class is minimally dispersed (maximum separation).

$max(a(\overline{x_1}-\overline{x_2}){)}^2,minvar(z_1),minvar(z_2)$
target: $maxJ=\frac{(a(\overline{x_1}-\overline{x_2}){)}^2}{var(z_1)+var(z_2)}$

Suppose there exists two class $w_1,w_2$
$z=a^{T}x$
$\widetilde{{\mu }_i}=\frac{1}{n_i}{\sum }_{z\in w_i}z$
${\mu }_i=\frac{1}{n_i}{\sum }_{x\in w_i}x,\widetilde{{\mu }_i}=a^T{\mu }_i$
$|\widetilde{{\mu }_1}-\widetilde{{\mu }_2}|=|a^T({\mu }_1-{\mu }_2)|$
${\widetilde{s_i}}^2={\sum }_{z\in w_i}(z-\widetilde{{\mu }_i}{)}^2$
$J(a)=\frac{(\widetilde{{\mu }_1}-\widetilde{{\mu }_2}{)}^2}{{\widetilde{s_1}}^2+{\widetilde{s_2}}^2}$

${\widetilde{s_i}}^2={\sum }_{y\in w_i}(y-\widetilde{{\mu }_i}{)}^2={\sum }_{x\in w_i}(a^{T}x-a^T{\mu }_i{)}^2={\sum }_{x\in w_i}(a^{T}x-a^T{\mu }_i)(a^{T}x-a^T{\mu }_i{)}^T={\sum }_{x\in w_i}{a}^T(x-{\mu }_i)(x-{\mu }_i{)}^{T}a=a^T{S}_{i}a$

within-in class scatter matrix: $S_W=S_1+S_2,{\widetilde{s_1}}^2+{\widetilde{s_2}}^2=a^T{S}_{W}a$

$(\widetilde{{\mu }_1}-\widetilde{{\mu }_2}{)}^2=(a^T{\mu }_1-a^T{\mu }_2{)}^2=a^T({\mu }_1-{\mu }_2)({\mu }_1-{\mu }_2{)}^{T}a=a^T{S}_{B}a$

between-class scatter matrix: $S_B=({\mu }_1-{\mu }_2)({\mu }_1-{\mu }_2{)}^T$

$J(a)=\frac{a^T{S}_{B}a}{a^T{S}_{W}a}$
$S_{B}a=\lambda S_{W}a$
$S_W^{-1}{S}_{B}a=\lambda a$

Chapter Four

FP mining

itemset: A set of one or more items
k-itemset X = { x 1 , … , x k } X = \{x_1, …, x_k\} X={x1​,…,xk​}
(absolute) support, or, support count of X: Frequency or occurrence of an itemset $X$;
(relative) support, s, is the fraction of transactions that contains $X$ (i.e., the probability that a transaction contains $X$).
An itemset $X$ is frequent if $X$’s support is no less than a minsup threshold.

Find all the rules $X\to Y$ with minimum support and confidence.
support, s, probability that a transaction contains $X\cup Y$;
confidence, c, conditional probability that a transaction having $X$ also contains $Y$.

closed-patterns and max-patterns
An itemset $X$ is closed if X is frequent and there exists no super-pattern $Y\supset X$, with the same support as $X$;
An itemset $X$ is a max-pattern if $X$ is frequent and there exists no frequent super-pattern $Y\supset X$.

So a max-pattern is a closed-pattern.

Apriori

An important property: ** Any subset of a frequent itemset must be frequent**.
Apriori pruning principle: If there is any itemset which is infrequent, its superset should not be generated/tested!

Method:

1. Initially, scan DB once to get frequent 1-itemset;
2. Generate length (k+1) candidate itemsets from length k frequent itemsets;
3. Test the candidates against DB;
4. Terminate when no frequent or candidate set can be generated;

Example:

Pseudo-code:
Ck: Candidate itemset of size k
Lk : frequent itemset of size k

L1 = {frequent items};
for (k = 1; Lk !=null; k++) do begin
    Ck+1 = candidates generated from Lk;
    for each transaction t in database do
  increment the count of all candidates in Ck+1 that are contained in t
    Lk+1  = candidates in Ck+1 with min_support
    end
return union(Lk);

Major computational challenges:

1. Multiple scans of transaction database
2. Huge number of candidates
3. Tedious workload of support counting for candidates

Improving Apriori: general ideas:

1. Reduce passes of transaction database scans
2. Shrink number of candidates
3. Facilitate support counting of candidates

FP-growth

Here, we link an website, it[step 3, pages 28] says that “Recursively mine conditional FP‐trees and grow frequent patterns obtained so far. If the conditional FP‐tree contains a single path, simply enumerate all the patterns”.

Mining sequential patterns

sequential patterns:

GSP

Chapter Five

Decision Tree
Bayes Classification Methods
Support Vector Machines

Decision Tree

It is derivated in the aspect of propability. We can calculate the propability of every output with the given input. If we assume every condition is independent, then $P(X|C)=\prod P(X_i|C)$, then $logP(X|C)=\sum logP(X_i|C)$, so we let the cost Function be $log$. To understand better, we can use the concept of thermodynamics, which is called entropy.

$H(Y)=-{\sum }_{i=1}^m{p}_{i}log(p_i)$ where $p_i=P(Y=y_i)$
$H(Y|X)=-{\sum }_{x}p(x)H(Y|X=x)$

$Info(D)=-{\sum }_{i=1}^m{p}_{i}lo{g}_2(p_i)$
$Inf{o}_A(D)=-{\sum }_{j=1}^v\frac{|D_j|}{|D|}\times Info(D_j)$
$Gain(A)=Info(D)-Inf{o}_A(D)$

Bayes Classification Methods

First, we know that $P(B)={\sum }_{i=1}^{M}P(B|A_i)P(A_i)$, and $P(H|X)=\frac{P(X|H)P(H)}{P(X)}$
Assume all condition is independent, then $P(X|C_i)={\prod }_{k=1}^{n}P(x_k|C_i)$

Naïve Bayesian prediction requires each conditional prob. be non-zero. Otherwise, the predicted prob. will be zero.
Use Laplacian correction:

1. Adding 1 to each case
2. The “corrected” prob. estimates are close to their “uncorrected” counterparts

Support Vector Machines

SVM原理-1
SVM原理-2

Model Evaluation and Selection

Confusion Matrix:

Actual class/ Predicted class	$C_1$	$\neg C_1$
$C_1$	True Positive(TP)	False Negative(FN)
$\neg C_1$	False Positive(FP)	True Negative(TN)

Accuracy: $\frac{TP+TN}{ALL}$
Error rate: $\frac{FP+FN}{ALL}$
Sensitivity: $\frac{TP}{P}$
Specificity: $\frac{TN}{N}$

Precision: $\frac{TP}{TP+FP}$
Recall: $\frac{TP}{TP+FN}$
F measure: $\frac{2\times Precision\times Recall}{Precision+Recall}$
F-beta measure: $\frac{(1+{\beta }^2)\times Precision\times Recall}{\beta \times Precision+Recall}$

Holdout method

Cross-validation

Bootstrap

Estimating Confidence Intervals

t-test

ROC curves

Chapter Six

K-means

K-medoids

choose the closet point of the K-means center.

Q