Data Mining Week2 - L2 ROC方法二元分

预测准确性:confusion matrix 

	预估阳性	预估阴性
实际阳性	真阳性	假阴性 Type2	阳性总数
实际阴性	假阳性Type1	真阴性	阴性总数
	预估阳性总数	预估阴性总数	总数N

• FP: “false alarm”, “type I error”
• FN: “miss”, “type II error”
• Accuracy = (TP+TN)/N = [Pos*TP rate + Neg*(1- FP rate)] / (Pos+Neg)

• Sensitivity(aka Recall) = TP/Pos = TP rate
• Specificity = TN/Neg = TN rate 在阴性的情况下，我们identify它的概率
• Precision = Positive Predictive Value = TP/(TP+FP)预测是阳性的情形下，有多少是真的阳性
• False Discovery Rate = FP/(TP+FP)预测是阳性的情形下,错了的概率

• TP also called “hit”, TN: “correct rejection”

• TP rate = TP/Pos = TP/(TP+FN)

• FN rate = FN/Pos = 1 – TP rate //真阳和假阴都是actual pos

• FP rate = FP/Neg= FP/(TN+FP)

• TN rate = TN/Neg= 1 – FP rate

(= % of correct prediction = 真阳和真阴预测rate的加权平均)

• Sensitivity = Recall = TP/Pos = TP rate在阳性的情况下，我们identify它的概率

通常失败的预测（FN（误诊错过时机）和FP（吓坏病人））的代价是不同的，同时现实中阳性和阴性的概率也一般不一样。

所以classification的目的最大化预测的准确性同时最小化误判的预期代价

Receiver Operating Characteristic (ROC) approach

二元Classification exp:根据血压判断健康

Threshold 的值同时影响了TP和FP，T越大假阳性越小，代价就是TP也会随着减小。给定一个TP就会有一个FP，也就能画出一个二维图。

RO Curve

越向左上角就说明classification越完美，对角线是随机分的情况，因为TP和FP都是0.5

右上角是永远说P，这样只要是P就是对的，只要是N就是错的；

左下角相反。

选择ROC图

不同的评估方法会得到不同的ROC，如果是：

傻子都知道A好。

如果是：就要考虑报假警和错过病情的cost

成本期望 E(cost)= E(P)*(1-TPr)*Cost(FN)+E(N)*FPr*Cost(FP)

E(P)是Pos/N，E(N)类似
TPr = TP/Pos = TP/(TP+FN)  //有病情况下错过的概率 

FPr = FP/ Neg = FP/(TN+FP) //没病的情况下误诊的概率

在总数里P的比例 X 有病情况下错过的概率 X 代价 

+ 在总数里N的比例 X 没病情况下误诊的概率 X 代价

Iso-performance line is a line in the ROC space such that all points on the line give the same profit/loss.

两个点(FPr1,TPr1) and (FPr2,TPr2) 在一个ISO-P线中则 ：

斜率 a = (TPr2 - TPr1)/(FPr2 -FPr1) 

= (E(N)/E(P))*(Cost(FN)/Cost(FP))

所以根据EN和EP的比率（Pos/Neg的赔率）与FN成本和FP成本比率的乘积可以使用不同的斜率a

举例：乳癌的诊断有两种方式，透视和复查

透视适用于大规模基本阴性，复查适合阳性概率大一些的人群

A就适合于透视，以为斜率大于1很多，保证E（N）很大的情况下FP很少；B就适合复查，不会错过病情。

Constructing ROC graphs for rating classifiers

Here is how to construct an empirical ROC for a rating classifier:
1. 对training样本做一个分类估计得到估计值（估计值0-1，越高代表越pos）
2. 按估计值降序排列
3. 画出ROC 坐标，从（0,0）开始，到（Neg,Pos ）也就是说每一格其实是1/Neg或1/Pos

4. 取出2排列表的最高值（估计最为pos的):
如果该预估的真值是pos，往上画一格到（1,0）

反之，划到（0,1）
5. 如果有一堆一样的估计值，则计数pos和neg的数量，画到长宽（neg，pos）矩形对角线

6. 画完为止

Package ‘ROCR’ for r

Description 

ROC graphs, sensitivity/specificity curves, lift charts,and precision/recall plots are popular examples of trade-off visualizations for specific pairs of performance measures. 

ROCR is a flexible tool for creating cutoff-parameterized 2D performance curves by freely combining two from over 25 performance measures (new performance measures can be added using a standard interface). 

Curves from different cross-validation or bootstrapping runs can be averaged by different methods, and standard deviations, standard errors or box plots can be used to visualize the variability across the runs. 

The parameterization can be visualized by printing cutoff values at the corresponding curve positions, or by coloring the curve according to cutoff. 

All components of a performance plot can be quickly adjusted using a flexible parameter dispatching mechanism. Despite its flexibility, 

performance(prediction.obj, measure, x.measure="cutoff", ...)

prediction.obj An object of class prediction.
measure Performance measure to use for the evaluation. 

A complete list of the performance measures that are available for measure and x.measure is given inthe ’Details’ section.
x.measure A second performance measure. If different from the default, a two-dimensional  curve, withx.measure taken to be the unit in direction of the x axis, and  measure to be the unit in directionof the y axis, is created. This curve is  parametrized with the cutoff.
... Optional arguments (specific to individual performance measures).

Here is how to call ’performance’ to create some standard evaluation plots:
ROC curves: measure="tpr", x.measure="fpr".
Precision/recall graphs: measure="prec", x.measure="rec".
Sensitivity/specificity plots: measure="sens", x.measure="spec".
Lift charts: measure="lift", x.measure="rpp".

Examples
## computing a simple ROC curve (x-axis: fpr, y-axis: tpr)
library(ROCR)
data(ROCR.simple)
pred <- prediction( ROCR.simple$predictions, ROCR.simple$labels)
perf <- performance(pred,"tpr","fpr")
plot(perf)
## precision/recall curve (x-axis: recall, y-axis: precision)
perf1 <- performance(pred, "prec", "rec")
plot(perf1)
## sensitivity/specificity curve (x-axis: specificity,y-axis: sensitivity)
perf1 <- performance(pred, "sens", "spec")
plot(perf1)

Slots
x.name: Performance measure used for the x axis.
y.name: Performance measure used for the y axis.
x.values: A list in which each entry contains the x values of the curve of this particular crossvalidation run. x.values[[i]], y.values[[i]], and alpha.values[[i]] correspond to each other.
y.values: A list in which each entry contains the y values of the curve of this particular crossvalidation run.
alpha.values: A list in which each entry contains the cutoff values of the curve of this particular cross-validation run.

alpha.name: Name of the unit that is used to create the parametrized curve. Currently, curves can only be parametrized by cutoff, so alpha.name is either none or cutoff.