Oracle R Enterprise中大型数据集的回归

有关ORACLE R的详细资料参考：www.oracle.com/goto/R

 

来自 这篇< R 中大型数据集的回归>
http://cos.name/2013/08/regression-of-large-dataset-in-r/
我用Oracle R Enterprise改写一下 .也来解决 R客户端内存不足的问题(只是无法用到并行这个特性)

#生成数据
set.seed(123);
n = 5000000;
p = 5;
system.time(x <- matrix(rnorm(n * p), n, p));
x = cbind(1, x);
bet = c(2, rep(1, p));
y = c(x %*% bet) + rnorm(n);
gc();
dat = as.data.frame(x);
rm(x);
gc();
dat$y = y;
rm(y);
gc();
colnames(dat) = c(paste("x", 0:p, sep = ""), "y");
gc();

将数据保存到Oracle数据库中

#在此之前已经用ORE( oracle R enterprise连接上一个数据库),登录 方式如下
#ore.connect(user,sid,host,password,port,all=TRUE) 

#将数据保存到Oracle数据库中
#将dat数据框的数据保存在Oracle数据库中表名为oracle_tab_dat;
system.time(ore.create(dat,table="ORACLE_TAB_DAT"))

 

(1)用Oracle 的ore.lm函数做回归,(注因为oracle_tab_dat是oracle中的表,所以要用ore.lm函数,而不用常规的lm函数)

   system.time( mod <-  ore.lm(  y ~ .,  ORACLE_TAB_DAT))

以下是输出

 

ORE> system.time(mod <- ore.lm( y ~., ORACLE_TAB_DAT))
user system elapsed
0.263 0.007 50.474

ORE> summary(mod)

Call:
ore.lm(formula = y ~ ., data = ORACLE_TAB_DAT)

Residuals:
Min 1Q Median 3Q Max
-101.320 -6.283 -0.106 6.088 113.854 

Coefficients: (1 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.361706 0.005354 7351 < 2e-16 ***
x0 NA NA NA NA
x1 19.681085 0.005354 3676 < 2e-16 ***
x2 19.678239 0.005355 3675 < 2e-16 ***
x3 19.680580 0.005351 3678 < 2e-16 ***
x4 23.861695 0.005354 4457 < 2e-16 ***
x5 23.936075 0.005352 4473 4.44e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Residual standard error: 11.97 on 4999994 degrees of freedom
Multiple R-squared: 0.9414, Adjusted R-squared: 0.9414
F-statistic: 1.606e+07 on 5 and 4999994 DF, p-value: < 2.2e-16 

ORE> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 753819 40.3 1166886 62.4 1073225 57.4
Vcells 920709 7.1 1598044 12.2 1125030 8.6
ORE> ls()
[1] "mod"
ORE>

可以发现mod对象当用40MB的空间

(2)用biglm函数调用Oracle中的ORACLE_TAB_DAT

#运行ore.tableApply函数
modList <- ore.tableApply(
   X=ORACLE_TAB_DAT,
   function(dat) {
     library(biglm)
     bigmod <- biglm(y ~ x0+x1+x2+x3+x4+x5, dat)
     summary(bigmod)
   }); 

#将结果保存在R客户端本地
modList_local <- ore.pull(modList)

 

#在执行上面ore.tableApply函数时,oracle会在数据库中创建一个类似ORE$185_117,用来保存结果

ORE> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 742514 39.7 1166886 62.4 899071 48.1
Vcells 908294 7.0 1598044 12.2 1125030 8.6
ORE> modList_local <- ore.pull(modList)
ORE> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 742894 39.7 1590760 85.0 1026996 54.9
Vcells 70907371 541.0 143830781 1097.4 141748327 1081.5

 

发现modList_local占用本地内存近500Mb,而ORE$185_117表的大小有近600MB

ORE> modList_local$mat
Coef (95% CI) SE p
(Intercept) 39.36171 39.35100 39.37241 0.005354317 0
x0 NA NA NA NA NA
x1 19.68108 19.67038 19.69179 0.005353924 0
x2 19.67824 19.66753 19.68895 0.005354583 0
x3 19.68058 19.66988 19.69128 0.005351436 0
x4 23.86169 23.85099 23.87240 0.005353987 0
x5 23.93607 23.92537 23.94678 0.005351736 0
ORE> modList_local$rsq
[1] 0.9413915
ORE> 

 

~~HAVE FUN~~~