基于前面几篇的准备工作,我们已经具备了初始模型、流式向量训练数据、流式向量预测数据,如下图蓝色节点所示。接下来,我们会进入该系列文章的关键时刻,演示如何接入FTRL在线训练模块及对应的在线预测模块。
FTRL在线模型训练的代码如下,在FtrlTrainStreamOp的构造函数中输入初始模型initModel,随后是设置各种参数,并“连接“流式向量训练数据。
# ftrl train
model = FtrlTrainStreamOp(initModel) \
.setVectorCol(vecColName) \
.setLabelCol(labelColName) \
.setWithIntercept(True) \
.setAlpha(0.1) \
.setBeta(0.1) \
.setL1(0.01) \
.setL2(0.01) \
.setTimeInterval(10) \
.setVectorSize(numHashFeatures) \
.linkFrom(feature_pipelineModel.transform(train_stream_data))
FTRL在线预测的代码如下,需要“连接”FTRL在线模型训练输出的模型流,和流式向量预测数据。
# ftrl predict
predResult = FtrlPredictStreamOp(initModel) \
.setVectorCol(vecColName) \
.setPredictionCol("pred") \
.setReservedCols([labelColName]) \
.setPredictionDetailCol("details") \
.linkFrom(model, feature_pipelineModel.transform(test_stream_data))
我们可以如下设置流式结果的打印,由于数据较多,打印前先对流式数据进行采样。注意,对于流式的任务,print()方法不能触发流式任务的执行,必须调用StreamOperator.execute()方法,才能开始执行。
predResult.sample(0.0001).print();
StreamOperator.execute();
在执行的过程中,会先运行批式的初始模型训练,待批式任务执行结束,再启动流式任务。得到的结果显示片段如下,
......
click|pred|details
-----|----|-------
......
collect model : 27
collect model : 27
collect model : 27
collect model : 27
collect model : 27
collect model : 27
collect model : 27
collect model : 27
6 load model : 27
0|0|{"0":"0.9955912589178986","1":"0.0044087410821014306"}
3 load model : 27
0 load model : 27
2 load model : 27
5 load model : 27
4 load model : 27
7 load model : 27
1 load model : 27
0|0|{"0":"0.8264317979578765","1":"0.17356820204212353"}
0|0|{"0":"0.9620885206519035","1":"0.037911479348096466"}
0|0|{"0":"0.7733924667279566","1":"0.22660753327204342"}
0|0|{"0":"0.8502672431715895","1":"0.14973275682841047"}
0|0|{"0":"0.9422313239589072","1":"0.057768676041092815"}
0|0|{"0":"0.8540319447494245","1":"0.14596805525057555"}
1|0|{"0":"0.7956910587819983","1":"0.2043089412180017"}
collect model : 28
collect model : 28
collect model : 28
1 load model : 28
collect model : 28
7 load model : 28
collect model : 28
collect model : 28
6 load model : 28
4 load model : 28
collect model : 28
collect model : 28
5 load model : 28
0 load model : 28
3 load model : 28
2 load model : 28
0|0|{"0":"0.794857507111827","1":"0.205142492888173"}
0|0|{"0":"0.7489915122615897","1":"0.25100848773841034"}
0|0|{"0":"0.9145883964932835","1":"0.0854116035067165"}
0|0|{"0":"0.9699130297461115","1":"0.030086970253888512"}
0|0|{"0":"0.8633425927307238","1":"0.13665740726927622"}
1|0|{"0":"0.5067251707884466","1":"0.4932748292115534"}
0|0|{"0":"0.9197477679857682","1":"0.08025223201423182"}
0|0|{"0":"0.8754429175320314","1":"0.1245570824679686"}
0|0|{"0":"0.9027103601565077","1":"0.09728963984349226"}
0|0|{"0":"0.9396522264624441","1":"0.06034777353755594"}
0|0|{"0":"0.7870435722294925","1":"0.2129564277705075"}
......
在打印显示的文本中混合了两种信息,一种为各流式预测节点,加载新模型的情况,比如:"5 load model : 27",代表5号节点成功加载模型流中的第27号模型。另一种是预测结果数据,前面打印信息中的
click|pred|details
-----|----|-------
为打印的流式数据的列名信息,第1列是原始的“click”信息,第二列为预测结果列,第三列为预测的详细信息列。对应的预测结果形式为
0|0|{"0":"0.7870435722294925","1":"0.2129564277705075"}
最后,我们再将预测结果流predResult,接入流式二分类评估组件EvalBinaryClassStreamOp,并设置相应的参数,由于每次评估结果给出的是Json格式,为了便于显示,还可以在后面上Json内容提取组件JsonValueStreamOp。代码如下:
// ftrl eval
predResult
.link(
new EvalBinaryClassStreamOp()
.setLabelCol(labelColName)
.setPredictionCol("pred")
.setPredictionDetailCol("details")
.setTimeInterval(10)
)
.link(
new JsonValueStreamOp()
.setSelectedCol("Data")
.setReservedCols(new String[] {"Statistics"})
.setOutputCols(new String[] {"Accuracy", "AUC", "ConfusionMatrix"})
.setJsonPath(new String[] {"$.Accuracy", "$.AUC", "$.ConfusionMatrix"})
)
.print();
StreamOperator.execute();
注意:流式的组件“连接”完成后,需要调用流式任务执行命令,即StreamOperator.execute(),开始执行。显示结果如下:
......
Statistics|Accuracy|AUC|ConfusionMatrix
----------|--------|---|---------------
......
window|0.839781746031746|0.6196235914061319|[[140,174],[6609,35413]]
all|0.839781746031746|0.6196235914061319|[[140,174],[6609,35413]]
......
window|0.8396464236640808|0.6729843274019895|[[206,220],[14601,77400]]
all|0.8396889353902778|0.6559248735416315|[[346,394],[21210,112813]]
......
window|0.8389366017867161|0.7125709974947197|[[328,233],[14695,77428]]
all|0.8393823616051211|0.6792648375138227|[[674,627],[35905,190241]]
......
上面Statistics列有两个值all和window,all表示从开始运行到现在的所有预测数据的评估结果;wiodow表示时间窗口(当前设置为10秒)的所有预测数据的评估结果。