lightGBM专题4:pyspark平台下lightgbm模型保存

最新推荐文章于 2023-11-15 09:56:59 发布
最新推荐文章于 2023-11-15 09:56:59 发布
阅读量2.1k 收藏 9
点赞数
分类专栏： 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/fangfanglovezhou/article/details/118339783
版权

之前的文章（pyspark lightGBM1和pyspark lightGBM2）介绍了pyspark下lightGBM算法的实现，本文将重点介绍下如何保存训练好的模型，直接上代码：

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer
#配置spark,创建SparkSession对象
spark = SparkSession.builder.master('yarn').appName('StringIndexerDemo').getOrCreate()
#创建简单的DataFrame
df = spark.createDataFrame([
    (0, "a", "s" ), (1, "b", "o"), 
    (0, "a", "g"), (1, "b", "l"), 
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u")
    ],
    ["id", "category", "name"])
indexer1 = StringIndexer(inputCol ='category', outputCol= 'categoryIndex')
indexer2 = StringIndexer(inputCol ='name', outputCol= 'nameIndex')
from pyspark.ml import Pipeline
pipeline = Pipeline(stages=[indexer1,indexer2])
model = pipeline.fit(df)
df = model.transform(df)
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(
  inputCols = [
  "id",
  "nameIndex"
  ],
  outputCol = "features")
df = assembler.transform(df)
# Generate classifier
from mmlspark import LightGBMClassifier
classifier = LightGBMClassifier(
  featuresCol="features",
  labelCol="categoryIndex",
  learningRate=0.3,
  numIterations=10,
  numLeaves=100)
glbModel = classifier.fit(df)
pred = glbModel.transform(df)
pred.show()
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(labelCol="categoryIndex", predictionCol="prediction")
accuracy = evaluator.evaluate(pred)
print("Accuracy = %g" % accuracy)
glbModel.saveNativeModel("/home/shenzhou/test_frame_model.txt",overwrite=True)

执行结果：

+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
| id|category|name|categoryIndex|nameIndex| features|       rawPrediction|         probability|prediction|
+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
|  0|       a|   s|          0.0|      2.0|[0.0,2.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   o|          1.0|      5.0|[1.0,5.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   g|          0.0|      3.0|[0.0,3.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   l|          1.0|      4.0|[1.0,4.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
Accuracy = 1

这里glbModel的类型为mmlspark.LightGBMClassifier.LightGBMClassificationModel ，调用起成员函数saveNativeModel可以保存模型，保存文件夹为hdfs下可访问的文件夹地址。

调用模型部分代码：

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer
#配置spark,创建SparkSession对象
spark = SparkSession.builder.master('yarn').appName('StringIndexerDemo').getOrCreate()
#创建简单的DataFrame
df = spark.createDataFrame([
    (0, "a", "s" ), (1, "b", "o"), 
    (0, "a", "g"), (1, "b", "l"), 
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u"),
    (0, "a", "p"), (1, "b", "u")
    ],
    ["id", "category", "name"])
indexer1 = StringIndexer(inputCol ='category', outputCol= 'categoryIndex')
indexer2 = StringIndexer(inputCol ='name', outputCol= 'nameIndex')
from pyspark.ml import Pipeline
pipeline = Pipeline(stages=[indexer1,indexer2])
model = pipeline.fit(df)
df = model.transform(df)

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(
  inputCols = [
  "id",
  "nameIndex"
  ],
  outputCol = "features")
df = assembler.transform(df)

from mmlspark.LightGBMClassifier import LightGBMClassificationModel
lgbModel=LightGBMClassificationModel.loadNativeModelFromFile("/home/reco/shenzhou/test_frame_model.txt/")
pred=lgbModel.transform(df)
pred.show()
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(labelCol="categoryIndex", predictionCol="prediction")
accuracy = evaluator.evaluate(pred)
print("Accuracy = %g" % accuracy)

打印结果：

+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
| id|category|name|categoryIndex|nameIndex| features|       rawPrediction|         probability|prediction|
+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
|  0|       a|   s|          0.0|      2.0|[0.0,2.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   o|          1.0|      5.0|[1.0,5.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   g|          0.0|      3.0|[0.0,3.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   l|          1.0|      4.0|[1.0,4.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
|  0|       a|   p|          0.0|      1.0|[0.0,1.0]|[3.78240462265661...|[0.97773895925226...|       0.0|
|  1|       b|   u|          1.0|      0.0|[1.0,0.0]|[-3.7824046226566...|[0.02226104074773...|       1.0|
+---+--------+----+-------------+---------+---------+--------------------+--------------------+----------+
Accuracy = 1

这里调用loadNativeModelFromFile加载模型，注意其为静态函数，所以要直接用LightGBMClassificationModel调用函数loadNativeModelFromFile返回对象。

I_belong_to_jesus
关注
  • 0
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
利用pyspark.ml训练lightgbm模型的流程
cxy7228484的博客
12-27 1万+
在spark上训练模型的优势: (1)机器学习算法一般都有很多个步骤迭代计算的过程，机器学习的计算需要在多次迭代后获得足够小的误差或者足够收敛才会停止，迭代时如果使用一般的Hadoop分布式计算框架，每次计算都要读 / 写磁盘以及任务的启动等工作，这回导致非常大的 I/O 和 CPU 消耗。而 Spark 基于内存的计算模型天生就擅长迭代计算，多个步骤计算直接在内存中完成. (2)从通信的角度...
LightGBM如何保存模型？
炼丹笔记
03-30 4810
用lgb保存模型遇到了几个坑，在这里记录一下。在用Lightgbm.LGBMClassifier训练生成模型时，Scikit-learn 官网上建议的两种方式：1.pickle方式这里我写了保存和加载两种方式：import pickle def pkl_save(filename,file): output = open(filename, 'wb') pickle.dump(fi...
lightgbm保存模型参数
kyle1314608的博客
12-16 1108
机器学习模型的可解释性是个让人头痛的问题。在使用LightGBM模型的肯定对生成的GBDT的结构是好奇的，我也好奇，所以就解析一个LightGBM的模型文件看看，通过这个解析，你可以看懂GBDT的结构。 另外，了解模型文件，能够在此基础上做模型的监控、评估。 训练配置： 6307410个样本做训练集 num_trees = 2 // 树的棵树 categorical_feature=1,2,3,5,6,8,299 //类别特征编号 num_leaves = 6 // 每棵树的叶子数 max_
lightGBM文件保存
sa726663676的博客
06-10 2866
lightGBM和sklearn中的lightGBM模型保存
Lightgbm模型两种保存方式
热门推荐
u012666689的专栏
12-17 2万+
一、原生形式使用lightgbm(import lightgbm as lgb) # 模型训练 gbm = lgb.train(params, lgb_train, num_boost_round=20, valid_sets=lgb_eval, early_stopping_rounds=5) # 模型保存 gbm.save_model('model.txt') # 模型加载 gbm =...
python pyspark-submit 保存模型到hdfs（全网之最）
WGS.
06-30 2983
文章目录
lightgbm模型保存为pmml文件 机器学习lgbm模型存为pmml文件 通过java调用jar
01-23
首先，LightGBM模型的保存和加载是模型持久化的重要步骤。在Python中，我们可以使用`lightgbm`库提供的`save_model`和`load_model`函数来实现这一过程。例如，一旦模型训练完成，可以使用以下代码保存模型： ```...
LightGBM.jl：LightGBM.jl为Microsoft的LightGBM提供了高性能的Julia界面
02-04
总结来说，LightGBM.jl是将LightGBM的高效机器学习能力引入Julia生态系统的重要桥梁，它结合了两种工具的优点，为数据科学家和机器学习工程师提供了一个强大且灵活的平台，用于解决复杂的分类和回归问题。...
LightGBM-MATLAB.rar_foundyt4_lightGBM_matlab lightgbm_termeaa_轻量
07-15
在MATLAB环境中集成LightGBM，可以使得数据科学家和研究人员在熟悉的编程环境下利用其强大的机器学习能力。 MATLAB是数学计算和工程应用的强大工具，它内置了多种机器学习算法，但LightGBM因其优化的特性，如直方图...
lightgbm算法原理.pptx
05-14
* 带深度限制的 Leaf-wise 算法：LightGBM 使用 Leaf-wise 算法来控制模型的复杂度，防止过拟合。 * 单边梯度采样算法 (GOSS)：LightGBM 使用 GOSS 算法来减少样本的数量，提高训练速度。 * 互斥特征捆绑算法 (EFB)...
ungil/LightGBM-MATLAB:LightGBM 的 MATLAB 包装器-matlab开发
05-29
LightGBM是一款高效、分布式、优化的梯度提升框架，由微软亚洲研究院开发，主要用于机器...总的来说，"LightGBM 的 MATLAB 包装器"为MATLAB用户提供了一种无缝集成高效机器学习模型的途径，简化了跨平台工作的复杂性。
LightGBM模型保存save_model后加载
最新发布
罟霖同学的站点
11-15 1147
LightGBM模型的加载通常会使用外置的joblib接口来实现，但有时候我们也能看到使用其自带接口save_model来实现。但网上对于这一接口下保存的模型该如何读取，并没有相关详尽的介绍。本文将给出保存后后该如何读取。
LGB　模型保存及应用
fang156239305的博客
06-09 6121
1 原生模式 # 模型训练 gbm = lgb.train(params, lgb_train, num_boost_round=20, valid_sets=lgb_eval, early_stopping_rounds=5) # 模型保存 gbm.save_model('model.txt') # 模型加载 gbm = lgb.Booster(model_file='model.txt') # 模型预测 y_pred = gbm.predict(X_test, num_iteration=.
lightgbm 保存模型 过大_lightgbm介绍
weixin_39906521的博客
01-02 334
lightgbm由微软开发的boosting工具包，顾明思议light，这个包计算速度更快，占用内存更少，而准确度和xgboost相当，也是一大神器。lightgbm进行了如下优化：1. 直方图算法1.1 创建直方图把连续的浮点特征值离散化成k个整数，同时构造一个宽度为k的直方图。在 遍历数据的时候，根据离散化后的值作为索引在直方图中累积统计量，当遍 历一次数据后，直方图累积了需要的统计量，然后根...
pyspark使用ML库并用pyspark2pmml把模型保存为pmml格式
weixin_42051109的博客
11-20 5891
一、准备工作        使用Idea开发pyspark程序，在这之前我们需要做一些准备工作， 如同PyCharm开发pyspark程序一样，我们需要先对python项目进行配置： 1. 使用anaconda3里面的python     打开File -> Project Structure -> Project ,其中Project SDK的路径设为anaconda3的pyt...
LGB模型保存及使用
kuxingseng123的博客
01-26 2342
会自己进行调参。
lightgbm 保存模型 过大_LightGBM原理分析
weixin_32122595的博客
12-25 192
1. Abstract在之前我介绍过XGB模型，这次想跟大家分享一下LightGBM这个模型。LightGBM论文的标题为A Highly Efficient Gradient Boosting Decision Tree。这说明LightGBM它是对于XGB提升性能的版本。而LightGBM相对于其他GBM来说具有相近的准确率而且是其训练速度20倍。2. Comparse with XGBLig...
lightGBM专题2:基于pyspark在spark平台下lightgbm训练详解
fangfanglovezhou的博客
06-27 2399
这里以数据集flight_weather.csv为例，首先读取数据对数据集进行查看： from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType, TimestampType from pyspark.sql import SparkSession sqlContext = SparkSession.builder.enableHiveSupport().getOrCreat...
Lightgbm
深挖技术点滴
03-14 825
来爱奇艺这边实习，分到了BI算法岗，要开始接触微软开源的Lightgbm了，号称比xgboost更快更狠的开源；开这篇博客，记录爬坑点滴！后续会更新的；2018/3/15这个算法对XGBOOST进行了进一步的优化（速度上）；而且精度损失不大，并且有GPU等加速方法，但是调节的参数非常多，也是一个棘手的问题，具体可以去论文介绍地址里面看；有兴趣了解XGBOOST的同学可以去看一下陈天奇大神的介绍；我...
解决LightGBM报错OSError: [WinError 126] - 三种解决方案
在Python编程中，LightGBM是一个流行的梯度提升框架，用于构建高效、灵活且可移植的机器学习模型。然而，有时在尝试导入lightgbm模块时，可能会遇到`OSError: [WinError 126] 找不到指定的模块`这样的错误。这个错误...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值