spark随机森林算法的应用

最新推荐文章于 2023-07-11 16:36:57 发布
最新推荐文章于 2023-07-11 16:36:57 发布
阅读量236 收藏 2
点赞数
文章标签： 大数据 数据结构与算法 人工智能
原文链接：https://my.oschina.net/u/3455048/blog/1031391
版权

应用场景：预测反欺诈用户

1、构建用户画像，用户画像由多种业务指标组成（例如用户申请后回访次数、ip城市与gps城市是否一致等等）

2、提取用户画像的业务指标--->将指标数值变成向量和矩阵

3、先验数据集：总数据条数：75568 正常用户条数：72723  确定欺诈用户条数：2845

     训练数据集条数：60162     预测数据集：15406

155722_LI9b_3455048.png

4、使用RandomForestClassifier算法

代码如下

package mllib

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.RandomForestClassifier
import org.apache.spark.ml.feature._
import org.apache.spark.sql.{DataFrame, SparkSession}

/**
  * Created by dongdong on 17/6/16.
  */

case class Feature(cid: String, label: String, f2: Double, f3: Double, f4: Double, f5: Double, f6: Double, f7: Double,
                   f8: Double, f9: Double, f10: Double, f11: Double, f12: Double, f13: Double, f14: Double,
                   f15: Double, f16: Double, f17: Double, f18: Double, f19: Double, f20: Double, f21: Double,
                   f22: Double, f23: Double, f24: Double, f25: Double, f26: Double, f27: Double, f28: Double,
                   f29: Double, f30: Double, f31: Double, f32: Double, text: String, f38: Double, f39: Double
                  )

object UserProfile_Forest {
  def main(args: Array[String]): Unit = {
    val inpath = "/user/hive/warehouse/user_profile_tmp_db.db/t_cid_feature/*"

    val spark = SparkSession
      .builder()
      .master("local[3]")
      .appName("UserProfile_Forest")
      .getOrCreate()
    import spark.implicits._

    //Build a dataset and read data 
    val originalData = spark.sparkContext
      .textFile(inpath)
      .map(line => {
        val arr = line.split("\001")
        val cid = arr(0)
        val f1 = arr(1)
        val f2 = arr(2).replace("\\N", "0").toDouble
        val f3 = arr(3).replace("\\N", "0").toDouble
        val f4 = arr(4).replace("\\N", "0").toDouble
        val f5 = arr(5).replace("\\N", "0").toDouble
        val f6 = arr(6).replace("\\N", "0").toDouble
        val f7 = arr(7).replace("\\N", "0").toDouble
        val f8 = arr(8).replace("\\N", "0").toDouble
        val f9 = arr(9).replace("\\N", "0").toDouble
        val f10 = arr(10).replace("\\N", "0").toDouble
        val f11 = arr(11).replace("\\N", "0").toDouble
        val f12 = arr(12).replace("\\N", "0").toDouble
        val f13 = arr(13).replace("\\N", "0").toDouble
        val f14 = arr(14).replace("\\N", "0").toDouble
        val f15 = arr(15).replace("\\N", "0").toDouble
        val f16 = arr(16).replace("\\N", "0").toDouble
        val f17 = arr(17).replace("\\N", "0").toDouble
        val f18 = arr(18).replace("\\N", "0").toDouble
        val f19 = arr(19).replace("\\N", "0").toDouble
        val f20 = arr(20).replace("\\N", "0").toDouble
        val f21 = arr(21).replace("\\N", "0").toDouble
        val f22 = arr(22).replace("\\N", "0").toDouble
        val f23 = arr(23).replace("\\N", "0").toDouble
        val f24 = arr(24).replace("\\N", "0").toDouble
        val f25 = arr(25).replace("\\N", "0").toDouble
        val f26 = arr(26).replace("\\N", "0").toDouble
        val f27 = arr(27).replace("\\N", "0").toDouble
        val f28 = arr(28).replace("\\N", "0").toDouble
        val f29 = arr(29).replace("\\N", "0").toDouble
        val f30 = arr(35).replace("\\N", "0").toDouble
        val f31 = arr(36).replace("\\N", "0").toDouble
        val f32 = arr(37).replace("\\N", "0").toDouble
        val text = arr(40) + "|" + arr(41)

        //val f35 = arr(44).replace("\\N", "0").toDouble
        //val f36 = arr(45).replace("\\N", "0").toDouble
        // val f37 = arr(46).replace("\\N", "0").toDouble
        val f38 = arr(47).replace("\\N", "0").toDouble
        val f39 = arr(48).replace("\\N", "0").toDouble
        Feature(cid, f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, f12, f13, f14, f15, f16, f17, f18, f19, f20, f21, f22, f23, f24, f25, f26, f27
          , f28, f29, f30, f31, f32, text, f38, f39)
      }
      ).toDS

    //build label
    val labelIndexer = new StringIndexer()
      .setInputCol("label")
      .setOutputCol("indexedLabel")
      .fit(originalData)

    val tokenizer = new RegexTokenizer()
      .setInputCol("text")
      .setOutputCol("words")
      .setPattern("\\|")

    val word2Vec = new Word2Vec()
      .setInputCol("words")
      .setOutputCol("feature_one")
      .setVectorSize(100)
      //.setMinCount(1)
      .setMaxIter(20)

    val arr = Array("f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17", "f18", "f19", "f20", "f21", "f22", "f23", "f24", "f25", "f26", "f27", "f28", "f29", "f30", "f31", "f32", "feature_one", "f38", "f39")
    val vectorAssembler = new VectorAssembler()
      .setInputCols(arr)
      .setOutputCol("featureVector")

    //set setnumtrees 1
    val rfClassifier = new RandomForestClassifier()
      .setLabelCol("indexedLabel")
      .setFeaturesCol("featureVector")
      .setNumTrees(1)

   //predict label to string
    val labelConverter = new IndexToString()
      .setInputCol("prediction")
      .setOutputCol("predictedLabel")
      .setLabels(labelIndexer.labels)

    val Array(trainingData, testData) = originalData.randomSplit(Array(0.8, 0.2))

   //build pipeline 
    val pipeline = new Pipeline().setStages(Array(labelIndexer, tokenizer, word2Vec, vectorAssembler, rfClassifier, labelConverter))
     
    //train data
    val model = pipeline.fit(trainingData)

    //predict data
    val predictionResultDF = model.transform(testData)
   
    //predict corrector bad user
    val correctcount_baduser = predictionResultDF.select("cid", "label", "predictedLabel")
      .filter($"label" === $"predictedLabel")
      .filter($"label" === 1).count()

    //predict corrector good user
    val correctcount_gooduser = predictionResultDF.select("cid", "label", "predictedLabel")
      .filter($"label" === $"predictedLabel")
      .filter($"label" === 0).count()
    spark.stop()

  }

}

debug时的一些向量特征如下

+--------------------+-----+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+----+---+----+---+---+----+---+---+---+-----+---+---+------------+--------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-----------------------------------------+----------+--------------+
|cid                 |label|f2 |f3 |f4 |f5 |f6 |f7 |f8 |f9 |f10|f11|f12|f13|f14|f15|f16|f17|f18|f19|f20|f21|f22|f23|f24 |f25|f26 |f27|f28|f29 |f30|f31|f32|text |f38|f39|indexedLabel|words   |feature_one          |featureVector                                                                                                                                  |rawPrediction                            |probability                              |prediction|predictedLabel|
+--------------------+-----+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+----+---+----+---+---+----+---+---+---+-----+---+---+------------+--------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-----------------------------------------+----------+--------------+
|2**60327000*0017**12|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**050300000031**55|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**051800000043**09|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**051900000044**35|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**052100000047**47|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**052600000051**75|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**053100000057**95|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**060100000057**90|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**060300000060**02|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**061500000072**13|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**061700000073**10|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**061700000074**37|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**061900000077**27|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**062100000080**02|1    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|1.0|0.0|0.0|0.0|1.0|1.0|0.0|0.0|21.0|1.0|20.0|0.0|0.0|18.0|1.0|1.0|1.0|\N|\N|0.5|1.0|1.0         |[\n, \n]|[-0.6971070766448975]|(34,[14,18,19,22,23,24,27,28,29,30,31,32,33],[1.0,1.0,1.0,21.0,1.0,20.0,18.0,1.0,1.0,1.0,-0.6971070766448975,0.5,1.0])                         |[0.8320209973753281,0.1679790026246719]  |[0.8320209973753281,0.1679790026246719]  |0.0       |0             |
|20**062400000083**16|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**062400000084**81|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**070500000098**50|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**070600000099**12|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
|20**070900000102**72|0    |1.0|0.0|0.0|0.0|0.0|0.0|1.0|0.0|0.0|1.0|0.0|0.0|0.0|1.0|1.0|1.0|0.0|0.0|1.0|1.0|0.0|0.0|44.0|1.0|43.0|0.0|1.0|14.0|1.0|1.0|1.0|\N|高 |0.0|0.0|0.0         |[\n, 高] |[-0.8806669116020203]|(34,[0,6,9,13,14,15,18,19,22,23,24,26,27,28,29,30,31],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,44.0,1.0,43.0,1.0,14.0,1.0,1.0,1.0,-0.8806669116020203])|[0.9938829787234043,0.006117021276595745]|[0.9938829787234043,0.006117021276595745]|0.0       |0             |
|20**071700000112**30|0    |0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0 |0.0|0.0 |0.0|0.0|0.0 |0.0|0.0|0.0|\N|\N|0.0|0.0|0.0         |[\n, \n]|[-0.6971070766448975]|(34,[31],[-0.6971070766448975])                                                                                                                |[0.964332892998679,0.035667107001321]    |[0.964332892998679,0.035667107001321]    |0.0       |0             |
+--------------------+-----+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+----+---+----+---+---+----+---+---+---+-----+---+---+------------+--------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-----------------------------------------+----------+--------------+

结果：

程序预测为欺诈用户为73人,其中正确为欺诈用户为57人，16人预测不正确(本身是正常用户被预测为欺诈用户)

程序预测为正常用户为14995人，其中正确为正常用户为14498人，其中407预测不正确

优化方向：1、gooduser 数据量和baduser数据量分布不均匀，可以提取更多baduser的先验数据

               2、用户画像中的指标数据量不全，导致有些用户是指标为null

               3、根据业务设置更多有用指标

总结：使用过lr算法和kmeas，但是效果不怎么好

 

转载于:https://my.oschina.net/u/3455048/blog/1031391

chengujun7940
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
随机森林原理、示例、应用
10-16
本资源主要简单介绍随机森林的原理，随机森林的应用特点，以及它的实例和在python上的实现。
Spark 随机森林算法原理、源码分析及案例实战
FZ的博客
01-13 6503
1.Spark 随机森林算法原理是什么？ 2.随机森林算法源码做了哪些分析？ 3.随机森林算法本文例举了什么案例？ 本文首先对决策树算法的原理进行分析并指出其存在的问题，进而介绍随机森林算法。同单机环境下的随机森林构造不同的是，分布式环境下的决策树构建如果不进行优化的话，会带来大量的网络 IO 操作，算法效率将非常低，为此本文给出了随机森林在分布式环境下的具体优化策略，然
spark机器学习实现之随机森林
wangkai198911的博客
12-26 1746
还是按照以前交代的，通过接口配置sc，忘记的可查看 http://blog.csdn.net/wangkai198911/article/details/78728449 通过sqlcontext直接创建dataframe，数据格式的为libsvm， label: double, features: vector label即为分类标签，features是特征，以vector的方式进行保存 //
随机森林（Random Forest）算法原理及Spark MLlib调用实例（Scala/Java/python）
liulingyuan6的博客
12-01 1万+
随机森林分类器： 算法简介：         随机森林是决策树的集成算法。随机森林包含多个决策树来降低过拟合的风险。随机森林同样具有易解释性、可处理类别特征、易扩展到多分类问题、不需特征缩放等性质。        随机森林分别训练一系列的决策树，所以训练过程是并行的。因算法中加入随机过程，所以每个决策树又有少量区别。通过合并每个树的预测结果来减少预测的方差，提高在测试集上的性能表现。
Spark实战：基于Spark的随机森林分类算法分析
竹石破岩
06-14 7885
基于Spark的随机森林分类算法分析 一．随机森林算法原理 随机森林是由多个决策树构成的森林，算法分类结果由决策树的投票结果得到，其属于集成学习中的bagging方法。算法的主要原理如下： 1)假设为一棵决策树，其中每个决策树的抽样方式为重 抽样，独立抽样次，每次随机抽取个样本，就可组成个训练集数据集，且它们是相互独立同分布的。 2)单棵决策树的生长原则遵循：每次从数据集的...
基于Spark的机器学习应用框架研究与实现
04-27
K-means算法和随机森林算法都存在一些限制和缺点。K-means算法中 的群组数目K值需要使用者预先设定，这对使用者提出了较高的要求， 经验不足的使用者设定的K值的准确性也存在一定的问题;随机森林算 法进行分类决策时...
常用机器学习算法应用案例合集（包含全部源码及数据）.zip
最新发布
09-22
常用机器学习算法应用案例合集（包含全部源码及数据）.zip 回归分析（基于邓肯的声望数据集） 基于K近邻算法实现分类预测（基于不同水果的外观特征进行水果类别识别） 基于Spark的大数据分析样例 基于随机森林模型...
如何在MLlib中实现随机森林和梯度提升树（GBTs）？
03-03
Spark1.2在MLlib中引入了随机森林和梯度提升树(GBTs).这两种机器学习方法适用于分类和回归，且是在机器学习算法中应用得最多和最成功的算法。随机森林和GBTs都是集成学习算法，它们通过集成多棵决策树来实现强分类器...
benchm-ml：用于二进制分类的顶级机器学习算法（随机森林，Rus，Python scikit-learn，H2O，xgboost，Spark MLlib等）的可扩展性，速度和准确性的最低基准。梯度增强树，深度神经网络等）
02-04
该项目旨在为一些机器学习算法的常用实现的可伸缩性，速度和准确性提供一个最低基准。 这项研究的目标是使用数字和分类输入（具有有限的基数，即不是很稀疏）并且没有丢失的数据进行二进制分类，这可能是业务应用...
spark实现随机森林
baidu_41605403的博客
10-12 1079
# 1. 导入需要的包： import pyspark from pyspark.sql import SQLContext from pyspark import SparkContext from pyspark.sql import Row,functions from pyspark.mllib.linalg import Vector,Vectors from pyspark.ml.ev...
Spark-MLlib 学习入门到掌握-RandomForestClassifier随机森林分类器[20]
qq_41610493的博客
06-24 753
随机森林是一个典型的分类和回归方法 将数据集其拆分为训练集和测试集，分别用于训练和评估。 def randomForestTest(): Unit ={ import org.apache.spark.ml.Pipeline import org.apache.spark.ml.classification.{RandomForestClassificationModel, RandomForestClassifier} import org.apache.spark.ml..
educoder-Spark机器学习
qq_48664727的博客
06-08 2834
头歌Spark机器学习实训答案代码
机器学习之随机森林（Random forest）
热门推荐
十年以上架构设计经验，专注于软件架构和人工智能领域，对机器视觉、NLP、音视频等领域都有涉猎
07-11 1万+
随机森林是一种监督式算法，使用由众多决策树组成的一种集成学习方法，输出是对问题最佳答案的共识。随机森林可用于分类或回归，是一种主流的集成学习算法。
随机森林
lml77的博客
03-24 1万+
随机森林就是通过集成学习的思想将多棵树集成的一种算法，它的基本单元是决策树，而它的本质属于机器学习的一大分支——集成学习（Ensemble Learning）方法。随机森林的名称中有两个关键词，一个是“随机”，一个就是“森林”。“森林”我们很好理解，一棵叫做树，那么成百上千棵就可以叫做森林了，这样的比喻还是很贴切的，其实这也是随机森林的主要思想–集成思想的体现。 其实从直观角度来解释，每棵决策树都...
随机森林案例
ZJH
01-11 2429
解决思路 定义问题——>数据收集——>数据清洗——>分析——>建模——>验证——>优化 该问题下前两步已经完成，本文从数据清洗开始 数据集的特征解读 英 中 分析 PassengeID 乘客ID 乘客ID不影响Survive与否，但可标识身份 Pclass 船舱等级 高等船舱靠近甲板，更容易Survive Name 姓名 标记 Sex 性别 男女生理差别，对Survive有影响 Age 年龄 同上 SibSp 兄弟配偶数 有无同行兄弟o
分类算法-随机森林实战案例
m0_56839722的博客
12-15 9903
随机森林、调参、特征筛选
机器学习----集成学习----随机森林
weixin_41395763的博客
02-22 1426
决策树 1.决策树与随机森林都属于机器学习中监督学习的范畴，主要用于分类问题。决策树算法有这几种：ID3、C4.5、CART，基于决策树的算法有bagging、随机森林、GBDT等。 决策树是一种利用树形结构进行决策的算法，对于样本数据根据已知条件或叫特征进行分叉，最终建立一棵树，树的叶子结节标识最终决策。新来的数据便可以根据这棵树进行判断。随机森林是一种通过多棵决策树进行优化决策的算法。 ...
【随机森林算法及其行业应用】
Nibiruhhg的博客
04-28 1690
随机森林是一种集成学习算法，它由多个决策树组成。每个决策树都是一个分类器，对于输入的数据，随机森林通过投票方式来决定它的类别。
【随机森林】原理，优缺点和应用场景
qq_44910978的博客
06-04 4463
随机森林是以决策树作为基学习器的集成模型，特点是低方差低偏差，分成两部分，随机和森林。原理/步骤：优点：缺点：应用场景：既可以用于分类也可以用于回归问题，不适用于需要高实时的场景。...
spark随机森林预测
08-21
接下来，可以使用Spark的机器学习库中的随机森林算法来训练模型。 在训练过程中，随机森林会随机选择数据集的子集，并针对每个子集构建一个决策树。这些决策树会根据特征的重要性进行划分，直到达到停止条件（例如...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值