spark1.统计句子中特定内容

最新推荐文章于 2019-10-07 11:21:57 发布

weixin_30565199

最新推荐文章于 2019-10-07 11:21:57 发布

阅读量98

点赞数

原文链接：http://www.cnblogs.com/wcLT/p/6623635.html

版权

    val logFile = "./README.md"  // Should be some file on your server.
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
//    val numAs = logData.filter(line => line.contains("h")).count()
//    val numBs = logData.filter(line => line.contains("j")).count()
    var  params = List("h","j","c","w");

    var searchAnylisay =   params.map(item => logData.filter(line => line.contains(item)).count() )

    println("searchAnylisay length  : %s,".format(searchAnylisay.length))

    searchAnylisay.foreach( x => println(x))

转载于:https://www.cnblogs.com/wcLT/p/6623635.html