Spark2.4.8实现网页日志分析

最新推荐文章于 2022-07-25 08:44:01 发布

若兰幽竹

最新推荐文章于 2022-07-25 08:44:01 发布

阅读量1k

点赞数 9

分类专栏： Spark 文章标签： scala spark java

本文链接：https://blog.csdn.net/sujiangming/article/details/120425609

版权

Spark 专栏收录该内容

34 篇文章 14 订阅

订阅专栏

Spark2.4.8实现网页日志分析

一、测试数据

测试数据格式如下：

niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89912 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89923 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89933 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89933 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89944 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89944 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89912 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89923 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89933 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"
niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89933 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"

测试数据下载地址：点击下载

二、任务要求

【任务一】：实现网页浏览次数统计
【任务二】：实现网页按照浏览量降序排名（TopN）
【任务三】：将排名结果存入MySQL数据库

三、编码实现

添加依赖，在pom.xml中添加如下：

<packaging>jar</packaging>

<properties>
    <scala.version>2.11.8</scala.version>
    <spark.version>2.4.8</spark.version>
    <spark.artifact.version>2.12</spark.artifact.version>
    <hadoop.version>2.7.3</hadoop.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${spark.artifact.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>
    <!-- 使用scala2.11.8进行编译和打包 -->
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
    </dependency>
    <!-- 引入MySQL驱动 -->
    <dependency>
	    <groupId>mysql</groupId>
	    <artifactId>mysql-connector-java</artifactId>
	    <version>5.1.6</version>
	</dependency>
</dependencies>

<build>
    <!-- 指定scala源代码所在的目录 -->
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>

    <plugins>
        <!--对src/main/java下的后缀名为.java的文件进行编译 -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
            </configuration>
        </plugin>
        <!-- scala的打包插件 -->
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>4.5.4</version>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <scalaVersion>${scala.version}</scalaVersion>
            </configuration>
        </plugin>
    </plugins>
</build>

【任务一】编码实现：

思路：由于url地址基本都是相同的，唯一的区别在于每个url后的id不同，因此，我们可以先找到url，再对url进行分词，取出id，然后按照wordcount的思路进行求和即可

实现代码

import org.apache.spark.{SparkConf, SparkContext}

object PageCount {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf()
      .setAppName(PageCount.getClass.getName)
      .setMaster("local[2]")

    val sc = new SparkContext(sparkConf)

    sc.textFile("d:/Tools/access2.log")
      .map(line => {
        val strings = line.split(" ")
        val url = strings(6)
        val parts = url.split("id=")
        val id = parts(1)
        (id,1)
      })
      .reduceByKey(_+_)
      .collect()
      .foreach(println)

    sc.stop()
  }
}

结果如下

【任务二】编码实现:

思路：只需将结果进行排序，按照降序排即可

代码：

import org.apache.spark.{SparkConf, SparkContext}

object PageCountSort {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf()
      .setAppName(PageCountSort.getClass.getName)
      .setMaster("local[2]")

    val sc = new SparkContext(sparkConf)

    sc.textFile("d:/Tools/access2.log")
      .map(line => {
        val strings = line.split(" ")
        val url = strings(6)
        val parts = url.split("id=")
        val id = parts(1)
        (id,1)
      })
      .reduceByKey(_+_)
      .sortBy(tuple2 => {
        tuple2._2
      },false)
      .collect()
      .foreach(println)

    sc.stop()
  }
}

结果：

【任务三】编码实现：

思路：将分析结果写入到MySQL表中
原理：

步骤：

1、在数据库中创建结果表

CREATE TABLE `page_result` (
  `id` varchar(255) NOT NULL COMMENT 'url中的ID',
  `count` int(11) NOT NULL COMMENT '总浏览数'
) ENGINE=InnoDB DEFAULT CHARSET=utf8

2、在代码中添加连接MySQL的代码

易错点：MySQL数据库的连接代码是不能进行序列化的，因此需要将数据库连接放置在一个Task代码中，即放在触发action算子内部，作为整体的Task

代码：

import java.net.InetAddress
import java.sql.DriverManager

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object SparkAccessLog {
  def main(args: Array[String]): Unit = {
    //1. 创建sparkConf和SparkContext对象
    val conf = new SparkConf().setAppName("SparkAccessLog").setMaster("local[2]")
    val sc = new SparkContext(conf)
    //2. 读取数据分析
    val resultRDD = sc.textFile("d:/Tools/access2.log")
      .map(line => {
        val strings = line.split(" ")
        val url = strings(6)
        val parts = url.split("id=")
        val id = parts(1)
        (id,1)
      }).reduceByKey(_ + _).sortBy(_._2, false) 

    // 获取下面这段代码所在被执行的主机
    val hostName = InetAddress.getLocalHost.getHostName
    val address = InetAddress.getLocalHost.getAddress
    val theadId = Thread.currentThread().getId
    val threadName = Thread.currentThread().getName

    println("==main==" + hostName + ", address: " + address + ", theadId: " + theadId + ", threadName: " + threadName)

    //3. 将分析的结果写入MySQL中
    saveToMySQL(resultRDD)
    //4. 关闭资源
    sc.stop()
  }

  // 定义函数saveToMySQL,用于MySQL的写入操作
  def saveToMySQL(resultRDD:RDD[(String,Int)]): Unit ={
    //3.1 将MySQL的驱动依赖添加进来
    resultRDD.foreach(t=>{
      // 获取下面这段代码所在被执行的主机
      val hostName = InetAddress.getLocalHost.getHostName
      val address = InetAddress.getLocalHost.getAddress
      val theadId = Thread.currentThread().getId
      val threadName = Thread.currentThread().getName

      println("==foreach==" + hostName + ", address: " + address + ", theadId: " + theadId + ", threadName: " + threadName)

      val url = t._1
      val value = t._2
      //3.2 调用MySQL驱动程序api进行编写业务逻辑
      val connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/niit?characterEncoding=UTF-8", "root", "123456")
      // create table page_result(url varchar(50),total int)
      val preparedStatement = connection.prepareStatement("insert into page_result values (?,?)")
      preparedStatement.setString(1,url)
      preparedStatement.setInt(2,value)
      // insert into log values("/shop/detail.html?id=402857036a2831e001kshdksdsdk89933",4)
      val result = preparedStatement.executeUpdate()
      // 关闭资源
      if (preparedStatement != null){
        preparedStatement.close()
      }
      if (connection != null){
        connection.close()
      }
    })
  }
}

结果：

四、实验要求

请使用不同的算子或实现方式完成上述任务
请将代码打包，上传至服务器，查看【任务三】的日志情况，看看代码中的注释会在哪里被打印出来，如下图是在从节点上查看到的日志：

若兰幽竹

关注

9
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
Spark2.4.8实现网页日志分析

Spark2.4.8实现网页日志分析一、测试数据二、任务要求三、编码实现四、实验要求一、测试数据测试数据格式如下：niit110,192.168.215.131 - - [28/May/2019:18:11:44 +0800] "GET /shop/detail.html?id=402857036a2831e001kshdksdsdk89912 HTTP/1.0" 200 4391 "-" "ApacheBench/2.3" "-"niit110,192.168.215.131 - - [28/M
复制链接

扫一扫