Spark项目---- 模拟互联网网站用户行为实时分析系统（第一部分）

最新推荐文章于 2024-05-13 13:55:26 发布

置顶

hailunw

最新推荐文章于 2024-05-13 13:55:26 发布

阅读量567

点赞数 2

分类专栏：软件工程文章标签： spark hbase

本文链接：https://blog.csdn.net/hailunw/article/details/118992135

版权

本文介绍了如何搭建一个实时分析系统，使用Python生成模拟数据，通过Flume发送到Kafka，再由Spark Stream进行处理。详细步骤包括Python数据生成、Linux调度、Flume与Kafka配置，以及Spark Stream读取Kafka数据时的Scala版本调整。

摘要由CSDN通过智能技术生成

简介

https://www.bilibili.com/video/BV1k4411e7j8

1）使用Python脚本生成dummy的实时数据到文件

2）将Python脚本传到Linux服务器上，然后创建Linux脚本调用它，然后设置crontab每个30分钟执行一次

[user@NewBieMaster ~]$ cat ~/log_generator2.sh 
python /home/user/generate2.py
[user@NewBieMaster ~]$

[user@NewBieMaster ~]$ crontab -l
30 * * * * /home/user/log_generator2.sh
[user@NewBieMaster ~]$

3）配置生产端flume，启动Hadoop，Kafka集群，以及生产端flume，写入数据到Kafka集群。

4）配置Spark，从Kafka集群中读取数据

4.1）使用IDEA创建Scala Maven项目

4.2）配置pom文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.example</groupId>
  <artifactId>com.exampler.spark</artifactId>
  <version>1.0-SNAPSHOT</version>
  <inceptionYear>2008</inceptionYear>
  <properties>
    <scala.version>2.13.6</scala.version>
    <kafka.version>2.8.0</kafka.version>
    <spark.version>3.1.2</spark.version>
    <hadoop.version>3.2.2</hadoop.version>
    <hbase.version>1.2.0</hbase.version>
  </properties>

<dependencies>
<!-- scala依赖 -->
  <dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.13.6</version>
  </dependency>
<!-- hbase依赖 -->
  <dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>2.4.3</version>
  </dependency>
  <dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-server</artifactId>
    <version>2.4.3</version>
  </dependency>
<!--hadoop依赖-->
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>${hadoop.version}</version>
  </dependency>


<!--kafka依赖-->
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.13</artifactId>
      <version>${kafka.version}</version>
    </dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.12</artifactId>
      <version>3.1.2</version>
    </dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
      <version>3.1.2</version>
    </dependency>
  <!-- https://mvnrepository.com/artifact/org.scala-tools/maven-scala-plugin -->
  <dependency>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <version>2.15.2</version>
  </dependency>

  <!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-eclipse-plugin -->
  <dependency>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <version>2.10</version>
  </dependency>
  </dependencies>

<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
  <plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
    <executions>
      <execution>
        <goals>
          <goal>compile</goal>
          <goal>testCompile</goal>
        </goals>
      </execution>
    </executions>
    <configuration>
      <scalaVersion>${scala.version}</scalaVersion>
      <args>
        <arg>-target:jvm-1.8</arg>
      </args>
    </configuration>
  </plugin>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <version>2.5.1</version>
    <configuration>
      <downloadSources>true</downloadSources>
      <buildcommands>
        <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
      </buildcommands>
      <additionalProjectnatures>
        <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
      </additionalProjectnatures>
      <classpathContainers>
        <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
        <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
      </classpathContainers>
    </configuration>
  </plugin>
</plugins>
</build>
<reporting>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <configuration>
          <scalaVersion>2.13.6</scalaVersion>
        </configuration>
      </plugin>
    </plugins>
</reporting>
</project>

4.3）创建Kafka集群的SparkStream读取类

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html

目前有错误信息如下

"C:\Program Files\Java\jdk1.8.0_291\bin\java.ex

最低0.47元/天解锁文章