Spark大数据分析与可视化实战案例-CSDN博客

本文链接：https://blog.csdn.net/Gx_2647664843/article/details/148129063

一、项目背景与需求分析

本案例基于Spark大数据处理框架，对美妆商品的销售数据进行全面分析，并通过Pyecharts实现可视化展示。
主要完成以下分析需求：
商品价格分析：找出每个商品小类中价格最高的前5个商品
月度销售趋势：统计每月订单的订购数量和消费金额
地域销售分布：分析各城市的订购数量排行TOP20
商品需求分析：统计各类型美妆产品的订购数量排行
省份需求分析：分析各省的美妆订购数量
客户价值分析：通过RFM模型挖掘高价值客户

二、技术栈与准备工作

2.1 主要技术组件

Spark：分布式计算框架，用于大数据处理
PySpark：Spark的Python API
Pandas：数据分析和处理库
Pyecharts：数据可视化库

2.2 环境准备

# 安装必要的Python库
sudo pip install pandas==1.1.5
sudo pip install numpy
sudo pip install pyecharts==1.9.0
sudo pip install pyspark-stubs==2.4.0

2.3 数据准备

将数据文件上传至HDFS：
hdfs dfs -mkdir -p /datas
hdfs dfs -put ~/mydata/beauty/beauty_prod_*.csv /datas

三、核心代码解析

3.1 数据清洗与预处理

# 创建SparkSession
spark = SparkSession.builder.appName('SparkBeautyProduct').getOrCreate()

# 读取商品信息和销售数据
df1 = spark.read.csv('hdfs://localhost:9000/datas/beauty_prod_info.csv',
encoding='UTF-8', header=True, sep=',',
schema='prod_id string, product string, cataB string, cataA string, price float')

df2 = spark.read.csv('hdfs://localhost:9000/datas/beauty_prod_sales.csv',
encoding='UTF-8', header=True, sep=',',
schema='''order_id string, od_date string, cust_id string,
cust_region string, cust_province string, cust_city string,
prod_id string, od_quantity string, od_price string, od_amount float''')

# 数据清洗
df_prod_info = df1.dropna().dropDuplicates(['prod_id'])

df_prod_sales = df2 \
.dropna() \
.distinct() \
.withColumn("od_date", regexp_replace('od_date', '#', '-')) \
.withColumn("od_quantity", regexp_replace('od_quantity','个','')) \
.withColumn("od_price", regexp_replace('od_price','元','')) \
.withColumn('cust_province', regexp_replace('cust_province','自治区|维吾尔|回族|壮族|省|市','')) \
.withColumn("od_date", col('od_date').cast('date')) \
.withColumn("od_quantity", col('od_quantity').cast('integer')) \
.withColumn("od_price", col('od_price').cast('float')) \
.withColumn("od_month", F.month('od_date')) \
.filter("od_date<'2025-12-31'")

3.2 关键分析实现

3.2.1 商品小类价格TOP5（窗口函数应用）

df_top5_product = spark.sql(
'SELECT *,dense_rank() over (partition by cataB order by price DESC) as rank FROM prod')\
.where('rank<=5')
这里使用了Spark SQL的窗口函数dense_rank()，按商品小类分组后按价格降序排名，取前5名。

3.2.2 月度销售分析

df_month_sale = spark.sql(
'''SELECT od_month,
sum(od_quantity) as total_quantity,
sum(od_amount) as total_amount
FROM sales
GROUP BY od_month
ORDER BY od_month''')

3.2.3 城市销售TOP20

df_city_sale = spark.sql(
'''SELECT cust_city,
sum(od_quantity) as total_quantity
FROM sales
GROUP BY cust_city
ORDER BY total_quantity DESC
LIMIT 20''')

3.2.4 RFM客户价值分析

# 计算RFM基础指标
df = spark.sql(
'''SELECT cust_id,
max(od_date) as od_latest,
count(order_id) as total_count,
sum(od_amount) as total_amount
FROM sales
GROUP BY cust_id''') \
.withColumn('cust_all', lit(1))

# 计算归一化RFM值
df = spark.sql(
'''SELECT cust_id, od_latest,total_count,total_amount,
percent_rank() over (partition by cust_all order by od_latest) as R,
percent_rank() over (partition by cust_all order by total_count) as F,
percent_rank() over (partition by cust_all order by total_amount) as M
FROM customer''')

# 计算综合得分
df_customerRFM = df\
.withColumn('score', col('R')*20+col('F')*30+col('M')*50) \
.withColumn('score', F.round(col('score'),1)) \
.orderBy(F.desc('score'))

3.3 数据可视化实现

# 月度销售可视化
month_sale = pd.read_csv('/home/spark/mydata/month_sale.csv')
x = [f'{v}月' for v in month_sale['订单月份'].tolist()]
y1 = [round(v/10000, 2) for v in month_sale['订购数量'].tolist()]
y2 = [round(v/10000/10000, 2) for v in month_sale['金额'].tolist()]

c = (
Bar()
.add_xaxis(x)
.add_yaxis("订购数量（万件）", y1)
.add_yaxis("金额（亿元）", y2)
.set_global_opts(
title_opts=opts.TitleOpts(title="每月订购情况"),
yaxis_opts=opts.AxisOpts(name="金额(亿元)"),
xaxis_opts=opts.AxisOpts(name="月份"),
)
.render("/home/spark/month_sale.html")
)

# 城市销售TOP20可视化
city_sale = pd.read_csv('/home/spark/mydata/city_sale.csv')
x = city_sale['所在地市'].tolist()
y = [round(v/10000, 2) for v in city_sale['订购数量'].tolist()]
x.reverse()
y.reverse()

c = (
Bar()
.add_xaxis(x)
.add_yaxis("订购量", y,
label_opts=opts.LabelOpts(position="right", formatter='{@[1]/} 万'))
.reversal_axis()
.set_global_opts(
title_opts=opts.TitleOpts("订购数量排行 TOP20")
)
.render("/home/spark/city_sale.html")
)