spark的相关rdd操作

最新推荐文章于 2024-10-02 18:26:26 发布

大仙s

最新推荐文章于 2024-10-02 18:26:26 发布

阅读量47

点赞数

文章标签： spark 数学建模大数据

本文链接：https://blog.csdn.net/qq1849238919/article/details/131376154

版权

这段代码展示了PySpark在不同场景下的数据操作。首先，它演示了如何从文本文件中读取数据并计算商品类别平均价格；接着，它使用SparkSession进行SQL查询，分析考试成绩；然后，处理实时日志数据，找出GET请求最多的IP；最后，利用随机森林模型对蘑菇数据集进行分类预测。

摘要由CSDN通过智能技术生成

from pyspark import SparkContext

sc = SparkContext("local","fruit")

data = sc.textFile("products.txt")

header = data.first()
data = data.filter(lambda x: x != header)

fruits = data.map(lambda x: (x.split(",")[3], float(x.split(",")[2])))

avg_price = fruits.aggregateByKey((0, 0), lambda acc, price: (acc[0] + price, acc[1] + 1),
lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])) \
.mapValues(lambda x: x[0]/x[1])

for category, price in avg_price.collect():
print("Category:", category, "\tAverage Price:", price)

from pyspark.sql import SparkSession
from pyspark.sql.functions import avg,count

spark = SparkSession.builder.appName("Scores Analysis").getOrCreate()

data = spark.read.csv("exam.txt", header=True, inferSchema=True)

city_stats = data.groupBy("city").agg(count("*").alias("num_students"), avg("score").alias("avg_score"))

top_cities = city_stats.filter("avg_score > 85")

top_cities.show()

spark.stop()

from pyspark.streaming import StreamingContext
from pyspark.sql import SparkSession, Window
from pyspark.sql.functions import count, first
spark = SparkSession.builder.appName("IP Analysis").getOrCreate()

ssc = StreamingContext(spark.sparkContext, batchDuration=10)

data = ssc.textFileStream("web.log")

logs = data.map(lambda x: x.split(" ")) \
.map(lambda x: (x[0], x[3].lstrip('"'), x[5], x[6], x[8]))

fields = ["ip", "timestamp", "method", "path", "status"]

logs_df = logs.toDF(fields)

result = logs_df.filter(logs_df.method == "GET") \
.groupBy(logs_df.timestamp, logs_df.ip) \
.agg(count("*").alias("num_requests")) \
.withColumn("max_requests", count("*").over(w)) \
.where("num_requests = max_requests") \
.groupBy("timestamp") \
.agg(first("ip").alias("ip"), first("max_requests").alias("max_requests"))

result.printSchema()

query = result.writeStream.outputMode("complete").format("console").start()

query.awaitTermination()

spark.stop()
ssc.stop()

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer, VectorAssembler

spark = SparkSession.builder.appName("mushrooms").master("local[2]").getOrCreate()

df1 = spark.read.csv("mushrooms.csv", header=True, inferSchema=True)
df1.show(10)

vectorAssembler = VectorAssembler().setInputCols(["new_" + col for col in df1.columns if col != "class"]).setOutputCol("features")
df3 = vectorAssembler.transform(df2).select("features", "label")
df3.show(10)

(trainingData, testData) = df3.randomSplit([0.7, 0.3], seed=2023)

from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier().setLabelCol("label").setFeaturesCol("features").setNumTrees(10)
model = rf.fit(trainingData)
predictions = model.transform(testData)

predictions.select("label", "probability", "prediction").show(2)

==========================================================