Pyspark 基础

最新推荐文章于 2022-06-23 14:27:00 发布
最新推荐文章于 2022-06-23 14:27:00 发布
阅读量308 收藏 2
点赞数
分类专栏： 大数据 文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_42948291/article/details/106856260
版权

一、SparkSession

from pyspark.sql import SparkSession

## 设置要连接的Spark主节点URL,"local"表示在本地运行,"local[2]"表示在本地调用2个核心运行
spark = SparkSession.builder.master('local[2]').appName('Basics').getOrCreate()

二、DataFrame

DataFrame 是带有 schema 信息的 RDD，类似于传统数据库中的二位表格。

## 从 csv 文件中读取数据
df1 = spark.read.csv('appl_stock.csv',inferSchema=True,header=True)

## 从 json 文件中读取数据
df2 = spark.read.json('people.json')

1、基本操作

## 显示前五行基本信息
df1.show(5)

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|               Low|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

df2.show(5)

+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

## 显示表结构（包含字段和类型）
df1.printSchema()

root
 |-- Date: timestamp (nullable = true)
 |-- Open: double (nullable = true)
 |-- High: double (nullable = true)
 |-- Low: double (nullable = true)
 |-- Close: double (nullable = true)
 |-- Volume: integer (nullable = true)
 |-- Adj Close: double (nullable = true)

df2.printSchema()

root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)

## 直接显示字段名
df1.columns

['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']

df2.columns

['age', 'name']

## 显示描述性统计信息
df1.describe().show()

+-------+------------------+------------------+------------------+-----------------+-------------------+------------------+
|summary|              Open|              High|               Low|            Close|             Volume|         Adj Close|
+-------+------------------+------------------+------------------+-----------------+-------------------+------------------+
|  count|              1762|              1762|              1762|             1762|               1762|              1762|
|   mean| 313.0763111589103| 315.9112880164581| 309.8282405079457|312.9270656379113|9.422577587968218E7| 75.00174115607275|
| stddev|185.29946803981522|186.89817686485767|183.38391664371008|185.1471036170943|6.020518776592709E7| 28.57492972179906|
|    min|              90.0|         90.699997|         89.470001|        90.279999|           11475900|         24.881912|
|    max|        702.409988|        705.070023|        699.569977|       702.100021|          470249500|127.96609099999999|
+-------+------------------+------------------+------------------+-----------------+-------------------+------------------+

df2.describe().show()

+-------+------------------+-------+
|summary|               age|   name|
+-------+------------------+-------+
|  count|                 2|      3|
|   mean|              24.5|   null|
| stddev|7.7781745930520225|   null|
|    min|                19|   Andy|
|    max|                30|Michael|
+-------+------------------+-------+

## 比使用.describe()增加了四分位数信息
df1.summary().show()

+-------+------------------+------------------+------------------+------------------+-------------------+------------------+
|summary|              Open|              High|               Low|             Close|             Volume|         Adj Close|
+-------+------------------+------------------+------------------+------------------+-------------------+------------------+
|  count|              1762|              1762|              1762|              1762|               1762|              1762|
|   mean| 313.0763111589103| 315.9112880164581| 309.8282405079457| 312.9270656379113|9.422577587968218E7| 75.00174115607275|
| stddev|185.29946803981522|186.89817686485767|183.38391664371008| 185.1471036170943|6.020518776592709E7| 28.57492972179906|
|    min|              90.0|         90.699997|         89.470001|         90.279999|           11475900|         24.881912|
|    25%|        115.199997|        116.349998|             114.0|        115.190002|           49161400|         50.260037|
|    50%|        317.990002|320.18001200000003|        316.340004|318.21000699999996|           80500000| 72.95419100000001|
|    75%|470.94001799999995|478.55001799999997|468.05001799999997|472.69001799999995|          121095800|        100.228673|
|    max|        702.409988|        705.070023|        699.569977|        702.100021|          470249500|127.96609099999999|
+-------+------------------+------------------+------------------+------------------+-------------------+------------------+

df2.summary().show(5)

+-------+------------------+----+
|summary|               age|name|
+-------+------------------+----+
|  count|                 2|   3|
|   mean|              24.5|null|
| stddev|7.7781745930520225|null|
|    min|                19|Andy|
|    25%|                19|null|
+-------+------------------+----+
only showing top 5 rows

2、处理缺失值

df2.show()

+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

# 去除具有空值的行
df2.na.drop().show()

+---+------+
|age|  name|
+---+------+
| 30|  Andy|
| 19|Justin|
+---+------+

# 仅保留至少具有一个非空值的行
df2.na.drop(thresh=1).show()

+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

# 去除 age 字段具有空值的行
df2.na.drop(subset=["age"]).show()

+---+------+
|age|  name|
+---+------+
| 30|  Andy|
| 19|Justin|
+---+------+

# 去除空值的方式为 any
df2.na.drop(how='any').show() ## or 'all'

+---+------+
|age|  name|
+---+------+
| 30|  Andy|
| 19|Justin|
+---+------+

# 去除 name 字段具有空值的行
df2.na.fill(0, subset=['name']).show()

+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

3、DataFrame 函数操作

df = spark.read.csv('appl_stock.csv',inferSchema=True,header=True)

df.show(5)

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|               Low|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

（1）filter（筛选合适的行）

df.filter("Close < 500").show(5)
df.filter(df['Close']<500).show(5)

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|               Low|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|               Low|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

df.filter("Close < 500 AND Open > 500").show(5)
df.filter((df['Close']<500) & (df['Open']>500)).show(5)

+-------------------+----------+------------------+------------------+------------------+---------+---------+
|               Date|      Open|              High|               Low|             Close|   Volume|Adj Close|
+-------------------+----------+------------------+------------------+------------------+---------+---------+
|2012-02-15 00:00:00|514.259995|        526.290016|496.88998399999997|        497.669975|376530000|64.477899|
|2013-09-05 00:00:00|500.250008|500.67997699999995|493.63997699999993|495.26997400000005| 59091900|65.977837|
|2013-09-10 00:00:00|506.199997|        507.450012|        489.500015|494.63999900000005|185798900|65.893915|
|2014-01-30 00:00:00|502.539993|506.49997699999994|         496.70002|        499.779984|169625400|66.967353|
+-------------------+----------+------------------+------------------+------------------+---------+---------+

+-------------------+----------+------------------+------------------+------------------+---------+---------+
|               Date|      Open|              High|               Low|             Close|   Volume|Adj Close|
+-------------------+----------+------------------+------------------+------------------+---------+---------+
|2012-02-15 00:00:00|514.259995|        526.290016|496.88998399999997|        497.669975|376530000|64.477899|
|2013-09-05 00:00:00|500.250008|500.67997699999995|493.63997699999993|495.26997400000005| 59091900|65.977837|
|2013-09-10 00:00:00|506.199997|        507.450012|        489.500015|494.63999900000005|185798900|65.893915|
|2014-01-30 00:00:00|502.539993|506.49997699999994|         496.70002|        499.779984|169625400|66.967353|
+-------------------+----------+------------------+------------------+------------------+---------+---------+

（2）drop（去除指定的列）

df.drop('Low').show(5)

+-------------------+----------+----------+------------------+---------+------------------+
|               Date|      Open|      High|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+---------+------------------+
only showing top 5 rows

（3）withColumn（通过旧列构造新列）

df_new = df.withColumn('Low_plus',df['Low']+1)
df_new.select("Low_plus", "Low").show(5)

+------------------+------------------+
|          Low_plus|               Low|
+------------------+------------------+
|213.38000099999996|212.38000099999996|
|        214.249994|        213.249994|
|        211.750004|        210.750004|
|        210.050005|        209.050005|
|210.06000500000002|209.06000500000002|
+------------------+------------------+
only showing top 5 rows

## withColumnRenamed 直接将旧列重命名
df.withColumnRenamed('Low','Low_new').show(5)

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|           Low_new|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

（4）groupBy（按列分组聚合）

df.groupBy('Date').mean().show(5)

+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|               Date|         avg(Open)|         avg(High)|  avg(Low)|avg(Close)|avg(Volume)|    avg(Adj Close)|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|2012-03-12 00:00:00| 548.9799879999999|        551.999977|547.000023|551.999977| 1.018206E8|         71.516869|
|2012-11-23 00:00:00|        567.169991|        572.000008|562.600006|571.500023|  6.82066E7|         74.700825|
|2013-02-19 00:00:00|461.10000599999995|        462.730003|453.850014|459.990021| 1.089459E8|60.475753000000005|
|2013-10-08 00:00:00|        489.940025|490.64001500000006|480.540024| 480.93998|  7.27293E7|         64.068854|
|2015-05-18 00:00:00|        128.380005|        130.720001|128.360001|130.190002|  5.08829E7|        125.697198|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
only showing top 5 rows

df.groupBy('Date').min().show(5)

+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|               Date|         min(Open)|         min(High)|  min(Low)|min(Close)|min(Volume)|    min(Adj Close)|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|2012-03-12 00:00:00| 548.9799879999999|        551.999977|547.000023|551.999977|  101820600|         71.516869|
|2012-11-23 00:00:00|        567.169991|        572.000008|562.600006|571.500023|   68206600|         74.700825|
|2013-02-19 00:00:00|461.10000599999995|        462.730003|453.850014|459.990021|  108945900|60.475753000000005|
|2013-10-08 00:00:00|        489.940025|490.64001500000006|480.540024| 480.93998|   72729300|         64.068854|
|2015-05-18 00:00:00|        128.380005|        130.720001|128.360001|130.190002|   50882900|        125.697198|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
only showing top 5 rows

df.groupBy('Date').max().show(5)

+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|               Date|         max(Open)|         max(High)|  max(Low)|max(Close)|max(Volume)|    max(Adj Close)|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|2012-03-12 00:00:00| 548.9799879999999|        551.999977|547.000023|551.999977|  101820600|         71.516869|
|2012-11-23 00:00:00|        567.169991|        572.000008|562.600006|571.500023|   68206600|         74.700825|
|2013-02-19 00:00:00|461.10000599999995|        462.730003|453.850014|459.990021|  108945900|60.475753000000005|
|2013-10-08 00:00:00|        489.940025|490.64001500000006|480.540024| 480.93998|   72729300|         64.068854|
|2015-05-18 00:00:00|        128.380005|        130.720001|128.360001|130.190002|   50882900|        125.697198|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
only showing top 5 rows

df.groupBy('Date').sum().show(5)

+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|               Date|         sum(Open)|         sum(High)|  sum(Low)|sum(Close)|sum(Volume)|    sum(Adj Close)|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
|2012-03-12 00:00:00| 548.9799879999999|        551.999977|547.000023|551.999977|  101820600|         71.516869|
|2012-11-23 00:00:00|        567.169991|        572.000008|562.600006|571.500023|   68206600|         74.700825|
|2013-02-19 00:00:00|461.10000599999995|        462.730003|453.850014|459.990021|  108945900|60.475753000000005|
|2013-10-08 00:00:00|        489.940025|490.64001500000006|480.540024| 480.93998|   72729300|         64.068854|
|2015-05-18 00:00:00|        128.380005|        130.720001|128.360001|130.190002|   50882900|        125.697198|
+-------------------+------------------+------------------+----------+----------+-----------+------------------+
only showing top 5 rows

df.groupBy('Date').count().show(5)

+-------------------+-----+
|               Date|count|
+-------------------+-----+
|2012-03-12 00:00:00|    1|
|2012-11-23 00:00:00|    1|
|2013-02-19 00:00:00|    1|
|2013-10-08 00:00:00|    1|
|2015-05-18 00:00:00|    1|
+-------------------+-----+
only showing top 5 rows

（5）orderBy（按列排序）

df.orderBy('Date').show(5)

+-------------------+----------+----------+------------------+------------------+---------+------------------+
|               Date|      Open|      High|               Low|             Close|   Volume|         Adj Close|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
|2010-01-04 00:00:00|213.429998|214.499996|212.38000099999996|        214.009998|123432400|         27.727039|
|2010-01-05 00:00:00|214.599998|215.589994|        213.249994|        214.379993|150476200|27.774976000000002|
|2010-01-06 00:00:00|214.379993|    215.23|        210.750004|        210.969995|138040000|27.333178000000004|
|2010-01-07 00:00:00|    211.75|212.000006|        209.050005|            210.58|119282800|          27.28265|
|2010-01-08 00:00:00|210.299994|212.000006|209.06000500000002|211.98000499999998|111902700|         27.464034|
+-------------------+----------+----------+------------------+------------------+---------+------------------+
only showing top 5 rows

df.orderBy(df['Date'].desc()).show(5)
df.orderBy('Date',ascending=False).show(5)

+-------------------+----------+----------+----------+----------+--------+------------------+
|               Date|      Open|      High|       Low|     Close|  Volume|         Adj Close|
+-------------------+----------+----------+----------+----------+--------+------------------+
|2016-12-30 00:00:00|116.650002|117.199997|    115.43|    115.82|30586300|         115.32002|
|2016-12-29 00:00:00|116.449997|117.110001|116.400002|116.730003|15039500|        116.226096|
|2016-12-28 00:00:00|117.519997|118.019997|116.199997|116.760002|20905900|116.25596499999999|
|2016-12-27 00:00:00|116.519997|117.800003|116.489998|117.260002|18296900|116.75380600000001|
|2016-12-23 00:00:00|115.589996|116.519997|115.589996|116.519997|14249500|        116.016995|
+-------------------+----------+----------+----------+----------+--------+------------------+
only showing top 5 rows

+-------------------+----------+----------+----------+----------+--------+------------------+
|               Date|      Open|      High|       Low|     Close|  Volume|         Adj Close|
+-------------------+----------+----------+----------+----------+--------+------------------+
|2016-12-30 00:00:00|116.650002|117.199997|    115.43|    115.82|30586300|         115.32002|
|2016-12-29 00:00:00|116.449997|117.110001|116.400002|116.730003|15039500|        116.226096|
|2016-12-28 00:00:00|117.519997|118.019997|116.199997|116.760002|20905900|116.25596499999999|
|2016-12-27 00:00:00|116.519997|117.800003|116.489998|117.260002|18296900|116.75380600000001|
|2016-12-23 00:00:00|115.589996|116.519997|115.589996|116.519997|14249500|        116.016995|
+-------------------+----------+----------+----------+----------+--------+------------------+
only showing top 5 rows

（6）agg（聚合函数，可对多列执行聚合操作）

df.agg({'Volume':'sum'}).show()

+------------+
| sum(Volume)|
+------------+
|166025817100|
+------------+

df.groupBy('Date').agg({'Volume':'mean'}).show(5)

+-------------------+-----------+
|               Date|avg(Volume)|
+-------------------+-----------+
|2012-03-12 00:00:00| 1.018206E8|
|2012-11-23 00:00:00|  6.82066E7|
|2013-02-19 00:00:00| 1.089459E8|
|2013-10-08 00:00:00|  7.27293E7|
|2015-05-18 00:00:00|  5.08829E7|
+-------------------+-----------+
only showing top 5 rows

df.agg({'Volume':'max','Volume':'min'}).show()

+-----------+
|min(Volume)|
+-----------+
|   11475900|
+-----------+

from pyspark.sql import functions as F
df.agg(F.min(df.Volume),F.min(df.Open)).show()

+-----------+---------+
|min(Volume)|min(Open)|
+-----------+---------+
|   11475900|     90.0|
+-----------+---------+

spark.stop()
一只齐刘海的猫
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
PySpark启动过程解密
Hadoop_SC的博客
03-01 1283
原文作者：李海强，来自平安银行零售大数据团队 ​ 前言 作为数据工程师，你可能会碰到过很多种启动PySpark的方法，可能搞不懂这些方法有什么共同点、有什么区别，不同的方法对程序开发、部署有什么影响，今天我们一起分析一下这些启动PySpark的方法。 以下代码分析都是基于spark-2.4.4版本展开的，为了避免歧义，务必对照这个版本的Spark深入理解。 启动PySpark的方法 启动PySp...
pyspark基础操作
小猪课堂的博客
08-08 1088
最近在公司经常会用到一些spark,一般都是使用python进行处理,一开始使用还是有点陌生,看了几篇往上的文章也简单的操作了几天,通过写这篇文章再去巩固一下基础知识~持续更新 提示(tips) 这里要说一下spark中每一个环节都是分开的; spark函数使用的是驼峰命名法; 代码写在一行上,如果过长换行必须有"",否则会报错(此坑已踩!!); 基础查询 select where ->> where/filter group by ->> groupby case when 函
Spark源码分析：多种部署方式之间的区别与联系
sdujava2011
01-26 588
作者：过往记忆 从官方的文档我们可以知道， Spark 的部署方式有很多种：local、Standalone、Mesos、YARN…..不同部署方式的后台处理进程是不一样的，但是如果我们从代码的角度来看，其实流程都差不多。 从代码中，我们可以得知其实 Spark 的部署方式其实比官方文档中介绍的还要多，这里我来列举一下： 1、local：这种方式是在本地启
Spark Final-Examination Note
橘嘉禾
06-23 191
重点 4567，Special 4,5 3道编程题，分值 35(基本书上，不超纲) 选填各 20分，简答 5 题/25分，编程题 3题/35分、 简答题参考每章课后习题 第 1 章简答题 1． 请阐述大数据处理的基本流程。 数据采集与预处理 数据存储与管理 数据处理与分析 数据可视化 2． 请阐述大数据的计算模式及其代表产品。(记 Title) 批处理计算 流计算 图计算 查询分析计算 ...
pyspark 启动命令汇总 local、yarn、standalone
a8131357leo的博客
09-07 5911
Table of Contents 0. 启动Pyspark 1.Hadoop Yarn 启动 pyspark 2.Spark Standalone模式运行 pyspark 3. 从Jupyter启动Pyspark 1. spark-submit 启动命令实在是太多了。。记录下 0. 启动Pyspark 默认情况下，pyspark 会以 spark-shell启动 ...
pyspark启动与简单使用----本地模式（local）----shell
热门推荐
莫安逸无风雪
09-15 1万+
在Spark中采用本地模式启动pyspark的命令主要包含以下参数：–master：这个参数表示当前的pyspark要连接到哪个master，如果是local[*]，就是使用本地模式启动pyspark，其中，中括号内的星号表示需要使用几个CPU核心(core)；–jars： 这个参数用于把相关的JAR包添加到CLASSPATH中；如果有多个jar包，可以使用逗号分隔符连接它们； 1比如，要采用本...
pyspark启动遇到的问题及解决方法
不懂语言的猿
10-08 1万+
本文主要解决pyspark启动时常见的两种问题，以及pyspark启动后Tab无法补全的问题。
Pyspark入门
weixin_38087754的博客
05-01 183
RDD（Resilent Distributed Dataset） Resilent对丢失节点，丢失数据集的修复 Distributed分布式，运行在不同节点上 Dataset数据集 RDD是spark最基本的抽象，是不可变的（每个RDD生成后就不再改变，所有的操作都是生成一个新的RDD，因为是并行化计算，如果在原有基础上进行修改，那么不得不浪费时间在同步通信上） 可以进行分割的， 分...
pyspark基础知识
最新发布
10-12
pyspark基础知识
PySpark基础入门与快速上手（完整视频+课件+代码+软件工具）
11-10
关于Python+Spark 的入门安装、配置，示例与快速入门与基础案例等的视频讲解，配置讲义、代码、笔记、软件等，原价值千元内部培训教程现免费对外公开。
distributed-computing-pyspark:使用PySpark进行分布式计算
04-02
分布式计算-PySpark 该存储库包含有关在Python中使用Spark进行分布式计算的微型项目。 文本分析：PySpark中的逐点相互信息 计算文本文件中出现的所有单词的一个或多个标记的PMI。 图/网络分析：PySpark中的个性化PageRank算法 实现PageRank算法的修改版本，其中参照给定的源节点执行排名。 修改有两个方面： 随机仅跳到源节点 由于节点悬空而造成的质量损失将完全转移到源节点，而不是在整个图形上重新分配 使用Spark数据帧和Spark SQL查询TPCH
Learning PySpark
11-14
首先，书中详细讲解了PySpark的基础知识，包括安装和配置环境，以及如何启动Spark集群。读者将了解如何使用`pyspark` shell进行交互式数据分析，掌握Spark的核心概念，如RDD（弹性分布式数据集）和DataFrame。 接着...
Pyspark
03-17
大数据的Python和火花 适用于Python的课程笔记本和适用于大数据的Spark 课程大纲： 课程介绍 促销/介绍视频 课程大纲 Spark，RDD和Spark 2.0简介 课程设置 设置概述 EC2安装指南 ...带有PySpark示例
Learning PySpark英文版PDF+书中代码+安装教程
04-19
**PySpark基础知识** - **Spark核心概念**：Spark是一个分布式计算框架，其核心理念是提供快速、通用且可扩展的数据处理。它支持批处理、交互式查询（通过Spark SQL）和实时流处理。 - **Resilient Distributed ...
PySpark大数据分析(6)：数据读写
唐犁的博客
05-25 757
PySpark可以直接将一个文本文件读取进来，变成一个RDD，将文件的每一行作为RDD中的一个元素。使用saveAsTextFile()可以将数据保存为文本文件。PySpark会将传入路径作为文件夹，从多个节点在其下并行输出多个文件，因此数据的每一个部分输出到哪个文件中往往是不受控的。
Spark启动时的master参数以及Spark的部署方式
Zsigner的博客
03-13 323
推荐大家去看原文博主的文章，条理清晰阅读方便，转载是为了方便以后查阅 https://www.jianshu.com/p/aaac505908dd 我们在初始化SparkConf时，或者提交Spark任务时，都会有master参数需要设置，如下： conf = SparkConf().setAppName(appName).setMaster(master) sc = SparkConte...
Spark 用代码实现求分位数Percentile（Quentile）的方法
元元的李树专栏
03-16 1万+
参考下文得到的启发 https://stackoverflow.com/questions/28805602/how-to-compute-percentiles-in-apache-spark 简单说明下分位数的定义 Scala求分位数的方法： /** * compute percentile from an unsorted Spark RDD * @param ...
pyspark求近似分位数。添加index
qq_26921849的博客
11-30 3312
求近似分位数： DataFrame.approxQuantile()方法   对DataFrame添加index： 先建窗口，再用sql.functions的row_number() 例： from pyspark.sql import functions as F w = Window.orderBy("aggressive") withIndexDF = tmpDF.withCo...
python求四分位数_pyspark根据ID计算四分位数，并根据四分位数范围进行分类
weixin_39957318的博客
12-29 1049
我使用的是pyspark 1.5.2。我有一个列为“ID”和“Height”的pyspark数据帧，如下所示：| ID1| ID2| height|---------------------------------------------| 1| 000| 44|| ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值