PySpark！！！

* Spark 主从结构

在集群上运行 Spark 应用的详细过程

在集群上运行 Spark 应用的详细过程 　　　
(1) 用户通过 spark-submit 脚本提交应用。 　　　
(2) spark-submit 脚本启动驱动器程序，调用用户定义的 main() 方法。 　　　
(3) > 驱动器程序与集群管理器通信，申请资源以启动执行器节点。 　　　
(4) 集群管理器为驱动器程序启动执行器节点。 　　　
(5) 驱动器进程执行用户应用中的操作。根据程序中所定义的对 RDD 的转化操作和行动操作，驱动器节点把工作以任务的形式发送到执行器进程。
(6) 任务在执行器程序中进行计算并保存结果。 　　　
(7) 如果驱动器程序的 main() 方法退出，或者调用了SparkContext.stop() ，驱动器程序会终止执行器进程，并且通过集群管理器释放资源。

Yarn

deploy-mode: yarn-cluster & yarn-client
！详解 Understanding the chemistry between Spark, YARN and HDFS

How to choose? spark-yarn-cluster vs client

Spark supports two modes for running on YARN, “yarn-cluster” mode and “yarn-client” mode. Broadly, yarn-cluster mode makes sense for production jobs, while yarn-client mode makes sense for interactive and debugging uses where you want to see your application’s output immediately.

In yarn-cluster mode, the driver program will run on the node where application master is running where as in yarn-client mode the driver program will run on the node on which job is submitted on centralized gateway node.

The yarn-cluster mode is not well suited to using Spark interactively, but the yarn-client mode is. Spark applications that require user input, like spark-shell and PySpark, need the Spark driver to run inside the client process that initiates the Spark application.
In yarn-client mode, the Application Master is merely present to request executor containers from YARN

启动/停止Hadoop集群：start-all.sh stop-all.sh

1. 启动NameNode, DataNode
2. 启动JournalNode, JournalNode在hdfs-site.xml中指定edits log存储的位置，主备NameNode共享数据，方便同步
3. 启动DFSZKFailoverController, HA会启用ZooKeeper Failover Controller
4. 启动YARN守护进程ResourceManager, NodeManager

启动/停止HDFS集群：start-dfs.sh stop-dfs.sh

1. 启动NameNode, DataNode
2. 启动JournalNode, JournalNode在hdfs-site.xml中指定edits log存储的位置，主备NameNode共享数据，方便同步
3. 启动DFSZKFailoverController, HA会启用ZooKeeper Failover Controller
   启动/停止YARN集群：start-yarn.sh stop-yarn.sh

hadoop - JournalNode

JournalNode图解
两个NameNode为了数据同步，会通过一组称作JournalNodes的独立进程进行相互通信。

* Start from wordcount

首先确保hadoop启动，hdfs可以使用
/etc/profile中设置pyspark使用miniconda中虚拟环境thjj的python

export PYSPARK_PYTHON=/home/jj/miniconda3/envs/thjj/bin/python  

# 打印 hdfs 根目录下的所有文件
jj@JJ:~/Desktop$ hadoop dfs -l

jj@JJ:~/Desktop$ pyspark
Python 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
2021-08-21 15:37:09,218 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.2
      /_/

Using Python version 3.6.13 (default, Jun  4 2021 14:25:59)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local, app id = local-1629531429971).
SparkSession available as 'spark'.
>>> from pyspark import SparkContext, SparkConf
>>>logFile=sc.textFile("hdfs://localhost:9000/wordcount/exm_hadoop.txt")
### 也可以这样写：logFile=sc.textFile("/wordcount/exm_hadoop.txt")
### 注      //是访问hdfs文件， ///是访问本地文件
>>> logFile.count()
7

* spark-submit

#!/usr/bin/env python
#-*-conding:utf-8-*-

# here: spark_wordcount.py

import logging
from operator import add
from pyspark import SparkContext
import sys

logging.basicConfig(format='%(message)s', level=logging.INFO)

files = "/wordcount"

def main():
    sc = SparkContext("local","exm_hadoop")
    # 将hdfs中的文件成rdd
    rdd = sc.textFile(files)
    # 统计单词个数
    counts = rdd.flatMap(lambda line: line.split(" ")).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
    # 在控制台中打印输出
    results = counts.collect()
    for result in results:
        print("%s\t\t%d" % (result[0],result[1]))
    print("Jobs Done!")
    sc.stop()
if __name__=="__main__":
       main()

shell ：

# standalnoe
jj@JJ:~/Documents/JJ-Files$ spark-submit --master spark://jj:7077 spark_wordcount.py
# yarn-client
jj@JJ:~/Documents/JJ-Files$ spark-submit --master yarn --deploy-mode client spark_wordcount.py

spark on yarn

【conf/spark-env.sh】
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/home/jj/Downloads/hadoop-3.1.2
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export SPARK_MASTER_IP=172.16.158.143
export SPARK_LOCAL_IP=127.0.0.1
export SPAPK_LOCAL_DIRS=/home/jj/Downloads/spark-3.1.2-bin-hadoop3.2
export SPARK_DRIVER_MEMORY=1G

【conf/spark-defaults.conf】
spark.master=local

/etc/profile