Spark-Doris-Connecter By pySpark_pyspark doris-CSDN博客

本文链接：https://blog.csdn.net/wangleigiser/article/details/124726219

准备Spark-Doris-Connecter环境，具体参照：

https://blog.csdn.net/wangleigiser/article/details/124724390?spm=1001.2014.3001.5501

输入如下命令进入pyspark

[root@xxx ~]# pyspark
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/05/12 10:29:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/05/12 10:29:26 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.2
      /_/

Using Python version 3.6.9 (default, Dec  8 2021 21:08:43)
Spark context Web UI available at http://jiafeng:4041
Spark context available as 'sc' (master = local[*], app id = local-1652322566766).
SparkSession available as 'spark'.
>>>

通过pysprk从Doris读取数据.

dorisSparkDF = spark.read.format("doris").option("doris.table.identifier", "mongo_doris.data_sync_test").option("doris.fenodes", "127.0.0.1:8030").option("user", "root").option("password", "").load()
# 显示5行数据
dorisSparkDF.show(5)

4 . 完成运行结果如下:

>>> dorisSparkDF = spark.read.format("doris").option("doris.table.identifier", "mongo_doris.data_sync_test").option("doris.fenodes", "127.0.0.1:8030").option("user", "root").option("password", "").load()
>>> dorisSparkDF.show(5)
+---+---+---------+--------------------+----+------+------------+-----------+----------+
|_id| id|user_name|         member_list| age|height|lucky_number|create_time|is_married|
+---+---+---------+--------------------+----+------+------------+-----------+----------+
|  3|  1|     alex|[Document{{key1=1...|20.0|  3.14|    123456.0| 2022-05-10|     false|
|  4|  1|     alex|Document{{key1=1.0}}|20.0|  3.14|    123456.0| 2022-05-10|     false|
|  2|  1|     alex|     [1.0, 2.0, 3.0]|20.0|  3.14|    123456.0| 2022-05-09|     false|
+---+---+---------+--------------------+----+------+------------+-----------+----------+