1-读取小文件的函数wholeTextFiles

读取小文件的函数wholeTextFiles

测试有四个文件a.txt,b.txt,c.txt,d.txt

• # !/usr/bin/env python
  # -*-coding:utf-8 -*-
  
  """
  # File       : _03_create_rdd.py
  # Time       ：2022/4/18 14:27
  # Author     ：author name
  # version    ：python 3.6
  # Description：
  """
  from pyspark import SparkConf, SparkContext
  
  if __name__ == '__main__':
      print("处理相关小文件的问题")
      # 构建SparkContext对象
      conf = SparkConf().setMaster("local[*]").setAppName("03_create_rdd")
      sc = SparkContext(conf=conf)
      # 读取外部的数据源
      rdd_res1 = sc.textFile("C:\\Users\\aaa\\PycharmProjects\\pyspark_parent\\_02_pyspark_core\\data")
      rdd_res2 = sc.wholeTextFiles('C:\\Users\\aaa\\PycharmProjects\\pyspark_parent\\_02_pyspark_core\\data')
      print(rdd_res1.getNumPartitions())
      print(rdd_res1.collect())
      print("--------------------")
      print(rdd_res2.getNumPartitions())
      print(rdd_res2.collect())
      # 关闭相关的sc 服务
      sc.stop()
  
  

• 相关结果如下

  • 4
    ['a.txt', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'b.txt', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'c.txt', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'd.txt', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop', 'hello hello hive hive', 'sqoop sqoop hadoop hadoop']
    --------------------
    2
    [('file:/C:/Users/aaa/PycharmProjects/pyspark_parent/_02_pyspark_core/data/a.txt', 'a.txt\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop'), ('file:/C:/Users/aaa/PycharmProjects/pyspark_parent/_02_pyspark_core/data/b.txt', 'b.txt\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop'), ('file:/C:/Users/aaa/PycharmProjects/pyspark_parent/_02_pyspark_core/data/c.txt', 'c.txt\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop'), ('file:/C:/Users/aaa/PycharmProjects/pyspark_parent/_02_pyspark_core/data/d.txt', 'd.txt\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop\r\nhello hello hive hive\r\nsqoop sqoop hadoop hadoop')]
    
    

    从上面的操作来看，总结如下：

    用textFile时，它的partition的数量是与文件夹下的**文件数量（实例中用4个xxx.txt文件）**相关，一个文件就是一个partition(既然4个文件就是：partition=4)。
    wholeTextFiles的partition数量是根据用户指定或者文件大小来（文件内的数据量少 有hdfs源码默认确定的）确定，与hdfs目录下的文件数量无关！ 所以说：wholeTextFile通常用于读取许多小文件的需求。