spark读取文件源码分析-3

最新推荐文章于 2023-04-28 16:11:02 发布
最新推荐文章于 2023-04-28 16:11:02 发布
阅读量1.1k 收藏 1
点赞数
分类专栏： spark 文章标签： spark 大数据 hadoop scala
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u013200380/article/details/108578335
版权

本篇是spark read一个parquet源码分析的第三篇，这一篇主要介绍spark的默认的partition的设置逻辑，当然，这一篇实际上算不上源码分析了
第一篇
第二篇

1 . userProfileSource 的partition数量决定因素

这一块儿之前总是看到说是由文件的大小决定的，每个block是一个partition（一般是128M,可以在hdfs上设置），但是分析一般是对于单个文件做的分析，这里尝试对多个文件进行分析。

1. 增加调试代码

为了调试rdd partition的具体信息，增加了下面的代码

userProfileSource.javaRDD().getNumPartitions();

代码变成


public class UserProfileTest {

    static String filePath = "hdfs://test:9000/user/daily/20200828/*.parquet";
    public static void main(String[] args) {
        SparkConf sparkConf = new SparkConf()
                .setMaster("local")
                .setAppName("user_profile_test")
                .set(ConfigurationOptions.ES_NODES, "")
                .set(ConfigurationOptions.ES_PORT, "")
                .set(ConfigurationOptions.ES_MAPPING_ID, "uid");

        SparkSession sparkSession = SparkSession.builder().config(sparkConf).getOrCreate();
        Dataset<Row> userProfileSource = sparkSession.read().parquet(filePath);
        userProfileSource.javaRDD().getNumPartitions();
        userProfileSource.count();
        userProfileSource.write().parquet("hdfs:///user/daily/result2020091008/");
    }
}

2. 对应目录下的文件信息


# hsl -h hdfs://test:9000/user/daily/20200828/ |head
    184.2 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00000-41ca-c000.snappy.parquet
    182.2 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00001-41ca-c000.snappy.parquet
    183.3 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00002-41ca-c000.snappy.parquet
    183.2 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00003-41ca-c000.snappy.parquet
    183.6 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00004-41ca-c000.snappy.parquet
    182.7 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00005-41ca-c000.snappy.parquet
    183.0 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00006-41ca-c000.snappy.parquet
    182.1 M 2020-09-02 21:19 hdfs://test:9000/user/daily/20200828/part-00007-41ca-c000.snappy.parquet

可以看到文件的大小基本都是180M,总共有100个这样的文件。

3. rdd的partition信息

在调试中和程序运行中也可以看到(对应的job图)，rdd有150个partition,为了方便把前文中的DAG图再这里再粘贴一遍。下图中count的第一个stage就是有150个task运行。
在这里插入图片描述

这里我就不再仔细的调试rdd生成的逻辑了，直接给出rdd的分区信息,这个分区信息是通过debug模式从rdd中获取到的，下面显示了150个partition的具体情况。


result = {Partition[150]@13053}
 0 = {FilePartition@13054} "FilePartition(0,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00000-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 1 = {FilePartition@13055} "FilePartition(1,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00001-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 2 = {FilePartition@13056} "FilePartition(2,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00002-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 3 = {FilePartition@13057} "FilePartition(3,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00003-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 4 = {FilePartition@13058} "FilePartition(4,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00004-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 5 = {FilePartition@13059} "FilePartition(5,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00005-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 6 = {FilePartition@13060} "FilePartition(6,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00006-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 7 = {FilePartition@13061} "FilePartition(7,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00007-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 8 = {FilePartition@13062} "FilePartition(8,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00008-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 9 = {FilePartition@13063} "FilePartition(9,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00009-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 10 = {FilePartition@13064} "FilePartition(10,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00010-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 11 = {FilePartition@13065} "FilePartition(11,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00011-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 12 = {FilePartition@13066} "FilePartition(12,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00012-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 13 = {FilePartition@13067} "FilePartition(13,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00013-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 14 = {FilePartition@13068} "FilePartition(14,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00014-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 15 = {FilePartition@13069} "FilePartition(15,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00015-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 16 = {FilePartition@13070} "FilePartition(16,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00016-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 17 = {FilePartition@13071} "FilePartition(17,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00017-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 18 = {FilePartition@13072} "FilePartition(18,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00018-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 19 = {FilePartition@13073} "FilePartition(19,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00019-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 20 = {FilePartition@13074} "FilePartition(20,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00020-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 21 = {FilePartition@13075} "FilePartition(21,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00021-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 22 = {FilePartition@13076} "FilePartition(22,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00022-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 23 = {FilePartition@13077} "FilePartition(23,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00023-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 24 = {FilePartition@13078} "FilePartition(24,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00024-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 25 = {FilePartition@13079} "FilePartition(25,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00025-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 26 = {FilePartition@13080} "FilePartition(26,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00026-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 27 = {FilePartition@13081} "FilePartition(27,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00027-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 28 = {FilePartition@13082} "FilePartition(28,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00028-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 29 = {FilePartition@13083} "FilePartition(29,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00029-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 30 = {FilePartition@13084} "FilePartition(30,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00030-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 31 = {FilePartition@13085} "FilePartition(31,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00031-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 32 = {FilePartition@13086} "FilePartition(32,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00032-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 33 = {FilePartition@13087} "FilePartition(33,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00033-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 34 = {FilePartition@13088} "FilePartition(34,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00034-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 35 = {FilePartition@13089} "FilePartition(35,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00035-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 36 = {FilePartition@13090} "FilePartition(36,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00036-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 37 = {FilePartition@13091} "FilePartition(37,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00037-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 38 = {FilePartition@13092} "FilePartition(38,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00038-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 39 = {FilePartition@13093} "FilePartition(39,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00039-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 40 = {FilePartition@13094} "FilePartition(40,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00040-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 41 = {FilePartition@13095} "FilePartition(41,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00041-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 42 = {FilePartition@13096} "FilePartition(42,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00042-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 43 = {FilePartition@13097} "FilePartition(43,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00043-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 44 = {FilePartition@13098} "FilePartition(44,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00044-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 45 = {FilePartition@13099} "FilePartition(45,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00045-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 46 = {FilePartition@13100} "FilePartition(46,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00046-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 47 = {FilePartition@13101} "FilePartition(47,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00047-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 48 = {FilePartition@13102} "FilePartition(48,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00048-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 49 = {FilePartition@13103} "FilePartition(49,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00049-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 50 = {FilePartition@13104} "FilePartition(50,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00050-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 51 = {FilePartition@13105} "FilePartition(51,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00051-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 52 = {FilePartition@13106} "FilePartition(52,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00052-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 53 = {FilePartition@13107} "FilePartition(53,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00053-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 54 = {FilePartition@13108} "FilePartition(54,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00054-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 55 = {FilePartition@13109} "FilePartition(55,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00055-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 56 = {FilePartition@13110} "FilePartition(56,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00056-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 57 = {FilePartition@13111} "FilePartition(57,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00057-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 58 = {FilePartition@13112} "FilePartition(58,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00058-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 59 = {FilePartition@13113} "FilePartition(59,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00059-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 60 = {FilePartition@13114} "FilePartition(60,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00060-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 61 = {FilePartition@13115} "FilePartition(61,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00061-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 62 = {FilePartition@13116} "FilePartition(62,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00062-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 63 = {FilePartition@13117} "FilePartition(63,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00063-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 64 = {FilePartition@13118} "FilePartition(64,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00064-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 65 = {FilePartition@13119} "FilePartition(65,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00065-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 66 = {FilePartition@13120} "FilePartition(66,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00066-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 67 = {FilePartition@13121} "FilePartition(67,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00067-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 68 = {FilePartition@13122} "FilePartition(68,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00068-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 69 = {FilePartition@13123} "FilePartition(69,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00069-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 70 = {FilePartition@13124} "FilePartition(70,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00070-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 71 = {FilePartition@13125} "FilePartition(71,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00071-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 72 = {FilePartition@13126} "FilePartition(72,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00072-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 73 = {FilePartition@13127} "FilePartition(73,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00073-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 74 = {FilePartition@13128} "FilePartition(74,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00074-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 75 = {FilePartition@13129} "FilePartition(75,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00075-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 76 = {FilePartition@13130} "FilePartition(76,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00076-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 77 = {FilePartition@13131} "FilePartition(77,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00077-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 78 = {FilePartition@13132} "FilePartition(78,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00078-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 79 = {FilePartition@13133} "FilePartition(79,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00079-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 80 = {FilePartition@13134} "FilePartition(80,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00080-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 81 = {FilePartition@13135} "FilePartition(81,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00081-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 82 = {FilePartition@13136} "FilePartition(82,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00082-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 83 = {FilePartition@13137} "FilePartition(83,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00083-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 84 = {FilePartition@13138} "FilePartition(84,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00084-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 85 = {FilePartition@13139} "FilePartition(85,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00085-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 86 = {FilePartition@13140} "FilePartition(86,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00086-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 87 = {FilePartition@13141} "FilePartition(87,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00087-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 88 = {FilePartition@13142} "FilePartition(88,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00088-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 89 = {FilePartition@13143} "FilePartition(89,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00089-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 90 = {FilePartition@13144} "FilePartition(90,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00090-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 91 = {FilePartition@13145} "FilePartition(91,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00091-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 92 = {FilePartition@13146} "FilePartition(92,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00092-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 93 = {FilePartition@13147} "FilePartition(93,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00093-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 94 = {FilePartition@13148} "FilePartition(94,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00094-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 95 = {FilePartition@13149} "FilePartition(95,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00095-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 96 = {FilePartition@13150} "FilePartition(96,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00096-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 97 = {FilePartition@13151} "FilePartition(97,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00097-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 98 = {FilePartition@13152} "FilePartition(98,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00098-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 99 = {FilePartition@13153} "FilePartition(99,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00099-41ca-c000.snappy.parquet, range: 0-134217728, partition values: [empty row]))"
 100 = {FilePartition@13265} "FilePartition(100,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00020-41ca-c000.snappy.parquet, range: 134217728-195123990, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00009-41ca-c000.snappy.parquet, range: 134217728-195046711, partition values: [empty row]))"
 101 = {FilePartition@13266} "FilePartition(101,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00051-41ca-c000.snappy.parquet, range: 134217728-194146721, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00078-41ca-c000.snappy.parquet, range: 134217728-193958286, partition values: [empty row]))"
 102 = {FilePartition@13267} "FilePartition(102,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00072-41ca-c000.snappy.parquet, range: 134217728-193883957, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00017-41ca-c000.snappy.parquet, range: 134217728-193740729, partition values: [empty row]))"
 103 = {FilePartition@13268} "FilePartition(103,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00052-41ca-c000.snappy.parquet, range: 134217728-193733952, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00035-41ca-c000.snappy.parquet, range: 134217728-193630805, partition values: [empty row]))"
 104 = {FilePartition@13269} "FilePartition(104,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00060-41ca-c000.snappy.parquet, range: 134217728-193595463, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00040-41ca-c000.snappy.parquet, range: 134217728-193582950, partition values: [empty row]))"
 105 = {FilePartition@13270} "FilePartition(105,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00097-41ca-c000.snappy.parquet, range: 134217728-193553823, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00088-41ca-c000.snappy.parquet, range: 134217728-193318554, partition values: [empty row]))"
 106 = {FilePartition@13271} "FilePartition(106,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00028-41ca-c000.snappy.parquet, range: 134217728-193247136, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00081-41ca-c000.snappy.parquet, range: 134217728-193238350, partition values: [empty row]))"
 107 = {FilePartition@13272} "FilePartition(107,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00083-41ca-c000.snappy.parquet, range: 134217728-193192582, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00057-41ca-c000.snappy.parquet, range: 134217728-193169659, partition values: [empty row]))"
 108 = {FilePartition@13273} "FilePartition(108,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00000-41ca-c000.snappy.parquet, range: 134217728-193154555, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00096-41ca-c000.snappy.parquet, range: 134217728-193056440, partition values: [empty row]))"
 109 = {FilePartition@13274} "FilePartition(109,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00027-41ca-c000.snappy.parquet, range: 134217728-193040434, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00089-41ca-c000.snappy.parquet, range: 134217728-193010886, partition values: [empty row]))"
 110 = {FilePartition@13275} "FilePartition(110,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00012-41ca-c000.snappy.parquet, range: 134217728-192989901, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00099-41ca-c000.snappy.parquet, range: 134217728-192957150, partition values: [empty row]))"
 111 = {FilePartition@13276} "FilePartition(111,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00021-41ca-c000.snappy.parquet, range: 134217728-192890064, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00091-41ca-c000.snappy.parquet, range: 134217728-192871474, partition values: [empty row]))"
 112 = {FilePartition@13277} "FilePartition(112,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00030-41ca-c000.snappy.parquet, range: 134217728-192862266, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00058-41ca-c000.snappy.parquet, range: 134217728-192846129, partition values: [empty row]))"
 113 = {FilePartition@13278} "FilePartition(113,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00092-41ca-c000.snappy.parquet, range: 134217728-192831718, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00049-41ca-c000.snappy.parquet, range: 134217728-192826946, partition values: [empty row]))"
 114 = {FilePartition@13279} "FilePartition(114,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00087-41ca-c000.snappy.parquet, range: 134217728-192797029, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00053-41ca-c000.snappy.parquet, range: 134217728-192723879, partition values: [empty row]))"
 115 = {FilePartition@13280} "FilePartition(115,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00064-41ca-c000.snappy.parquet, range: 134217728-192715391, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00042-41ca-c000.snappy.parquet, range: 134217728-192710676, partition values: [empty row]))"
 116 = {FilePartition@13281} "FilePartition(116,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00055-41ca-c000.snappy.parquet, range: 134217728-192685175, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00033-41ca-c000.snappy.parquet, range: 134217728-192652367, partition values: [empty row]))"
 117 = {FilePartition@13282} "FilePartition(117,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00056-41ca-c000.snappy.parquet, range: 134217728-192639026, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00036-41ca-c000.snappy.parquet, range: 134217728-192636886, partition values: [empty row]))"
 118 = {FilePartition@13283} "FilePartition(118,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00077-41ca-c000.snappy.parquet, range: 134217728-192628108, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00037-41ca-c000.snappy.parquet, range: 134217728-192619533, partition values: [empty row]))"
 119 = {FilePartition@13284} "FilePartition(119,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00016-41ca-c000.snappy.parquet, range: 134217728-192610267, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00004-41ca-c000.snappy.parquet, range: 134217728-192553300, partition values: [empty row]))"
 120 = {FilePartition@13285} "FilePartition(120,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00038-41ca-c000.snappy.parquet, range: 134217728-192550224, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00079-41ca-c000.snappy.parquet, range: 134217728-192528242, partition values: [empty row]))"
 121 = {FilePartition@13286} "FilePartition(121,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00048-41ca-c000.snappy.parquet, range: 134217728-192518314, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00014-41ca-c000.snappy.parquet, range: 134217728-192492734, partition values: [empty row]))"
 122 = {FilePartition@13287} "FilePartition(122,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00043-41ca-c000.snappy.parquet, range: 134217728-192488828, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00075-41ca-c000.snappy.parquet, range: 134217728-192457423, partition values: [empty row]))"
 123 = {FilePartition@13288} "FilePartition(123,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00080-41ca-c000.snappy.parquet, range: 134217728-192436303, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00086-41ca-c000.snappy.parquet, range: 134217728-192422791, partition values: [empty row]))"
 124 = {FilePartition@13289} "FilePartition(124,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00010-41ca-c000.snappy.parquet, range: 134217728-192401383, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00031-41ca-c000.snappy.parquet, range: 134217728-192390802, partition values: [empty row]))"
 125 = {FilePartition@13290} "FilePartition(125,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00073-41ca-c000.snappy.parquet, range: 134217728-192384328, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00068-41ca-c000.snappy.parquet, range: 134217728-192373709, partition values: [empty row]))"
 126 = {FilePartition@13291} "FilePartition(126,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00045-41ca-c000.snappy.parquet, range: 134217728-192324641, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00074-41ca-c000.snappy.parquet, range: 134217728-192320288, partition values: [empty row]))"
 127 = {FilePartition@13292} "FilePartition(127,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00054-41ca-c000.snappy.parquet, range: 134217728-192310762, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00013-41ca-c000.snappy.parquet, range: 134217728-192309053, partition values: [empty row]))"
 128 = {FilePartition@13293} "FilePartition(128,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00029-41ca-c000.snappy.parquet, range: 134217728-192307665, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00085-41ca-c000.snappy.parquet, range: 134217728-192300557, partition values: [empty row]))"
 129 = {FilePartition@13294} "FilePartition(129,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00023-41ca-c000.snappy.parquet, range: 134217728-192287705, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00026-41ca-c000.snappy.parquet, range: 134217728-192244675, partition values: [empty row]))"
 130 = {FilePartition@13295} "FilePartition(130,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00002-41ca-c000.snappy.parquet, range: 134217728-192242883, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00032-41ca-c000.snappy.parquet, range: 134217728-192218855, partition values: [empty row]))"
 131 = {FilePartition@13296} "FilePartition(131,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00062-41ca-c000.snappy.parquet, range: 134217728-192200827, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00050-41ca-c000.snappy.parquet, range: 134217728-192191229, partition values: [empty row]))"
 132 = {FilePartition@13297} "FilePartition(132,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00069-41ca-c000.snappy.parquet, range: 134217728-192169417, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00098-41ca-c000.snappy.parquet, range: 134217728-192160782, partition values: [empty row]))"
 133 = {FilePartition@13298} "FilePartition(133,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00025-41ca-c000.snappy.parquet, range: 134217728-192152594, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00003-41ca-c000.snappy.parquet, range: 134217728-192146313, partition values: [empty row]))"
 134 = {FilePartition@13299} "FilePartition(134,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00095-41ca-c000.snappy.parquet, range: 134217728-192096023, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00047-41ca-c000.snappy.parquet, range: 134217728-192084102, partition values: [empty row]))"
 135 = {FilePartition@13300} "FilePartition(135,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00008-41ca-c000.snappy.parquet, range: 134217728-192046162, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00046-41ca-c000.snappy.parquet, range: 134217728-192036123, partition values: [empty row]))"
 136 = {FilePartition@13301} "FilePartition(136,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00090-41ca-c000.snappy.parquet, range: 134217728-191994182, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00066-41ca-c000.snappy.parquet, range: 134217728-191989716, partition values: [empty row]))"
 137 = {FilePartition@13302} "FilePartition(137,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00006-41ca-c000.snappy.parquet, range: 134217728-191934315, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00061-41ca-c000.snappy.parquet, range: 134217728-191865624, partition values: [empty row]))"
 138 = {FilePartition@13303} "FilePartition(138,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00015-41ca-c000.snappy.parquet, range: 134217728-191799380, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00065-41ca-c000.snappy.parquet, range: 134217728-191686604, partition values: [empty row]))"
 139 = {FilePartition@13304} "FilePartition(139,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00011-41ca-c000.snappy.parquet, range: 134217728-191638428, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00094-41ca-c000.snappy.parquet, range: 134217728-191613916, partition values: [empty row]))"
 140 = {FilePartition@13305} "FilePartition(140,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00093-41ca-c000.snappy.parquet, range: 134217728-191569247, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00005-41ca-c000.snappy.parquet, range: 134217728-191566051, partition values: [empty row]))"
 141 = {FilePartition@13306} "FilePartition(141,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00070-41ca-c000.snappy.parquet, range: 134217728-191540698, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00022-41ca-c000.snappy.parquet, range: 134217728-191510871, partition values: [empty row]))"
 142 = {FilePartition@13307} "FilePartition(142,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00024-41ca-c000.snappy.parquet, range: 134217728-191501951, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00041-41ca-c000.snappy.parquet, range: 134217728-191431095, partition values: [empty row]))"
 143 = {FilePartition@13308} "FilePartition(143,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00019-41ca-c000.snappy.parquet, range: 134217728-191423121, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00063-41ca-c000.snappy.parquet, range: 134217728-191381412, partition values: [empty row]))"
 144 = {FilePartition@13309} "FilePartition(144,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00067-41ca-c000.snappy.parquet, range: 134217728-191378487, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00039-41ca-c000.snappy.parquet, range: 134217728-191304847, partition values: [empty row]))"
 145 = {FilePartition@13310} "FilePartition(145,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00018-41ca-c000.snappy.parquet, range: 134217728-191205335, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00034-41ca-c000.snappy.parquet, range: 134217728-191131303, partition values: [empty row]))"
 146 = {FilePartition@13311} "FilePartition(146,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00084-41ca-c000.snappy.parquet, range: 134217728-191112887, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00001-41ca-c000.snappy.parquet, range: 134217728-191061715, partition values: [empty row]))"
 147 = {FilePartition@13312} "FilePartition(147,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00044-41ca-c000.snappy.parquet, range: 134217728-191023822, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00007-41ca-c000.snappy.parquet, range: 134217728-190977337, partition values: [empty row]))"
 148 = {FilePartition@13313} "FilePartition(148,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00059-41ca-c000.snappy.parquet, range: 134217728-190961642, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00071-41ca-c000.snappy.parquet, range: 134217728-190749907, partition values: [empty row]))"
 149 = {FilePartition@13314} "FilePartition(149,WrappedArray(path: hdfs://test:9000/user/daily/20200828/part-00076-41ca-c000.snappy.parquet, range: 134217728-190698048, partition values: [empty row], path: hdfs://test:9000/user/daily/20200828/part-00082-41ca-c000.snappy.parquet, range: 134217728-190665081, partition values: [empty row]))"

这里我们首先关注前100个partition,这些partition对应的文件都只有一个，对应的range都是range: 0-134217728,对应的是128M，也就是hdfs存储的一个block。

再看后面的50个partition的情况
这里的每个partition对应的都是两个文件，拿最后一个partition,partition149来说

 149 = {FilePartition@13314}
"FilePartition(149,WrappedArray(

path: hdfs://test:9000/user/daily/20200828/part-00076-41ca-c000.snappy.parquet, range: 134217728-190698048, partition values: [empty row],
path: hdfs://test:9000/user/daily/20200828/part-00082-41ca-c000.snappy.parquet, range: 134217728-190665081, partition values: [empty row]

))"

对应的是两个文件，分别是range: 134217728-19069804853.8M 和range: 134217728-19066508153.8M
对应的是两个文件的第二个block，猜测一下为何这两个文件会被分配到同一个partition呢，我想应该是因为他们在hdfs上面属于同一个data-node，这样的话可以认为两个数据是放在一起的。

夜月行者
关注
专栏目录
Hudi Spark SQL源码学习总结-Create Table
主要分享大数据相关的知识，如Spark、Hudi
07-22 1702
简要总结HudiSparkSql源码执行逻辑，从建表开始。其实从去年开始接触Hudi的时候就研究学习了HudiSparkSQL的部分源码，并贡献了几个PR，但是完整的逻辑有些地方还没有完全梳理清楚，所以现在想要从头开始学习，搞懂一些知识难点，这样以后看相关源码的时候就不会导致因为一些关键点不懂影响进度。由于本人能力和精力有限，本文只讲解自己觉得比较关键的点，主要目的是梳理整个流程。搞懂了SparkSQL默认的sqlParser为,那么Hudi是一样的吗？那我们就需要看一下开始的这里的f为为。...
源码分析 --- Spark如何确定Splits和分区数
杨鑫newlife的专栏
05-29 1331
一、结论: 默认情况下Splits和Task和Partition数是一一对应的，数量是一样的； 1.如果在没有指定分区数的时候，默认的是sc.defaultMinPartitions=min(sc.defaultParallelism,2)。也就是sc.defaultMinPartitions只有两个值1和2，当sc.defaultParallelism>1时值为2，当sc.defaultParallelism=1时，值为1； 2.如果是读取HDFS的文件的话，分区是当我们计
Spark写Parquet源码分析
orisonchan的博客
01-23 2353
Spark层面 Spark层面我们从dataframe.write.parquet("path")说起。 In DataFrame Dataset中的write()方法： /** * Interface for saving the content of the non-streaming Dataset out into external storage. * * @group...
spark读取文件源码分析-1
寒夜
09-10 2639
文章目录1. 问题背景2. 测试代码3. 生成的DAG图1. job02. job14. job0 产生的时机源码分析1. 调用DataFrameReader.load，DataFrameReader.loadV1Source2. 调用DataSoure.resolveRelation方法3. 调用DataSource.getOrInferFileFormatSchema()4. InMemoryFileIndex 初始化5. 调用InMemoryFileIndex.bulkListLeafFiles 方法
Spark读取hdfs分片数据源码剖析
程序员小李的博客
05-11 609
val lines: RDD[String] = sparkContext.textFile(args(0)) 我们通过 sparkContext.textFile来读取hdfs中某一数据 正常来说，应该是有几个文件就创建几个分区 然而我们dubug的时候发现，有时候3个文件却创建了4个分区 其实这是由于spark中对于hdfs分片数据拉取的源码 首先 我们在用testFile时，若不指定分区数量，则默认使用最小值=2 long goalSize = totalSize / (long)(num.
67、Spark SQL之Parquet数据源之自动分区推断
ZJ__ZFH的博客
12-29 1479
自动分区推断 表分区是一种常见的优化方式，比如Hive中就提供了表分区的特性。在一个分区表中，不同分区的数据通常存储在不同的目录中，分区列的值通常就包含在了分区目录的目录名中。Spark SQL中的Parquet数据源，支持自动根据目录名推断出分区信息。例如，如果将人口数据存储在分区表中，并且使用性别和国家作为分区列。那么目录结构可能如下所示： tableName    |- gender=m...
Spark sql 读文件的源码分析
rover
08-17 1437
从spark jobs监控页面上经常看到这种job： Listing leaf files and directories for 100 paths: 如图： 这其实是spark sql在读一大堆文件。 最简单的demo语句，这样读文件： val df = session.read.json("path/to/your/resources/data.json") 或者 session.read.parquet(file_path) 或者 session.read.c...
使用Spark进行流量日志分析-内含源码以及设计说明书(可以自己运行复现).zip
最新发布
05-08
源码可能包含以下几个部分：数据读取（如从HDFS或本地文件系统读取日志文件），数据处理（执行上述的预处理和分析操作），以及结果的保存或可视化。 为了运行和复现这个项目，你需要一个Spark环境，可以是本地模式...
安卓毕业设计app项目源码6-ml-labs-spark-python:ml-labs-spark-python
06-06
【标题】"安卓毕业设计app项目源码6-ml-labs-spark-python:ml-labs-spark-python" 提供的是一个结合了安卓应用开发与大数据分析的毕业设计项目。这个项目利用了Apache Spark和Python来处理和分析数据，旨在展示如何...
Spark3.0 Sql 使用HiveTableScanExec 读取Hive orc表源码分析及参数调优
u014034497的专栏
10-29 3005
默认情况下因此，默认情况下，切片大小=blocksize。因为一般文件的block大小为128M, 是小于mapreduce.input.fileinputformat.split.maxsize默认值的根据orc文件的stripe，结合参数配置，最终生成split结论：因此使用能够控制生成split个数，进而控制读取的map task数量以下为设置为"67108864"时，spark sql 读取orc文件的示意图。
spark读取文件机制 源码剖析
weixin_30484247的博客
06-22 166
Spark数据分区调研 Spark以textFile方式读取文件源码 textFile方法位于 spark-core_2.11/org.apache.spark.api.java/JavaSparkContent.scala 参数分析： path ：String是一个URI，可以是HDFS、本地文件、或者其他Hadoop支持的文件系统 minPartitio...
spark源码阅读一-spark读写文件代码分析
David的专栏
08-01 2219
1.读取文件 从本地文件读取sparkcontext.textFile(“abc.txt”) 从hdfs文件读取sparkcontext.textFile("hdfs://s1:8020/user/hdfs/input”)
MySQL性能优化(五)：分区
每天叫醒自己的不是闹铃 是娟 是梦 是理想。
06-10 3075
由于公司使用的是mysql数据库，在存储hive表数据结构时，数量已经达到了千万级别，需要进行分区存储，对这方面在之前的hive外部分区表有些了解，mysql是不是相同的道理，这里我在同事那里看到一些文章，作为学习使用； 一: 分区简介 分区是根据一定的规则，数据库把一个表分解成多个更小的、更容易管理的部分。就访问数据库应用而言，逻辑上就只有一个表或者一个索引，但实际上这个表可能有N个物理分区对象组成，每个分区都是一个独立的对象，可以独立处理，可以作为表的一部分进行处理。分区对应用来说是完全透明的，不影
MySQL分区相关sql语法
IT_心如止水的博客
08-29 2149
分区的sql语句：  partition BY RANGE (store_id) (     partition p0 VALUES LESS THAN (6),     partition p1 VALUES LESS THAN (11),     partition p2 VALUES LESS THAN (16),     partition p3 VALUES LESS THAN (21)...
ORACLE 分区表 PARTITION table
12-27 4290
1.1 分区表PARTITION table在ORACLE里如果遇到特别大的表，可以使用分区的表来改变其应用程序的性能。1.1.1 分区表的建立： 某公司的每年产生巨大的销售记录，DBA向公司建议每季度的数据放在一个分区内，以下示范的是该公司1999年的数据(假设每月产生30M的数据)，操作如下： 范围分区表：CREATE TABLE sales (invoice_no
Spark基础知识
qq_33115589的博客
03-03 804
1 Apache Spark 是什么? Spark是一个在集群上运行的统一计算引擎以及一组并行数据处理软件库。 下面展示了Spark给用户提供的组件： 2 Spark的设计哲学 统一平台 Spark主要目标是为编写大数据应用程序提供统一的平台，例如，JavaEE的Spring框架。Spark的统一API使得编写大数据应用变得简单且高效。 计算引擎 Spark对比Hadoop这样即解决分布式存储（HDFS）又提供了计算范式（MapReduce）的大数据平台，它主要专注于计算引擎。Spark从
【Spark练习】val、var、range等基本操作练习
weixin_40433003的博客
04-28 371
要求1: 将操作结果进行截图，并上传到gitee中要求2：编写readme.md。
干货|Spark优化之高性能Range Join
大数据星球-浪尖
12-02 376
作者|张兴超编辑|林颖供稿|ADI Carmel Team本文共3884字，预计阅读时间10分钟导 读Carmel是eBay内部基于Apache Spark打造的一款SQL-on-Had...
Spark2.X分布式弹性数据集
weixin_30788239的博客
03-18 289
跑一下这个结果 参考代码 package com.spark.test import org.apache.spark.sql.SparkSession import org.apache.spark.{SparkConf, SparkContext} object Test { def main(args: Array[S...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值