spark sql并行读取实践

spark sql 并行查询

第一种使用指定分区列的方式

http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases
partitionColumn must be a numeric, date, or timestamp column from the table in question.

partitionColumn, lowerBound, upperBound These options must all be specified if any of them is specified. In addition, numPartitions must be specified. They describe how to partition the table when reading in parallel from multiple workers. partitionColumn must be a numeric column from the table in question. Notice that lowerBound and upperBound are just used to decide the partition stride, not for filtering the rows in table. So all rows in the table will be partitioned and returned. This option applies only to reading.

numPartitions The maximum number of partitions that can be used for parallelism in table reading and writing. This also determines the maximum number of concurrent JDBC connections. If the number of partitions to write exceeds this limit, we decrease it to this limit by calling coalesce(numPartitions) before writing.

fetchsize The JDBC fetch size, which determines how many rows to fetch per round trip. This can help performance on JDBC drivers which default to low fetch size (eg. Oracle with 10 rows). This option applies only to reading.

batchsize The JDBC batch size, which determines how many rows to insert per round trip. This can help performance on JDBC drivers. This option applies only to writing. It defaults to 1000.

Dataset<Row> pgJdbcDF2 = spark.read()
    .format("jdbc")
    .option("url", "jdbc:postgresql://ip:15400/postgres?gssEncMode=disable")
    .