域嵌套太深_pyspark如何修改嵌套结构域

域嵌套太深

In our adventures trying to build a data lake, we are using dynamically generated spark cluster to ingest some data from MongoDB, our production database, to BigQuery. In order to do that, we use PySpark data frames and since mongo doesn’t have schemas, we try to infer the schema from the data.

在尝试建立数据湖的冒险中，我们使用动态生成的火花集群将一些数据从生产数据库MongoDB提取到BigQuery。 为此，我们使用PySpark数据帧，并且由于mongo没有架构，因此我们尝试从数据中推断出架构。

collection_schema = spark.read.format(“mongo”) \ 
.option(“database”, db) \ 
.option(“collection”, coll) \ 
.option(‘sampleSize’, 50000) \ 
.load() \ 
.schema ingest_df = spark.read.format(“mongo”) \ 
.option(“database”, db) \ 
.option(“collection”, coll) \ .load(schema=fix_spark_schema(collection_schema))

Our fix_spark_schema method just converts NullType columns to String.

我们的fix_spark_schema方法仅将NullType列转换为String。

In the users collection, we have the groups field, which is an array, because users can join multiple groups.

在users集合中，我们拥有groups字段，它是一个数组，因为用户可以加入多个group。

root
 |-- groups: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- programs: struct (nullable = true)
 |    |    |    |-- **{ program id }**: struct (nullable = true)
 |    |    |    |    |-- Date: timestamp (nu