This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

def create_spark():
    spark = SparkSession. \
        builder.master('local'). \
        appName('pipelinedemo'). \
        getOrCreate()
    return spark
    

报错：
WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Setting default log level to “WARN”.

改：

def create_spark():
    spark = SparkSession. \
        builder.master('local'). \
        appName('pipelinedemo'). \
        ***config("spark.debug.maxToStringFields", "100").\***
        getOrCreate()

    return spark