PySpark DataFrame show() 设置显示小数点的位数

PySpark DataFrame show() 设置显示小数点的位数

How do you set the display precision in PySpark when calling .show()?

1 源码及输出示例

from math import sqrt
import pyspark.sql.functions as f

data = zip(
    map(lambda x: sqrt(x), range(100, 105)),
    map(lambda x: sqrt(x), range(200, 205))
)
df = sqlCtx.createDataFrame(data, ["col1", "col2"])
df.select([f.avg(c).alias(c) for c in df.columns]).show()

#+------------------+------------------+
#|              col1|              col2|
#+------------------+------------------+
#|10.099262230352151|14.212583322380274|
#+------------------+------------------+

2 解决办法

2.1 Round

用pyspark.sql.functions.round()解决。

from pyspark.sql.functions import avg, round
df.select([round(avg(c), 3).alias(c) for c in df.columns]).show()
#+------+------+
#|  col1|  col2|
#+------+------+
#|10.099|14.213|
#+------+------+

2.2 Format Number

from pyspark.sql.functions import avg, format_number 
df.select([format_number(avg(c), 3).alias(c) for c in df.columns]).show()
#+------+------+
#|  col1|  col2|
#+------+------+
#|10.099|14.213|
#+------+------+

Ref:

How to set display precision in PySpark Dataframe show

https://stackoverflow.com/questions/48832493/how-to-set-display-precision-in-pyspark-dataframe-show