python print rdd,查看RDD内容在Python火花？

Running a simple app in pyspark.

f = sc.textFile("README.md")

wc = f.flatMap(lambda x: x.split(' ')).map(lambda x: (x, 1)).reduceByKey(add)

I want to view RDD contents using foreach action:

wc.foreach(print)

This throws a syntax error:

SyntaxError: invalid syntax

What am I missing?

解决方案

This error is because print isn't a function in Python 2.6.

You can either define a helper UDF that performs the print, or use the __future__ library to treat print as a function:

>>> from operator import add

>>> f = sc.textFile("README.md")

>>> def g(x):

... print x

...

>>> wc.foreach(g)

or

>>> from __future__ import print_function

>>> wc.foreach(print)

However, I think it would be better to use collect() to bring the RDD contents back to the driver, because foreach executes on the worker nodes and the outputs may not necessarily appear in your driver / shell (it probably will in local mode, but not when running on a cluster).

>>> for x in wc.collect():

... print x