pythonsparkpickle_腌制Spark RDD并将其读入Python

我试图通过pickling来序列化Spark RDD,并将pickled文件直接读入Python.

a = sc.parallelize(['1','2','3','4','5'])

a.saveAsPickleFile('test_pkl')

然后我将test_pkl文件复制到我的本地.我怎样才能直接将它们读入Python？当我尝试正常的pickle包时,当我尝试读取’test_pkl’的第一个pickle部分时,它会失败：

pickle.load(open('part-00000','rb'))

Traceback (most recent call last):

File "", line 1, in

File "/usr/lib64/python2.6/pickle.py", line 1370, in load

return Unpickler(file).load()

File "/usr/lib64/python2.6/pickle.py", line 858, in load

dispatch[key](self)

File "/usr/lib64/python2.6/pickle.py", line 970, in load_string

raise ValueError, "insecure string pickle"

ValueError: insecure string pickle

我假设spark使用的酸洗方法与python pickle方法不同(如果我错了,请纠正我).有什么方法可以让我从Spark中腌制数据并直接从文件中读取这个pickle对象到python中吗？

解决方法:

with open("/path/to/file", "rb") as f:

print(sparkpickle.load(f))

标签：python,apache-spark,pickle,pyspark