我试图通过pickling来序列化Spark RDD,并将pickled文件直接读入Python.
a = sc.parallelize(['1','2','3','4','5'])
a.saveAsPickleFile('test_pkl')
然后我将test_pkl文件复制到我的本地.我怎样才能直接将它们读入Python?当我尝试正常的pickle包时,当我尝试读取’test_pkl’的第一个pickle部分时,它会失败:
pickle.load(open('part-00000','rb'))
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python2.6/pickle.py", line 1370, in load
return Unpickler(file).load()
File "/usr/lib64/python2.6/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib64/python2.6/pickle.py", line 970, in load_string
raise ValueError, "insecure string pickle"
ValueError: insecure string pickle
我假设spark使用的酸洗方法与python pickle方法不同(如果我错了,请纠正我).有什么方法可以让我从Spark中腌制数据并直接从文件中读取这个pickle对象到python中吗?
解决方法:
with open("/path/to/file", "rb") as f:
print(sparkpickle.load(f))
标签:python,apache-spark,pickle,pyspark