python和机器学习代码中遇到的问题

1.pycharm运行pyspark代码，没有Hadoop环境

Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

解决方案

解压一份Hadoop包，配置HADOOP_HOME并加入Path变量中。

2.读取文件编码问题

SyntaxError: (unicode error) 'unicodeescape' codec can't decode
 bytes in position 2-3: truncated \xXX escape

错误代码

文件路径用的是斜杠

lines = spark.textFile("C:\xin\code\temp\ratings.dat")

解决方案

应该用反斜杠

lines = spark.textFile("C:/xin/code/temp/ratings.dat")

3.Rating参数个数问题

__new__() takes 4 positional arguments but 5 were given
代码

model = ALS.train(training, rank=50, iterations=10, lambda_=0.01)

报错信息

  File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\mllib\recommendation.py", line 233, in <lambda>
    ratings = ratings.map(lambda x: Rating(*x))
TypeError: __new__() takes 4 positional arguments but 5 were given

报错说，要4个参数，但是给你5个。

看源码

（ File "C:ProgramDataAnaconda3libsite-packagespysparkmllibrecommendation.py", line 30）

class Rating(namedtuple("Rating", ["user", "product", "rating"])):

这里其实要的是3个参数。

4.同时存在Python2和Python3，运行pyspark报错

报错信息

File "/usr/lib/python2.7/site-packages/pyspark-2.3.1-py2.7.egg/pyspark/worker.py", line 176, in main
    ("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with
different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

解决方案

在"~/.bashrc"添加配置（重启虚拟机）。

export PATH="/root/anaconda3/bin:$PATH"