项目场景:
python3.7下载pyspark
问题描述:
C:\Users\e123>pip install pyspark==2.3.3 -i https://pypi.douban.com/simple
Looking in indexes: https://pypi.douban.com/simple
Collecting pyspark==2.3.3
Using cached https://pypi.doubanio.com/packages/af/7a/fa556cd533f106747e115c74c2f6d2fc864dc1dad25fdf991b06ab2c9403/pyspark-2.3.3.tar.gz (211.9 MB)
ERROR: Command errored out with exit status 1:
command: 'd:\python\python37\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\pip-install-jn9lxybr\\pyspark_3e7f6123657a46a89187bc62803de0d6\\setup.py'"'"'; __file__='"'"'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\pip-install-jn9lxybr\\pyspark_3e7f6123657a46a89187bc62803de0d6\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\KiloYe\AppData\Local\Temp\pip-pip-egg-info-g3ci27t5'
cwd: C:\Users\KiloYe\AppData\Local\Temp\pip-install-jn9lxybr\pyspark_3e7f6123657a46a89187bc62803de0d6\
Complete output (47 lines):
Could not import pypandoc - required to package PySpark
WARNING: The wheel package is not available.
ERROR: Command errored out with exit status 1:
command: 'd:\python\python37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\pip-wheel-t9kotk1b\\pypandoc_59519f743b2540cca0333c4d7e5a44d9\\setup.py'"'"'; __file__='"'"'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\pip-wheel-t9kotk1b\\pypandoc_59519f743b2540cca0333c4d7e5a44d9\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\KiloYe\AppData\Local\Temp\pip-wheel-s47kz_dh'
cwd: C:\Users\KiloYe\AppData\Local\Temp\pip-wheel-t9kotk1b\pypandoc_59519f743b2540cca0333c4d7e5a44d9\
Complete output (8 lines):
no pandoc found, building platform unspecific wheel...
use 'python setup.py download_pandoc' to download pandoc.
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
ERROR: Failed building wheel for pypandoc
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
File "d:\python\python37\lib\site-packages\setuptools\installer.py", line 128, in fetch_build_egg
subprocess.check_call(cmd)
File "d:\python\python37\lib\subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['d:\\python\\python37\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\tmpgxftys98', '--quiet', 'pypandoc']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\KiloYe\AppData\Local\Temp\pip-install-jn9lxybr\pyspark_3e7f6123657a46a89187bc62803de0d6\setup.py", line 224, in <module>
'Programming Language :: Python :: Implementation :: PyPy']
File "d:\python\python37\lib\site-packages\setuptools\__init__.py", line 143, in setup
_install_setup_requires(attrs)
File "d:\python\python37\lib\site-packages\setuptools\__init__.py", line 138, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "d:\python\python37\lib\site-packages\setuptools\dist.py", line 698, in fetch_build_eggs
replace_conflicting=True,
File "d:\python\python37\lib\site-packages\pkg_resources\__init__.py", line 783, in resolve
replace_conflicting=replace_conflicting
File "d:\python\python37\lib\site-packages\pkg_resources\__init__.py", line 1066, in best_match
return self.obtain(req, installer)
File "d:\python\python37\lib\site-packages\pkg_resources\__init__.py", line 1078, in obtain
return installer(requirement)
File "d:\python\python37\lib\site-packages\setuptools\dist.py", line 754, in fetch_build_egg
return fetch_build_egg(self, req)
File "d:\python\python37\lib\site-packages\setuptools\installer.py", line 130, in fetch_build_egg
raise DistutilsError(str(e))
distutils.errors.DistutilsError: Command '['d:\\python\\python37\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'C:\\Users\\KiloYe\\AppData\\Local\\Temp\\tmpgxftys98', '--quiet', 'pypandoc']' returned non-zero exit status 1.
----------------------------------------
WARNING: Discarding https://pypi.doubanio.com/packages/af/7a/fa556cd533f106747e115c74c2f6d2fc864dc1dad25fdf991b06ab2c9403/pyspark-2.3.3.tar.gz#sha256=8c6cc6b489de5ccea2e8e0f93cc907c1c155ef8ee363c8e4929c7dd0d4038cae (from https://pypi.doubanio.com/simple/pyspark/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyspark==2.3.3 (from versions: 2.1.2, 2.1.3, 2.2.0.post0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.1, 3.1.2)
ERROR: No matching distribution found for pyspark==2.3.3
原因分析:
注意到报错的代码
Could not import pypandoc - required to package PySpark
说明缺少了依赖
解决方案:
手动安装缺少的依赖
pip install pypandoc
再次安装就成功了