from py4j.protocol import Py4JError ModuleNotFoundError 错误

最新推荐文章于 2024-05-15 21:15:21 发布
最新推荐文章于 2024-05-15 21:15:21 发布
阅读量928 收藏 1
点赞数 2
分类专栏： Python编程 文章标签： spark python
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/sinat_38432876/article/details/106270352
版权

from py4j.protocol import Py4JError ModuleNotFoundError: No module named 'py4j'报错了！！！

学习林子雨老师的pySpark慕课时，按照给的链接安装时，运行python3 ~/test.py程序发现了from py4j.protocol import Py4JError ModuleNotFoundError: No module named 'py4j'的错误了。

后来发现了在cd /usr/local/spark的工作目录，即

cd /usr/local/spark
vim ~/.bashrc

设置环境变量时是这样设置的

export JAVA_HOME=/usr/lib/jvm/default-java
export HADOOP_HOME=/usr/local/hadoop
export SPARK_HOME=/usr/local/spark
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH
export PYSPARK_PYTHON=python3
export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH

其中第四行为export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH

而进入我们的 /usr/local/spark/python/lib 目录下输入 ls 会发现我们的版本号不对，我们的是

将上边的/usr/local/spark/bashrc里的环境路径改为如下

export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PYTHONPATH

然后在输入

cd /usr/local/spark
source ~/.bashrc

在运行 python3 ~/test.py时就没有错了。

Surupor
关注
  • 2
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
VOT Toolkit工具配置和使用--Python版本（csdn）————程序.pdf
12-04
VOT Toolkit工具配置和使用--Python版本（csdn）————程序
msgpack-python-0.4.2.tar
09-19
======================= MessagePack for Python ======================= :author: INADA Naoki :version: 0.4.1 :date: 2014-02-17 .. image:: https://secure.travis-ci.org/msgpack/msgpack-python.png :target: https://travis-ci.org/#!/msgpack/msgpack-python What's this ------------ `MessagePack <http://msgpack.org/>`_ is a fast, compact binary serialization format, suitable for similar data to JSON. This package provides CPython bindings for reading and writing MessagePack data. Install --------- You can use ``pip`` or ``easy_install`` to install msgpack:: $ easy_install msgpack-python or $ pip install msgpack-python PyPy ^^^^^ msgpack-python provides pure python implementation. PyPy can use this. Windows ^^^^^^^ When you can't use binary distribution, you need to install Visual Studio or Windows SDK on Windows. (NOTE: Visual C++ Express 2010 doesn't support amd64. Windows SDK is recommanded way to build amd64 msgpack without any fee.) Without extension, using pure python implementation on CPython runs slowly. Notes ----- Note for msgpack 2.0 support ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ msgpack 2.0 adds two types: *bin* and *ext*. *raw* was bytes or string type like Python 2's ``str``. To distinguish string and bytes, msgpack 2.0 adds *bin*. It is non-string binary like Python 3's ``bytes``. To use *bin* type for packing ``bytes``, pass ``use_bin_type=True`` to packer argument. >>> import msgpack >>> packed = msgpack.packb([b'spam', u'egg'], use_bin_type=True) >>> msgpack.unpackb(packed, encoding='utf-8') ['spam', u'egg'] You shoud use it carefully. When you use ``use_bin_type=True``, packed binary can be unpacked by unpackers supporting msgpack-2.0. To use *ext* type, pass ``msgpack.ExtType`` object to packer. >>> import msgpack >>> packed = msgpack.packb(msgpack.ExtType(42, b'xyzzy')) >>> msgpack.unpackb(packed) ExtType(code=42, data='xyzzy') You can use it with ``default`` and ``ext_hook``. See below. Note for msgpack 0.2.x users ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The msgpack 0.3 have some incompatible changes. The default value of ``use_list`` keyword argument is ``True`` from 0.3. You should pass the argument explicitly for backward compatibility. `Unpacker.unpack()` and some unpack methods now raises `OutOfData` instead of `StopIteration`. `StopIteration` is used for iterator protocol only. How to use ----------- One-shot pack & unpack ^^^^^^^^^^^^^^^^^^^^^^ Use ``packb`` for packing and ``unpackb`` for unpacking. msgpack provides ``dumps`` and ``loads`` as alias for compatibility with ``json`` and ``pickle``. ``pack`` and ``dump`` packs to file-like object. ``unpack`` and ``load`` unpacks from file-like object. :: >>> import msgpack >>> msgpack.packb([1, 2, 3]) '\x93\x01\x02\x03' >>> msgpack.unpackb(_) [1, 2, 3] ``unpack`` unpacks msgpack's array to Python's list, but can unpack to tuple:: >>> msgpack.unpackb(b'\x93\x01\x02\x03', use_list=False) (1, 2, 3) You should always pass the ``use_list`` keyword argument. See performance issues relating to use_list_ below. Read the docstring for other options. Streaming unpacking ^^^^^^^^^^^^^^^^^^^ ``Unpacker`` is a "streaming unpacker". It unpacks multiple objects from one stream (or from bytes provided through its ``feed`` method). :: import msgpack from io import BytesIO buf = BytesIO() for i in range(100): buf.write(msgpack.packb(range(i))) buf.seek(0) unpacker = msgpack.Unpacker(buf) for unpacked in unpacker: print unpacked Packing/unpacking of custom data type ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is also possible to pack/unpack custom data types. Here is an example for ``datetime.datetime``. :: import datetime import msgpack useful_dict = { "id": 1, "created": datetime.datetime.now(), } def decode_datetime(obj): if b'__datetime__' in obj: obj = datetime.datetime.strptime(obj["as_str"], "%Y%m%dT%H:%M:%S.%f") return obj def encode_datetime(obj): if isinstance(obj, datetime.datetime): return {'__datetime__': True, 'as_str': obj.strftime("%Y%m%dT%H:%M:%S.%f")} return obj packed_dict = msgpack.packb(useful_dict, default=encode_datetime) this_dict_again = msgpack.unpackb(packed_dict, object_hook=decode_datetime) ``Unpacker``'s ``object_hook`` callback receives a dict; the ``object_pairs_hook`` callback may instead be used to receive a list of key-value pairs. Extended types ^^^^^^^^^^^^^^^ It is also possible to pack/unpack custom data types using the msgpack 2.0 feature. >>> import msgpack >>> import array >>> def default(obj): ... if isinstance(obj, array.array) and obj.typecode == 'd': ... return msgpack.ExtType(42, obj.tostring()) ... raise TypeError("Unknown type: %r" % (obj,)) ... >>> def ext_hook(code, data): ... if code == 42: ... a = array.array('d') ... a.fromstring(data) ... return a ... return ExtType(code, data) ... >>> data = array.array('d', [1.2, 3.4]) >>> packed = msgpack.packb(data, default=default) >>> unpacked = msgpack.unpackb(packed, ext_hook=ext_hook) >>> data == unpacked True Advanced unpacking control ^^^^^^^^^^^^^^^^^^^^^^^^^^ As an alternative to iteration, ``Unpacker`` objects provide ``unpack``, ``skip``, ``read_array_header`` and ``read_map_header`` methods. The former two read an entire message from the stream, respectively deserialising and returning the result, or ignoring it. The latter two methods return the number of elements in the upcoming container, so that each element in an array, or key-value pair in a map, can be unpacked or skipped individually. Each of these methods may optionally write the packed data it reads to a callback function: :: from io import BytesIO def distribute(unpacker, get_worker): nelems = unpacker.read_map_header() for i in range(nelems): # Select a worker for the given key key = unpacker.unpack() worker = get_worker(key) # Send the value as a packed message to worker bytestream = BytesIO() unpacker.skip(bytestream.write) worker.send(bytestream.getvalue()) Note about performance ------------------------ GC ^^ CPython's GC starts when growing allocated object. This means unpacking may cause useless GC. You can use ``gc.disable()`` when unpacking large message. `use_list` option ^^^^^^^^^^^^^^^^^^ List is the default sequence type of Python. But tuple is lighter than list. You can use ``use_list=False`` while unpacking when performance is important. Python's dict can't use list as key and MessagePack allows array for key of mapping. ``use_list=False`` allows unpacking such message. Another way to unpacking such object is using ``object_pairs_hook``. Test ---- MessagePack uses `pytest` for testing. Run test with following command: $ py.test .. vim: filetype=rst
pyspark报错解决：py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTime
m0_68131322的博客
03-26 271
使用pyspark运行时，报错下面内容。
protobuf-3.20.1-py2.py3-none-any.whl
04-21
protobuf-3.20.1 解决树莓派import mediapipe 出现以下问题 TypeError: Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are: 1. Downgrade the protobuf package to 3.20.x or lower. 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). 使用pip3 install即可
pySpark API 使用过程中出现 “ImportError: No module named 'py4j'“错误
每天都要有进步鸭！
06-07 4781
首先确认 ~/.bashrc 里面有： export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH 2. 进入 $SPARK_HOME/python/lib/ 里面看一看py4j 的版本号和你...
pyspark报错Py4JJavaError
砍柴人的博客
02-16 2723
Py4JJavaError Traceback (most recent call last) <ipython-input-14-063f7e74ac7a> in <module> ----> 1 words_df = article_data.rdd.mapPartitions(segmentation).toDF(['article_id', 'channel_id', 'words']) Py4JJavaError: An error occurred while
ModuleNotFoundError: No module named ‘py4j‘
zsxbd的博客
10-20 5059
Ubuntu配置Jupyter Notebook实现和PySpark交互，在运行实例的时候出现ModuleNotFoundError: No module named 'py4j’问题 学习厦门大学数据库实验室的spark教程中它的环境配置中py4j的版本是0.10.4，而我下载的py4j版本是0.10.7。 如果不知道自己的py4j版本，可以在命令行中输入cd $SPARK_HOME/python/lib，这表示进入到py4j所在的文件目录下 然后再输入ls 这样就能看到自己py4j所对应的版本，如图所示
解决python3和jupyter-notebook中的报错No module named pyspark和No module named ‘py4j’
weixin_41108545的博客
09-11 8415
背景描述： 在centos7–CDH6下配置了spark2.4和hive2.3，在linux-shell中输入pyspark可以正常启动，执行下列语句可正常显示 from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Python Spark SQL Hive integration example").enableHiveSupport().getOrCreate() spark.sql("show databa
windows版本的spark中，出现no mudule named ‘py4j‘错误
weixin_44004835的博客
07-13 1010
我看了很多网上的一些帖子，都是说linux环境下spark为什么出现错误，我在这里就写一下，windows环境出现no mudule named 'py4j’应该怎么处理 一开始，在写代码的时候，软件是会出现pyspark的提示框。（如果在写代码的时候，就没有提示pyspark，那么说明你还没有引入pyspark包） 但是，运行出来，确实显示no mudule named ‘py4j’。 后来发现，由于我装的spark是3.2版本的（注意，3.0版本以后就有可能出现以下情况），只有一个pyspark的文
在linux中使用python命令或者脚本启动py文件显示ImportError: No module named
qq_41911460的博客
04-17 5010
背景 公司的代码放在linux中，所以需要使用网络驱动器或者linux挂载共享的方式将wins和linux文件进行关联。日常开发使用pycharm，ide启动程序是没有问题的，但是在linux中使用python命令或者脚本启动py文件就会显示ImportError: No module named。 解决方案 | --public |–init.py |–a.py | --zk_log |–ini...
defi-sdk-py:适用于DeFi SDK的Python包装器
05-08
安装$ pip3 install defisdk用法初始化DeFiSDK from defisdk import DeFiSDKINFURA_API_KEY = '<MY>'ETHEREUM_NODE_URL = f'https://mainnet.infura.io/v3/ { INFURA_API_KEY } 'defi_sdk = DeFiSDK ( ETHEREUM_NODE...
assetexchange-opensource：Asset Ninja使用的资产交换协议实现
01-30
资产交换 资产忍者使用的资产交换协议实现。 Python 包扩展 # prerequisites ... # Step 1: import the packed assetexchange.py from . import assetexchange # Step 2: now you can import the mo
py4j.protocol.Py4JJavaError错误
热门推荐
Eat_shopping的博客
10-16 1万+
py4j.protocol.Py4JJavaError: An error occurred while calling o30.toDebugString. : org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://192.168.1.84:9000/user/root/shakespea
Spark 报错 ImportError: No module named py4j.java_gateway
jk123vip的专栏
03-08 8446
vi .bashrc export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH
from py4j.protocol import Py4JError ModuleNotFoundError: No module named 'py4j'
一个程序媛的成长记录❤
05-30 4823
报错了，开始还想不明白。 后来发现错在环境变量配置时 我开始是这样写的 export JAVA_HOME=/usr/lib/jvm/default-java export HADOOP_HOME=/usr/local/hadoop export SPARK_HOME=/usr/local/spark export PYTHONPATH=$SPARK_HOME/python:$SPARK_H...
使用Pycharm运行spark实例时没有pyspark包（ModuleNotFoundError: No module named ‘py4j‘）
哇咔咔哇咔的博客
04-23 531
ModuleNotFoundError: No module named 'py4j'
【Python大数据】PySpark
最新发布
qq_45951891的博客
05-15 1549
Apache Spark是用于大规模数据(large-scala data)处理的统一(unified)分析引擎简单来说，Spark是一款分布式的计算框架，用于调度成百上千的服务器集群，计算TB、PB乃至EB级别的海量数据。
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
05-31
这个错误通常是由于Spark环境配置不正确或者Spark和Python版本不匹配导致的。以下是几种常见的情况和解决方法： 1. Spark环境配置不正确 请确保你已经正确地设置了SPARK_HOME环境变量，并且在Python代码中正确地初始化了SparkContext。如果你正在使用PySpark，可以使用以下代码创建SparkContext： ```python from pyspark import SparkContext, SparkConf conf = SparkConf().setAppName("myApp") sc = SparkContext(conf=conf) ``` 2. Spark和Python版本不匹配 请确保你的Spark和Python版本是兼容的。例如，如果你正在使用Spark 2.4，则需要使用Python 2.7或Python 3.4及以上版本。如果版本不匹配，则可能会出现各种错误。 3. Py4J版本不匹配 请确保Py4J的版本与Spark版本兼容。例如，如果你正在使用Spark 2.4，则需要使用Py4J版本0.10.x。你可以使用以下命令来安装正确的Py4J版本： ```bash pip install py4j==0.10.* ``` 如果你仍然遇到问题，请检查Spark和Python的日志文件以获取更多详细信息。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值