python打开方式错误_python – 使用Kafka在应用程序上打开太多文件错误

我正在使用Kafka和Spark Streaming构建应用程序.输入数据来自第三部分流媒体,并在kafka主题上发布.此代码显示了Stream Proxy模块：这是我从流式传输获得结果的方式以及我如何将它们发送到KafkaPublisher(它只显示了草图)：

def on_result_response(self,*args):

self.kafkaPublisher.pushMessage(str(args[0]))

KafkaPublisher使用以下两种方法实现：

class KafkaPublisher:

def __init__(self,address,port,topic):

self.kafka = KafkaClient(str(address)+":"+str(port))

self.producer = SimpleProducer(self.kafka)

self.topic=topic

def pushMessage(self,message):

self.producer.send_messages(self.topic, message)

self.producer = SimpleProducer(self.kafka, async=True)

该应用程序由这个主要推出：

from StreamProxy import StreamProxy

streamProxy=StreamProxy("localhost",9092,"task1")

streamProxy.getStreaming(20) #seconds of streaming

经过一些批处理(或多或少10秒)之后,它启动了以下例外：

Exception in thread Thread-2354:

Traceback (most recent call last):

File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner

File “/usr/lib/python2.7/threading.py”, line 754, in run

File “/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py”, line 164, in _send_upstream

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 649, in send_produce_request

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 253, in _send_broker_aware_request

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 74, in _get_conn

File “/usr/local/lib/python2.7/dist-packages/kafka/conn.py”, line 236, in connect

error: [Errno 24] Too many open files

Exception in thread Thread-2355:

Traceback (most recent call last):

File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner

File “/usr/lib/python2.7/threading.py”, line 754, in run

File “/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py”, line 164, in _send_upstream

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 649, in send_produce_request

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 253, in _send_broker_aware_request

File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 74, in _get_conn

File “/usr/local/lib/python2.7/dist-packages/kafka/conn.py”, line 236, in connect

error: [Errno 24] Too many open files

请注意,有相同的消息有许多不同的例外,当然问题是发布者方面.