我正在使用Kafka和Spark Streaming构建应用程序.输入数据来自第三部分流媒体,并在kafka主题上发布.此代码显示了Stream Proxy模块:这是我从流式传输获得结果的方式以及我如何将它们发送到KafkaPublisher(它只显示了草图):
def on_result_response(self,*args):
self.kafkaPublisher.pushMessage(str(args[0]))
KafkaPublisher使用以下两种方法实现:
class KafkaPublisher:
def __init__(self,address,port,topic):
self.kafka = KafkaClient(str(address)+":"+str(port))
self.producer = SimpleProducer(self.kafka)
self.topic=topic
def pushMessage(self,message):
self.producer.send_messages(self.topic, message)
self.producer = SimpleProducer(self.kafka, async=True)
该应用程序由这个主要推出:
from StreamProxy import StreamProxy
streamProxy=StreamProxy("localhost",9092,"task1")
streamProxy.getStreaming(20) #seconds of streaming
经过一些批处理(或多或少10秒)之后,它启动了以下例外:
Exception in thread Thread-2354:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
File “/usr/lib/python2.7/threading.py”, line 754, in run
File “/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py”, line 164, in _send_upstream
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 649, in send_produce_request
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 253, in _send_broker_aware_request
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 74, in _get_conn
File “/usr/local/lib/python2.7/dist-packages/kafka/conn.py”, line 236, in connect
error: [Errno 24] Too many open files
Exception in thread Thread-2355:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
File “/usr/lib/python2.7/threading.py”, line 754, in run
File “/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py”, line 164, in _send_upstream
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 649, in send_produce_request
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 253, in _send_broker_aware_request
File “/usr/local/lib/python2.7/dist-packages/kafka/client.py”, line 74, in _get_conn
File “/usr/local/lib/python2.7/dist-packages/kafka/conn.py”, line 236, in connect
error: [Errno 24] Too many open files
请注意,有相同的消息有许多不同的例外,当然问题是发布者方面.