I'm using a generator function from the requests module in a QT-Application, pretty much the same as in the requests-streaming example:
import json
import requests
def get_stream():
r = requests.get('http://httpbin.org/stream/20', stream=True)
for line in r.iter_lines():
if line:
yield json.loads(line)
def consume_stream():
for message in get_stream():
#do something
However, when there is no incoming response (f.e. irregularly incoming tweets from Twitters Streaming API), the generator get_stream will block the consume_stream method.
This might occur in any situation where a generator does not yield immediately, but hast to wait for incoming messages etc., and therefore blocks the consumer.
Is there any pattern in Python where you can consume a generator in a non-blocking way, i.e. if the generator yields, process it's results, otherwise do something else until the next results are occuring?
解决方案
Take a look at the producer-consumer pattern. It is commonly implemented in python using a Queue.
The producer, typically running in a thread or another process (Queue supports either), simply puts messages in the queue. The consumer, whenever it feels like it, pops messages from the queue. This operation supports a timeout argument.