I have a simple Producer-Consumer setup: 1 producer(as a thread) and 2 consumers(as 2 processes). The run method of producer:
def run(self):
producer = KafkaProducer(bootstrap_servers=self.bootstrap_servers,
api_version=(0, 10))
while not self.stop_event.is_set():
self.logger.info("Checking for new changes")
self.check_for_new_changes(producer)
self.logger.info("Sleeping for {minutes}
minutes...".format(minutes=self.time_to_sleep / 60))
time.sleep(self.time_to_sleep)
producer.close()
Basically it checks for changes, sends messages if new changes found and then goes to sleep for 5 minutes.
The run method:
def run(self):
if self.group_id:
consumer = KafkaConsumer(bootstrap_servers=self.bootstrap_servers,
consumer_timeout_ms=1000,
api_version=(0, 10),