KaBoom - A High Performance Consumer Client for Kafka
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop.
The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.
This is a hadoop job for incremental loading of kafka topics into hdfs