kafka实践

安全终端模拟软件使用：xshell+sftp

集群部署：1 部署zookeeper集群

                     2 配置kafka/conf/server.properties 

                       配置 broker的id. 每个broker的id必须是唯一的

                       配置log.dirs存放log的目录

                       配置zookeeper.connect的连接,zookeeper集群,按逗号分开即可

                       配置host.name,为IP即可

kafka启动了，关闭运行窗口，kafka立刻停止运行，启动命令如下：

 bin/kafka-server-start.sh config/server.properties > stdout.log 2>&1 &

同事用这个命令启动，关闭运行窗口kafka后台会继续运行，百思不得其解

最后使用另外一种启动方式

nohup bin/kafka-server-start.sh config/server.properties > stdout.log 2>&1 &

就可以解决此问题

2016-08-26 14:27:50,240 [ERROR][main][com.ucloudlink.ass.container.Main.main(Main.java:86)] - Error creating bean with name 'balanceReminderConsumer' defined in URL [jar:file:/opt/crm/123-456-interface/lib/123-456-interface-0.0.1.jar!/com/u/ass/reminder/BalanceReminderConsumer.class]: Invocation of init method failed; nested exception is kafka.common.ConsumerRebalanceFailedException: BALANCEREMINDERGROP_sdf1472192861490-64d4c7e8 can't rebalance after 4 retries 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'balanceReminderConsumer' defined in URL [jar:file:/opt/crm/ucl-crm-interface/lib/ucl-crm-interface-0.0.1.jar!/com/u/ass/reminder/BalanceReminderConsumer.class]: Invocation of init method failed; nested exception is kafka.common.ConsumerRebalanceFailedException: BALANCEREMINDERGROP_sdf1472192861490-64d4c7e8 can't rebalance after 4 retries 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) ~[spring-beans-4.0.1.RELEASE.jar:4.0.1.RELEASE] 

解决方案：

zkclient-0.1.jar

zkclient-0.7.jar

冲突，删除zkclient-0.1.jar即可

生产者、消费者、重写分区接口

public class KafkaStatleConsumer

{
private final ConsumerConnector consumer;

private final String topic = "topictest";

private final String zk = "xxx:2181";

private final int numThreads = 3;

public static void main(String[] args)
{
KafkaStatleConsumer consumer = new KafkaStatleConsumer();
consumer.run();
}

public Properties getConsumerProps()
{
Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", zk);
return consumerProps;
}

public KafkaStatleConsumer()
{
consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig());
}

private ConsumerConfig createConsumerConfig()
{
// 不同主题的不同groupID的consumerProps，需要不同设置，不能用同一个单例consumerProps
Properties props = getConsumerProps();
props.put("group.id", "testConsumer");
props.put("zookeeper.session.timeout.ms", "10000");
return new ConsumerConfig(props);
}

public void run()
{
Map<String, Integer> topickMap = new HashMap<String, Integer>();
topickMap.put(topic, numThreads);
Map<String, List<KafkaStream<byte[], byte[]>>> streamMap = consumer.createMessageStreams(topickMap);

List<KafkaStream<byte[], byte[]>> streams = streamMap.get(topic);
ExecutorService executor = Executors.newFixedThreadPool(numThreads);

/**
* 多线程消费分区
*/
for (final KafkaStream<byte[], byte[]> stream : streams)
{
Runnable runnable = new Runnable()
{
public void run()
{
ConsumerIterator<byte[], byte[]> it = stream.iterator();
while (it.hasNext())
{
MessageAndMetadata<byte[], byte[]> mam = it.next();
System.out.println(Thread.currentThread().getName()
+ ": partition[" + mam.partition() + "],"
+ "offset[" + mam.offset() + "], "
+ new String(mam.message()));
consumer.commitOffsets();
}
}
};
executor.submit(runnable);
}
}

}

public class KafkaStatleProducer
{
private final String server = "xxxx:9092";
private final static String topicName = "topictest";
private Properties props = new Properties();
private Producer<String, String> producer;

public static Properties getProductProps()
{
Properties producterProps;
producterProps = new Properties();
producterProps.setProperty("metadata.broker.list", "xxx:9092");
return producterProps;
}

public static void main(String[] args)
{
Properties props = new Properties();
props.put("metadata.broker.list", brokerList);
// 是否等待消息被提交的通知
props.put("request.required.acks", "1");

                //异步发送
props.put("producer.type", "async");
props.put("key.serializer.class", "kafka.serializer.StringEncoder");
//props.put("partitioner.class", BalanceReminderConstants.PATITIONER);
props.put("zk.connectiontimeout.ms", CONNECTION_TIMEOUT);
props.put("group.id", BalanceReminderConstants.BALANCEREMINDER_GROP);
props.put("queue.buffer.max.ms", "1000");
props.put("queue.buffering.max.ms", "1000");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
try
{
int i = 0;
while (i < 100)
{
String msg = "sdfsdfsdf: " + i;
KeyedMessage<String, String> data = new KeyedMessage<String, String>(
topicName, i + "", msg);
producer.send(data);


System.out.println(data);
data = null;
i++;
}
} catch (Exception e)
{
e.printStackTrace();
}
producer.close();
}
}

public class PartitionerExt extends DefaultPartitioner
{

public PartitionerExt(VerifiableProperties props)
{
super(props);
}
public int partition(Object arg0, int arg1)
{
int result = Math.abs(arg0.hashCode()) % arg1;
System.out.println("current DefaultPartitioner :" + result);
return result;
}
}

如果你的分区数是N，那么最好线程数也保持为N，这样通常能够达到最大的吞吐量。超过N的配置只是浪费系统资源，因为多出的线程不会被分配到任何分区.

topic下的一个分区只能被同一个consumer group下的一个consumer线程来消费，但反之并不成立，即一个consumer线程可以消费多个分区的数据.即一个消费线程可以对应若干个分区，但一个分区只能被具体某一个消费线程消费

Kafka 0.8.2之后推出了Java版的全新的producer，这个producer有个参数batch.size，默认是16KB。它会为每个分区缓存消息，一旦满了就打包将消息批量发出

生产消息时不用重写DefaultPartitioner，指定发送时的Key就可以保证发送到特定分区，kafka内部保证

可通过 get /consumers/BALANCEREMINDERGROP/offsets/BALANCEREMINDERTOPICNAME/1 在zookeeper客户端获取kafka特定分区是否消费和offsets的值

一般使用单拉取线程，多线程消费，如果要使用多线程还要按用户保证消费顺序的话，可使用队列，将特定用户放入对列（每个线程配置一个队列）

在生产消息时可指定key,不用重写partition,即可实现用一个用户发送到特定分区

在server.properties中可配置分区数，在生产者生产消息时自动自动按配置分区数根据TOPIC创建分区

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat --describe --zookeeper localhost:2181 --topic BALANCEREMINDERTOPICNAME

kafka-console-producer.bat --broker-list localhost:9092 --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat zookeeper   localhost:2181 -alter --partitions 3 --topic BALANCEREMINDERTOPICNAME

kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat --delete --zookeeper localhost:2181 --topic BALANCEREMINDERTOPICNAME

zookeeper查看节点是否消费，get /consumers/grop/offsets/toppicName

groups的list列表
bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --list


bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --group console-consumer-11967 --describe
其中依次展示group名称、消费的topic名称、partition id、consumer group最后一次提交的offset、最后提交的生产消息offset、消费offset与生产offset之间的差值、当前消费topic-partition的group成员id(不一定包含hostname)

kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group REMINDERGROP  --topic REMINDERTOPICNAME  --zookeeper 10.1.114.174:2181

GROUP     TOPIC     PID       OFFSET             LOGSIZE   LAG

消费者组  话题id    分区id    当前已消费的条数   总条数    未消费的条数

错误解决

can't rebalance after 3 retries

consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").

• If your consumer subscribes to many topics and your ZK server is busy, this could be caused by consumers not having enough time to see a consistent view of all consumers in the same group. If this is the case, try Increasing rebalance.max.retries and rebalance.backoff.ms.
• Another reason could be that one of the consumers is hard killed. Other consumers during rebalancing won't realize that consumer is gone after zookeeper.session.timeout.ms time. In the case, make sure that rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms.

rebalance.backoff.ms时间设置过短就会导致old consumer还没有来得及释放资源，new consumer重试失败多次到达阀值就退出了。

确保rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms