kafka容灾设计（使用开关切换单双kafka）

目录

1.maven引入kafka

2.application.propertise配置（两个kafka）

3.消费者配置

4.消费者使用

5.生产者配置

6.生产者使用

7.其他

---

1.maven引入kafka

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

2.application.propertise配置（两个kafka）

配置两个kafka，其中active为容灾未触发时使用的单kafka，passive为容灾触发时，开启的支持其它区域业务的kafka。使用开关kafka.switch进行控制，kafka.switch=normal代表容灾未触发，kafka.switch=all代表容灾触发。active.region对应active kafka消息，passive.region对应passive kafka消息。

#switch
spring.switch=xxxxxx

#region
spring.active.region=xxxxxx
spring.passive.region=xxxxxx

#kafka active
spring.kafka.active.bootstrap-servers=xxxxxx
spring.kafka.active.group.id=xxxxxx
spring.kafka.active.username=xxxxxx
spring.kafka.active.password=xxxxxx
spring.kafka.active.truststore.password=xxxxxx

#kafka passive
spring.kafka.active.bootstrap-servers=xxxxxx
spring.kafka.active.group.id=xxxxxx
spring.kafka.active.username=xxxxxx
spring.kafka.active.password=xxxxxx
spring.kafka.active.truststore.password=xxxxxx

#kafka common
spring.kafka.config=xxxxxx
spring.kafka.truststore.location=xxxxxx
spring.kafka.topic=xxxxxx

3.消费者配置

主要准备两个实例，然后使用时可以根据开关选择是开启单个实例还是多个实例。

ActiveKafkaConsumerConfig 配置

@Configuration
public class ActiveKafkaConsumerConfig extends BaseKafkaConfig{
    @Autowired
    ActiveKafkaProperties activeKafkaProperties;

    @Bean("activeKafkaListenerContainerFactory")
    ConcurrentKafkaListenerContainerFactory<Integer,String> kafkaListenerContainerFactory(){
        return buildListenerContainerFactory();
    }


    @Override
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(
             consumerFactory(
                     activeKafkaProperties.getServer(),
                     activeKafkaProperties.getGroupId(),
                     activeKafkaProperties.getTruststoreLocation(),
                     activeKafkaProperties.getStorePwd(),
                     activeKafkaProperties.getUserName(),
                     activeKafkaProperties.getPwd()));
    }
}

 PassiveKafkaConsumerConfig 配置

@Configuration
public class PassiveKafkaConsumerConfig extends BaseKafkaConfig{
    @Autowired
    PassiveKafkaProperties PassiveKafkaProperties;

    @Bean("PassiveKafkaListenerContainerFactory")
    ConcurrentKafkaListenerContainerFactory<Integer,String> kafkaListenerContainerFactory(){
        return buildListenerContainerFactory();
    }


    @Override
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(
             consumerFactory(
                     PassiveKafkaProperties.getServer(),
                     PassiveKafkaProperties.getGroupId(),
                     PassiveKafkaProperties.getTruststoreLocation(),
                     PassiveKafkaProperties.getStorePwd(),
                     PassiveKafkaProperties.getUserName(),
                     PassiveKafkaProperties.getPwd()));
    }
}

4.消费者使用

消费者通过switch的值，确定是否开启容灾。如果容灾开启，就同时监听两个kafka实例，如果容灾未开启就监听一个kafka实例。

监听active实例

@Component
public class KafkaActiveMessageRectiver extends BaseKafkaMessageReceiver{
    @KafkaListener(topics = {"${spring.kafka.topic}"},
                   containerFactory = "activeKafkaListenerContainerFactory")
    public void processQuantity(String message){
        processMessage(message);
    }
}

 监听passive实例

@ConditionalOnExpression("'${spring.switch}'.equal('all')")
@Component
public class KafkaPassiveMessageRectiver extends BaseKafkaMessageReceiver{
    @KafkaListener(topics = {"${spring.kafka.topic}"},
                   containerFactory = "passiveKafkaListenerContainerFactory")
    public void processQuantity(String message){
        processMessage(message);
    }
}

5.生产者配置

ActiveKafkaProducerConfig 配置

@Configuration
public class ActiveKafkaProducerConfig {
    @Autowired
    ActiveKafkaProperties activeKafkaProperties;
    
    @Bean
    @Qualifier("activeKafkaTemplate")
    public KafkaTemplate<String,String> activeKafkTemplate(){
        return new KafkaTemplate<>(producerFactory());
    }

    protected ProducerFactory<String,String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(
                producerConfig(
                        activeKafkaProperties.getServer(),
                        activeKafkaProperties.getGroupId(),
                        activeKafkaProperties.getTruststoreLocation(),
                        activeKafkaProperties.getStorePwd(),
                        activeKafkaProperties.getUserName(),
                        activeKafkaProperties.getPwd()));
    }
}

PassiveKafkaProducerConfig 配置 

@Configuration
public class PassiveKafkaProducerConfig {
    @Autowired
    PassiveKafkaProperties passiveKafkaProperties;
    
    @Bean
    @Qualifier("passiveKafkaTemplate")
    public KafkaTemplate<String,String> passiveKafkTemplate(){
        return new KafkaTemplate<>(producerFactory());
    }

    protected ProducerFactory<String,String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(
                producerConfig(
                        passiveKafkaProperties.getServer(),
                        passiveKafkaProperties.getGroupId(),
                        passiveKafkaProperties.getTruststoreLocation(),
                        passiveKafkaProperties.getStorePwd(),
                        passiveKafkaProperties.getUserName(),
                        passiveKafkaProperties.getPwd()));
    }
}

6.生产者使用

生产者生产消息，可以通过switch开关的值来确定是否开启容灾。然后通过region来生产不同区域的kafka消息，然后获取对应region的kafka实例，将该区域的消息发送至正确kafka上。

获取active实例：

@Component
public class KafkaActiveMessageSender extends BaseKafkaMessageSender{
    @Autowired
    @Qualifier("passiveKafkaTemplate")
    private KafkaTemplate<String,String> kafkaTemplate;
    
    @Override
    protected KafkaTemplate<String,String> getKafkaTemlate(){
        return kafkaTemplate;
    }
}

获取passive实例 

@Component
public class KafkaPassiveMessageSender extends BaseKafkaMessageSender{
    @Autowired
    @Qualifier("passiveKafkaTemplate")
    private KafkaTemplate<String,String> kafkaTemplate;
    
    @Override
    protected KafkaTemplate<String,String> getKafkaTemlate(){
        return kafkaTemplate;
    }
}

 通过region选取不同kafka实例：

private IkafkaSender getKafkaMessageSender(String region){
        if(region.equals("passiveRegion")){
            return passiveKafkaMessageSender;
        }
        return activeKafkaMessageSender;
    }

7.其他

大概的思路就是这样，配置两个kafka实例，如果是生产者需要根据容灾开关和区域生产不同区域的kafka消息发往不同区域的kafka上。如果是消费者，需要根据容灾开关来确定是监听一个kafka还是两个kafka实例。某一些类没有贴上来，可以根据自己的项目特点，再补全。