五、Kafka，构建TB级异步消息系统

五、Kafka，构建TB级异步消息系统

5.1、阻塞队列

代码演示：

public class BlockingQueueTests {
    public static void main(String[] args){
        BlockingQueue queue = new ArrayBlockingQueue(10);
        new Thread(new Producer(queue)).start();
        //一个生产者生产数据，三个消费者同时并发地消费数据
        new Thread(new Consumer(queue)).start();
        new Thread(new Consumer(queue)).start();
        new Thread(new Consumer(queue)).start();
    }
}
 
 
class Producer implements Runnable{
    private BlockingQueue<Integer> queue;
    public Producer(BlockingQueue<Integer> queue){
        this.queue = queue;
    }
    @Override
    public void run(){
        try{
            for(int i = 0; i < 100; i++){
                Thread.sleep(20);
                //不断地往队列中添加数据
                queue.put(i);
                System.out.print(Thread.currentThread().getName() 
                                 + "生产:" + queue.size());
            }
        }catch (Exception e){
            e.printStackTrace();
        }
    }
}
 
class Consumer implements Runnable{
    private BlockingQueue<Integer> queue;
    public Consumer(BlockingQueue<Integer> queue){
        this.queue = queue;
    }

    @Override
    public void run(){
        try{
            while (true){
                //消费者的消费能力没有生产者的那么快,消费者消费的时间间隔是随机的
                Thread.sleep(new Random().nextInt(1000));
                queue.take();
                System.out.println(Thread.currentThread().getName()
                                   + "消费:" + queue.size());
            }
        }catch (Exception e){
            e.printStackTrace();
        }
    }
}

5.2、Kafka入门

http://kafka.apache.org

• Broker：缓存代理，Kafka集群中的一台或多台服务器统称broker
• Zookeeper：一个分布式的，开放源码的分布式应用程序协调服务。它是一个为分布式应用提供一致性服务的软件，提供的功能包括：配置维护、域名服务、分布式同步、组服务等。
• Topic：Kafka处理资源的消息源的不同分类：点对点、发布订阅
• Partition：Topic物理上的分组，一个topic可以分为多个partion,每个partion是一个有序的队列。partion中每条消息都会被分配一个 有序的Id(offset)。提高并发性能
• Offset：表示消息在分区中的偏移起始位置（索引序列），消费者读取消息是按照偏移量来读取的
• 一个分区有多个副本(Kafka是一个分布式的消息引擎， Leader Replica主副本 Follower Replica:从副本)

启动:

先启动zookeeper 需要指定配置文件（新启动一个cmd窗口)

bin\windows\zookeeper-server-start.bat config\zookeeper.properties

启动kafka（重新打开一个终端）

bin\windows\kafka-server-start.bat config\server.properties

使用kafka:

不管是创建主题还是查看主题都需要指定服务器。

创建主题（重新打开一个终端): 代表一个位置，同时也代表一种消息的类型

kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

主题名字test

查看主题: 指定需要查看的是哪台服务器

kafka-topics.bat --list --bootstrap-server localhost:9092

生产者发送消息：

kafka-console-producer.bat --broker-list localhost:9902 --topic test
hello
world

broker-list:要指定服务器列表和主题 ,向哪台服务器的哪些主题发消息

消费者读取消息:

（重新打开一个终端）从哪个服务器的哪些主题中读取消息

kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
hello
world

5.3、Spring整合Kafka

配置

对kafka做相应的配置application.properties

# KafkaProperties
spring.kafka.bootstrap-servers=localhost:9092 
spring.kafka.consumer.group-id=community-consumer-group 
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=3000

配置server、消费者的分组id（注意:修改之后需要重启服务）、配置是否自动提交(偏移量)、配置自动提交的频率

演示代码

生产者发送消息是主动的，但消费者消费消息是被动的。(所以可能会有一点点的延迟)

@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)
public class KafkaTests {
    @Autowired
    private KafkaProducer kafkaProducer;
    @Test
    public void testKafka() throws InterruptedException {
        kafkaProducer.sendMessage("test","你好");
        kafkaProducer.sendMessage("test","在吗");
        Thread.sleep(1000*10);
    }
}
@Component
class KafkaProducer{
    @Autowired
    private KafkaTemplate kafkaTemplate;
 
    public void sendMessage(String topics, String content){//消息的主题和内容
        kafkaTemplate.send(topics, content);
    }
}
@Component
class KafkaConsumer{
    @KafkaListener(topics={"test"}) //关注需要监听的主题，处理监听到消息的方法
    public void handleMessage(ConsumerRecord record){//消息封装成ConsumerRecord
        System.out.println(record.value());
    }
}

5.4、发布系统消息

因为评论、点赞、关注都是频繁的操作，为了提高性能，需要用到消息队列。把这件事包装成一个事件，仍到消息队列里去。后续的事情由消费者去处理，消费者和生产者**并发异步执行**。

**在Kafka的基础上，以事件驱动编程。**事件对象中包含了消息中的所有数据，而不是只有一个字符串，因此更具有扩展性。消费事件最终是把消息插入数据库。

封装事件Event实体

Event.java

public class Event {

    private String topic;
    private int userId;        //事件的触发者
    private int entityType;    //实体类型
    private int entityId;      //实体id
    private int entityUserId;  //实体的作者id
    private Map<String, Object> data = new HashMap<>();//数据都放在map之中，以便今后的扩展

    public Event setTopic(String topic) {
        this.topic = topic;
        return this;
    }

    public Event setUserId(int userId) {
        this.userId = userId;
        return this;
    }

    public Event setEntityType(int entityType) {
        this.entityType = entityType;
        return this;
    }

    public Event setEntityId(int entityId) {
        this.entityId = entityId;
        return this;
    }

    public Event setEntityUserId(int entityUserId) {
        this.entityUserId = entityUserId;
        return this;
    }

    public Event setData(String key, Object value) {
        this.data.put(key, value);
        return this;
    }
}

在set方法中返回了事件对象，链式编程，灵活。setData函数中只传递一个(key, value)，很方便。

事件生产者EventProducer

@Component
public class EventProducer {

    @Autowired
    private KafkaTemplate kafkaTemplate;

    // 处理事件
    public void fireEvent(Event event) {
        // 将事件发布到指定的主题
        kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
    }
}

事件消费者EventConsumer

CommunityConstant 定义主题类型

    /**
     * 主题: 评论
     */
    String TOPIC_COMMENT = "comment";
    /**
     * 主题: 点赞
     */
    String TOPIC_LIKE = "like";
    /**
     * 主题: 关注
     */
    String TOPIC_FOLLOW = "follow";
    /**
     * 系统用户ID
     */
    int SYSTEM_USER_ID = 1;

message表中，若是系统发送的消息，from_id = 1，conversation_id = 主题，content = 要显示的消息的json字符串

消费事件最终是把消息存入数据库

eventProducer是主动调用的，eventConsumer是被动接收，不需要调用

@Component
public class EventConsumer implements CommunityConstant {

    private static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);

    @Autowired
    private MessageService messageService;

    @KafkaListener(topics = {TOPIC_COMMENT, TOPIC_LIKE, TOPIC_FOLLOW})
    public void handleCommentMessage(ConsumerRecord record) {
        if (record == null || record.value() == null) {
            logger.error("消息的内容为空!");
            return;
        }
		//json转字符串再转为event
        Event event = JSONObject.parseObject(record.value().toString(), Event.class);
        if (event == null) {
            logger.error("消息格式错误!");
            return;
        }

        // 发送站内通知
        //构造一个Message对象插入数据库，由系统发送，所以fromId为1
        Message message = new Message();
        message.setFromId(SYSTEM_USER_ID);//系统用户ID=1
        message.setToId(event.getEntityUserId());//被评论/赞/点赞的UserID
        message.setConversationId(event.getTopic());//主题
        message.setCreateTime(new Date());
		
        //用内容去拼接实际显示的通知 比如页面显示的：
        //用户nowcoder(事件的触发者)评论了你的帖子(实体的数据，因为之后在前台需要链接过去
        Map<String, Object> content = new HashMap<>();
        content.put("userId", event.getUserId());//事件触发者的id
        content.put("entityType", event.getEntityType());//实体类型：评论 赞 关注 
        content.put("entityId", event.getEntityId());//被通知的用户id

        if (!event.getData().isEmpty()) {//其他的信息
            for (Map.Entry<String, Object> entry : event.getData().entrySet()) {
                content.put(entry.getKey(), entry.getValue());
            }
        }

        message.setContent(JSONObject.toJSONString(content));
        messageService.addMessage(message);
    }
}

Controller

（1）CommentController

@Controller
@RequestMapping("/comment")
public class CommentController implements CommunityConstant {

    @Autowired
    private CommentService commentService;

    @Autowired
    private HostHolder hostHolder;

    @Autowired
    private EventProducer eventProducer;

    @Autowired
    private DiscussPostService discussPostService;

    @RequestMapping(path = "/add/{discussPostId}", method = RequestMethod.POST)
    public String addComment(@PathVariable("discussPostId") int discussPostId, Comment comment) {
        comment.setUserId(hostHolder.getUser().getId());
        comment.setStatus(0);
        comment.setCreateTime(new Date());
        commentService.addComment(comment);

        // ==============触发评论事件================
        Event event = new Event()
                .setTopic(TOPIC_COMMENT)
                .setUserId(hostHolder.getUser().getId())//当前登录者触发的消息
                .setEntityType(comment.getEntityType())
                .setEntityId(comment.getEntityId())
                .setData("postId", discussPostId);//帖子的id，查看详情时需要
        if (comment.getEntityType() == ENTITY_TYPE_POST) {//帖子
            DiscussPost target = 
                discussPostService.findDiscussPostById(comment.getEntityId());
            event.setEntityUserId(target.getUserId());
        } else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {//评论
            Comment target = commentService.findCommentById(comment.getEntityId());
            event.setEntityUserId(target.getUserId());
        }
        eventProducer.fireEvent(event);//异步并发

        return "redirect:/discuss/detail/" + discussPostId;
    }
}

（2）LikeControlloer

@Controller
public class LikeController implements CommunityConstant {

    @Autowired
    private LikeService likeService;

    @Autowired
    private HostHolder hostHolder;

    @Autowired
    private EventProducer eventProducer;

    @RequestMapping(path = "/like", method = RequestMethod.POST)
    @ResponseBody
    public String like(int entityType, int entityId, int entityUserId, int postId) {
        User user = hostHolder.getUser();

        // 点赞
        likeService.like(user.getId(), entityType, entityId, entityUserId);

        // 数量
        long likeCount = likeService.findEntityLikeCount(entityType, entityId);
        // 状态
        int likeStatus = likeService.findEntityLikeStatus(user.getId(), entityType, entityId);
        // 返回的结果
        Map<String, Object> map = new HashMap<>();
        map.put("likeCount", likeCount);
        map.put("likeStatus", likeStatus);

        // 触发点赞事件
        if (likeStatus == 1) {//如果是点赞才会触发事件，取消点赞不会通知
            Event event = new Event()
                    .setTopic(TOPIC_LIKE)
                    .setUserId(hostHolder.getUser().getId())
                    .setEntityType(entityType)
                    .setEntityId(entityId)
                    .setEntityUserId(entityUserId)
                    .setData("postId", postId);//传入的参数中就加入帖子的id
            eventProducer.fireEvent(event);
        }

        return CommunityUtil.getJSONString(0, null, map);
    }

}

（3）FollowController

    @RequestMapping(path = "/follow", method = RequestMethod.POST)
    @ResponseBody
    public String follow(int entityType, int entityId) {
        User user = hostHolder.getUser();

        followService.follow(user.getId(), entityType, entityId);

        // 触发关注事件
        Event event = new Event()
                .setTopic(TOPIC_FOLLOW)
                .setUserId(hostHolder.getUser().getId())
                .setEntityType(entityType)
                .setEntityId(entityId)
                .setEntityUserId(entityId);
        eventProducer.fireEvent(event);

        return CommunityUtil.getJSONString(0, "已关注!");
    }

测试出现的bug

ServiceLogAspect.java 拦截器：出现空指针异常

HttpServletRequest request = attributes.getRequest();

    @Pointcut("execution(* com.nowcoder.community.service.*.*(..))")
    public void pointcut() {

    }

    @Before("pointcut()")
    public void before(JoinPoint joinPoint) {
        // 用户[1.2.3.4],在[xxx],访问了[com.nowcoder.community.service.xxx()].
        ServletRequestAttributes attributes = 
            (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
        if (attributes == null) {
            return;//直接返回，不做后面的日志
        }
        HttpServletRequest request = attributes.getRequest();
        String ip = request.getRemoteHost();
        String now = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
        String target = joinPoint.getSignature().getDeclaringTypeName()
            + "." + joinPoint.getSignature().getName();
        logger.info(String.format("用户[%s],在[%s],访问了[%s].", ip, now, target));
    }

attributes 是和请求有关的对象，当前的AOP拦截的都是service层（Pointcut），之前所有对service的访问都是通过controller访问的，但**消费者EventConsumer中直接调用了messageService.addMessage(message)，没有通过controller，因此并不存在HttpServletRequest对象，所以出现空指针异常**，再加一个非空判断即可。

5.5、显示系统通知

（1）通知列表

dao数据访问层

MessageMapper

    //查询某个主题下最新的一个通知
    Message selectLatestNotice(int userId, String topic);
 
    //查询某个主题所包含的通知的数量
    int selectNoticeCount(int userId, String topic);
 
    //查询某个主题未读的通知数量
    int selectNoticeUnreadCount(int userId, String topic);

message-mapper.xml

    <select id="selectLatestNotice" resultType="Message">
        select <include refid="selectFields"></include>
        from message
        where id in(
            select max(id) from message
            where status != 2    //删除的状态
            and from_id = 1
            and to_id = #{userId}
            and conversation_id = #{topic}
        )
    </select>
 
    <select id="selectNoticeCount" resultType="int">
        select count(id) from message
        where status != 2
        and from_id = 1
        and to_id = #{userId}
        and conversation_id = #{topic}
    </select>
 
    <select id="selectNoticeUnreadCount" resultType="int">
        select count(id) from message
        where status = 0 //未读
        and from_id = 1
        and to_id = #{userId}
        <if test="topic!=null">
            and conversation_id = #{topic} //不传主题，查询的就是所有主题的未读消息数量
        </if>
    </select>

service业务层

MessageService

    public Message findLatestNotice(int userId, String topic){
        return messageMapper.selectLatestNotice(userId, topic);
    }
 
    public int findNoticeCount(int userId, String topic){
        return messageMapper.selectNoticeCount(userId, topic);
    }
 
    public int findNoticeUnreadCount(int userId, String topic){
        return messageMapper.selectNoticeUnreadCount(userId, topic);
    }

Controller层

MessageController

    @RequestMapping(path="/notice/list", method = RequestMethod.GET)
    public String getNoticeList(Model model){
        User user = hostHolder.getUser();
 
        //===========================查询评论通知============================
        Message message = 
            messageService.findLatestNotice(user.getId(), TOPIC_COMMENT);
        Map<String, Object> messageVO = new HashMap<>();
        if(message != null){
            messageVO.put("message",message);
 
            //去掉，转义字符
            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
 
            //这里存放的是最新的一个对象
            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));
 
            //还需要查询这一类通知的总数量和未读通知数量
            int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT);
         	messageVO.put("count", count);
 
            int unread = messageService.findNoticeUnreadCount(
                user.getId(), TOPIC_COMMENT);
            messageVO.put("unread", unread);
        }
        model.addAttribute("commentNotice", messageVO);
 
        //=============================查询点赞类通知==============================
        message = messageService.findLatestNotice(user.getId(), TOPIC_LIKE);
        messageVO = new HashMap<>();
        if(message != null){
            messageVO.put("message",message);
 
            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
 
            //这里存放的是最新的一个对象
            messageVO.put("user", userService.findUserById(
                (Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));
 
            int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE);
            messageVO.put("count", count);
 
            int unread = messageService.findNoticeUnreadCount(
                user.getId(), TOPIC_LIKE);
            messageVO.put("unread", unread);
        }
        model.addAttribute("likeNotice", messageVO);
 
        //============================查询关注类通知==================================
        message = messageService.findLatestNotice(user.getId(), TOPIC_FOLLOW);
        messageVO = new HashMap<>();
        if(message != null){
            messageVO.put("message", message);
            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
 
            //这里存放的是最新的一个对象
            messageVO.put("user", userService.findUserById(
                (Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
 
            int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW);
            messageVO.put("count", count);
 
            int unread = messageService.findNoticeUnreadCount(
                user.getId(), TOPIC_FOLLOW);
            messageVO.put("unread", unread);
        }
        model.addAttribute("followNotice", messageVO);
 
        //============================查询所有未读消息数量========================
        int letterUnreadCount = messageService.findLetterUnreadCount(
            user.getId(), null);
        model.addAttribute("letterUnreadCount", letterUnreadCount);
 
        //返回的是通知的数量
        int noticeUnreadCount = messageService.findNoticeUnreadCount(
            user.getId(), null);
        model.addAttribute("noticeUnreadCount", noticeUnreadCount);
 
        return "/site/notice";
    }

notice.html

（2）通知详情

dao层

MessageMapper

//查询某个主题所包含的通知列表
List<Message> selectNotices(int userId, String topic, int offset, int limit);

message-mapper.xml

    <select id="selectNotices" resultType="Message">
        select <include refid="selectFields"></include>
        from message
        where status != 2
        and from_id = 1
        and to_id = #{userId}
        and conversation_id = #{topic}
        order by create_time desc
        limit #{offset}, #{limit}
    </select>

service层

MessageService

    public List<Message> findNotices(int userId, String topic, int offset, int limit){
        return messageMapper.selectNotices(userId, topic, offset, limit);
    }

controller层

MessageController

    @RequestMapping(path = "/notice/detail/{topic}", method = RequestMethod.GET)
    public String getNoticeDetail(@PathVariable("topic") String topic, Page page, Model model){
        User user = hostHolder.getUser();
 
        //对分页进行设置
        page.setLimit(5);
        page.setPath("/notice/detail/" + topic);
        page.setRows(messageService.findNoticeCount(user.getId(), topic));
 
        List<Message> noticeList = messageService.findNotices(
            user.getId(), topic, page.getOffset(), page.getLimit());
        List<Map<String, Object>> noticeVoList = new ArrayList<>();
        if(noticeVoList != null){
            for(Message notice: noticeList){
                //传递聚合数据
                Map<String, Object> map = new HashMap<>();
                //通知
                map.put("notice", notice);
                //内容
                String content = HtmlUtils.htmlUnescape(notice.getContent());
                Map<String, Object> data = JSONObject.parseObject(
                    content, HashMap.class);
                map.put("user", userService.findUserById(
                    (Integer) data.get("userId")));
                map.put("entityType", data.get("entityType"));
                map.put("entityId", data.get("entityId"));
                map.put("postId", data.get("postId"));
 
                //发布通知的作者
                map.put("fromUser", userService.findUserById(notice.getFromId()));
 
                noticeVoList.add(map);
            }
        }
        model.addAttribute("notices", noticeVoList);
 
        //设置已读
        List<Integer> ids = getLetterIds(noticeList);
        if(!ids.isEmpty()){
            messageService.readMessage(ids);
        }
        return "/site/notice-detail";
    }

（3）显示头部的总的未读消息总数，利用拦截器实现

创建MessageInterceptor

消息总数量为未读私信数量和未读通知数量，重写postHandle方法。

 
@Component
public class MessageInterceptor implements HandlerInterceptor {
    @Autowired
    private HostHolder hostHolder;
 
    @Autowired
    private MessageService messageService;
 
    @Override
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
        User user = hostHolder.getUser();
        if(user != null && modelAndView != null){
            int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
            int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
            modelAndView.addObject("allUnreadCount", letterUnreadCount + noticeUnreadCount);
        }
    }
}

WebMvcConfig配置拦截器

不拦截静态请求，但拦截所有的动态请求

        registry.addInterceptor(messageInterceptor)
                .excludePathPatterns("/**/*.css", "/**/*.js", "/**/*.jpg", "/**/*.jpeg", "/**/*.png");

在index.html中的header中显示数据

<li class="nav-item ml-3 btn-group-vertical" th:if="${loginUser!=null}">
	<a class="nav-link position-relative" th:href="@{/letter/list}">消息
		<span class="badge badge-danger" th:text="${allUnreadCount!=0?allUnreadCount:''}">12</span>
	</a>
</li>