5.Kafka，构建TB级异步消息系统

最新推荐文章于 2024-04-22 20:43:05 发布

ShayneCh

最新推荐文章于 2024-04-22 20:43:05 发布

阅读量2k

点赞数

分类专栏：牛客社区项目笔记文章标签： java 后端

本文链接：https://blog.csdn.net/weixin_44712574/article/details/123917927

版权

牛客社区项目笔记专栏收录该内容

8 篇文章 1 订阅

订阅专栏

文章目录

Kafka，构建TB级异步消息系统

项目源码可以在 https://gitee.com/ShayneC/community获取

Kafka，构建TB级异步消息系统

1. 阻塞队列

BlockingQueue
- 解决线程通信的问题。
- 阻塞方法：put、take。

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WSaBqEjB-1648876326522)(img\image-20191114195406.png)]$

生产者消费者模式
- 生产者：产生数据的线程。
- 消费者：使用数据的线程。
实现类
- ArrayBlockingQueue
- LinkedBlockingQueue
- PriorityBlockingQueue、SynchronousQueue、DelayQueue等。

2. Kafka入门

Kafka简介
- Kafka是一个分布式的流媒体平台。
- 应用：消息系统、日志收集、用户行为追踪、流式处理。
Kafka特点
- 高吞吐量、消息持久化、高可靠性、高扩展性。
Kafka术语
- Broker、Zookeeper
- Topic、Partition、Offset
- Leader Replica 、Follower Replica

配置zookeeper 修改zookeeper.properties

dataDir=D:/work/zookeeper

配置kafka 修改server.properties

log.dirs=D:/work/kafka-logs

启动zookeeper

E:\MyDownloads\Download\kafka_2.13-3.1.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties

启动kafka

E:\MyDownloads\Download\kafka_2.13-3.1.0>bin\windows\kafka-server-start.bat config\server.properties

创建topic

E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

列出所有的topic

E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-topics.bat --list --bootstrap-server localhost:9092

启动kafka中的生产者发送消息

E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test

启动kafka中的消费者接收消息

E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning

3. Spring整合Kafka

引入依赖
- spring-kafka
配置Kafka
- 配置server、consumer
访问Kafka
- 生产者
  kafkaTemplate.send(topic, data);
- 消费者
  @KafkaListener(topics = {“test”})
  public void handleMessage(ConsumerRecord record) {}

4. 发送系统通知

触发事件
- 评论后，发布通知
- 点赞后，发布通知
- 关注后，发布通知
处理事件
- 封装事件对象
- 开发事件的生产者
- 开发事件的消费者

新建Event实体类

package com.nowcoder.community.entity;

import java.util.HashMap;
import java.util.Map;

public class Event {

    private String topic;
    private int userId;
    private int entityType;
    private int entityId;
    private int entityUserId;
    private Map<String, Object> data = new HashMap<>();

    public String getTopic() {
        return topic;
    }

    public Event setTopic(String topic) {
        this.topic = topic;
        return this;
    }

    public int getUserId() {
        return userId;
    }

    public Event setUserId(int userId) {
        this.userId = userId;
        return this;
    }

    public int getEntityType() {
        return entityType;
    }

    public Event setEntityType(int entityType) {
        this.entityType = entityType;
        return this;
    }

    public int getEntityId() {
        return entityId;
    }

    public Event setEntityId(int entityId) {
        this.entityId = entityId;
        return this;
    }

    public int getEntityUserId() {
        return entityUserId;
    }

    public Event setEntityUserId(int entityUserId) {
        this.entityUserId = entityUserId;
        return this;
    }

    public Map<String, Object> getData() {
        return data;
    }

    public Event setData(String key, Object value) {
        this.data.put(key, value);
        return this;
    }
}

在src/main/java/com/nowcoder/community下新建event包，并新建生产者和消费者组件

新建EventProducer类

@Component
public class EventProducer {

    @Autowired
    private KafkaTemplate kafkaTemplate;

    // 处理事件
    public void fireEvent(Event event) {
        // 将事件发布到指定的主题
        kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
    }
}

新建EventConsumer类

@Component
public class EventConsumer implements CommunityConstant {

    private static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);

    @Autowired
    private MessageService messageService;

    @KafkaListener(topics = {TOPIC_COMMENT, TOPIC_LIKE, TOPIC_FOLLOW})
    public void handleCommentMessage(ConsumerRecord record) {
        if (record == null || record.value() == null) {
            logger.error("消息的内容为空");
            return;
        }

        Event event = JSONObject.parseObject(record.value().toString(), Event.class);
        if (event == null) {
            logger.error("消息格式错误");
            return;
        }

        // 发送站内通知
        Message message = new Message();
        message.setFromId(SYSTEM_USER_ID);
        message.setToId(event.getEntityUserId());
        message.setConversationId(event.getTopic());
        message.setCreateTime(new Date());

        Map<String, Object> content = new HashMap<>();
        content.put("userId", event.getUserId());
        content.put("entityType", event.getEntityType());
        content.put("entityId", event.getEntityId());

        if (!event.getData().isEmpty()) {
            for (Map.Entry<String, Object> entry : event.getData().entrySet()) {
                content.put(entry.getKey(), entry.getValue());
            }
        }

        message.setContent(JSONObject.toJSONString(content));
        messageService.addMessage(message);
    }
}

修改CommentController中的发送帖子的请求方法，在帖子发送后触发系统通知

@RequestMapping(path = "/add/{discussPostId}", method = RequestMethod.POST)
public String addComment(@PathVariable("discussPostId") int discussPostId, Comment comment) {
    comment.setUserId(hostHolder.getUser().getId());
    comment.setStatus(0);
    comment.setCreateTime(new Date());
    commentService.addComment(comment);

    // 触发评论事件
    Event event = new Event()
            .setTopic(TOPIC_COMMENT)
            .setUserId(hostHolder.getUser().getId())
            .setEntityType(comment.getEntityType())
            .setEntityId(comment.getEntityId())
            .setData("postId", discussPostId);

    if (comment.getEntityType() == ENTITY_TYPE_POST) {
        DiscussPost targe = discussPostService.findDiscussPostById(comment.getEntityId());
        event.setEntityUserId(targe.getUserId());
    } else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {
        Comment target = commentService.findCommentById(comment.getEntityId());
        event.setEntityUserId(target.getUserId());
    }
    eventProducer.fireEvent(event);

    return "redirect:/discuss/detail/" + discussPostId;
}

修改LikeController中点赞的请求，在点赞后发起系统通知

@RequestMapping(path = "/like", method = RequestMethod.POST)
@ResponseBody
public String like(int entityType, int entityId, int entityUserId, int postId) {
    User user = hostHolder.getUser();

    // 点赞
    likeService.like(user.getId(), entityType, entityId, entityUserId);
    // 点赞数量
    long likeCount = likeService.findEntityLikeCount(entityType, entityId);
    // 点赞状态
    int likeStatus = likeService.findEntityLikeStatus(user.getId(), entityType, entityId);

    Map<String, Object> map = new HashMap<>();
    map.put("likeCount", likeCount);
    map.put("likeStatus", likeStatus);

    // 触发点赞事件
    if (likeStatus == 1) {
        Event event = new Event()
                .setTopic(TOPIC_LIKE)
                .setUserId(user.getId())
                .setEntityType(entityType)
                .setEntityId(entityId)
                .setEntityUserId(entityUserId)
                .setData("postId", postId);
        eventProducer.fireEvent(event);
    }

    return CommunityUtil.getJSONString(200, null, map);
}

修改FollowController中关注用户的请求方法，在关注后发起系统通知

@RequestMapping(path = "/follow", method = RequestMethod.POST)
@ResponseBody
public String follow(int entityType, int entityId) {
    User user = hostHolder.getUser();

    followService.follow(user.getId(), entityType, entityId);

    // 触发关注事件
    Event event = new Event()
            .setTopic(TOPIC_FOLLOW)
            .setUserId(user.getId())
            .setEntityType(entityType)
            .setEntityId(entityId)
            .setEntityUserId(entityId);
    eventProducer.fireEvent(event);

    return CommunityUtil.getJSONString(200, "已关注");
}

最后修改discuss-detail.html和discuss.js两个文件

由于在EventConsumer中注入了MessageService组件，但由于调用它的时候并没有发起请求，就会导致ServiceLogAspect中请求的attributes为空，所以如果为空就直接返回。

5. 显示系统通知

通知列表
- 显示评论、点赞、关注三种类型的通知
通知详情
- 分页显示某一类主题所包含的通知
未读消息
- 在页面头部显示所有的未读消息数量

在MessageMapper中新增查询通知的接口

// 查询某个主题下最新的通知
Message selectLatestNotice(int userId, String topic);

// 查询某个主题下所包含的通知数量
int selectNoticeCount(int userId, String topic);

// 查询未读的通知数量
int selectNoticeUnreadCount(int userId, String topic);

// 查询某个主题所包含的通知列表
List<Message> selectNotices(int userId, String topic, int offset, int limit);

并在message-mapper.xml中实现接口的sql语句

<select id="selectLatestNotice" resultType="Message">
    select <include refid="selectFields"></include>
    from message
    where id in (
        select max(id) from message
        where status != 2
        and from_id = 1
        and to_id = #{userId}
        and conversation_id = #{topic}
    )
</select>

<select id="selectNoticeCount" resultType="int">
    select count(id) from message
    where status != 2
    and from_id = 1
    and to_id = #{userId}
    and conversation_id = #{topic}
</select>

<select id="selectNoticeUnreadCount" resultType="int">
    select count(id) from message
    where status = 0
    and from_id = 1
    and to_id = #{userId}
    <if test="topic!=null">
        and conversation_id = #{topic}
    </if>
</select>

<select id="selectNotices" resultType="Message">
    select <include refid="selectFields"></include>
    from message
    where status != 2
    and from_id = 1
    and to_id = #{userId}
    and conversation_id = #{topic}
    order by create_time desc
    limit #{offset}, #{limit}
</select>

在MessageService中实现查询通知信息的业务方法

public Message findLatestNotice(int userId, String topic) {
    return messageMapper.selectLatestNotice(userId, topic);
}

public int findNoticeCount(int userId, String topic) {
    return messageMapper.selectNoticeCount(userId, topic);
}

public int findNoticeUnreadCount(int userId, String topic) {
    return messageMapper.selectNoticeUnreadCount(userId, topic);
}

public List<Message> findNotices(int userId, String topic, int offset, int limit) {
    return messageMapper.selectNotices(userId, topic, offset, limit);
}

在MessageController中增加访问系统通知的请求方法和显示通知详情的请求方法

@RequestMapping(path = "/notice/list", method = RequestMethod.GET)
public String getNoticeList(Model model) {
    User user = hostHolder.getUser();

    // 查询评论类通知
    Message message = messageService.findLatestNotice(user.getId(), TOPIC_COMMENT);
    Map<String, Object> messageVo = new HashMap<>();
    if (message != null) {
        messageVo.put("message", message);

        String content = HtmlUtils.htmlUnescape(message.getContent());
        Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

        messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
        messageVo.put("entityType", data.get("entityType"));
        messageVo.put("entityId", data.get("entityId"));
        messageVo.put("postId", data.get("postId"));

        int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT);
        messageVo.put("count", count);

        int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_COMMENT);
        messageVo.put("unread", unread);

        model.addAttribute("commentNotice", messageVo);
    }

    // 查询点赞通知
    message = messageService.findLatestNotice(user.getId(), TOPIC_LIKE);
    messageVo = new HashMap<>();
    if (message != null) {
        messageVo.put("message", message);

        String content = HtmlUtils.htmlUnescape(message.getContent());
        Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

        messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
        messageVo.put("entityType", data.get("entityType"));
        messageVo.put("entityId", data.get("entityId"));
        messageVo.put("postId", data.get("postId"));

        int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE);
        messageVo.put("count", count);

        int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_LIKE);
        messageVo.put("unread", unread);

        model.addAttribute("likeNotice", messageVo);
    }

    // 查询关注类通知
    message = messageService.findLatestNotice(user.getId(), TOPIC_FOLLOW);
    messageVo = new HashMap<>();
    if (message != null) {
        messageVo.put("message", message);

        String content = HtmlUtils.htmlUnescape(message.getContent());
        Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

        messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
        messageVo.put("entityType", data.get("entityType"));
        messageVo.put("entityId", data.get("entityId"));

        int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW);
        messageVo.put("count", count);

        int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_FOLLOW);
        messageVo.put("unread", unread);

        model.addAttribute("followNotice", messageVo);
    }

    // 查询未读消息数量
    int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
    model.addAttribute("letterUnreadCount", letterUnreadCount);
    int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
    model.addAttribute("noticeUnreadCount", noticeUnreadCount);

    return "/site/notice";

}

@RequestMapping(path = "/notice/detail/{topic}", method = RequestMethod.GET)
public String getNoticeDetail(@PathVariable("topic") String topic, Page page, Model model) {
    User user = hostHolder.getUser();

    page.setLimit(5);
    page.setPath("/notice/detail/" + topic);
    page.setRows(messageService.findNoticeCount(user.getId(), topic));

    List<Message> noticeList = messageService.findNotices(user.getId(), topic, page.getOffset(), page.getLimit());
    List<Map<String, Object>> noticeVoList = new ArrayList<>();
    if (noticeList != null) {
        for (Message notice : noticeList) {
            Map<String, Object> map = new HashMap<>();
            // 通知
            map.put("notice", notice);
            // 内容
            String content = HtmlUtils.htmlUnescape(notice.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
            map.put("user", userService.findUserById((Integer) data.get("userId")));
            map.put("entityType", data.get("entityType"));
            map.put("entityId", data.get("entityId"));
            map.put("postId", data.get("postId"));
            // 通知作者
            map.put("fromUser", userService.findUserById(notice.getFromId()));

            noticeVoList.add(map);
        }
    }
    model.addAttribute("notices", noticeVoList);

    // 设置已读
    List<Integer> ids = getLetterIds(noticeList);
    if (!ids.isEmpty()) {
        messageService.readMessage(ids);
    }
    return "/site/notice-detail";
}

新建MessageInterceptor，并在WebMvcConfig中配置，用于显示未读消息数量

@Component
public class MessageInterceptor implements HandlerInterceptor {

    @Autowired
    private HostHolder hostHolder;

    @Autowired
    private MessageService messageService;

    @Override
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
        User user = hostHolder.getUser();
        if (user != null && modelAndView != null) {
            int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
            int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
            modelAndView.addObject("allUnreadCount", letterUnreadCount + noticeUnreadCount);
        }
    }
}

最后修改index.html，notice.html，notice-detail.html
HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if (user != null && modelAndView != null) {
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
modelAndView.addObject(“allUnreadCount”, letterUnreadCount + noticeUnreadCount);
}
}
}


最后修改index.html，notice.html，notice-detail.html

ShayneCh

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
5.Kafka，构建TB级异步消息系统

文章目录Kafka，构建TB级异步消息系统1. 阻塞队列2. Kafka入门3. Spring整合Kafka4. 发送系统通知5. 显示系统通知项目源码可以在https://gitee.com/ShayneC/community获取Kafka，构建TB级异步消息系统1. 阻塞队列BlockingQueue解决线程通信的问题。阻塞方法：put、take。生产者消费者模式生产者：产生数据的线程。消费者：使用数据的线程。实现类ArrayBlockingQueueL
复制链接

扫一扫