项目源码可以在 https://gitee.com/ShayneC/community获取
Kafka,构建TB级异步消息系统
1. 阻塞队列
- BlockingQueue
- 解决线程通信的问题。
- 阻塞方法:put、take。
-
生产者消费者模式
- 生产者:产生数据的线程。
- 消费者:使用数据的线程。
-
实现类
- ArrayBlockingQueue
- LinkedBlockingQueue
- PriorityBlockingQueue、SynchronousQueue、DelayQueue等。
2. Kafka入门
- Kafka简介
- Kafka是一个分布式的流媒体平台。
- 应用:消息系统、日志收集、用户行为追踪、流式处理。
- Kafka特点
- 高吞吐量、消息持久化、高可靠性、高扩展性。
- Kafka术语
- Broker、Zookeeper
- Topic、Partition、Offset
- Leader Replica 、Follower Replica
配置zookeeper 修改zookeeper.properties
dataDir=D:/work/zookeeper
配置kafka 修改server.properties
log.dirs=D:/work/kafka-logs
启动zookeeper
E:\MyDownloads\Download\kafka_2.13-3.1.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
启动kafka
E:\MyDownloads\Download\kafka_2.13-3.1.0>bin\windows\kafka-server-start.bat config\server.properties
创建topic
E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
列出所有的topic
E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-topics.bat --list --bootstrap-server localhost:9092
启动kafka中的生产者 发送消息
E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
启动kafka中的消费者 接收消息
E:\MyDownloads\Download\kafka_2.13-3.1.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
3. Spring整合Kafka
- 引入依赖
- spring-kafka
- 配置Kafka
- 配置server、consumer
- 访问Kafka
- 生产者
kafkaTemplate.send(topic, data); - 消费者
@KafkaListener(topics = {“test”})
public void handleMessage(ConsumerRecord record) {}
- 生产者
4. 发送系统通知
- 触发事件
- 评论后,发布通知
- 点赞后,发布通知
- 关注后,发布通知
- 处理事件
- 封装事件对象
- 开发事件的生产者
- 开发事件的消费者
新建Event实体类
package com.nowcoder.community.entity;
import java.util.HashMap;
import java.util.Map;
public class Event {
private String topic;
private int userId;
private int entityType;
private int entityId;
private int entityUserId;
private Map<String, Object> data = new HashMap<>();
public String getTopic() {
return topic;
}
public Event setTopic(String topic) {
this.topic = topic;
return this;
}
public int getUserId() {
return userId;
}
public Event setUserId(int userId) {
this.userId = userId;
return this;
}
public int getEntityType() {
return entityType;
}
public Event setEntityType(int entityType) {
this.entityType = entityType;
return this;
}
public int getEntityId() {
return entityId;
}
public Event setEntityId(int entityId) {
this.entityId = entityId;
return this;
}
public int getEntityUserId() {
return entityUserId;
}
public Event setEntityUserId(int entityUserId) {
this.entityUserId = entityUserId;
return this;
}
public Map<String, Object> getData() {
return data;
}
public Event setData(String key, Object value) {
this.data.put(key, value);
return this;
}
}
在src/main/java/com/nowcoder/community下新建event包,并新建生产者和消费者组件
新建EventProducer类
@Component
public class EventProducer {
@Autowired
private KafkaTemplate kafkaTemplate;
// 处理事件
public void fireEvent(Event event) {
// 将事件发布到指定的主题
kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
}
}
新建EventConsumer类
@Component
public class EventConsumer implements CommunityConstant {
private static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);
@Autowired
private MessageService messageService;
@KafkaListener(topics = {TOPIC_COMMENT, TOPIC_LIKE, TOPIC_FOLLOW})
public void handleCommentMessage(ConsumerRecord record) {
if (record == null || record.value() == null) {
logger.error("消息的内容为空");
return;
}
Event event = JSONObject.parseObject(record.value().toString(), Event.class);
if (event == null) {
logger.error("消息格式错误");
return;
}
// 发送站内通知
Message message = new Message();
message.setFromId(SYSTEM_USER_ID);
message.setToId(event.getEntityUserId());
message.setConversationId(event.getTopic());
message.setCreateTime(new Date());
Map<String, Object> content = new HashMap<>();
content.put("userId", event.getUserId());
content.put("entityType", event.getEntityType());
content.put("entityId", event.getEntityId());
if (!event.getData().isEmpty()) {
for (Map.Entry<String, Object> entry : event.getData().entrySet()) {
content.put(entry.getKey(), entry.getValue());
}
}
message.setContent(JSONObject.toJSONString(content));
messageService.addMessage(message);
}
}
修改CommentController中的发送帖子的请求方法,在帖子发送后触发系统通知
@RequestMapping(path = "/add/{discussPostId}", method = RequestMethod.POST)
public String addComment(@PathVariable("discussPostId") int discussPostId, Comment comment) {
comment.setUserId(hostHolder.getUser().getId());
comment.setStatus(0);
comment.setCreateTime(new Date());
commentService.addComment(comment);
// 触发评论事件
Event event = new Event()
.setTopic(TOPIC_COMMENT)
.setUserId(hostHolder.getUser().getId())
.setEntityType(comment.getEntityType())
.setEntityId(comment.getEntityId())
.setData("postId", discussPostId);
if (comment.getEntityType() == ENTITY_TYPE_POST) {
DiscussPost targe = discussPostService.findDiscussPostById(comment.getEntityId());
event.setEntityUserId(targe.getUserId());
} else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {
Comment target = commentService.findCommentById(comment.getEntityId());
event.setEntityUserId(target.getUserId());
}
eventProducer.fireEvent(event);
return "redirect:/discuss/detail/" + discussPostId;
}
修改LikeController中点赞的请求,在点赞后发起系统通知
@RequestMapping(path = "/like", method = RequestMethod.POST)
@ResponseBody
public String like(int entityType, int entityId, int entityUserId, int postId) {
User user = hostHolder.getUser();
// 点赞
likeService.like(user.getId(), entityType, entityId, entityUserId);
// 点赞数量
long likeCount = likeService.findEntityLikeCount(entityType, entityId);
// 点赞状态
int likeStatus = likeService.findEntityLikeStatus(user.getId(), entityType, entityId);
Map<String, Object> map = new HashMap<>();
map.put("likeCount", likeCount);
map.put("likeStatus", likeStatus);
// 触发点赞事件
if (likeStatus == 1) {
Event event = new Event()
.setTopic(TOPIC_LIKE)
.setUserId(user.getId())
.setEntityType(entityType)
.setEntityId(entityId)
.setEntityUserId(entityUserId)
.setData("postId", postId);
eventProducer.fireEvent(event);
}
return CommunityUtil.getJSONString(200, null, map);
}
修改FollowController中关注用户的请求方法,在关注后发起系统通知
@RequestMapping(path = "/follow", method = RequestMethod.POST)
@ResponseBody
public String follow(int entityType, int entityId) {
User user = hostHolder.getUser();
followService.follow(user.getId(), entityType, entityId);
// 触发关注事件
Event event = new Event()
.setTopic(TOPIC_FOLLOW)
.setUserId(user.getId())
.setEntityType(entityType)
.setEntityId(entityId)
.setEntityUserId(entityId);
eventProducer.fireEvent(event);
return CommunityUtil.getJSONString(200, "已关注");
}
最后修改discuss-detail.html和discuss.js两个文件
由于在EventConsumer中注入了MessageService组件,但由于调用它的时候并没有发起请求,就会导致ServiceLogAspect中请求的attributes为空,所以如果为空就直接返回。
5. 显示系统通知
- 通知列表
- 显示评论、点赞、关注三种类型的通知
- 通知详情
- 分页显示某一类主题所包含的通知
- 未读消息
- 在页面头部显示所有的未读消息数量
在MessageMapper中新增查询通知的接口
// 查询某个主题下最新的通知
Message selectLatestNotice(int userId, String topic);
// 查询某个主题下所包含的通知数量
int selectNoticeCount(int userId, String topic);
// 查询未读的通知数量
int selectNoticeUnreadCount(int userId, String topic);
// 查询某个主题所包含的通知列表
List<Message> selectNotices(int userId, String topic, int offset, int limit);
并在message-mapper.xml中实现接口的sql语句
<select id="selectLatestNotice" resultType="Message">
select <include refid="selectFields"></include>
from message
where id in (
select max(id) from message
where status != 2
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
)
</select>
<select id="selectNoticeCount" resultType="int">
select count(id) from message
where status != 2
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
</select>
<select id="selectNoticeUnreadCount" resultType="int">
select count(id) from message
where status = 0
and from_id = 1
and to_id = #{userId}
<if test="topic!=null">
and conversation_id = #{topic}
</if>
</select>
<select id="selectNotices" resultType="Message">
select <include refid="selectFields"></include>
from message
where status != 2
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
order by create_time desc
limit #{offset}, #{limit}
</select>
在MessageService中实现查询通知信息的业务方法
public Message findLatestNotice(int userId, String topic) {
return messageMapper.selectLatestNotice(userId, topic);
}
public int findNoticeCount(int userId, String topic) {
return messageMapper.selectNoticeCount(userId, topic);
}
public int findNoticeUnreadCount(int userId, String topic) {
return messageMapper.selectNoticeUnreadCount(userId, topic);
}
public List<Message> findNotices(int userId, String topic, int offset, int limit) {
return messageMapper.selectNotices(userId, topic, offset, limit);
}
在MessageController中增加访问系统通知的请求方法和显示通知详情的请求方法
@RequestMapping(path = "/notice/list", method = RequestMethod.GET)
public String getNoticeList(Model model) {
User user = hostHolder.getUser();
// 查询评论类通知
Message message = messageService.findLatestNotice(user.getId(), TOPIC_COMMENT);
Map<String, Object> messageVo = new HashMap<>();
if (message != null) {
messageVo.put("message", message);
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
messageVo.put("entityType", data.get("entityType"));
messageVo.put("entityId", data.get("entityId"));
messageVo.put("postId", data.get("postId"));
int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT);
messageVo.put("count", count);
int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_COMMENT);
messageVo.put("unread", unread);
model.addAttribute("commentNotice", messageVo);
}
// 查询点赞通知
message = messageService.findLatestNotice(user.getId(), TOPIC_LIKE);
messageVo = new HashMap<>();
if (message != null) {
messageVo.put("message", message);
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
messageVo.put("entityType", data.get("entityType"));
messageVo.put("entityId", data.get("entityId"));
messageVo.put("postId", data.get("postId"));
int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE);
messageVo.put("count", count);
int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_LIKE);
messageVo.put("unread", unread);
model.addAttribute("likeNotice", messageVo);
}
// 查询关注类通知
message = messageService.findLatestNotice(user.getId(), TOPIC_FOLLOW);
messageVo = new HashMap<>();
if (message != null) {
messageVo.put("message", message);
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
messageVo.put("user", userService.findUserById((Integer) data.get("userId")));
messageVo.put("entityType", data.get("entityType"));
messageVo.put("entityId", data.get("entityId"));
int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW);
messageVo.put("count", count);
int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_FOLLOW);
messageVo.put("unread", unread);
model.addAttribute("followNotice", messageVo);
}
// 查询未读消息数量
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
model.addAttribute("letterUnreadCount", letterUnreadCount);
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
model.addAttribute("noticeUnreadCount", noticeUnreadCount);
return "/site/notice";
}
@RequestMapping(path = "/notice/detail/{topic}", method = RequestMethod.GET)
public String getNoticeDetail(@PathVariable("topic") String topic, Page page, Model model) {
User user = hostHolder.getUser();
page.setLimit(5);
page.setPath("/notice/detail/" + topic);
page.setRows(messageService.findNoticeCount(user.getId(), topic));
List<Message> noticeList = messageService.findNotices(user.getId(), topic, page.getOffset(), page.getLimit());
List<Map<String, Object>> noticeVoList = new ArrayList<>();
if (noticeList != null) {
for (Message notice : noticeList) {
Map<String, Object> map = new HashMap<>();
// 通知
map.put("notice", notice);
// 内容
String content = HtmlUtils.htmlUnescape(notice.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
map.put("user", userService.findUserById((Integer) data.get("userId")));
map.put("entityType", data.get("entityType"));
map.put("entityId", data.get("entityId"));
map.put("postId", data.get("postId"));
// 通知作者
map.put("fromUser", userService.findUserById(notice.getFromId()));
noticeVoList.add(map);
}
}
model.addAttribute("notices", noticeVoList);
// 设置已读
List<Integer> ids = getLetterIds(noticeList);
if (!ids.isEmpty()) {
messageService.readMessage(ids);
}
return "/site/notice-detail";
}
新建MessageInterceptor,并在WebMvcConfig中配置,用于显示未读消息数量
@Component
public class MessageInterceptor implements HandlerInterceptor {
@Autowired
private HostHolder hostHolder;
@Autowired
private MessageService messageService;
@Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if (user != null && modelAndView != null) {
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
modelAndView.addObject("allUnreadCount", letterUnreadCount + noticeUnreadCount);
}
}
}
最后修改index.html,notice.html,notice-detail.html
HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if (user != null && modelAndView != null) {
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
modelAndView.addObject(“allUnreadCount”, letterUnreadCount + noticeUnreadCount);
}
}
}
最后修改index.html,notice.html,notice-detail.html