五、Kafka,构建TB级异步消息系统
5.1、阻塞队列
代码演示:
public class BlockingQueueTests {
public static void main(String[] args){
BlockingQueue queue = new ArrayBlockingQueue(10);
new Thread(new Producer(queue)).start();
//一个生产者生产数据,三个消费者同时并发地消费数据
new Thread(new Consumer(queue)).start();
new Thread(new Consumer(queue)).start();
new Thread(new Consumer(queue)).start();
}
}
class Producer implements Runnable{
private BlockingQueue<Integer> queue;
public Producer(BlockingQueue<Integer> queue){
this.queue = queue;
}
@Override
public void run(){
try{
for(int i = 0; i < 100; i++){
Thread.sleep(20);
//不断地往队列中添加数据
queue.put(i);
System.out.print(Thread.currentThread().getName()
+ "生产:" + queue.size());
}
}catch (Exception e){
e.printStackTrace();
}
}
}
class Consumer implements Runnable{
private BlockingQueue<Integer> queue;
public Consumer(BlockingQueue<Integer> queue){
this.queue = queue;
}
@Override
public void run(){
try{
while (true){
//消费者的消费能力没有生产者的那么快,消费者消费的时间间隔是随机的
Thread.sleep(new Random().nextInt(1000));
queue.take();
System.out.println(Thread.currentThread().getName()
+ "消费:" + queue.size());
}
}catch (Exception e){
e.printStackTrace();
}
}
}
5.2、Kafka入门
- Broker:缓存代理,Kafka集群中的一台或多台服务器统称broker
- Zookeeper:一个分布式的,开放源码的分布式应用程序协调服务。它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、域名服务、分布式同步、组服务等。
- Topic:Kafka处理资源的消息源的不同分类:点对点、发布订阅
- Partition:Topic物理上的分组,一个topic可以分为多个partion,每个partion是一个有序的队列。partion中每条消息都会被分配一个 有序的Id(offset)。提高并发性能
- Offset:表示消息在分区中的偏移起始位置(索引序列),消费者读取消息是按照偏移量来读取的
- 一个分区有多个副本(Kafka是一个分布式的消息引擎, Leader Replica主副本 Follower Replica:从副本)
启动:
先启动zookeeper 需要指定配置文件(新启动一个cmd窗口)
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
启动kafka(重新打开一个终端)
bin\windows\kafka-server-start.bat config\server.properties
使用kafka:
不管是创建主题还是查看主题都需要指定服务器。
创建主题(重新打开一个终端): 代表一个位置,同时也代表一种消息的类型
kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
主题名字test
查看主题: 指定需要查看的是哪台服务器
kafka-topics.bat --list --bootstrap-server localhost:9092
生产者发送消息:
kafka-console-producer.bat --broker-list localhost:9902 --topic test
hello
world
broker-list:要指定服务器列表和主题 ,向哪台服务器的哪些主题发消息
消费者读取消息:
(重新打开一个终端)从哪个服务器的哪些主题中读取消息
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
hello
world
5.3、Spring整合Kafka
配置
对kafka做相应的配置application.properties
# KafkaProperties
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=community-consumer-group
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=3000
配置server、消费者的分组id(注意:修改之后需要重启服务)、配置是否自动提交(偏移量)、配置自动提交的频率
演示代码
生产者发送消息是主动的,但消费者消费消息是被动的。(所以可能会有一点点的延迟)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)
public class KafkaTests {
@Autowired
private KafkaProducer kafkaProducer;
@Test
public void testKafka() throws InterruptedException {
kafkaProducer.sendMessage("test","你好");
kafkaProducer.sendMessage("test","在吗");
Thread.sleep(1000*10);
}
}
@Component
class KafkaProducer{
@Autowired
private KafkaTemplate kafkaTemplate;
public void sendMessage(String topics, String content){//消息的主题和内容
kafkaTemplate.send(topics, content);
}
}
@Component
class KafkaConsumer{
@KafkaListener(topics={"test"}) //关注需要监听的主题,处理监听到消息的方法
public void handleMessage(ConsumerRecord record){//消息封装成ConsumerRecord
System.out.println(record.value());
}
}
5.4、发布系统消息
因为评论、点赞、关注都是频繁的操作,为了提高性能,需要用到消息队列。把这件事包装成一个事件,仍到消息队列里去。后续的事情由消费者去处理,消费者和生产者**并发异步执行**。
**在Kafka的基础上,以事件驱动编程。**事件对象中包含了消息中的所有数据,而不是只有一个字符串,因此更具有扩展性。消费事件最终是把消息插入数据库。
封装事件Event实体
Event.java
public class Event {
private String topic;
private int userId; //事件的触发者
private int entityType; //实体类型
private int entityId; //实体id
private int entityUserId; //实体的作者id
private Map<String, Object> data = new HashMap<>();//数据都放在map之中,以便今后的扩展
public Event setTopic(String topic) {
this.topic = topic;
return this;
}
public Event setUserId(int userId) {
this.userId = userId;
return this;
}
public Event setEntityType(int entityType) {
this.entityType = entityType;
return this;
}
public Event setEntityId(int entityId) {
this.entityId = entityId;
return this;
}
public Event setEntityUserId(int entityUserId) {
this.entityUserId = entityUserId;
return this;
}
public Event setData(String key, Object value) {
this.data.put(key, value);
return this;
}
}
在set方法中返回了事件对象,链式编程,灵活。setData函数中只传递一个(key, value),很方便。
事件生产者EventProducer
@Component
public class EventProducer {
@Autowired
private KafkaTemplate kafkaTemplate;
// 处理事件
public void fireEvent(Event event) {
// 将事件发布到指定的主题
kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
}
}
事件消费者EventConsumer
CommunityConstant 定义主题类型
/**
* 主题: 评论
*/
String TOPIC_COMMENT = "comment";
/**
* 主题: 点赞
*/
String TOPIC_LIKE = "like";
/**
* 主题: 关注
*/
String TOPIC_FOLLOW = "follow";
/**
* 系统用户ID
*/
int SYSTEM_USER_ID = 1;
message表中,若是系统发送的消息,from_id = 1,conversation_id = 主题,content = 要显示的消息的json字符串
消费事件最终是把消息存入数据库
eventProducer是主动调用的,eventConsumer是被动接收,不需要调用
@Component
public class EventConsumer implements CommunityConstant {
private static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);
@Autowired
private MessageService messageService;
@KafkaListener(topics = {TOPIC_COMMENT, TOPIC_LIKE, TOPIC_FOLLOW})
public void handleCommentMessage(ConsumerRecord record) {
if (record == null || record.value() == null) {
logger.error("消息的内容为空!");
return;
}
//json转字符串再转为event
Event event = JSONObject.parseObject(record.value().toString(), Event.class);
if (event == null) {
logger.error("消息格式错误!");
return;
}
// 发送站内通知
//构造一个Message对象插入数据库,由系统发送,所以fromId为1
Message message = new Message();
message.setFromId(SYSTEM_USER_ID);//系统用户ID=1
message.setToId(event.getEntityUserId());//被评论/赞/点赞的UserID
message.setConversationId(event.getTopic());//主题
message.setCreateTime(new Date());
//用内容去拼接实际显示的通知 比如页面显示的:
//用户nowcoder(事件的触发者)评论了你的帖子(实体的数据,因为之后在前台需要链接过去
Map<String, Object> content = new HashMap<>();
content.put("userId", event.getUserId());//事件触发者的id
content.put("entityType", event.getEntityType());//实体类型:评论 赞 关注
content.put("entityId", event.getEntityId());//被通知的用户id
if (!event.getData().isEmpty()) {//其他的信息
for (Map.Entry<String, Object> entry : event.getData().entrySet()) {
content.put(entry.getKey(), entry.getValue());
}
}
message.setContent(JSONObject.toJSONString(content));
messageService.addMessage(message);
}
}
Controller
(1)CommentController
@Controller
@RequestMapping("/comment")
public class CommentController implements CommunityConstant {
@Autowired
private CommentService commentService;
@Autowired
private HostHolder hostHolder;
@Autowired
private EventProducer eventProducer;
@Autowired
private DiscussPostService discussPostService;
@RequestMapping(path = "/add/{discussPostId}", method = RequestMethod.POST)
public String addComment(@PathVariable("discussPostId") int discussPostId, Comment comment) {
comment.setUserId(hostHolder.getUser().getId());
comment.setStatus(0);
comment.setCreateTime(new Date());
commentService.addComment(comment);
// ==============触发评论事件================
Event event = new Event()
.setTopic(TOPIC_COMMENT)
.setUserId(hostHolder.getUser().getId())//当前登录者触发的消息
.setEntityType(comment.getEntityType())
.setEntityId(comment.getEntityId())
.setData("postId", discussPostId);//帖子的id,查看详情时需要
if (comment.getEntityType() == ENTITY_TYPE_POST) {//帖子
DiscussPost target =
discussPostService.findDiscussPostById(comment.getEntityId());
event.setEntityUserId(target.getUserId());
} else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {//评论
Comment target = commentService.findCommentById(comment.getEntityId());
event.setEntityUserId(target.getUserId());
}
eventProducer.fireEvent(event);//异步并发
return "redirect:/discuss/detail/" + discussPostId;
}
}
(2)LikeControlloer
@Controller
public class LikeController implements CommunityConstant {
@Autowired
private LikeService likeService;
@Autowired
private HostHolder hostHolder;
@Autowired
private EventProducer eventProducer;
@RequestMapping(path = "/like", method = RequestMethod.POST)
@ResponseBody
public String like(int entityType, int entityId, int entityUserId, int postId) {
User user = hostHolder.getUser();
// 点赞
likeService.like(user.getId(), entityType, entityId, entityUserId);
// 数量
long likeCount = likeService.findEntityLikeCount(entityType, entityId);
// 状态
int likeStatus = likeService.findEntityLikeStatus(user.getId(), entityType, entityId);
// 返回的结果
Map<String, Object> map = new HashMap<>();
map.put("likeCount", likeCount);
map.put("likeStatus", likeStatus);
// 触发点赞事件
if (likeStatus == 1) {//如果是点赞才会触发事件,取消点赞不会通知
Event event = new Event()
.setTopic(TOPIC_LIKE)
.setUserId(hostHolder.getUser().getId())
.setEntityType(entityType)
.setEntityId(entityId)
.setEntityUserId(entityUserId)
.setData("postId", postId);//传入的参数中就加入帖子的id
eventProducer.fireEvent(event);
}
return CommunityUtil.getJSONString(0, null, map);
}
}
(3)FollowController
@RequestMapping(path = "/follow", method = RequestMethod.POST)
@ResponseBody
public String follow(int entityType, int entityId) {
User user = hostHolder.getUser();
followService.follow(user.getId(), entityType, entityId);
// 触发关注事件
Event event = new Event()
.setTopic(TOPIC_FOLLOW)
.setUserId(hostHolder.getUser().getId())
.setEntityType(entityType)
.setEntityId(entityId)
.setEntityUserId(entityId);
eventProducer.fireEvent(event);
return CommunityUtil.getJSONString(0, "已关注!");
}
测试出现的bug
ServiceLogAspect.java 拦截器:出现空指针异常
HttpServletRequest request = attributes.getRequest();
@Pointcut("execution(* com.nowcoder.community.service.*.*(..))")
public void pointcut() {
}
@Before("pointcut()")
public void before(JoinPoint joinPoint) {
// 用户[1.2.3.4],在[xxx],访问了[com.nowcoder.community.service.xxx()].
ServletRequestAttributes attributes =
(ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
if (attributes == null) {
return;//直接返回,不做后面的日志
}
HttpServletRequest request = attributes.getRequest();
String ip = request.getRemoteHost();
String now = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
String target = joinPoint.getSignature().getDeclaringTypeName()
+ "." + joinPoint.getSignature().getName();
logger.info(String.format("用户[%s],在[%s],访问了[%s].", ip, now, target));
}
attributes 是和请求有关的对象,当前的AOP拦截的都是service层(Pointcut),之前所有对service的访问都是通过controller访问的,但**消费者EventConsumer中直接调用了messageService.addMessage(message),没有通过controller,因此并不存在HttpServletRequest对象,所以出现空指针异常**,再加一个非空判断即可。
5.5、显示系统通知
(1)通知列表
dao数据访问层
MessageMapper
//查询某个主题下最新的一个通知
Message selectLatestNotice(int userId, String topic);
//查询某个主题所包含的通知的数量
int selectNoticeCount(int userId, String topic);
//查询某个主题未读的通知数量
int selectNoticeUnreadCount(int userId, String topic);
message-mapper.xml
<select id="selectLatestNotice" resultType="Message">
select <include refid="selectFields"></include>
from message
where id in(
select max(id) from message
where status != 2 //删除的状态
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
)
</select>
<select id="selectNoticeCount" resultType="int">
select count(id) from message
where status != 2
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
</select>
<select id="selectNoticeUnreadCount" resultType="int">
select count(id) from message
where status = 0 //未读
and from_id = 1
and to_id = #{userId}
<if test="topic!=null">
and conversation_id = #{topic} //不传主题,查询的就是所有主题的未读消息数量
</if>
</select>
service业务层
MessageService
public Message findLatestNotice(int userId, String topic){
return messageMapper.selectLatestNotice(userId, topic);
}
public int findNoticeCount(int userId, String topic){
return messageMapper.selectNoticeCount(userId, topic);
}
public int findNoticeUnreadCount(int userId, String topic){
return messageMapper.selectNoticeUnreadCount(userId, topic);
}
Controller层
MessageController
@RequestMapping(path="/notice/list", method = RequestMethod.GET)
public String getNoticeList(Model model){
User user = hostHolder.getUser();
//===========================查询评论通知============================
Message message =
messageService.findLatestNotice(user.getId(), TOPIC_COMMENT);
Map<String, Object> messageVO = new HashMap<>();
if(message != null){
messageVO.put("message",message);
//去掉,转义字符
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
//这里存放的是最新的一个对象
messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
messageVO.put("entityType", data.get("entityType"));
messageVO.put("entityId", data.get("entityId"));
messageVO.put("postId", data.get("postId"));
//还需要查询这一类通知的总数量和未读通知数量
int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT);
messageVO.put("count", count);
int unread = messageService.findNoticeUnreadCount(
user.getId(), TOPIC_COMMENT);
messageVO.put("unread", unread);
}
model.addAttribute("commentNotice", messageVO);
//=============================查询点赞类通知==============================
message = messageService.findLatestNotice(user.getId(), TOPIC_LIKE);
messageVO = new HashMap<>();
if(message != null){
messageVO.put("message",message);
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
//这里存放的是最新的一个对象
messageVO.put("user", userService.findUserById(
(Integer) data.get("userId")));
messageVO.put("entityType", data.get("entityType"));
messageVO.put("entityId", data.get("entityId"));
messageVO.put("postId", data.get("postId"));
int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE);
messageVO.put("count", count);
int unread = messageService.findNoticeUnreadCount(
user.getId(), TOPIC_LIKE);
messageVO.put("unread", unread);
}
model.addAttribute("likeNotice", messageVO);
//============================查询关注类通知==================================
message = messageService.findLatestNotice(user.getId(), TOPIC_FOLLOW);
messageVO = new HashMap<>();
if(message != null){
messageVO.put("message", message);
String content = HtmlUtils.htmlUnescape(message.getContent());
Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
//这里存放的是最新的一个对象
messageVO.put("user", userService.findUserById(
(Integer) data.get("userId")));
messageVO.put("entityType", data.get("entityType"));
messageVO.put("entityId", data.get("entityId"));
int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW);
messageVO.put("count", count);
int unread = messageService.findNoticeUnreadCount(
user.getId(), TOPIC_FOLLOW);
messageVO.put("unread", unread);
}
model.addAttribute("followNotice", messageVO);
//============================查询所有未读消息数量========================
int letterUnreadCount = messageService.findLetterUnreadCount(
user.getId(), null);
model.addAttribute("letterUnreadCount", letterUnreadCount);
//返回的是通知的数量
int noticeUnreadCount = messageService.findNoticeUnreadCount(
user.getId(), null);
model.addAttribute("noticeUnreadCount", noticeUnreadCount);
return "/site/notice";
}
notice.html
(2)通知详情
dao层
MessageMapper
//查询某个主题所包含的通知列表
List<Message> selectNotices(int userId, String topic, int offset, int limit);
message-mapper.xml
<select id="selectNotices" resultType="Message">
select <include refid="selectFields"></include>
from message
where status != 2
and from_id = 1
and to_id = #{userId}
and conversation_id = #{topic}
order by create_time desc
limit #{offset}, #{limit}
</select>
service层
MessageService
public List<Message> findNotices(int userId, String topic, int offset, int limit){
return messageMapper.selectNotices(userId, topic, offset, limit);
}
controller层
MessageController
@RequestMapping(path = "/notice/detail/{topic}", method = RequestMethod.GET)
public String getNoticeDetail(@PathVariable("topic") String topic, Page page, Model model){
User user = hostHolder.getUser();
//对分页进行设置
page.setLimit(5);
page.setPath("/notice/detail/" + topic);
page.setRows(messageService.findNoticeCount(user.getId(), topic));
List<Message> noticeList = messageService.findNotices(
user.getId(), topic, page.getOffset(), page.getLimit());
List<Map<String, Object>> noticeVoList = new ArrayList<>();
if(noticeVoList != null){
for(Message notice: noticeList){
//传递聚合数据
Map<String, Object> map = new HashMap<>();
//通知
map.put("notice", notice);
//内容
String content = HtmlUtils.htmlUnescape(notice.getContent());
Map<String, Object> data = JSONObject.parseObject(
content, HashMap.class);
map.put("user", userService.findUserById(
(Integer) data.get("userId")));
map.put("entityType", data.get("entityType"));
map.put("entityId", data.get("entityId"));
map.put("postId", data.get("postId"));
//发布通知的作者
map.put("fromUser", userService.findUserById(notice.getFromId()));
noticeVoList.add(map);
}
}
model.addAttribute("notices", noticeVoList);
//设置已读
List<Integer> ids = getLetterIds(noticeList);
if(!ids.isEmpty()){
messageService.readMessage(ids);
}
return "/site/notice-detail";
}
(3)显示头部的总的未读消息总数,利用拦截器实现
创建MessageInterceptor
消息总数量为未读私信数量和未读通知数量,重写postHandle方法。
@Component
public class MessageInterceptor implements HandlerInterceptor {
@Autowired
private HostHolder hostHolder;
@Autowired
private MessageService messageService;
@Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if(user != null && modelAndView != null){
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);
modelAndView.addObject("allUnreadCount", letterUnreadCount + noticeUnreadCount);
}
}
}
WebMvcConfig配置拦截器
不拦截静态请求,但拦截所有的动态请求
registry.addInterceptor(messageInterceptor)
.excludePathPatterns("/**/*.css", "/**/*.js", "/**/*.jpg", "/**/*.jpeg", "/**/*.png");
在index.html中的header中显示数据
<li class="nav-item ml-3 btn-group-vertical" th:if="${loginUser!=null}">
<a class="nav-link position-relative" th:href="@{/letter/list}">消息
<span class="badge badge-danger" th:text="${allUnreadCount!=0?allUnreadCount:''}">12</span>
</a>
</li>