在日常的工作当中,服务器上面跑了非常多的东西,使用开源的监控组件去监控服务器上面的信息,发现服务器上面一些负载情况超标了就发送对应的告警信息,下面的案例是模仿服务器可能在一个时间段内发生多次告警信息,使用alertmanager将这一系列的告警信息路由给java后端接口来进行处理,使用rabbitmq缓冲告警消息,从而提高程序的性能
- 在pom.xml中导入依赖:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.7.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
2.编写配置文件
server:
port: 7003
spring:
application:
name: rabbitmq
rabbitmq:
host: 192.168.31.100
port: 5672
username: guest
password: guest
template:
retry: # 失败重试
enabled: true # 开启失败重试
initial-interval: 10000ms # 第一次重试的间隔时长
max-interval: 300000ms # 最长重试间隔,超过这个间隔将不再重试
multiplier: 2 # 下次重试间隔的倍数,此处是2即下次重试间隔是上次的2倍
exchange: topic.exchange # 缺省的交换机名称,此处配置后,发送消息如果不指定交换机就会使用这个
publisher-confirms: true # 生产者确认机制,确保消息会正确发送,如果发送失败会有错误回执,从而触发重试
publisher-returns: true # 可以确保消息在未被队列接收时返回
# 如果consumer只是接收消息而不发送,就不用配置template相关内容
3.编写Rabbitmq配置类
@Configuration
public class RabbitMQConfiguration {
public static final String EXCHANGE_NAME_DIRECT="direct.exchange";//topics类型交换机
public static final String DIRECT_EXCHANGE_WITH_QUEUE="direct_exchange_with_queue";
/**
* 配置RabbitMq 消息服务
*/
@Bean("connectionFactory")//默认和方法名一致
public ConnectionFactory rabbitConnectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
String rabbitmqHost = "192.168.31.100";
String rabbitmqPort = "5672";
String rabbitmqUsername = "guest";
String rabbitmqPassword = "guest";
String rabbitmqVirtualHost = "/mye";
connectionFactory.setHost(rabbitmqHost);
connectionFactory.setPort(Integer.parseInt(rabbitmqPort));
connectionFactory.setUsername(rabbitmqUsername);
connectionFactory.setPassword(rabbitmqPassword);
connectionFactory.setVirtualHost(rabbitmqVirtualHost);
connectionFactory.setPublisherConfirms(true); // 设置后才能显示调用
return connectionFactory;
}
/**
* 消息生产者
*/
@Bean(name = "rabbitTemplate")
//必须是prototype类型
@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(rabbitConnectionFactory());
}
/**
* 参数详解
* 1、directExchange() 路由模式,也是默认的
* 2、topicExchange() 通配符模式
* 3、fanoutExchange() 发布订阅模式
* 4、headersExchange() 头交换机(路由模式键只有是字符串,而头交换机可以是整型和哈希值)
* 5、durable(true) 持久化,mq重启后交换机还在
* @return 交换机
*/
//声明交换机
@Bean(EXCHANGE_NAME_DIRECT)
public Exchange exchange() {
return ExchangeBuilder.directExchange(EXCHANGE_NAME_DIRECT).durable(true).build();
}
/**
* 参数详解
* public Queue(String name, boolean durable, boolean exclusive, boolean autoDelete, Map<String, Object> arguments)
* 1、name: 队列的名称
* 2、durable: 是否持久化,如果持久化,mq重启后队列还在
* 3、exclusive: 是否独占连接,队列只允许在该连接中访问,如果connection连接关闭队列则自动删除,如果将此参数设置true可用于临时队列的创建,默认false
* 4、autoDelete:是否自动删除,队列不再使用时是否自动删除此队列,如果将此参数和exclusive参数设置为true就可以实现临时队列(队列不用了就自动删除)默认false
* 5、arguments:参数,可以设置一个队列的扩展参数,比如:可设置存活时间
*
* @return
* @throws UnknownHostException
*/
//声明队列
@Bean("queue")
public Queue queue() throws UnknownHostException {
获取本机(或者服务器ip地址)
InetAddress inetAddress = InetAddress.getLocalHost();
return new Queue(inetAddress.getHostName(),true, false, true);
}
//队列绑定交换机
@Bean
public Binding binding(@Qualifier("queue") Queue queue,@Qualifier(EXCHANGE_NAME_DIRECT) Exchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(DIRECT_EXCHANGE_WITH_QUEUE).noargs();
}
@Bean("customContainerFactory")
public SimpleRabbitListenerContainerFactory containerFactory(SimpleRabbitListenerContainerFactoryConfigurer configurer,
@Autowired ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConcurrentConsumers(1); //设置线程数
factory.setMaxConcurrentConsumers(1); //最大线程数
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);//开启手动确认消息
configurer.configure(factory, connectionFactory);
return factory;
}
}
一般在实际的生产环境中,要考虑非常多的情况,比如一些网络的抖动,服务器可能宕机了等等一系列不可抗拒的因素都有可能影响消息的投递,因此会单独建立一张表来记载消息的一些情况,消息表字段如下:
对应实体类
@Data
public class MsgLog implements Serializable {
private String id;
private String msg;
private String exchange;
private String routingKey;
private Integer status;
private Integer tryCount;
private Long nextTryTime;
private Long createTime;
private Long updateTime;
private String alertId;
消息发送确认
发送的消息怎么样才算失败或成功?如何确认?
- 当消息无法路由到队列时,确认消息路由失败。消息成功路由时,当需要发送的队列都发送成功后,进行确认消息,对于持久化队列意味着写入磁盘,对于镜像队列意味着所有镜像接收成功
ConfirmCallback
- 通过实现 ConfirmCallback 接口,消息发送到 Broker 后触发回调,确认消息是否到达 Broker 服务器,也就是只确认是否正确到达 Exchange 中
@Component
public class ConfirmCallbackService implements RabbitTemplate.ConfirmCallback {
private static final Logger logger = LoggerFactory.getLogger(ConfirmCallbackService.class);
@Autowired
private MsgLogBiz msgLogBiz;
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
if (!ack) {
logger.error("消息发送异常!");
} else {
String msgId = correlationData.getId();
msgLogBiz.updateStatus(msgId, MsgLogStatusEnum.DELIVER_SUCCESS.getKey(),System.currentTimeMillis());
logger.info("发送者已经收到确认,correlationData={} ,ack={}, cause={}", correlationData.getId(), ack, cause);
}
}
}
ReturnCallback
- 通过实现 ReturnCallback 接口,启动消息失败返回,比如路由不到队列时触发回调
@Component
public class ReturnCallbackService implements RabbitTemplate.ReturnCallback {
private static final Logger logger = LoggerFactory.getLogger(ConfirmCallbackService.class);
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
logger.error("returnedMessage ===> replyCode={} ,replyText={} ,exchange={} ,routingKey={}", replyCode, replyText, exchange, routingKey);
System.out.println("消息主体 message : "+message);
System.out.println("消息主体 message : "+replyCode);
System.out.println("描述:"+replyText);
System.out.println("消息使用的交换器 exchange : "+exchange);
System.out.println("消息使用的路由键 routing : "+routingKey);
}
}
编一个枚举类用于记录消息的投递状态
public enum MsgLogStatusEnum {
/**
* 投递中
**/
DELIVERING(0, "delivering"),
/**
* 投递成功
**/
DELIVER_SUCCESS(1, "deliver_success"),
/**
* 投递失败
**/
DELIVER_FAIL(2, "deliver_fail"),
/**
* 消费成功
**/
CONSUMED_SUCCESS(3, "consumed_success");
private final int key;
private final String value;
public int getKey() {
return key;
}
public String getValue() {
return value;
}
MsgLogStatusEnum(int key, String value) {
this.key = key;
this.value = value;
}
/**
* 判断数值是否属于枚举类的值
*
* @param key
* @return
*/
public static boolean isInclude(int key) {
boolean include = false;
for (MsgLogStatusEnum e : MsgLogStatusEnum.values()) {
if (e.key == key) {
include = true;
break;
}
}
return include;
}
public static String getName(int key) {
String value = "";
for (MsgLogStatusEnum e : MsgLogStatusEnum.values()) {
if (e.key == key) {
value = e.getValue();
break;
}
}
return value;
}
}
接下来是生产者代码投递消息的代码
@RestController
@RequestMapping("/api/webhook")
public class WebHookController extends ApiController {
@Autowired
private RabbitTemplate rabbitTemplate;
@Autowired
private ConfirmCallbackService confirmCallbackService;
@Autowired
private ReturnCallbackService returnCallbackService;
@Autowired
private MsgLogBiz msgLogBiz;
private static final Logger logger = LoggerFactory.getLogger(WebHookController.class);
@RequestMapping(value = "/dcim1", produces = "application/json;charset=UTF-8")
@ResponseBody
public void getAlarm2(@RequestBody String alarmJson) {
logger.info("收到原始告警信息: {}", alarmJson);
List<Alert> alertList;
try {
JSONObject jsonObject = JSONUtil.parseObj(alarmJson);
JSONArray alerts = jsonObject.getJSONArray("alerts");
alertList = alerts.toList(Alert.class);
logger.info("alertList: {}", alertList);
} catch (Exception e) {
logger.error("收到告警信息Json格式化出现问题: " + alarmJson);
return;
}
if (CollectionUtil.isEmpty(alertList)) {
return;
}
this.sendAlert(alertList);
}
private void sendAlert(List<Alert> alertList) {
String msgId = "alert-" + UUIDUtil.generateUUID();
MsgLog msgLog = new MsgLog(msgId, JSONUtil.toJsonStr(alertList), "alertDirectExchange", "alert", MsgLogStatusEnum.DELIVERING.getKey(), 0, System.currentTimeMillis() + 2 * 60 * 1000L, System.currentTimeMillis(), 0L);
msgLogBiz.insert(msgLog);
/**
* 当mandatory标志位设置为true时
* 如果exchange根据自身类型和消息routingKey无法找到一个合适的queue存储消息
* 那么broker会调用basic.return方法将消息返还给生产者
* 当mandatory设置为false时,出现上述情况broker会直接将消息丢弃
*/
rabbitTemplate.setMandatory(true);
//confirmCallback接口用于实现消息发送到rabbitmq交换机后接受ack回调
rabbitTemplate.setConfirmCallback(confirmCallbackService);
//returnCallback接口用于实现消息发送到rabbitmq交换器。但无相应队列与交换机绑定式的回调
rabbitTemplate.setReturnCallback(returnCallbackService);
rabbitTemplate.convertAndSend("alertDirectExchange", "alert", JSONUtil.toJsonStr(msgLog), message -> {
// 消息的持久化,生产者到broker消息不丢失(消息持久化到磁盘)
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}, new CorrelationData(msgId));
}
}
在消息投递失败时需要有一个补偿机制,这里编写一个定时任务,在生产者消息投递失败时继续投递,直到达到最大投递次数,如果还是失败那就不再投递
@Component
public class ResendMsg {
private static final Logger logger = LoggerFactory.getLogger(ResendMsg.class);
@Autowired
private MsgLogBiz msgLogBiz;
@Autowired
private RabbitTemplate rabbitTemplate;
// 最大投递次数
private static final int MAX_TRY_COUNT = 3;
/**
* 每30s拉取投递失败的消息, 重新投递
*/
@Scheduled(cron = "0/30 * * * * ?")
public void resend() {
logger.info("开始执行定时任务(重新投递消息)");
long currentTimeMillis = System.currentTimeMillis();
List<MsgLog> msgLogs = msgLogBiz.selectTimeoutMsg(currentTimeMillis);
if (CollectionUtil.isEmpty(msgLogs)) {
logger.info("开始执行定时任务(重新投递消息),数据为空终止执行.");
return;
}
msgLogs.forEach(msgLog -> {
String msgId = msgLog.getId();
if (msgLog.getTryCount() >= MAX_TRY_COUNT) {
msgLogBiz.updateStatus(msgId, MsgLogStatusEnum.DELIVER_FAIL.getKey(), currentTimeMillis);
logger.info("超过最大重试次数, 消息投递失败, msgId: {}", msgId);
} else {
int count = msgLog.getTryCount() + 1;
msgLog.setTryCount(count);
Long nextTryTime = msgLog.getNextTryTime();
if (nextTryTime == 0) {
nextTryTime = currentTimeMillis;
}
msgLog.setNextTryTime(nextTryTime + count * 60 * 1000L);
msgLog.setUpdateTime(currentTimeMillis);
msgLogBiz.updateTryCount(msgLog);
CorrelationData correlationData = new CorrelationData(msgId);
rabbitTemplate.convertAndSend(msgLog.getExchange(), msgLog.getRoutingKey(), JSONUtil.toJsonStr(msgLog), correlationData);// 重新投递
logger.info("第 " + (msgLog.getTryCount()) + " 次重新投递消息: " + msgId);
}
});
logger.info("定时任务执行结束(重新投递消息)");
}
}
其中数据库持久层操作接口代码如下:
public interface MsgLogBiz {
// 插入一条数据
int insert(MsgLog msgLog);
// 根据id更新投递消息的状态
int updateStatus(String msgId,int status,long updateTime);
// 根据id查询消息详情
MsgLog selectByMsgId(String msgId);
// 根据当前时间查询所有正在投递中的
List<MsgLog> selectTimeoutMsg(long currentTime);
// 更新消息的重试次数
int updateTryCount(MsgLog msgLog);
}
对应mapper层代码
@Repository
@Mapper
public interface MsgLogMapper {
/**
* 消息表 表名称
**/
String MSG_LOG_TABLE_NAME = "msg_log";
/**
* 插入一条数据
**/
@Insert("INSERT INTO " + MSG_LOG_TABLE_NAME + " (id, msg,exchange,routingKey,status,tryCount,nextTryTime,createTime,updateTime,alertId) VALUES (" +
"#{id}, #{msg},#{exchange},#{routingKey},#{status}, #{tryCount},#{nextTryTime},#{createTime},#{updateTime},#{alertId})")
int insert(MsgLog msgLog);
@Update("update " + MSG_LOG_TABLE_NAME + " set status=#{status}," +
"updateTime=#{updateTime} where id=#{id}")
int updateStatus(@Param("id") String id, @Param("status") int status, @Param("updateTime") long updateTime);
@Select("SELECT * from " + MSG_LOG_TABLE_NAME + " where status = 0 AND nextTryTime <= #{currentTime} ")
List<MsgLog> selectTimeoutMsg(@Param("currentTime")long currentTime);
@Update("update " + MSG_LOG_TABLE_NAME + " set tryCount=#{tryCount}, nextTryTime=#{nextTryTime}, " +
"updateTime=#{updateTime} where id=#{id}")
int updateTryCount(MsgLog msgLog);
@Select("SELECT * from " + MSG_LOG_TABLE_NAME + " where id =#{id}")
MsgLog selectByMsgId(@Param("id") String id);
}
消费者端代码
@Component
public class WebHookMqAlertListener {
@Autowired
private RabbitTemplate rabbitTemplate;
@Autowired
private ConfirmCallbackService confirmCallbackService;
@Autowired
private ReturnCallbackService returnCallbackService;
@Autowired
private MsgLogBiz msgLogBiz;
private static final Logger logger = LoggerFactory.getLogger(WebHookMqAlertListener.class);
@RabbitListener(containerFactory = "customContainerFactory", bindings = @QueueBinding(
value = @Queue("alertDirectQueue"),
exchange = @Exchange(value = "alertDirectExchange", type = ExchangeTypes.DIRECT),
key = "alert"
))
public void process(@Payload String msg, Channel channel, Message message) {
logger.info("收到告警消息:" + msg);
MsgLog msgLog = JSONUtil.toBean(msg, MsgLog.class);
MsgLog msgLogOld = msgLogBiz.selectByMsgId(msgLog.getId());
if (null == msgLogOld || msgLogOld.getStatus().equals(MsgLogStatusEnum.CONSUMED_SUCCESS.getKey())) {
logger.info("重复消费, msgId: {}", msgLog.getId());
msgAck(channel, message, msgLog, false);
return;
}
System.out.println(JSONUtil.toList(msgLog.getMsg(), Alert.class));
System.out.println(msgLog.getId());
msgAck(channel, message, msgLog, false);
}
private void msgAck(Channel channel, Message message, MsgLog msgLog, boolean isRepetition) {
try {
if (!isRepetition) {
msgLogBiz.updateStatus(msgLog.getId(), MsgLogStatusEnum.CONSUMED_SUCCESS.getKey(), System.currentTimeMillis());
}
//deliveryTag:该消息的index
//false:是否批处理
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
logger.info("告警消息手动确认:" + message.getMessageProperties().getDeliveryTag());
} catch (Exception e) {
//获取消息中是否再次投递,true:再次投递
if (message.getMessageProperties().getRedelivered()) {
logger.error("告警消息已重复处理失败,拒绝再次接收...");
try {
//deliveryTag:该消息的index
//消息发送成功并且在broker落地,deliveryTag是唯一标志符,在channek上发布的消息的deliveryTag都会比之前加1
//requeue:被拒绝后是否重新加入队列
//channel.basicReject和channel.basicNack的区别是basicNack可以拒绝多条消息,而basicReject一次只能拒绝一条消息
channel.basicReject(message.getMessageProperties().getDeliveryTag(), false); // 拒绝消息
} catch (IOException ioException) {
ioException.printStackTrace();
}
} else {
logger.error("告警消息即将再次返回队列处理...");
try {
//deliveryTag:该消息的index
//multiple:是否批量true:将一次性拒绝所有小于deliveryTag的消息
//requeue:被拒绝后是否重新加入队列
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, true);
} catch (IOException ioException) {
ioException.printStackTrace();
}
}
}
}
告警信息内容如下
{
"alerts": [
{
"annotations": {
"description": "温度指标告警告警,当前值是: 28.3!",
"summary": "温度指标告警告警"
},
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "e467f0ec6d95a4b4",
"generatorURL": "http://d8a0df8cb5a1:9090/graph?g0.expr=temperature_guage%7BcomponentInstanceId%3D%2233%22%2CcomponentSignalId%3D%221%22%7D+%3C+30&g0.tab=1",
"labels": {
"alertname": "温度指标告警闪断震荡测试",
"componentId": "6",
"componentInstanceId": "33",
"componentSignalId": "1",
"description": "获取温度信息",
"dynamicId": "1",
"hostGroupId": "1",
"instance": "192.168.10.125:8112",
"job": "test-dcim",
"nameChs": "温度指标",
"oid": "1.3.6.1.4.1.123456.1.1.1.1.0",
"severity": "minor",
"signalCollectId": "1",
"snmpAddress": "udp:192.168.10.125/161",
"threshold": "27",
"thresholdInfo": "温度指标告警 < 30℃,当前值= 28.3℃"
},
"startsAt": "2021-08-17T08:06:38.885Z",
"status": "firing"
}
]
}