CAT具体的设计思想、实现原理在这我就不罗列了,本文主要是记录一下在使用CAT的过程中遇到的一些问题,比如分布式logview,Cache、DB埋点监控等,问题不多,但是比较典型。
我已经实现一个 CAT 监控相关的扩展实现,可供参考:
https://github.com/summerpotato/cat-monitor
(本文涉及的CAT版本为1.3.6)
1、分布式 logview 的日志树串联实现
目前使用过两种,一种是基于 dubbo 应用的 rpc 调用,一种是基于 http 请求的 rest 服务调用。首先说下 message tree 的实现,追踪跨服务的消息时,通过根消息 id 和父级消息 id 及子消息 id 三个属性进行消息串联,组成消息树。关键点在 tree 的三个 id 的获得和传递。
这里有两点,第一是 CAT 消息树生成原理:
我们需要实现 Cat 的 Context 上下文,然后通过 Cat.logRemoteCallClient(context) 生成包含节点数据的上下文对象(方法中通过创建消息树对象来获取各节点的消息 id,填充给上下文),当远程服务端接收到这个 context 时,使用 Cat.logRemoteCallServer(context) 方法,读取各节点消息 id,组建消息树。
第二是消息应如何传递:
dubbo 应用的 rpc 调用方式:调用过程要传递的 rpc 上下文,其中包含调用信息、参数以及状态信息等,可以把消息 id 信息放到 RpcContext 中,然后通过调用 Invocation 对象的 invoke 方法,将消息传递至服务端。最后,通过dubbo的 spi 拓展机制,实现 com.alibaba.dubbo.rpc.Filter,用来获取 rpcContext 的内容。
rest 风格的 http 请求方式:调用时,在服务请求方把消息 id 信息放到 Http-Header 中,在服务提供方,用 filter 拦截,并获得 http-header 中的消息 id,这样通过埋点,串联起消息树。
废话不多说了,上码吧。
1).dubbo 调用方式部分实现(首先要清楚 dubbo 的 spi 相关配置,CAT监控的配置等)
public class DubboCatFilter implements Filter {
private static final ThreadLocal<Cat.Context> CAT_CONTEXT = new ThreadLocal<Cat.Context>();
@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
URL url = invoker.getUrl();
String sideKey = url.getParameter(Constants.SIDE_KEY);
String loggerName = invoker.getInterface().getSimpleName() + "." + invocation.getMethodName();
String type = "PigeonCall";
if (Constants.PROVIDER_SIDE.equals(sideKey)) {
type = "PigeonService";
}
Transaction t = Cat.newTransaction(type, loggerName);
Result result = null;
try {
Cat.Context context = getContext();
if (Constants.CONSUMER_SIDE.equals(sideKey)) {
createConsumerCross(url, t);
Cat.logRemoteCallClient(context);
} else {
createProviderCross(url, t);
Cat.logRemoteCallServer(context);
}
setAttachment(context);
result = invoker.invoke(invocation);
if (result.hasException()) {
//给调用接口出现异常进行打点
Throwable throwable = result.getException();
Event event = null;
if (RpcException.class == throwable.getClass()) {
Throwable caseBy = throwable.getCause();
if (caseBy != null && caseBy.getClass() == TimeoutException.class) {
event = Cat.newEvent("DUBBO_TIMEOUT_ERROR", loggerName);
} else {
event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
}
} else if (RemotingException.class.isAssignableFrom(throwable.getClass())) {
event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
}else{
event = Cat.newEvent("DUBBO_BIZ_ERROR", loggerName);
}
event.setStatus(result.getException());
completeEvent(event);
t.addChild(event);
t.setStatus(result.getException().getClass().getSimpleName());
} else {
t.setStatus(Message.SUCCESS);
}
return result;
} catch (RuntimeException e) {
Event event = null;
if (RpcException.class == e.getClass()) {
Throwable caseBy = e.getCause();
if (caseBy !=null && caseBy.getClass() == TimeoutException.class) {
event = Cat.newEvent("DUBBO_TIMEOUT_ERROR", loggerName);
} else {
event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
}
} else {
event = Cat.newEvent("DUBBO_BIZ_ERROR", loggerName);
}
event.setStatus(e);
completeEvent(event);
t.addChild(event);
t.setStatus(e.getClass().getSimpleName());
if (result == null) {
throw e;
} else {
return result;
}
} finally {
t.complete();
CAT_CONTEXT.remove();
}
}
static class DubboCatContext implements Cat.Context {
private Map<String,String> properties = new HashMap<String, String>();
@Override
public void addProperty(String key, String value) {
properties.put(key,value);
}
@Override
public String getProperty(String key) {
return properties.get(key);
}
}
private void setAttachment(Cat.Context context) {
RpcContext.getContext().setAttachment(Cat.Context.ROOT,context.getProperty(Cat.Context.ROOT));
RpcContext.getContext().setAttachment(Cat.Context.CHILD,context.getProperty(Cat.Context.CHILD));
RpcContext.getContext().setAttachment(Cat.Context.PARENT,context.getProperty(Cat.Context.PARENT));
}
private Cat.Context getContext(){
Cat.Context context = CAT_CONTEXT.get();
if (context==null) {
context = initContext();
CAT_CONTEXT.set(context);
}
return context;
}
private Cat.Context initContext() {
Cat.Context context = new DubboCatContext();
Map<String,String> attachments = RpcContext.getContext().getAttachments();
if (attachments!=null&&attachments.size()>0) {
for (Map.Entry<String,String> entry:attachments.entrySet()) {
if (Cat.Context.CHILD.equals(entry.getKey()) || Cat.Context.ROOT.equals(entry.getKey()) || Cat.Context.PARENT.equals(entry.getKey())) {
context.addProperty(entry.getKey(),entry.getValue());
}
}
}
return context;
}
private void createConsumerCross(URL url, Transaction t) {
Event crossAppEvent = Cat.newEvent("PigeonCall.app", getProviderAppName(url));
Event crossServerEvent = Cat.newEvent("PigeonCall.server", url.getHost());
Event crossPortEvent = Cat.newEvent("PigeonCall.port", url.getPort() + "");
crossAppEvent.setStatus(Event.SUCCESS);
crossServerEvent.setStatus(Event.SUCCESS);
crossPortEvent.setStatus(Event.SUCCESS);
completeEvent(crossAppEvent);
completeEvent(crossPortEvent);
completeEvent(crossServerEvent);
t.addChild(crossAppEvent);
t.addChild(crossPortEvent);
t.addChild(crossServerEvent);
}
private void createProviderCross(URL url, Transaction t) {
String consumerAppName = RpcContext.getContext().getAttachment(Constants.APPLICATION_KEY);
if (StringUtils.isEmpty(consumerAppName)) {
consumerAppName = RpcContext.getContext().getRemoteHost() + ":" + RpcContext.getContext().getRemotePort();
}
Event crossAppEvent = Cat.newEvent("PigeonService.app", consumerAppName);
Event crossServerEvent = Cat.newEvent("PigeonService.client", url.getHost());
crossAppEvent.setStatus(Event.SUCCESS);
crossServerEvent.setStatus(Event.SUCCESS);
completeEvent(crossAppEvent);
completeEvent(crossServerEvent);
t.addChild(crossAppEvent);
t.addChild(crossServerEvent);
}
private void completeEvent(Event event) {
AbstractMessage message = (AbstractMessage) event;
message.setCompleted(true);
}
}
2).http-restful 调用方式部分实现
CatHttpClientProxy.java
public void requestByGet(String url) {
Transaction t = Cat.newTransaction("PigeonCall", "method000");
//创建默认的httpClient实例
CloseableHttpClient httpClient = HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(5000).setConnectionRequestTimeout(1000)
.setSocketTimeout(5000).build();
try {
HttpGet httpGet = new HttpGet(url);
httpGet.setConfig(requestConfig);
//串联埋点
Cat.Context context = new CatHttpContext();
this.createConsumerCross(url, t);
Cat.logRemoteCallClient(context);
httpGet.setHeader(Cat.Context.ROOT, context.getProperty(Cat.Context.ROOT));
httpGet.setHeader(Cat.Context.PARENT, context.getProperty(Cat.Context.PARENT));
httpGet.setHeader(Cat.Context.CHILD, context.getProperty(Cat.Context.CHILD));
System.out.println("执行get请求:...." + httpGet.getURI());
CloseableHttpResponse httpResponse = null;
//发送get请求
httpResponse = httpClient.execute(httpGet);//请求返回的Resp,含http的header和执行结果实体Entity
try {
//response实体
HttpEntity entity = httpResponse.getEntity();//不包含header
if (null != entity) {
System.out.println("响应状态码:"+ httpResponse.getStatusLine());
System.out.println("-------------------------------------------------");
System.out.println("响应内容:" + EntityUtils.toString(entity));
}
} finally {
httpResponse.close();
}
t.setStatus(Transaction.SUCCESS);
} catch (Exception e) {
e.printStackTrace();
t.setStatus(e.getClass().getSimpleName());
} finally {
t.complete();
try {
closeHttpClient(httpClient);
} catch (IOException e) {
e.printStackTrace();
}
}
}
private void createConsumerCross(String url, Transaction t){
Event crossAppEvent = Cat.newEvent("PigeonCall.app", "serverName");
Event crossServerEvent = Cat.newEvent("PigeonCall.server", "serverIp");
Event crossPortEvent = Cat.newEvent("PigeonCall.port", "serverPort");
crossAppEvent.setStatus(Event.SUCCESS);
crossServerEvent.setStatus(Event.SUCCESS);
crossPortEvent.setStatus(Event.SUCCESS);
completeEvent(crossAppEvent);
completeEvent(crossPortEvent);
completeEvent(crossServerEvent);
t.addChild(crossAppEvent);
t.addChild(crossPortEvent);
t.addChild(crossServerEvent);
}
private void completeEvent(Event event){
AbstractMessage message = (AbstractMessage) event;
message.setCompleted(true);
}
private void closeHttpClient(CloseableHttpClient client) throws IOException{
if (client != null) {
client.close();
}
}
2、CAT对 redis 缓存进行详细监控
CAT源码内部对于缓存的识别存在一个 convention 约定,是基于匹配 “Cache.” 字符串的,并且通过判断字符串 “Cache.memcached” 来支持 memcached 监控,可是没有对 redis 做显示支持,需要修改源码,增加判断字符串 “Cache.redis”;
1).修改类:cat-home - com.dianping.cat.report.page.statistics.task.utilization.TransactionReportVisitor.java
增加对 redis 的判断支持:
private static final String REDIS = "Cache.redis";
public TransactionReportVisitor() { m_types.add("URL"); m_types.add("Service"); m_types.add("PigeonService"); m_types.add("Call"); m_types.add("PigeonCall"); m_types.add("SQL"); m_types.add(MEMCACHED); m_types.add(REDIS); }
@Override public void visitType(TransactionType type) { String typeName = type.getId(); Domain domain = m_report.findOrCreateDomain(m_domain); if ("Service".equals(typeName)) { typeName = "PigeonService"; } else if ("Call".equals(typeName)) { typeName = "PigeonCall"; } else if (typeName.startsWith(MEMCACHED)) { typeName = MEMCACHED; } else if (typeName.startsWith(REDIS)){ typeName = REDIS; } ...... }
2).修改类:cat-core - com.dianping.cat.config.server.ServerConfigManager.java
增加对 redis 的判断支持:
public boolean isCacheTransaction(String type) { return StringUtils.isNotEmpty(type) && (type.startsWith("Cache.memcached") || type.startsWith("Cache.redis")); }
3).修改类:cat-consumer - com.dianping.cat.consumer.storage.StorageAnalyzer.java
增加对redis的判断支持:
private void processCacheTransaction(MessageTree tree, Transaction t) { String cachePrefix = "Cache."; String ip = "Default"; String domain = tree.getDomain(); String cacheType = t.getType().substring(cachePrefix.length()); String name = t.getName(); String method = name.substring(name.lastIndexOf(":") + 1); List<Message> messages = t.getChildren(); for (Message message : messages) { if (message instanceof Event) { String type = message.getType(); if (type.equals("Cache.memcached.server") || type.equals("Cache.redis.server")) { ip = message.getName(); int index = ip.indexOf(":"); if (index > -1) { ip = ip.substring(0, index); } } } } ...... }
3、CAT 对 DB 数据库进行详细监控
如果你的 orm 框架使用的 mybatis,可以考虑通过实现拦截器 Interceptor 来对DB进行底层监控,CAT对数据库的埋点也存在 convention,这里代码中存在 hard code。具体埋点如下:
MappedStatement mappedStatement = (MappedStatement) invocation.getArgs()[0];
//得到类名,方法
String[] strArr = mappedStatement.getId().split("\\.");
String methodName = strArr[strArr.length - 2] + "." + strArr[strArr.length - 1];
Transaction t = Cat.newTransaction("SQL", "methodName");
//获取SQL类型
SqlCommandType sqlCommandType = mappedStatement.getSqlCommandType();
Cat.logEvent("SQL.Method", sqlCommandType.name().toLowerCase());
String JDBC_CONNECTION = "jdbc:mysql://unknown:3306/%s?useUnicode=true";
Cat.logEvent("SQL.Database", String.format(JDBC_CONNECTION, serverIp, dbName));
spring配置如下:
<bean id="sessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="configLocation" value="classpath:mybatis.xml"/>
<!-- 插件配置 -->
<property name="plugins">
<array>
<bean class="com.kubbo.java.common.cat.CatMybatisPlugin"></bean>
</array>
</property>
</bean>
以上仅罗列了每个问题的一种实现方案,只是给正在研究CAT的同学一个参考思路,个人研究CAT也是刚开始,所说之处不免存在一些纰漏,欢迎指正和交流。
欢迎转载,但请务必注明来源!