接口偶尔出现卡顿排查解决

接口偶尔出现卡顿排查解决

接口路由节点: nginx > zuul > k8s svc > pod

排查问题1:

--通过域名访问(偶尔几秒偶尔0.2秒,(偶尔几秒偶尔0.2秒,不正常))

for((i=1;i<=100;i++));  
do 
curl "https://m8.XXXXX.com.cn/XXXXX/pe/coupon/queryGiftCouponByUser?ver=0.43384155157615356  -o /dev/null -s -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"
done


--直接访问k8s zuul  svc的应用url (偶尔几秒偶尔0.2秒,不正常)
for((i=1;i<=150;i++));  
do 
curl "http://svc-XXXXX-gate:8765/XXXXX/pe/coupon/queryGiftCouponByUser?ver=0.43384155157615356   -o /dev/null -s -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"
done

for((i=1;i<=100;i++));  (偶尔几秒偶尔0.2秒,不正常)
do 
curl "http://localhost:8765/XXXXX/pe/coupon/queryGiftCouponByUser?ver=0.43384155157615356   -o /dev/null -s -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"
done  

sleep 0.5
--直接访问k8s svc的应用url (一直0.2秒,正常)
for((i=1;i<=50;i++));  
do   
curl "http://svc-XXXXX-pe:9003/coupon/queryGiftCouponByUser?ver=0.43384155157615356   -o /dev/null -s -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"
done 

--直接访问docker内部的应用url,(一直是0.2秒,正常)
for((i=1;i<=50;i++));  
do 
curl "http://localhost:9003/coupon/queryGiftCouponByUser?ver=0.43384155157615356   -o /dev/null -s -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"
done 

排查问题2:给应用加资源cpu内存还是存在问题


排查问题3:查看应用jvm gc日志也是正常的

-----------------log------------------------------------------
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.366 | http-nio-8765-exec-9 | DEBUG | o.s.web.servlet.DispatcherServlet.processDispatchResult(1044) | Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.366 | http-nio-8765-exec-9 | DEBUG | o.s.web.servlet.DispatcherServlet.processRequest(1000) | Successfully completed request
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.366 | http-nio-8765-exec-9 | DEBUG | o.s.c.s.instrument.web.TraceFilter.detachOrCloseSpans(216) | Closing the span [Trace: 635fa133781b8221, Span: 635fa133781b8221, Parent: null, exportable:false] since the response was successful
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.366 | http-nio-8765-exec-9 | DEBUG | o.s.c.s.zipkin.ZipkinSpanListener.report(216) | The span [Trace: 635fa133781b8221, Span: 635fa133781b8221, Parent: null, exportable:false] will not be sent to Zipkin due to sampling
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.591 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.instrument.web.TraceFilter.doFilter(130) | Received a request to uri [/XXXXX/pe/coupon/queryGiftCouponByUser] that should not be sampled [false]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.591 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.instrument.web.TraceFilter.createSpan(316) | No parent span present - creating a new span
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.591 | http-nio-8765-exec-5 | DEBUG | o.s.web.servlet.DispatcherServlet.doService(865) | DispatcherServlet with name 'dispatcherServlet' processing GET request for [/XXXXX/pe/coupon/queryGiftCouponByUser]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.591 | http-nio-8765-exec-5 | DEBUG | o.s.web.servlet.DispatcherServlet.doDispatch(951) | Last-Modified value for [/XXXXX/pe/coupon/queryGiftCouponByUser] is: -1
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.592 | http-nio-8765-exec-5 | INFO  | c.d.cloud.gate.filter.PreZuulFilter.run(60) | /XXXXX/pe/coupon/queryGiftCouponByUser进入服务器:
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.592 | http-nio-8765-exec-5 | INFO  | c.d.cloud.gate.filter.PreZuulFilter.run(61) | GET >>> http://svc-XXXXX-gate:8765/XXXXX/pe/coupon/queryGiftCouponByUser
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.592 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.i.zuul.TracePreZuulFilter.runFilter(74) | Current span is [Trace: 02cdd3adda96d990, Span: 02cdd3adda96d990, Parent: null, exportable:true]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.592 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.i.zuul.TracePreZuulFilter.runFilter(82) | New Zuul Span is [Trace: 02cdd3adda96d990, Span: 90297cf6a00a9f48, Parent: 02cdd3adda96d990, exportable:true]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.592 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.i.zuul.TracePreZuulFilter.runFilter(86) | Result of Zuul filter is [SUCCESS]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.613 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.i.zuul.TracePostZuulFilter.run(58) | Closing current client span [Trace: 02cdd3adda96d990, Span: 90297cf6a00a9f48, Parent: 02cdd3adda96d990, exportable:true]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:33.653 | http-nio-8765-exec-5 | INFO  | c.d.cloud.gate.filter.PostZuulFilter.run(36) | /XXXXX/pe/coupon/queryGiftCouponByUser 返回响应, 耗时 5061


根据日志发现:
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:28.613 | http-nio-8765-exec-5 | DEBUG | o.s.c.s.i.zuul.TracePostZuulFilter.run(58) | Closing current client span [Trace: 02cdd3adda96d990, Span: 90297cf6a00a9f48, Parent: 02cdd3adda96d990, exportable:true]
XXXXX-gate | deploy-XXXXX-gate-59bf5f5f8c-ssj9p | 2019-04-12 13:40:33.653 | http-nio-8765-exec-5 | INFO  | c.d.cloud.gate.filter.PostZuulFilter.run(36) | /XXXXX/pe/coupon/queryGiftCouponByUser 返回响应, 耗时 5061

结论:根据测试发现是zuul问题

o.s.c.s.i.zuul.TracePostZuulFilter.run(58) 到  c.d.cloud.gate.filter.PostZuulFilter.run(36) 花5秒
 
=============解决方案=========================
spring:
    sleuth:
       enabled: false
       
=================解决后的log================
XXXXX-gate | deploy-XXXXX-gate-6d79bb6655-d6ph7 | 2019-04-12 16:03:53.377 | http-nio-8765-exec-7 | INFO  | c.d.cloud.gate.filter.PreZuulFilter.run(60) | /XXXXX/pe/coupon/queryGiftCouponByUser进入服务器:
XXXXX-gate | deploy-XXXXX-gate-6d79bb6655-d6ph7 | 2019-04-12 16:03:53.377 | http-nio-8765-exec-7 | INFO  | c.d.cloud.gate.filter.PreZuulFilter.run(61) | GET >>> http://svc-XXXXX-gate:8765/XXXXX/pe/coupon/queryGiftCouponByUser
XXXXX-gate | deploy-XXXXX-gate-6d79bb6655-d6ph7 | 2019-04-12 16:03:53.410 | http-nio-8765-exec-7 | INFO  | c.d.cloud.gate.filter.PostZuulFilter.run(36) | /XXXXX/pe/coupon/queryGiftCouponByUser 返回响应, 耗时 33
XXXXX-gate | deploy-XXXXX-gate-6d79bb6655-d6ph7 | 2019-04-12 16:03:53.412 | http-nio-8765-exec-7 | DEBUG | o.s.web.servlet.DispatcherServlet.processDispatchResult(1044) | Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
XXXXX-gate | deploy-XXXXX-gate-6d79bb6655-d6ph7 | 2019-04-12 16:03:53.412 | http-nio-8765-exec-7 | DEBUG | o.s.web.servlet.DispatcherServlet.processRequest(1000) | Successfully completed request
       
       
问题出现在:TracePostZuulFilter
spring-cloud-sleuth:1.1.0.RELEASE
spring-cloud-netflix:1.2.3.RELEASE


       

转载于:https://my.oschina.net/xiaominmin/blog/3036067

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
当Linux服务器出现顿的情况,我们可以采取以下步骤进行排查: 1.查看系统负载:使用命令"top"或者"uptime"可以查看服务器系统负载情况,观察CPU使用率、内存使用率和I/O等指标,找出是否有任何异常。 2.检查CPU占用率:使用命令"top"或者"htop"查看是否有某个进程或者服务占用了过高的CPU资源,这可能是导致服务器顿的原因。 3.检查内存使用情况:使用命令"free"查看服务器的内存使用情况,确认是否有内存泄漏或者内存不足的情况。 4.检查磁盘使用情况:使用命令"df"查看服务器磁盘使用情况,确认是否有某个分区占满了导致顿的情况。 5.检查进程和服务状态:使用命令"ps"查看当前运行的进程和服务的状态,确认是否有异常进程或者服务运行导致顿。 6.查看日志文件:检查系统日志文件,如/var/log/syslog或者/var/log/dmesg,查找是否有任何错误、警告或者异常信息。 7.网络连接情况:使用命令"netstat"或者"ss"查看服务器的网络连接状态,确认是否有过多的网络连接或者异常连接导致顿。 8.检查硬件健康状况:如果以上排查没有发现问题,可以进行硬件检查,如查看硬盘、内存、CPU等硬件是否正常工作。 以上是一些常见的排查步骤,希望可以帮助您解决Linux服务器顿的问题。如果问题依然存在,建议联系专业的系统管理员或者技术支持来协助解决
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值