consul envoy upstream connect error or disconnect

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

consul和envoy一起构建成的服务网格,上述的问题估计会经常发现,也头疼了我很久,还是总结列一下。

首先,应该从哪里开始检查?

1、看内容,就是tls的校验问题,根本不关你上游的服务什么事情。

2、要知道envoy之间,tls的控制来源是啥?consul!!!!!consul通过xds来动态配置envoy。所以,envoy出问题,很大程度是来源consul。不要单单看envoy的日志,其实感觉看envoy的日志,对于这种问题,感觉没啥卵用,直接去查一下,相关的envoy节点所关联的consul节点日志。

2023-06-26T11:25:35.211+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=10.70.51.58:47024 error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:25:35.211+0800 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=0
2023-06-26T11:25:35.211+0800 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=ingress-service-eai service_id=ingress-service-eai id=leaf error="error filling agent cache: rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:26:38.201+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=10.70.51.58:47004 error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:26:38.201+0800 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=0
2023-06-26T11:26:38.201+0800 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=ingress-service-eai service_id=ingress-service-eai id=leaf error="error filling agent cache: rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"

既然引期故障的日志找到了,就开始处理问题。

其实这个问题是因为consul connect 我采用的是vault,刚好昨天vault服务所在的服务器重启了,vault又没有配置自启,为啥又到了今天才报异常?因为刚好要更换leaf cert。consul感知到vault服务的异常,就所有的服务都gg了。

vault 这个蛋疼的服务,非必要,不要搞,老老实实用consul自带的cert。consul和envoy在国内的文档除了非常简单的,稍微深入一点的都没有,vault比它们更偏门,Google都不好使。

立刻把vault删了,把ca的配置改成用consul的,重启consul,完事

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值