Google Gmail 454 4.7.0 Too many login attempts, please try again later

最近服务上线，监控发现服务一直在重启 ，我勒个去，这个咋回事，之前在测试环境一直没有问题，咋上线就不行了，重启时一直报如下错误:

2021-03-25 09:12:08,696--[TID: ef8abccab685473abad00e4d03415ed6.85.16166635285550003]-- [http-nio-8000-exec-9] WARN  
org.springframework.boot.actuate.mail.MailHealthIndicator  - Mail health check failed
javax.mail.AuthenticationFailedException: 454 4.7.0 Too many login attempts, please try again later. z23sm1242156pgn.88 - gsmtp
        at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:947)
        at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:858)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:762)
        at javax.mail.Service.connect(Service.java:342)
        at org.springframework.mail.javamail.JavaMailSenderImpl.connectTransport(JavaMailSenderImpl.java:518)
        at org.springframework.mail.javamail.JavaMailSenderImpl.testConnection(JavaMailSenderImpl.java:398)
        at org.springframework.boot.actuate.mail.MailHealthIndicator.doHealthCheck(MailHealthIndicator.java:42)
        at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
        at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:95)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:43)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:108)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:119)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:105)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:83)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:70)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:81)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:70)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
        at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77)
        at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
        at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:305)
        at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:388)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke$original$T1LbfnTd(StandardHostValve.java:139)
        at org.apache.catalina.core.StandardHostValve.invoke$original$T1LbfnTd$accessor$TuAcUbqL(StandardHostValve.java)
        at org.apache.catalina.core.StandardHostValve$auxiliary$FxHoa1f0.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

首次看到这个错误，一脸的懵逼，NM什么情况? 怎么是 Too many login attempts？既然看不懂，先在本地看能不能复现吧?
一把辛酸泪啊? 本地也出现了这种情况了，可是一点规律都没有，而且还是偶发性的，这真是蛋疼的很。问题还是要解决的，跟进源码看一下吧。

先看重启的原因吧，看日志里面有
org.springframework.boot.actuate.mail.MailHealthIndicator - Mail health check failed
明显看到这是健康检查出了问题，在健康检查中有个mail的配置：

management:
  health:
    mail:
      enabled: true

健康检查的代码：

public class MailHealthIndicator extends AbstractHealthIndicator {

	private final JavaMailSenderImpl mailSender;
	
	public MailHealthIndicator(JavaMailSenderImpl mailSender) {
		super("Mail health check failed");
		this.mailSender = mailSender;
	}
	
	@Override
	protected void doHealthCheck(Builder builder) throws Exception {
		builder.withDetail("location", this.mailSender.getHost() + ":" + this.mailSender.getPort());
		this.mailSender.testConnection();
		builder.up();
	}
}

// testConnection() 方法中的connectTransport()
protected Transport connectTransport() throws MessagingException {
	String username = getUsername();
	String password = getPassword();
	if ("".equals(username)) {  // probably from a placeholder
		username = null;
		if ("".equals(password)) {  // in conjunction with "" username, this means no password to use
			password = null;
		}
	}

	Transport transport = getTransport(getSession());
	transport.connect(getHost(), getPort(), username, password);
	return transport;
}

由于k8s是根据健康检查的结果来进行服务重启，由上面日志和代码可明显的确定是connectTransport()出了问题。如果仅仅是解决重启的问题，将健康检查关掉即可，但是一发邮件就会失败。根本问题没有得到解决。
现在还是要看connectTransport()为什么出这个鬼问题。
跟了一遍，没有发现问题，😭😭，我太菜了
没得办法，去搜吧，刚好看到https://stackoverflow.com/有这个贴子，

The issue is because of Google Apps Security. You Can't use same E-mail ID with more than 2 Apps. 
as Gmail needs to sync it, and it is Obvious to receive such error.

我们在测试和生产共用了同一个E-mail ID，生产我们更换一个，更新配置重启服务，我去，服务起不来，欲哭无泪啊，然后发现下面还有：

> Since we are using gmail to send email, we need to enable "less secure access" on the gmail account.
>  Log out from all the Google accounts from the browser. Login in to the gmail account configured in application.properties. 
>  Then click on this link and turn on access for less secure apps. In some cases, you may need to access this link as well!
>   If you don't follow these steps, you will get the following error,

赶紧去打开这个安全设置，重启服务，服务起来了，观察一个多小时，也没有再发生服务重启的现象了，问题看似完满解决，可是我的疑问还是很大。
在我入职之前，这个服务在测试环境和生产环境已经运行，当时也是公用了同一个E-mail Id,但是服务运行是OK的，也没有发生重启的现象？为什么这次上线出现了这个问题？我们这次上线并没有改动邮件相关的代码和配置呀。
服务正常运行了，也松了一口气，但是问题还得继续定位，并没有找到具体的原因，不放心呀，谁知道是不是个定时炸弹。
在业务中，国内我们用的是阿里的邮件服务，国外用的是gmail(因为免费)，因为这个事现在考虑接入AWS的ses。
长路漫漫，唯有bug相伴啊😭😭😭😭