AmazonClientException: {“message“:“The security token included in the request is expired“}

最近部署在AWS Elasticbeanstalk EC2 Instance的服务调用某个第三方服务提供的接口（该服务基于AWS API Gateway和AWS Lambda实现）时出现了token过期的问题。

com.amazonaws.AmazonClientException: {"message":"The security token included in the request is expired"} 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1632) 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

这个问题一开始我还是很不解的，在使用其他服务的时候并没有出现这个问题，只有这个服务提供的接口会有这个问题，于是自己开始调研。之前看过aws的很多开发文档，记得里面有提到说token的更新在aws sdk里面都会自动进行，不需要开发者主动进行更新。于是便去官方文档查询，查到以下内容：
IAM roles for Amazon EC2 - Amazon Elastic Compute Cloud

Retrieve security credentials from instance metadata An application on the instance retrieves the security credentials provided by the role from the instance metadata item iam/security-credentials/role-name. The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.

意思就是EC2的token信息保存在metadata中，并且会自动进行更新。可以通过登录到ec2上执行以下命令进行查询：

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/${ec2_role_name}

${ec2_role_name}需要替换为你给EC2分配的role。

远程登录Elasticbeanstalk EC2 instance

Step1：创建 key pair

点击这里创建key pair：

1. 输入key pair名称
2. Key pair type选择RSA
3. Private key file format选择pem，因为我们是要ssh到ec2用的

创建成功后自动下载key文件，这个文件一定要保存好，ssh的时候要用

Step2：为EC2 instance指定key pair

1. 点击这里打开beanstalk的环境列表，
2. 选择instance所属的environment，
3. 点击左侧菜单栏的Configuration选项，
4. 在Configuration列表中选择Security进行编辑，
5. 在EC2 key pair下拉菜单中选择Step1中创建的key pair
6. 选择完成后点击Apply，之后beanstalk会重启

Step3:编辑EC2 instance的security group

• 找到ec2 instance 的security group，添加一条inbound rule用于允许本机对ec2 isntance的远程访问。

  SSH

  TCP

  22

  ${your.comupter.ip.address}/32

Step4:远程登录ec2 instance

查看一下ec2instance的Public IPv4 DNS，用Public IPv4 DNS进行远程登录,user_name根据ec2 instance的ami的类型会有不同，具体可以点击这里查看：

• For Amazon Linux 2 or the Amazon Linux AMI, the user name is ec2-user.

• For a CentOS AMI, the user name is centos or ec2-user.

• For a Debian AMI, the user name is admin.

• For a Fedora AMI, the user name is fedora or ec2-user.

• For a RHEL AMI, the user name is ec2-user or root.

• For a SUSE AMI, the user name is ec2-user or root.

• For an Ubuntu AMI, the user name is ubuntu.

• For an Oracle AMI, the user name is ec2-user.

• For a Bitnami AMI, the user name is bitnami.

• Otherwise, check with the AMI provider.

ssh -i "${key_pari_name}.pem" ${user_name}@*****.ap-southeast-1.compute.amazonaws.com

Setp5：查看token

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/${ec2_role_name}

token结构长这个样子：

{
  "Code" : "Success",
  "LastUpdated" : "2021-11-12T04:17:27Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "*****",
  "SecretAccessKey" : "******************************",
  "Token" : "************************************************************************",
  "Expiration" : "2021-11-12T10:51:36Z"
}

一段时间之后再次查看，token确实自动更新了。

那就说明EC2这边没什么问题，于是开始查看第三方服务的接口。

AWSCredentials

第三方服务的接口是这样写的：

public InfoClientImpl(String landscape, String accountId, String host, Optional<AWSCredentials> awsCredentials) {
        this.landscape = Landscape.of(landscape);
        this.accountId = accountId;
        this.host = host;
        this.gson = new Gson();
        this.awsCredentials = awsCredentials.orElseGet(() -> {
            DefaultAWSCredentialsProviderChain credProvider = DefaultAWSCredentialsProviderChain.getInstance();
            return credProvider.getCredentials();
        });
    }

private String request(Map<String, List<String>> parameters, String path) {
        AWS4Signer signer = new AWS4Signer();

        DefaultRequest<?> req = new DefaultRequest<>("execute-api");
        **************
        signer.sign(req, awsCredentials);

        **********************
    }

这个写法的意思就是只在创建client的实例的时候获取了一次AWSCredentials，后续每个request直接使用这个Credential，并没有进行更新，正确写法如下，应该在request的时候从DefaultAWSCredentialsProviderChain获取Credentials。

public InfoClientImpl(String landscape, String accountId, String host) {
        this.landscape = Landscape.of(landscape);
        this.accountId = accountId;
        this.host = host;
        this.gson = new Gson();
    }

private String request(Map<String, List<String>> parameters, String path) {
        AWS4Signer signer = new AWS4Signer();

        DefaultRequest<?> req = new DefaultRequest<>("execute-api");
        **************
        signer.sign(req, DefaultAWSCredentialsProviderChain.getInstance().getCredentials());

        **********************
    }

DefaultAWSCredentialsProviderChain包含的几个provider都由自动更新Credentials的逻辑，所以每次拿到的都是在有效期内的。对于EC2 instance使用到的provider是InstanceProfileCredentialsProvider，完整代码如下，可以看到其中有refresh的逻辑。

/*
 * Copyright 2012-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License").
 * You may not use this file except in compliance with the License.
 * A copy of the License is located at
 *
 *  http://aws.amazon.com/apache2.0
 *
 * or in the "license" file accompanying this file. This file is distributed
 * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
 * express or implied. See the License for the specific language governing
 * permissions and limitations under the License.
 */
package com.amazonaws.auth;

import com.amazonaws.AmazonClientException;
import com.amazonaws.SDKGlobalConfiguration;
import com.amazonaws.SdkClientException;
import com.amazonaws.internal.CredentialsEndpointProvider;
import com.amazonaws.internal.EC2CredentialsUtils;
import com.amazonaws.util.EC2MetadataUtils;
import java.io.Closeable;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/**
 * Credentials provider implementation that loads credentials from the Amazon EC2 Instance Metadata Service.
 *
 * <p>When using {@link InstanceProfileCredentialsProvider} with asynchronous refreshing it is
 * <b>strongly</b> recommended to explicitly call {@link #close()} to release the async thread.</p>
 */
public class InstanceProfileCredentialsProvider implements AWSCredentialsProvider, Closeable {

    private static final Log LOG = LogFactory.getLog(InstanceProfileCredentialsProvider.class);

    /**
     * The wait time, after which the background thread initiates a refresh to
     * load latest credentials if needed.
     */
    private static final int ASYNC_REFRESH_INTERVAL_TIME_MINUTES = 1;

    /**
     * The default InstanceProfileCredentialsProvider that can be shared by
     * multiple CredentialsProvider instance threads to shrink the amount of
     * requests to EC2 metadata service.
     */
    private static final InstanceProfileCredentialsProvider INSTANCE = new InstanceProfileCredentialsProvider();

    private final EC2CredentialsFetcher credentialsFetcher;

    /**
     * The executor service used for refreshing the credentials in the
     * background.
     */
    private volatile ScheduledExecutorService executor;

    private volatile boolean shouldRefresh = false;

    /**
     * @deprecated for the singleton method {@link #getInstance()}.
     */
    @Deprecated
    public InstanceProfileCredentialsProvider() {
        this(false);
    }

    /**
     * Spins up a new thread to refresh the credentials asynchronously if
     * refreshCredentialsAsync is set to true, otherwise the credentials will be
     * refreshed from the instance metadata service synchronously,
     *
     * <p>It is <b>strongly</b> recommended to reuse instances of this credentials provider, especially
     * when async refreshing is used since a background thread is created.</p>
     *
     * @param refreshCredentialsAsync
     *            true if credentials needs to be refreshed asynchronously else
     *            false.
     */
    public InstanceProfileCredentialsProvider(boolean refreshCredentialsAsync) {
        this(refreshCredentialsAsync, true);
    }

    /**
     * Spins up a new thread to refresh the credentials asynchronously.
     *
     * <p>It is <b>strongly</b> recommended to reuse instances of this credentials provider, especially
     * when async refreshing is used since a background thread is created.</p>
     *
     * @param eagerlyRefreshCredentialsAsync
     *            when set to false will not attempt to refresh credentials asynchronously
     *            until after a call has been made to {@link #getCredentials()} - ensures that
     *            {@link EC2CredentialsFetcher#getCredentials()} is only hit when this CredentialProvider is actually required
     */
    public static InstanceProfileCredentialsProvider createAsyncRefreshingProvider(final boolean eagerlyRefreshCredentialsAsync) {
        return new InstanceProfileCredentialsProvider(true, eagerlyRefreshCredentialsAsync);
    }

    private InstanceProfileCredentialsProvider(boolean refreshCredentialsAsync, final boolean eagerlyRefreshCredentialsAsync) {

        credentialsFetcher = new EC2CredentialsFetcher(new InstanceMetadataCredentialsEndpointProvider());

        if (!SDKGlobalConfiguration.isEc2MetadataDisabled()) {
            if (refreshCredentialsAsync) {
                executor = Executors.newScheduledThreadPool(1);
                executor.scheduleWithFixedDelay(new Runnable() {
                    @Override
                    public void run() {
                        try {
                            if (shouldRefresh) credentialsFetcher.getCredentials();
                        } catch (AmazonClientException ace) {
                            handleError(ace);
                        } catch (RuntimeException re) {
                            handleError(re);
                        }
                    }
                }, 0, ASYNC_REFRESH_INTERVAL_TIME_MINUTES, TimeUnit.MINUTES);
            }
        }
    }

    /**
     * Returns a singleton {@link InstanceProfileCredentialsProvider} that does not refresh credentials asynchronously.
     *
     * <p>
     * See {@link #InstanceProfileCredentialsProvider(boolean)} or {@link #createAsyncRefreshingProvider(boolean)} for
     * asynchronous credentials refreshing.
     * </p>
     */
    public static InstanceProfileCredentialsProvider getInstance() {
        return INSTANCE;
    }

    private void handleError(Throwable t) {
        refresh();
        LOG.error(t.getMessage(), t);
    }

    @Override
    protected void finalize() throws Throwable {
        if (executor != null) {
            executor.shutdownNow();
        }
    }


    /**
     * {@inheritDoc}
     *
     * @throws AmazonClientException if {@link SDKGlobalConfiguration#isEc2MetadataDisabled()} is true
     */
    @Override
    public AWSCredentials getCredentials() {
        if (SDKGlobalConfiguration.isEc2MetadataDisabled()) {
            throw new AmazonClientException("AWS_EC2_METADATA_DISABLED is set to true, not loading credentials from EC2 Instance "
                                         + "Metadata service");
        }
        AWSCredentials creds = credentialsFetcher.getCredentials();
        shouldRefresh = true;
        return creds;
    }

    @Override
    public void refresh() {
        if (credentialsFetcher != null) {
            credentialsFetcher.refresh();
        }
    }

    @Override
    public void close() throws IOException {
        if (executor != null) {
            executor.shutdownNow();
            executor = null;
        }
    }

    private static class InstanceMetadataCredentialsEndpointProvider extends CredentialsEndpointProvider {
        @Override
        public URI getCredentialsEndpoint() throws URISyntaxException, IOException {
            String host = EC2MetadataUtils.getHostAddressForEC2MetadataService();

            String securityCredentialsList = EC2CredentialsUtils.getInstance().readResource(new URI(host + EC2MetadataUtils.SECURITY_CREDENTIALS_RESOURCE));
            String[] securityCredentials = securityCredentialsList.trim().split("\n");
            if (securityCredentials.length == 0) {
                throw new SdkClientException("Unable to load credentials path");
            }

            return new URI(host + EC2MetadataUtils.SECURITY_CREDENTIALS_RESOURCE + securityCredentials[0]);
        }
    }
}

总结

只要出现The security token included in the request is expired的问题，那一定是token有问题，具体问题在哪里，需要根据实际情况进行排查。