kubernetes实践之十八:集群各模块之间的通信

一: 通信结构图

二:kubernetes客户端认证方式
Kubernetes提供管理三种级别的客户端身份认证方式:
1.
最严格的HTTPS证书认证:基于CA根证书签名的双向数字证书认证方式;
2.
HTTP Token认证:通过一个Token来识别合法用户;
3.
HTTP Base认证:通过用户名+密码的方式认证;

上图中展示的是使用HTTPS证书认证的方式。
三:各模块之间的通信方式
在开启了 TLS 的集群中,每当与集群交互的时候少不了的是身份认证,使用 kubeconfig(即证书) 和 token 两种认证方式是最简单也最通用的认证方式。

1.kubectl跟API Server之间采用的是kubeconfig进行TLS的认证
2.kubelet跟API Server之间采用Token(bootstrap token)和Kubeconfig相结合的方式进行TLS的认证
3.Pod跟API Server之间采用的是Token(Service Account Token)的方式进行TLS的认证。
4.集群外用户通过kubectl proxy 方式提供http访问(仅作测试用
5.scheduler,controller manager与API Server之间通过Http访问(scheduler,controller manager,API Server要求部署在同一台服务器上
6.ETCD之间通过TLS进行通信
7.API Server与ETCD之间通过TLS进行通信
8.API SERVER与kube-proxy之间通过kubeconfig进行TL的认证

四:kubectl通信设置

kubectl默认会从$HOME/.kube目录下查找文件名为 config 的文件,也可以通过设置环境变量 KUBECONFIG 或者通过设置 --kubeconfig 去指定其它 kubeconfig 文件。

点击(此处)折叠或打开

  1. export KUBE_APISERVER="https://masterIp:6443"
  2. # 设置集群参数
  3. kubectl config set-cluster kubernetes \
  4. --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  5. --embed-certs=true \
  6. --server=${KUBE_APISERVER}
  7. # 设置客户端认证参数
  8. kubectl config set-credentials admin \
  9. --client-certificate=/etc/kubernetes/ssl/admin.pem \
  10. --embed-certs=true \
  11. --client-key=/etc/kubernetes/ssl/admin-key.pem
  12. # 设置上下文参数
  13. kubectl config set-context kubernetes \
  14. --cluster=kubernetes \
  15. --user=admin
  16. # 设置默认上下文
  17. kubectl config use-context kubernetes
1. 生成的 kubeconfig 被保存到  ~/.kube/config  文件;配置文件描述了集群、用户和上下文
2.设置集群参数: set-cluster设置了需要访问的集群,如上面的kubernetes,这只是个名称,实际为--server指向的apiserver;   --certificate-authority设置了该集群的公钥; --embed-certs为true表示将--certificate-authority证书写入到kubeconfig中;--server则表示该集群的kube-apiserver地址
3.设置客户端认证参数: set - credentials admin 设置用户admin;--client-certificate设置客户证书;--client-key设置客户私钥;--embed-certs为true表示将证书写入kubeconfig中;admin.pem 证书 OU 字段值为 system:masters , kube-apiserver预定义的 RoleBinding cluster-admin 将 Group system:masters与 Role cluster-admin 绑定,该 Role 授予了调? kube-apiserver相关 API 的权限;
4.设置上下文参数:上下文名称为kubenetes,集群为kubenetes,用户为admin,表示使用admin的用户凭证来访问kubenetes集群的default命名空间,也可以增加--namspace来指定访问的命名空间。集群参数和用户参数可以同时设置多对;
5.kubectl config use-context kubernetes来使用名为kubenetes的环境项来作为配置。如果配置了多个环境项,可以通过切换不同的环境项名字来访问到不同的集群环境。

最终生成的config文件内容如下:

点击(此处)折叠或打开

  1. apiVersion: v1
  2. clusters:
  3. - cluster:
  4.     certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSVprZlNwNSs5REY3OUZEYXFhQ1haYXRsOG9Rd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURNeE1qQTFNall3TUZvWERUSXpNRE14TVRBMU1qWXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcERaU3NjQWF2N3Roa0JDUTZYN2cKWGZCZ3JOZWM2amc1YjdzVXEzSENyc1hYL3dlOW5POVNxVmxGdS8vY3VMTXNiWEhTdGhXYzJtTXBhdzlmQWVUdwpqK2gyOVZjTkJEamhWbzNhbUFZNk9pR1hOWWZQd0hsaHN4bVFXT0Zrc1NybnpOaGk4djc5K0R5bk1Bc3BSVHlxCk5KODdSUi93d0x6TDVUcFlka1U2U3dIK1ZsR01NRjNEMFdRSjV1ZHlNbFBiRnQyWVVnZE1VRXVCdDRVYWp3d2cKZE9jSVdEcFV0Q0pGNXUvU3RTaHA3RDdRb05rUDdKa2dodm5KMEx6enZxZnlkVFE2MW1HSXIvUHpYS0d3VkJoTgpEQ0hCMTJ1SDBEUkxwK3V2UE5pdmRYcS9wSzE0dWlxYjFoTGdRbzd6OS92d0xBMVg0c0FrU1o2UjUzQzBEZ3kvCnN3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVbG00R3RSbFgzZmYrWkFjLzI1cmdURHNhTEo0d0h3WURWUjBqQkJnd0ZvQVVsbTRHdFJsWAozZmYrWkFjLzI1cmdURHNhTEo0d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFISWVvY1lXM21ZVVdMbTVkR3hvCnN0a01DZlc3aDhDMEduZ2ZIOXFhbFZVTzk0b0RDd2hEMWlqOWtZVW9nZzF3a1BFbGZjSWEybERDOFZBckJtZ3UKWXhFOEJwSkpxMUtIaXAyM0xXSXhWTFB4VWs5ckFUNEtwbE9idmo1RG5adTFDTndlVXJpWU1QUEtRc01LaXROaApVQ3pBV0FVTUpIa2JwVHdsZGNBYmtGMHFyUThGRzZsa1EwakRpTUpKc0ZrelEzbU5YY1pzYXpnamNRWjJHczBxCkhySzlrQ2FHTWtHbzEvQUtDYms5VzZEZDBpL05QeWtvT2d3U1ExMzFSaHQrYmlGZVdVOWQwYjZST3JyYW9GeFoKd1hnZGRYZm11UHQ3eHhyeVFZTUN0NWxRYkQ4aS8wNzJZMGVNYVpQU09jc2xtTk1RQWpsN3Z6T0N0UGcyRzNKNAoxemM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  5.     server: https://-----------------
  6.   name: kubernetes
  7. contexts:
  8. - context:
  9.     cluster: kubernetes
  10.     user: admin
  11.   name: kubernetes
  12. current-context: kubernetes
  13. kind: Config
  14. preferences: {}
  15. users:
  16. - name: admin
  17.   user:
  18.     as-user-extra: {}
  19.     client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0lVVnR3RHBzZlRaN1RobUY1SUdYa2ZwWUhrZklzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURNeE1qQTJNamN3TUZvWERUSTRNRE13T1RBMk1qY3dNRm93YXpFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RnpBVgpCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVE4d0RRWURWUVFMRXdaVGVYTjBaVzB4RGpBTUJnTlZCQU1UCkJXRmtiV2x1TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0dStsZzRmYzlRT1cKQWxRaWdGTmRHWUZhQjhNaDBuMXRrOFBNUzU2VWRZQmxJMnVNOWNHVTZUdmdGcGJMOFMvMDI2RklmcXZRRlRpQgpqbS9aZEl6R05GbmlOOFk0M3JVWnFTdUtpL3J6czBUU2E3UG5RMnhvOHhiTGNmN24va2NYSEpUSENHbHNhb28xClRmOUdyZGR1bkdWdnhlMzFDT0ovY25LSFZKNVc1aTdoZTlYU21tMWxNdnlkT0FRVmZQT1pJVDNnUS9VV29TL2cKMWI2VEM3cmR5TzhwT2YvYmxOSExFbHRYSk9PcjJOb2ducnJPSk9WYTlTbnQ5S2FBUjNvNnFSM21QR1hnK2pBawpFZGswVzlwMm4rMWpjd094WUd2N1JtQm11Q3QwRnBvaDVBanZPNlhCQTYvSU10WnJkR0tKS1J6NTFZZDRLNHVwCnBQM05jcHhRV3dJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUNCYUF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZJWTNkNWlFK2VyKwo0M3lRaFVlWkhrRXRUM3JRTUI4R0ExVWRJd1FZTUJhQUZKWnVCclVaVjkzMy9tUUhQOXVhNEV3N0dpeWVNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCWGM2cWNSOHQ4dFhHYjRUK0l3clVCd0RvVHhVSWtRQ0p5M1NaT09jZ1AKcnRKalFtNGhPSDhMd1NFcHBHQ2JjV1JxeWRZTzVUQ1Y4K1kzbGQvUzd6TWJIUmNnU2IvczlBbERjbHZZR1daago3akk0TEs4K1lJSXIwVWJtTDdOZWlaMzAxcWorRmlaeElBNkVxN284Y29WWFhlYmpYQUtKKzFva0lqWVoxaGlxCmlBaHRTeXh0MTZQQVlla0NYdGl1U2tGTGNRSUZCYUk5REo4RlpuZDVnbkJnQUloLzlVUFVNR1hDUjhwdkJscFUKeWlGb0puOUtSY3YzQVEwSVJRejkwVzhUMHVYTHdXaU5SM2E2YU5uc1hVZmloaXAwY1VKWXNRODAvb0dIbHJHeApzQktvRVN1QXNSbnE3N25DWmlRUlliZmtEQVZpc0d3KzZHWTNpMnJVUXpDZwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
五:ETCD集群通信设置
  1. ETCD_CERT_FILE="/etc/kubernetes/ssl/kubernetes.pem"
  2. ETCD_KEY_FILE="/etc/kubernetes/ssl/kubernetes-key.pem"
  3. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/ssl/ca.pem"
  4. ETCD_PEER_CERT_FILE="/etc/kubernetes/ssl/kubernetes.pem"
  5. ETCD_PEER_KEY_FILE="/etc/kubernetes/ssl/kubernetes-key.pem"
  6. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/ssl/ca.pem"
--cert-file --key-file 设置ETCD的公私钥; --peer-cert-file, --peer-key-file设置ETCD 集群各节点之间通信的公私钥; --trusted-ca-file=设置客户端CA证书; --peer-trusted-ca-file设置ETCD集群各节点CA证书。

六:API Server与ETCD集群通信设置(apiserver配置文件

点击(此处)折叠或打开

  1. --etcd-cafile=/etc/kubernetes/ssl/ca.pem 
  2. --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem 
  3. --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem
七:API Server与Kubelet之间通信设置

1.Master节点生成token.csv文件;并在apiserver配置文件中由--token-auth-file参数指定。用于验证来自kubelet的bootstrap.kubeconfig文件.
token.csv为一个用户的描述文件,基本格式为 Token,用户名,UID,用户组;这个文件在 apiserver 启动时被 apiserver 加载,然后就相当于在集群内创建了一个这个用户;接下来就可以用 RBAC 给他授权;持有这个用户 Token 的组件访问 apiserver 的时候,apiserver 根据 RBAC 定义的该用户应当具有的权限来处理相应请求

点击(此处)折叠或打开

2c691a233d152b582f69de61e40dfc32,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
2.kubelet配置bootstrap.kubeconfig文件

点击(此处)折叠或打开

  1. export KUBE_APISERVER="https://masterIp:6443"
  2. # 设置集群参数
  3. kubectl config set-cluster kubernetes \
  4. --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  5. --embed-certs=true \
  6. --server=${KUBE_APISERVER} \
  7. --kubeconfig=bootstrap.kubeconfig

  8. # 设置客户端认证参数
  9. kubectl config set-credentials kubelet-bootstrap \
  10. --token=${BOOTSTRAP_TOKEN} \
  11. --kubeconfig=bootstrap.kubeconfig
  12. # 设置上下?参数
  13. kubectl config set-context default \
  14. --cluster=kubernetes \
  15. --user=kubelet-bootstrap \
  16. --kubeconfig=bootstrap.kubeconfig
  17. # 设置默认上下?
  18. kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
在 apiserver 配置中指定了一个  token.csv  文件,该文件中是一个预设的用户配置;同时该用户的 Token 和 apiserver 的 CA 证书被写入了 kubelet 所使用的  bootstrap.kubeconfig  配置文件中;这样在首次请求时,kubelet 使用  bootstrap.kubeconfig  中的 apiserver CA 证书来与 apiserver 建立 TLS 通讯,使用  bootstrap.kubeconfig  中的用户 Token 来向 apiserver 声明自己的 RBAC 授权身份.

3. 创建一个 ClusterRoleBinding,将预设用户  kubelet-bootstrap  与内置的 ClusterRole  system:node-bootstrapper  绑定到一起,使其能够发起 CSR 请求

点击(此处)折叠或打开

  1. cd /etc/kubernetes
  2. kubectl create clusterrolebinding kubelet-bootstrap \
  3. --clusterrole=system:node-bootstrapper \
  4. --user=kubelet-bootstrap
4. 在 kubelet 首次启动后,如果用户 Token 没问题,并且 RBAC 也做了相应的设置,那么此时在集群内应该能看到 kubelet 发起的 CSR 请求,可以使用 kubectl 手动签发(允许) kubelet 的证书,当成功签发证书后,目标节点的 kubelet 会将证书写入到 --cert-dir= 选项指定的目录中.

至此,API Server和kubelet之间可以通过TLS进行相互通信。

八:Pod与API Server之间通信设置

1.使用API Server私钥(controller-manager配置文件中的--service-account-private-key-file指定的文件)创建一个Token(JWT Token),并用该Token、CA证书(这个是API Server的CA公钥证书,来源于controller-manager配置文件中的root-ca-file),Namespace名称等三个信息产生一个新的Secret对象,然后放入创建的Service Account中

2.API Server收到Token以后,采用自己的私钥(实际是使用apiserver配置文件中的参数service-account-key-file指定的私钥,如果此参数没有设置,则默认采用tls-private-key-file指定的参数,即自己的私钥)对Token进行合法验证。

九:API Server与kube-proxy之间通信设置

点击(此处)折叠或打开

  1. export KUBE_APISERVER="https://masterIp:6443"
  2. # 设置集群参数
  3. kubectl config set-cluster kubernetes \
  4. --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  5. --embed-certs=true \
  6. --server=${KUBE_APISERVER} \
  7. --kubeconfig=kube-proxy.kubeconfig
  8. # 设置客户端认证参数
  9. kubectl config set-credentials kube-proxy \
  10. --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  11. --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  12. --embed-certs=true \
  13. --kubeconfig=kube-proxy.kubeconfig
  14. # 设置上下?参数
  15. kubectl config set-context default \
  16. --cluster=kubernetes \
  17. --user=kube-proxy \
  18. --kubeconfig=kube-proxy.kubeconfig
  19. # 设置默认上下?
  20. kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
kube-proxy.pem 证书中 CN 为 system:kube-proxy,kubeapiserver预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定,该 Role授予了调? kube-apiserver Proxy 相关 API 的权限;
生成的 kube-proxy . kubec onfig文件如下:

点击(此处)折叠或打开

  1. apiVersion: v1
  2. clusters:
  3. - cluster:
  4.     certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSVprZlNwNSs5REY3OUZEYXFhQ1haYXRsOG9Rd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURNeE1qQTFNall3TUZvWERUSXpNRE14TVRBMU1qWXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBcERaU3NjQWF2N3Roa0JDUTZYN2cKWGZCZ3JOZWM2amc1YjdzVXEzSENyc1hYL3dlOW5POVNxVmxGdS8vY3VMTXNiWEhTdGhXYzJtTXBhdzlmQWVUdwpqK2gyOVZjTkJEamhWbzNhbUFZNk9pR1hOWWZQd0hsaHN4bVFXT0Zrc1NybnpOaGk4djc5K0R5bk1Bc3BSVHlxCk5KODdSUi93d0x6TDVUcFlka1U2U3dIK1ZsR01NRjNEMFdRSjV1ZHlNbFBiRnQyWVVnZE1VRXVCdDRVYWp3d2cKZE9jSVdEcFV0Q0pGNXUvU3RTaHA3RDdRb05rUDdKa2dodm5KMEx6enZxZnlkVFE2MW1HSXIvUHpYS0d3VkJoTgpEQ0hCMTJ1SDBEUkxwK3V2UE5pdmRYcS9wSzE0dWlxYjFoTGdRbzd6OS92d0xBMVg0c0FrU1o2UjUzQzBEZ3kvCnN3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVbG00R3RSbFgzZmYrWkFjLzI1cmdURHNhTEo0d0h3WURWUjBqQkJnd0ZvQVVsbTRHdFJsWAozZmYrWkFjLzI1cmdURHNhTEo0d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFISWVvY1lXM21ZVVdMbTVkR3hvCnN0a01DZlc3aDhDMEduZ2ZIOXFhbFZVTzk0b0RDd2hEMWlqOWtZVW9nZzF3a1BFbGZjSWEybERDOFZBckJtZ3UKWXhFOEJwSkpxMUtIaXAyM0xXSXhWTFB4VWs5ckFUNEtwbE9idmo1RG5adTFDTndlVXJpWU1QUEtRc01LaXROaApVQ3pBV0FVTUpIa2JwVHdsZGNBYmtGMHFyUThGRzZsa1EwakRpTUpKc0ZrelEzbU5YY1pzYXpnamNRWjJHczBxCkhySzlrQ2FHTWtHbzEvQUtDYms5VzZEZDBpL05QeWtvT2d3U1ExMzFSaHQrYmlGZVdVOWQwYjZST3JyYW9GeFoKd1hnZGRYZm11UHQ3eHhyeVFZTUN0NWxRYkQ4aS8wNzJZMGVNYVpQU09jc2xtTk1RQWpsN3Z6T0N0UGcyRzNKNAoxemM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  5.     server: https://-----------------
  6.   name: kubernetes
  7. contexts:
  8. - context:
  9.     cluster: kubernetes
  10.     user: kube-proxy
  11.   name: default
  12. current-context: default
  13. kind: Config
  14. preferences: {}
  15. users:
  16. - name: kube-proxy
  17.   user:
  18.     as-user-extra: {}
  19.     client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzakNDQXNhZ0F3SUJBZ0lVTCtlZWYzVmJjenZiZFBueUxCdXlMTmJlWFVNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU0TURNeE1qQTJOVEl3TUZvWERUSTRNRE13T1RBMk5USXdNRm93YkRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUm93R0FZRFZRUURFeEZ6ZVhOMFpXMDZhM1ZpClpTMXdjbTk0ZVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHo5bzVJTk5jMWgKU1NSeEs5SU5TcmhPZm4wcHV4dlF0cUp2Yzl0UFNUVXQ3QkZLT1h6dXd2VkowY1hKV201M2xZTEtObm9oSnljdgpsTElZSUhBMXFDa1hOTnNrc2huRlJPYmNmTFZMWU50cXV6WDBoQm5vaUtPYU8vYzUrVEpIYW41a1R3YXhSNklNCmZNNitlZkxTYUU0VGUzenRwVHVMVjlUT2QyZEdJTHpJRm1BSDFmZW5VNnJvUEsxcEtHbjZjUmlKZXk5S1VSNzIKWHpsZXZONWplTCs5cXNWSitINVo3ZFdrd21iQ2VqdWxYNVpZUmxZYm5sdSsxZ2syRXN0Z0NEODUrdldvYklHVgpJTkZmS2Q2YW1jN0E2cnVhQTh4aW54RTNtOFpuQmQzaFMyNktZSUlQVUExTGxNb05KTHBUK0Z4ZlA2MFN2bXBIClZWQktXSURUSDlVQ0F3RUFBYU4vTUgwd0RnWURWUjBQQVFIL0JBUURBZ1dnTUIwR0ExVWRKUVFXTUJRR0NDc0cKQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUWR5M0pidTVYNQoxbE9PcTFWRzZRbDBuM1JxblRBZkJnTlZIU01FR0RBV2dCU1diZ2ExR1ZmZDkvNWtCei9ibXVCTU94b3NuakFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVcxWWNhdGlTVS9XVFZzc2R2dVNwdGpUQllleGhrZlhSMjNIU3VnYUIKM1J4T2JscG5ETEtWdVVxY2dZMnIrTEU2SEFKZG9scEVydlc0RDRISFgzbW8wVTdFdlZVNUlvU1JmQ2p4dTZmaApDK0VsbzcybmFPNW96Y2NVbXcwaisrKzN6UzFZTnE0LzF0ZzhTd1NBOWx0alI5SU9kOHY4ZS91UWJnN2ZuMHJnCmxsR28rb0dxa0JaSDlnQzROc0Frd1haYUJPWWxjcDJEN244S2dqRS9DaWsxWDZzSDJiL0tPb1ZHR3NiTDRwY1QKMUxlOFFROWk4dDRlclI3OFlFVFFwZnJVbkZjUndlZTkvbDJSeVBDRDROSVpiZ0U5UXVCWW9nMUc3bjNYWG9DNwpFNUtWdUFXRkJraFBEaG44MU9RczRkamtBWkFIaEF1c1ZSUTByTzlPeVpodG53PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=






来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/28624388/viewspace-2153070/,如需转载,请注明出处,否则将追究法律责任。

转载于:http://blog.itpub.net/28624388/viewspace-2153070/

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值