【k8s实践】私有kubernetes使用阿里云cloudprovider

0.简介

本文主要介绍v1.20版本k8s如何使用阿里云cloud-provider做负载均衡。

1.环境配置

k8s版本：v1.20.4

客户端访问流程：集群外的客户端 –> LoadBalancer –> NodeIP:NodePort –> ClusterIP:servicePort –> PortIP:containerPort

默认文档：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md

主机：使用阿里云ecs主机部署

默认认为已经有了k8s集群，这里不在介绍如何安装。

2.设置阿里云cloudprovider

修改/etc/kubernetes/manifests/目录下kube-apiserver.yaml与kube-controller-manager.yaml

添加

- --cloud-provider=external

3.在每台主机里都设置实例id与区域id

获取每台实例信息
META_EP=http://100.100.100.200/latest/meta-data
echo `curl -s $META_EP/region-id`.`curl -s $META_EP/instance-id`
配置
kubectl patch node cn-hangzhou.i-bp16uimj7fl6ze8q5rf3 -p '{"spec":{"providerID": "cn-hangzhou.i-bp16uimj7fl6ze8q5rf3"}}'

4.配置验证

 How to get AccessKey?

需要创建接入key

# base64 AccessKey & AccessKeySecret
$ echo -n "$AccessKeyID" |base64
$ echo -n "$AcceessKeySecret"|base64


cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  cloud-config.conf: |-
    {
        "Global": {
            "accessKeyID": "$your-AccessKeyID-base64",
            "accessKeySecret": "$your-AccessKeySecret-base64"
        }
    }
    
 kubectl create -f cloud-config.yaml

5.创建阿里云控制

默认server是你api地址

/etc/kubernetes/cloud-controller-manager.conf
kind: Config
contexts:
- context:
    cluster: kubernetes
    user: system:cloud-controller-manager
  name: system:cloud-controller-manager@kubernetes
current-context: system:cloud-controller-manager@kubernetes
users:
- name: system:cloud-controller-manager
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $CA_DATA
    server: https://192.168.1.76:6443
  name: kubernetes

$CA_DATA从下面命令获取

 cat /etc/kubernetes/pki/ca.crt|base64 -w 0

6.配置ds

下载

wget https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/examples/cloud-controller-manager.yml

这个地址改为创建是--service-cidr地址

- --cluster-cidr=172.20.0.0/16

7.创建阿里云公网负载均衡

vim whats-my-ip.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: whats-my-ip
  name: whats-my-ip
spec:
  replicas: 3
  selector:
    matchLabels:
      run: whats-my-ip
  template:
    metadata:
      labels:
        run: whats-my-ip
    spec:
      containers:
      - image: cloudnativelabs/whats-my-ip
        name: whats-my-ip
        ports:
        - containerPort: 8080

      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 20
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 20
      terminationGracePeriodSeconds: 30

应用与配置svc

kubectl apply -f whats-my-ip.yml

kubectl expose deployment whats-my-ip --type=LoadBalancer --name=whats-my-ip --port=80 --target-port=8080

默认使用kube-proxy的iptables，会在本地做转发

[root@iz0jl52ythz64abom43swwz manifests]# kubectl get svc
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes    ClusterIP      10.96.0.1       <none>          443/TCP        20h
whats-my-ip   LoadBalancer   10.108.49.210   39.101.68.154   80:30780/TCP   25m
[root@iz0jl52ythz64abom43swwz manifests]# netstat -tlnp|grep 30780
tcp        0      0 0.0.0.0:30780           0.0.0.0:*               LISTEN      7503/kube-proxy
[root@iz0jl52ythz64abom43swwz manifests]# kubectl describe svc whats-my-ip
Name:                     whats-my-ip
Namespace:                default
Labels:                   run=whats-my-ip
                          service.beta.kubernetes.io/hash=7db81c73a86d92bbc48c36f4dbcef64b5d75810e01614e91eabb4f26
Annotations:              <none>
Selector:                 run=whats-my-ip
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.108.49.210
IPs:                      10.108.49.210
LoadBalancer Ingress:     39.101.68.154
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30780/TCP
Endpoints:                192.168.32.11:8080,192.168.32.7:8080,192.168.32.9:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason            Age                From                Message
  ----     ------            ----               ----                -------
  Warning  NoBackend         16m (x2 over 16m)  service-controller  There are no available nodes for loadbalancer
  Normal   SuccessfulEnsure  16m (x2 over 16m)  service-controller  Ensure loadbalancer successfully

效果

[root@iz0jl52ythz64abom43swwz manifests]# curl 39.101.68.154
HOSTNAME:whats-my-ip-7c7bdc9f9b-rh2f4 IP:192.168.32.7
[root@iz0jl52ythz64abom43swwz manifests]# curl 39.101.68.154
HOSTNAME:whats-my-ip-7c7bdc9f9b-rh2f4 IP:192.168.32.7
[root@iz0jl52ythz64abom43swwz manifests]# curl 39.101.68.154
HOSTNAME:whats-my-ip-7c7bdc9f9b-5nwtv IP:192.168.32.9
[root@iz0jl52ythz64abom43swwz manifests]# curl 39.101.68.154
HOSTNAME:whats-my-ip-7c7bdc9f9b-pxzmt IP:192.168.32.11