【 Azure 】基于aks-engine的kubernetes集群部署

【 Azure 】基于aks-engine的kubernetes集群部署

AKS Engine provides tooling to quickly bootstrap Kubernetes clusters on Azure. By leveraging ARM (Azure Resource Manager), AKS Engine helps you create, destroy and maintain clusters provisioned with basic IaaS resources in Azure. AKS Engine is limited in its support for ongoing operational needs such as scaling, in-place upgrading, and extension management. The Cluster API Provider for Azure a.k.a. CAPZ provides more complete operational capabilities. AKS Engine remains the tool for managing Kubernetes clusters on Azure Stack Hub as CAPZ does not yet work there.

AKS engine和aks的区别

AKS engine和aks都是azure中管理部署kubernetes集群的，其中AKS engine是社区开源版的，aks是产品级别的。两者都是免费的，只需要支付集群中节点费用即可。相对于aks，aks engine支持的功能更多，技术上也更超前。

AKS engine功能简介

• 支持Azure Active Directory(AAD)集成，即可以通过AAD进行集群认证，以及权限配置
• 支持extension，即可以给集群虚拟机配置extension，比如收集系统日志的omsagent
• 支持配置GPU
• 支持public/private集群，以及可配置的CNI，比如Azure CNI，flannel，cilium等
• 支持集成azure的keyvault
• 支持部署集群到windows
• 支持集群的弹性伸缩
• 支持集群VMSS Node Pools更新
• 支持kubernetes集群升级

部署集群

前置条件

• An Azure Subscription
• The Azure CLI
• resource group
• Vnet/Subnet

部署集群

1. 下载aks-engine

wget https://github.com/Azure/aks-engine/releases/download/v0.55.4/aks-engine-v0.55.4-linux-amd64.tar.gz

tar -xvf aks-engine-v0.55.4-linux-amd64.tar.gz

cd aks-engine-v0.55.4-linux-amd64

2. 配置aks engine template
   template包含对azure资源的配置，以及kubernetes集群的配置，下面是个配置例子akse.json，包含了如下的需求

• 部署高可用的kubernetes1.17.11
• private集群，外部不可访问
• 选用standard的loadblancer
• cni选用flannel
• 节点启用MSI（useManagedIdentity）
• 安装跳板机用来访问管理集群
• 配置集群autoscaler
• 安装tiller
• 集成azure keyvault
• 集成container-monitoring来收集日志以及连接azure monitor
• 3 masters ，2 agent pools
• 开启AAD认证

akse.json

{
  "apiVersion": "vlabs",
  "location": "chinanorth2",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.17",
      "orchestratorVersion": "1.17.11",
      "kubernetesConfig": {
        "loadBalancerSku": "Standard",
	"excludeMasterFromStandardLB": true,
	"kubeProxyMode" : "ipvs",
	"networkPlugin": "flannel",
    "networkPolicy": "",
    "useManagedIdentity": true,
    "userAssignedID": "/subscriptions/xxxx-xxxxx-xxxxx-xxxx/resourceGroups/xxxx-xxxxx-xxxxx-xxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/xxxx-xxxxx-xxxxx-xxxx",    
        "privateCluster": {
          "enabled": true,
	  "jumpboxProfile": {
	     "name": "reh-corebe-jumpbox",
	     "vmSize": "Standard_B2ms",
	     "osDiskSizeGB": 30,
	     "username": "vmadmin",
	     "publickey": "ssh-rsa "
	   }
        },
	"addons": [
        {
            "name": "cluster-autoscaler",
            "enabled": true,
            "pools": [
              {
                "name": "pool1",
                "config": {
                  "min-nodes": "4",
                  "max-nodes": "500"
                }
              },
              {
                "name": "pool2",
                "config": {
                  "min-nodes": "3",
                  "max-nodes": "3"
                }
              }
            ],
            "config": {
              "scan-interval": "1m"
            }
          },
	  {
            "name": "heapster",
	    "enabled": true
          },
	  {
            "name": "dns-autoscaler",
            "enabled": false
          },
          {
            "name": "container-monitoring",
            "enabled": true,
            "config": {
              "logAnalyticsWorkspaceResourceId" :  "/subscriptions/xxxx-xxxxx-xxxxx-xxxx/resourceGroups/xxxx-xxxxx-xxxxx-xxxx/providers/Microsoft.OperationalInsights/workspaces/xxxx-xxxxx-xxxxx-xxxx"
          }
          },
          {
            "name": "tiller",
            "enabled": true,
	    "containers": [
              {
                "name": "tiller",
                "image": "gcr.azk8s.cn/kubernetes-helm/tiller:v2.13.0",
                "cpuRequests": "50m",
                "memoryRequests": "150Mi",
                "cpuLimits": "50m",
                "memoryLimits": "150Mi"
              }
            ],
            "config": {
              "max-history": "0"
            }
          },
          {
            "name": "blobfuse-flexvolume",
            "enabled": true
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": true
          },
          {
            "name": "kubernetes-dashboard",
            "enabled": true
          },
          {
            "name": "aci-connector",
            "enabled": false
          },
          {
            "name": "smb-flexvolume",
            "enabled": false
          },
          {
            "name": "rescheduler",
            "enabled": false
          },
          {
            "name": "nvidia-device-plugin",
            "enabled": false
          }
        ]
       }
    },
      "masterProfile": {
        "count": 3,
        "dnsPrefix": "master",
        "vmSize": "Standard_B12ms",
	"vnetSubnetId": "/subscriptions/xxxx-xxxxx-xxxxx-xxxx/resourceGroups/xxxx-xxxxx-xxxxx-xxxx/providers/Microsoft.Network/virtualNetworks/xxxx-xxxxx-xxxxx-xxxx/subnets/xxxx-xxxxx-xxxxx-xxxx",
	"firstConsecutiveStaticIP": "xxx.xxx.xxx.xxx",
	"distro": "aks-ubuntu-18.04",
            "customVMTags": {
            "Name": "k8s-master",
            "ProjectID": "ICTO-27027",
            "ApplicationID": "AKSE",
            "ApplicationVersion": "v0.50.0",
            "CostCenterID": "8120020",
            "Role": "Shareservice",
            "SupportContact": ""
        }
      },
      "agentPoolProfiles": [
        {
          "name": "pool1",
          "count": 4,
          "vmSize": "Standard_B12ms",
	  "vnetSubnetId": "/subscriptions/xxxx-xxxxx-xxxxx-xxxx/resourceGroups/xxxx-xxxxx-xxxxx-xxxx/providers/Microsoft.Network/virtualNetworks/xxxx-xxxxx-xxxxx-xxxx/subnets/xxxx-xxxxx-xxxxx-xxxx",
	  "distro": "aks-ubuntu-18.04",
          "availabilityProfile": "VirtualMachineScaleSets",
	  "storageProfile": "ManagedDisks",
	  "OSDiskSizeGB": 200,
                         "customVMTags": {
                "Name": "k8s-vmss",
                "ProjectID": "ICTO-27027",
                "ApplicationID": "AKSE",
                "ApplicationVersion": "v0.50.0",
                "CostCenterID": "8120020",
                "Role": "Shareservice",
                "SupportContact": ""
            }
        },
        {
          "name": "pool2",
          "count": 3,
          "vmSize": "Standard_B12ms",
	  "vnetSubnetId": "/subscriptions/xxxx-xxxxx-xxxxx-xxxx/resourceGroups/xxxx-xxxxx-xxxxx-xxxx/providers/Microsoft.Network/virtualNetworks/xxxx-xxxxx-xxxxx-xxxx/subnets/xxxx-xxxxx-xxxxx-xxxx",
	  "distro": "aks-ubuntu-18.04",
          "availabilityProfile": "VirtualMachineScaleSets",
	  "storageProfile": "ManagedDisks",
	  "OSDiskSizeGB": 200,
                         "customVMTags": {
                "Name": "k8s-vmss",
                "ProjectID": "ICTO-27027",
                "ApplicationID": "AKSE",
                "ApplicationVersion": "v0.50.0",
                "CostCenterID": "8120020",
                "Role": "Shareservice",
                "SupportContact": ""
            }
        }
      ],
      "linuxProfile": {
        "adminUsername": "vmadmin",
        "ssh": {
          "publicKeys": [
            {
              "keyData": "ssh-rsa "
            }
          ]
        }        
      },   
      "aadProfile": {
        "serverAppID": "xxxx-xxxxx-xxxxx-xxxx",
        "clientAppID": "xxxx-xxxxx-xxxxx-xxxx",
        "tenantID": "xxxx-xxxxx-xxxxx-xxxx",
        "adminGroupID": "xxxx-xxxxx-xxxxx-xxxx"
    },      
      "certificateProfile": {}
  }
}

3. 生成Azure Resource Manager template

aks-engine  generate akse.json

_output目录下会生成相关文件，建议备份这个目录下的文件，后面集群维护会使用相关配置文件或者证书。

4. 部署集群

aks-engine deploy --resource-group "xxxx-xxxx-xxxx-xxxx \
  --azure-env "AzureChinaCloud" \
  --location "${region}" \
  --subscription-id "${subscription_id}" \
  --client-id "${client_id}" \
  --client-secret "${client_secret}" \
  --api-model "_output/xxxx-xxxx-xxxx-xxxx/apimodel.json" \
  -o "deploy" \
  --debug -f

验证

登录跳本机，安装kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.11/bin/linux/amd64/kubectl

确认安装，第一次登录集群需要通过aad认证

sudo kubectl get nodes
To sign in, use a web browser to open the page https://microsoft.com/deviceloginchina and enter the code CRXP2BB5U to authenticate.
NAME                                STATUS   ROLES    AGE    VERSION
k8s-master-81692357-0               Ready    master   7m2s   v1.17.11
k8s-master-81692357-1               Ready    master   7m2s   v1.17.11
k8s-master-81692357-2               Ready    master   6m7s   v1.17.11
k8s-prdconapl-81692357-vmss000000   Ready    agent    7m2s   v1.17.11
k8s-prdconapl-81692357-vmss000001   Ready    agent    7m2s   v1.17.11
k8s-prdconapl-81692357-vmss000002   Ready    agent    7m2s   v1.17.11
k8s-prdconeny-81692357-vmss000000   Ready    agent    7m2s   v1.17.11