KubeFed环境搭建

kubernetes federation搭建

前置要求

软件以及版本需求

软件	版本
kubernetes	>= v1.13
minikube(可用于替代kubernetes)	只要安装的kubernetes版本同上即可(装最新的肯定没问题)
helm	>= 2.10
kubefedctl	这个kubefed版本选择即可
docker	看kubernetes需要的版本

软件环境搭建

docker环境搭建

• 海外版本

  apt-get update
  apt-get install -y docker.io
  systemctl start docker
  

• 国内版本

  apt-get update
  apt-get install -y apt-transport-https
  apt-get install -y ca-certificates
  apt-get install -y curl
  apt-get install -y software-properties-common
  curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
  add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
  sudo apt-get update
  apt-get install -y docker-ce
  
  systemctl start docker
  

minikube搭建 (ubuntu 18.04)

海外版本(可以直接访问google的版本)

• 安装kubectl
  • 直接下载二进制

    步骤	命令	example
    下载kubectl	curl -LO https://storage.googleapis.com/kubernetes-release/release/curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt/bin/linux/amd64/kubectl	curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl
    给kubectl添加可执行属性	chmod +x ./kubectl
    将kubectl移动到本地PATH中	sudo mv ./kubectl /usr/local/bin/kubectl
    测试kubectl的可用性	kubectl version --client

  • apt安装

    sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
    sudo apt-get update
    sudo apt-get install -y kubectl
    

  • yum 安装

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    EOF
    yum install -y kubectl
    

• 下载minikube可执行二进制文件

  curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
  

• 将minikube 二进制文件放入到本地PATH

  sudo mkdir -p /usr/local/bin/
  sudo install minikube /usr/local/bin/
  

• 启动minikube

  sudo apt-get install -y conntrack
  minikube start --driver=none
  

国内版本(通过阿里云安装)

curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.11.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
sudo apt-get install -y conntrack
minikube start --driver=none

helm安装

• 下载helm离线安装包

  # 下载地址
  # https://github.com/helm/helm/releases
  wget https://get.helm.sh/helm-v2.16.9-linux-amd64.tar.gz
  # 解压
  tar -zxvf helm-v2.16.9-linux-amd64.tar.gz
  # 将helm二进制可执行文件移动到本地path中
  mv linux-amd64/helm /usr/local/bin/helm
  

• 下载tiller image

  # 版本请自行修改，与helm的版本有关系，init的时候就知道要下载哪个了
  docker pull gcr.io/kubernetes-helm/tiller:v2.16.9
  

• 安装socat

  apt-get install -y socat
  

• helm init

  helm init --service-account tiller
  

kubefed安装

helm安装方法

过程	命令
安装repo	helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
查看已安装的repo	helm repo list
查询kubefed	helm search kubefed --devel
安装kubefed	helm install kubefed-charts/kubefed --name kubefed --version=<x.x.x> --namespace kube-federation-system --devel
卸载	kubectl -n kube-federation-system delete FederatedTypeConfig --all
	kubectl delete crd $(kubectl get crd
	helm delete --purge kubefed
查看安装后的api资源	kubectl apiresources

离线安装

• 下载kubefed的源代码包

  git clone https://github.com/kubernetes-sigs/kubefed.git
  

• 进入到kubefed的chart目录下

  cd charts/
  

• 使用helm离线安装

  helm install --name kubefed --namespace kube-federation-system ./kubefed/
  # 注意，此步骤还是需要通过途径获取到镜像quay.io/kubernetes-multicluster/kubefed:canary
  # 另外注意，如果通过helm delete --purge的方法删除了kubefed，其安装的FederatedDeployment等api resource是无法被删除的，目前还没有找到解决方案
  

kubefedctl 安装

• 下载网址

  https://github.com/kubernetes-sigs/kubefed/releases
  

• example

  wget https://github.com/kubernetes-sigs/kubefed/releases/download/v0.3.0/kubefedctl-0.3.0-linux-amd64.tgz
  tar -zxvf kubefedctl-0.3.0-linux-amd64.tgz
  mv linux-amd64/kubefedctl /usr/local/bin/
  

kubefed的使用指南

配置kubeconfig

需要在安装了kubefed的设备上配置到管理的集群的context

kubeconfig的路径和结构

• 路径 /root/.kube/config
• 文件结构
  • structure

    part	meaning
    apiVersion	api版本
    kind	配置类型，Config
    current-context	当前正在使用的context
    preferences
    clusters	集群配置
    users	用户配置
    contexts	context配置

    • clusters,users,contexts配置

      kind	subkind	configuration	meaning
      clusters	cluster	certificate-authority	kubernetes的API server的ca.crt
      		server	kubernetes的API server的https url
      		name	此kubernetes的api server的名字，我们可以随便儿起名字
      users	user	client-certificate	crt文件路径
      		client-key	key文件路径
      		name	用户名称，这个随便写，保持唯一性即可
      contexts	context	cluster	cluster的名称
      		user	user的名称
      		name	context的名称

  • example

    apiVersion: v1
    clusters:
    - cluster: # cluster 1
        certificate-autority: /root/.minikube/ca.crt
        server: https://${cluster 1 ip}:8443
        name: ${cluster 1 name}
    - cluster: # cluster2
        certificate-autority: ${cluster2's ca.crt path}
        server: https://${cluster 2 ip}:8443
        name: ${cluster 2 name}
    # other clusters
    contexts:
    - context:
        cluster: ${cluster 1 name}
        user: ${user 1 name}
    - context:
        cluster: ${cluster 2 name}
        user: ${user 2 name}
    # other contexts
    users:
    - name: ${user 1 name}
        user:
        client-certificate: ${user 1's client.crt}
        client-key: ${user 1's client.key}
    # other users
    

    • 注意，cluster和user要一一对应

配置kubeconfig

• 将kubefed之外的其他集群的以下几个文件下载到本地
  • 文件列表(以minikube举例)：

    文件作用	文件路径
    kubernetes API server的ca证书	/root/.minikube/ca.crt
    kubernetes 用户的crt	/root/.minikube/profiles/minikube/client.crt
    kubernetes 用户的key	/root/.minikube/profiles/minikube/client.key

• 举例
  • 下载来自另一台设备的对应文件

    # 创建文件夹
    mkdir -p /root/.cluster1/
    # 下载文件
    scp ${host}:/root/.minikube/ca.crt /root/.cluster1/
    scp ${host}:/root/.minikube/profiles/minikube/client.crt /root/.cluster1/
    scp ${host}:/root/.minikube/profiles/minikube/client.key /root/.cluster1/
    # 配置kubeconfig
    # 可以直接改/root/.kube/config，按照上面的文件结构加上cluster、context、user就行
    # 另一种命令方式如下
    ## 添加cluster
    kubectl config set-cluster cluster1 --server=https://${host}:8443 --certificate-authority=/root/.cluster1/ca.crt
    ## 添加user
    kubectl config set-credentials cluster1 --client-certificate=/root/.cluster1/client.crt --client-key=/root/.cluster1/client.key
    ## 添加context
    kubectl config set-context cluster1 --cluster cluster1 --user=cluster1
    

  • 配置完成后，使用如下命令做测试

    kubectl config use-context cluster1
    kubectl get namespaces
    

将集群纳入到fed集群的管辖范围内

# kubefedctl join ${hostname} --cluster-context ${hostname's k8s context} --host-cluster-context ${host machine's context} --v=2
kubefedctl join cluster1 --cluster-context cluster1 --host-cluster-context minikube --v=2

查看集群加入情况

kubectl get kubefedclusters -n kube-federation-system

在fed集群上创建一个namespace

kubectl create namespace federated-namespace

• 这一步必须做， namespace名可以自定义，只要保证后面步骤中的namespace和当前的namespace保持一致

创建FederatedNamespace

apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: federated-namespace
  namespace: federated-namespace
spec:
  placement:
    clusters:
    - name: cluster1
    - name: cluster2

查看namespace创建情况

# 查看cluster1上是否创建了federated-namespace
kubectl config use-context cluster1
kubectl get namespaces
# 查看cluster2上是否创建了federated-namespace
kubectl config use-context cluster2
kubectl get namespaces

创建FederatedDeployment

kind: FederatedDeployment
metadata:
  name: federated-nginx
  namespace: federated-namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx
            name: nginx
  placement:
    clusters:
    - name: cluster2
    - name: cluster1
  overrides:
  - clusterName: cluster2
    clusterOverrides:
    - path: "/spec/replicas"
      value: 5
    - path: "/spec/template/spec/containers/0/image"
      value: "nginx:1.17.0-alpine"
    - path: "/metadata/annotations"
      op: "add"
      value:
        foo: bar

查看deployment创建情况

# 查看cluster1上是否创建了federated-nginx
kubectl config use-context cluster1
kubectl get deployments -n federated-namespace
kubectl get pods -n federated-namespace
# 查看cluster2上是否创建了federated-nginx
kubectl config use-context cluster2
kubectl get deployments -n federated-namespace
kubectl get pods -n federated-namespace