VMWare + ubuntu + minikube 部署AWX

2023 Nov 部署单机k8s minikube以及部署AWX

docker / minikube / nginx (用于监听虚拟机端口转发进miniku内部service)

minikube 部署 ingress addons 以及 awx-operator / awx-demo

任务比较紧急，手头没有k8s，所以就紧急使用minikube部署个AWX，如下是踩坑经过。。。

 

Docker apt key & install

* Install docker

sudo curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

sudo apt install docker-ce -y

Docker setup aliyun mirror

sudo mkdir /etc/docker

sudo cat > /etc/docker/daemon.json <<EOF

{

    "exec-opts": ["native.cgroupdriver=systemd"],

    "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]

}

EOF

重新docker以及设置自启动

sudo systemctl daemon-reload

sudo systemctl restart docker

sudo systemctl enable docker

 

安装kubectl

sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

sudo cat <<EOF >/etc/apt/sources.list.d/kubernetes.list

deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main

EOF

apt update

sudo apt -y install  kubectl=1.28.2-00

 

安装minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb

sudo dpkg -i minikube_latest_amd64.deb

启动minikube

minikube start --cpus=4 --memory=6g --force

追加minikube addons ingress

这个命令minikube addons images ingress，先查看使用的名字和镜像对应关系

minikube addons images ingress

| IngressController        | ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3                        | registry.k8s.io  |

| KubeWebhookCertgenCreate | ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80 | registry.k8s.io  |

| KubeWebhookCertgenPatch  | ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80 | registry.k8s.io  |

addons的时候，指定--images name + image

minikube addons enable ingress --images="IngressController=registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.9.4,KubeWebhookCertgenCreate=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 ,KubeWebhookCertgenPatch=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 "

* ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.

You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS

  - Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0

  - Using image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0

  - Using image registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.9.4

* Verifying ingress addon...

* The 'ingress' addon is enabled

到这里，minikube k8s 部署完毕

root@k8su0:~# kubectl get pod -A

NAMESPACE       NAME                                        READY   STATUS      RESTARTS        AGE

ingress-nginx   ingress-nginx-admission-create-ncqkt        0/1     Completed   0               2m24s

ingress-nginx   ingress-nginx-admission-patch-f4fhx         0/1     Completed   3               2m24s

ingress-nginx   ingress-nginx-controller-56f5cd4fb9-qqbvc   1/1     Running     0               2m24s

kube-system     coredns-5dd5756b68-jdkk8                    1/1     Running     0               2m43s

kube-system     etcd-minikube                               1/1     Running     0               2m58s

kube-system     kube-apiserver-minikube                     1/1     Running     0               2m55s

kube-system     kube-controller-manager-minikube            1/1     Running     0               2m55s

kube-system     kube-proxy-h2p6d                            1/1     Running     0               2m43s

kube-system     kube-scheduler-minikube                     1/1     Running     0               2m55s

kube-system     storage-provisioner                         1/1     Running     1 (2m22s ago)   2m53s

 

minikube 部署awx

awx operator

vi kustomization.yaml

---

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization

resources:

  # Find the latest tag here: https://github.com/ansible/awx-operator/releases

  - github.com/ansible/awx-operator/config/default?ref=2.8.0

  -

# Set the image tags to match the git version from above

images:

  - name: quay.io/ansible/awx-operator

    newTag: 2.8.0

# Specify a custom namespace in which to install AWX

namespace: awx

---

第一次应用kustomization.yaml

kubectl apply -k .

kubectl -n awx get pod

awx-operator-controller-manager-xxx

镜像失败的话，edit这个deployment

我这里是gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0 pull不到

kubectl -n awx edit deployment.apps/awx-operator-controller-manager

#gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0更改成kubebuilder/kube-rbac-proxy:v0.15.0,去掉gcr.io/即可

全都成功以后，awx的operator pod都是running

awx-demo

vi awx-demo.yaml

---

apiVersion: awx.ansible.com/v1beta1

kind: AWX

metadata:

  name: awx-demo

spec:

  service_type: nodeport

---

第二次应用kustomization.yaml

# 再次vi kustomization.yaml,把github.com/ansible/awx-operator/config/default?ref=2.8.0去掉，更换成awx-demo.yaml

# 如果保留github.com/ansible/awx-operator/config/default?ref=2.8.0，deployment镜像问题需要再次更改。

kubectl apply -k .

# 成功了以后，由于minikube的pod是docker镜像内部，这里使用nginx进行转发

# 先取得nodeport service的Port是多少

root@k8su0:~# kubectl get service -n awx -o wide

NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE     SELECTOR

awx-demo-postgres-13                              ClusterIP   None            <none>        5432/TCP       5m5s    app.kubernetes.io/component=database,app.kubernetes.io/instance=postgres-13-awx-demo,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=postgres-13,app.kubernetes.io/part-of=awx-demo

awx-demo-service                                  NodePort    10.98.117.116   <none>        80:31606/TCP   3m33s   app.kubernetes.io/component=awx,app.kubernetes.io/managed-by=awx-operator,app.kubernetes.io/name=awx-demo-web

awx-operator-controller-manager-metrics-service   ClusterIP   10.101.184.90   <none>        8443/TCP       10m     control-plane=controller-manager

root@k8su0:~#

# 再取得minikube的内部nodeip

root@k8su0:~# kubectl get node -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME

minikube   Ready    control-plane   15m   v1.28.3   192.168.49.2   <none>        Ubuntu 22.04.3 LTS   5.15.0-89-generic   docker://24.0.7

root@k8su0:~# minikube ip

192.168.49.2

# 使用这两个编辑nginx.conf

sudo cp /home/setup/nginx.conf /etc/nginx/nginx.conf

sudo systemctl daemon-reload

sudo systemctl restart nginx

# 取得admin密码

kubectl get secret awx-demo-admin-password -o jsonpath="{.data.password}" -n awx | base64 --decode ; echo

# 访问主机地址，11.0.1.130:31606 admin + password

附录：

nginx.conf

#load_module /usr/lib64/nginx/modules/ngx_stream_module.so;

user nginx;

worker_processes auto;

error_log /var/log/nginx/error.log;

pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {

    worker_connections 1024;

}

 

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream minikube-awx {

       # ip_hash;

       server 192.168.49.2:31606 weight=10; # minikube ip + aws demo service nodeport's port

    }

   

    server {

       listen 31606;  # aws demo service 保持一直，这里主机也监听service nodeport's port

       proxy_pass minikube-awx;

    }

}

http {

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;

    tcp_nopush          on;

    tcp_nodelay         on;

    keepalive_timeout   65;

    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;

    default_type        application/octet-stream;

}