https://en.wikipedia.org/wiki/Kubernetes
Minikube runs a single-node Kubernetes cluster,可以用于单机上体会k8s的用法,参考了不少文档,包括官方的和阿里云的文档. 可以说缺点都挺多,一是没有不用VM 的示例,二是很多前提条件没有列出.所以我摸索了好久才完成这个实验.
目标架构:
Centos Linux: minikube -> docker
前提
确保docker 已经安装:
alex@abc]$sudo yum list docker*
删除过去安装的kubernets
[alex@abc]$sudo yum remove kuber*
[alex@~]$sudo yum remove etcd
查看已添加的iptables规则
root@alex:/home/alex #iptables -L -n
root@alex:/home/alex #systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.
root@alex:/home/alex #systemctl status iptables
● iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
Active: inactive (dead)
root@alex:/home/alex #systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
root@alex:/home/alex #systemctl stop firewalld
root@alex:/home/alex #systemctl mask firewalld
Created symlink from /etc/systemd/system/firewalld.service to /dev/null.
如下如果不做的话, minikube dashboard 会返回:unexpected response code: 503
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -P INPUT ACCEPT
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -F
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -X
root@alex:/home/alex/base/DevOps/minikube # iptables -NX
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -Z
root@alex:/home/alex/base/DevOps/minikube/nginx #systemctl restart iptables.service
add aliyun repo
-bash-4.2# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
install
[alex@abc]$sudo yum install kubectl kubeadm
先尝试原版minikube: https://github.com/kubernetes/minikube/releases
root@alex:/home/alex/abc #./minikube.sh
......
E1019 18:07:03.540155 8671 start.go:254] Error updating cluster: downloading binaries: downloading kubeadm: Error downloading kubeadm v1.10.0: failed to download: failed to download to temp file: failed to copy contents: read tcp 192.168.0.101:40246->216.58.197.112:443: read: connection reset by peer
================================================================================
由于无法指定mirror, 原版的minikube无法下载需要的软件,所以没法继续.
改用aliyun 版本的minikube
[alex@abc]$curl -Lo minikube http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v0.30.0/minikube-linux-amd64
碰到了kubelet 状态异常的问题.
troubleshooting
root@alex:/home/alex/abc #systemctl status kubelet -l
root@alex:/home/alex/abc #journalctl -xefu kubelet
kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
原因是kubelet与docker使用的 cgroupfs不同,首先尝试修改kubelet的配置
root@alex:/etc/systemd/system/kubelet.service.d #vi 10-kubeadm.conf
--cgroup-driver 修改为 =systemd
问题还是发生,因为发现kubeadm会restart kubelet
[alex@~]$systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2018-10-19 22:22:04 CST; 6s ago
Docs: https://kubernetes.io/docs/
Process: 5645 ExecStart=/usr/bin/kubelet --cluster-dns=10.96.0.10 --authorization-mode=Webhook --client-ca-file=/var/lib/minikube/certs/ca.crt --cadvisor-port=0 --fail-swap-on=false --pod-manifest-path=/etc/kubernetes/manifests --hostname-override=minikube --allow-privileged=true --cluster-domain=cluster.local --cgroup-driver=cgroupfs --kubeconfig=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf (code=exited, status=255)
Main PID: 5645 (code=exited, status=255)
只能修改 docker 的cgroup-driver配置,修改很成功
$ vim /lib/systemd/system/docker.service 将 --exec-opt native.cgroupdriver=systemd 修改为:--exec-opt native.cgroupdriver=cgroupfs
start again
root@alex:/home/alex/abc #./minikube.sh
.......
WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS
The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks
Loading cached images from config file.
检查一下状态
root@alex:/home/alex/abc #minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.0.101
可以看到minikube下载/运行了很多images
[alex@base]$docker images
[alex@base]$docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e87305073a6e registry.cn-hangzhou.aliyuncs.com/google_containers/coredns@sha256:a905b50f8ecddf67cd1ad92ef95f8547aea9fd1970359e00c16aa0ab1d517baa "/coredns -conf /e..." 28 minutes ago Up 28 minutes k8s_coredns_coredns-5bfd87b64b-8l4sp_kube-system_e1f8077c-d3f8-11e8-a152-00e06f69c642_0
2f70bdb9a83b registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64@sha256:1d2e1229a918f4bc38b5a3f9f5f11302b3e71f8397b492afac7f273a0008776a "/dashboard --inse..." 28 minutes ago Up 28 minutes k8s_kubernetes-dashboard_kubernetes-dashboard-866c7586d-bctvw_kube-system_e16384c4-d3f8-11e8-a152-00e06f69c642_0
b089eda4be39 registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a "/storage-provisioner" 28 minutes ago Up 28 minutes k8s_storage-provisioner_storage-provisioner_kube-system_e16e436b-d3f8-11e8-a152-00e06f69c642_0
a556ade3370c registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64@sha256:a08b7f5ce38f736da9aca5d9df8ea5e356a01a51ad9ffd70027aba62a941fdfb "/usr/local/bin/ku..." 28 minutes ago Up 28 minutes k8s_kube-proxy_kube-proxy-5ftfb_kube-system_df7c6342-d3f8-11e8-a152-00e06f69c642_0
e4c6b1a09dac k8s.gcr.io/pause-amd64:3.1 "/pause" 28 minutes ago Up 28 minutes k8s_POD_coredns-5bfd87b64b-8l4sp_kube-system_e1f8077c-d3f8-11e8-a152-00e06f69c642_0
3bb9a286eeab k8s.gcr.io/pause-amd64:3.1 "/pause" 28 minutes ago Up 28 minutes k8s_POD_storage-provisioner_kube-system_e16e436b-d3f8-11e8-a152-00e06f69c642_0
92fa0b8466f8 k8s.gcr.io/pause-amd64:3.1 "/pause" 28 minutes ago Up 28 minutes k8s_POD_kubernetes-dashboard-866c7586d-bctvw_kube-system_e16384c4-d3f8-11e8-a152-00e06f69c642_0
18a2204c0dec 6f7f2dc7fab5 "/sidecar --v=2 --..." 29 minutes ago Up 29 minutes k8s_sidecar_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
22a7b1f75236 c2ce1ffb51ed "/dnsmasq-nanny -v..." 29 minutes ago Up 29 minutes k8s_dnsmasq_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
49d935f03e80 80cc5ea4b547 "/kube-dns --domai..." 29 minutes ago Up 29 minutes k8s_kubedns_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
a9e52f3944a2 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
8acee2605b73 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_kube-proxy-5ftfb_kube-system_df7c6342-d3f8-11e8-a152-00e06f69c642_0
6729fbb00727 52920ad46f5b "etcd --listen-cli..." 29 minutes ago Up 29 minutes k8s_etcd_etcd-minikube_kube-system_297504360d343dfdf5aeec497ed68de6_0
44d59e849826 704ba848e69a "kube-scheduler --..." 29 minutes ago Up 29 minutes k8s_kube-scheduler_kube-scheduler-minikube_kube-system_6842c2514f06b2c32c75c69be9c6618a_0
98a224697f05 af20925d51a3 "kube-apiserver --..." 29 minutes ago Up 29 minutes k8s_kube-apiserver_kube-apiserver-minikube_kube-system_c0f55fe15a4c62b7baa15edc14baa3a2_0
6f219881f2a6 ad86dbed1555 "kube-controller-m..." 29 minutes ago Up 29 minutes k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_5a3aced2c0bbcdcc542d4e179bbb3593_0
8c8bbdad6172 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_etcd-minikube_kube-system_297504360d343dfdf5aeec497ed68de6_0
06dc3a7abda2 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_kube-scheduler-minikube_kube-system_6842c2514f06b2c32c75c69be9c6618a_0
a8449fda08f8 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_kube-apiserver-minikube_kube-system_c0f55fe15a4c62b7baa15edc14baa3a2_0
2adaa8ff3da6 k8s.gcr.io/pause-amd64:3.1 "/pause" 29 minutes ago Up 29 minutes k8s_POD_kube-controller-manager-minikube_kube-system_5a3aced2c0bbcdcc542d4e179bbb3593_0
bcc84c2a05d7 9c16409588eb "/opt/kube-addons.sh" 30 minutes ago Up 30 minutes k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_e7f9b63d8d56a24739db3329dc512f82_0
6ec73f69b30b k8s.gcr.io/pause-amd64:3.1 "/pause" 30 minutes ago Up 30 minutes k8s_POD_kube-addon-manager-minikube_kube-system_e7f9b63d8d56a24739db3329dc512f82_0
[alex@base]$
检查网络的话可以看到 none driver
[alex@base]$docker network ls
NETWORK ID NAME DRIVER SCOPE
ed6663bfdf39 bridge bridge local
7c26b74fdab4 host host local
b4cb21683a90 none null local
you can use the --extra-config
flag on the minikube start
command.
minikube start --extra-config=apiserver.v=10 --extra-config=kubelet.max-pods=100
实际是传给 The kubeadm bootstrapper
[命令行]尝试创建一个test deployment:hello
root@alex:/home/alex/base #kubectl run hello --image=docker.io/mongo
dashboard 可以看到deployment/pods/Replica Sets 增加了 hello 相关内容
[命令行]创建service
root@alex:/home/alex/base # kubectl expose deployment hello --port=80 --target-port=8000
部署mongodb
kubectl run mongodb --image=docker.io/mongo --env="p='27017:27017'" --env="v='/home/alex/mongodb:/data'"
kubectl expose deployment mongodb --port=27017 --target-port=27017
从网页上找到 service 对应的 ip 地址: 10.109.21.194
测试mongodb端口,成功
用python 访问mongodb数据库成功.
minikube addon: kubernetes/minikube
root@minikube:/home/alex/base/DevOps/minikube #minikube addons list
- addon-manager: enabled
- coredns: enabled
- dashboard: enabled
.............
用dashboard deploy 一个app(2 pods)
对应到两个container
[alex@~]$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db5f927cf0e3 docker.io/mongo@sha256:c4e8225e68348b18283cf5c523f99122426fe1e15d883104e24c61bedb4b2ca7 "docker-entrypoint..." 10 minutes ago Up 10 minutes k8s_mongodba_mongodba-6d5d6ddf64-r6f69_default_8e227c01-d5d2-11e8-a0d4-00e06f69c642_0
e0506d12d49e docker.io/mongo@sha256:c4e8225e68348b18283cf5c523f99122426fe1e15d883104e24c61bedb4b2ca7 "docker-entrypoint..." 10 minutes ago Up 10 minutes k8s_mongodba_mongodba-6d5d6ddf64-jw4fv_default_8e23d7e3-d5d2-11e8-a0d4-00e06f69c642_0
内部只是以 tcp port 来区分.
尝试用deploymet-scale 到3个pods 或一个pods
按 kubernetes/minikube 用 yaml 语法创建volume
attach to the pod:
[alex@~]$kubectl proxy --port=8080
then:
[alex@minikube]$kubectl get pods
NAME READY STATUS RESTARTS AGE
mongodba-6d5d6ddf64-jw4fv 1/1 Running 0 16h
可能因为minikube 是root身份start的,如下命令也只能以root 运行.
root@minikube:/home/alex/base/DevOps/minikube #kubectl exec -it mongodba-6d5d6ddf64-jw4fv bash
some query
root@minikube:/home/alex/base/DevOps/minikube #kubectl config view
root@minikube:/home/alex/base/DevOps/minikube #kubectl cluster-info
查namesapce
root@minikube:/home/alex/base/DevOps/minikube #kubectl get po,svc -n kube-system
point the ‘docker’ command to your Minikube’s Docker daemon 失败
root@minikube:/home/alex/base/DevOps/minikube #(minikube docker-env)
'none' driver does not support 'minikube docker-env' command
enable heapster:
root@alex:/home/alex/base/DevOps/minikube #minikube addons enable heapster
heapster was successfully enabled
然后dashboard 就可以看到如下load 统计了:
stop minikube cluster:
root@alex:/home/alex/base/DevOps #minikube stop
Stopping local Kubernetes cluster...
Machine stopped.
docker⽤户过渡到kubectl命令⾏指南
$ docker run -d --restart=always -e DOMAIN=cluster --name nginxapp -p 80:80 nginx
kubectl
$ kubectl run --image=nginx nginx-app --port=80 --env="DOMAIN=cluster"
$ kubectl expose deployment nginx-app --port=80 --name=nginx-http
对应docker exec
$ kubectl exec -ti nginx-app-5jyvm -- /bin/sh
最终宝典:
https://github.com/kubernetes/minikube/tree/master/docs