的driver_minikube with NONE (VM )driver

https://en.wikipedia.org/wiki/Kubernetes

Minikube runs a single-node Kubernetes cluster,可以用于单机上体会k8s的用法,参考了不少文档,包括官方的和阿里云的文档. 可以说缺点都挺多,一是没有不用VM 的示例,二是很多前提条件没有列出.所以我摸索了好久才完成这个实验.

目标架构:

Centos Linux: minikube -> docker

前提

确保docker 已经安装:
alex@abc]$sudo yum list docker*

删除过去安装的kubernets 
[alex@abc]$sudo yum remove  kuber*
[alex@~]$sudo yum remove etcd

查看已添加的iptables规则
root@alex:/home/alex #iptables -L -n

root@alex:/home/alex #systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.

root@alex:/home/alex #systemctl status iptables
● iptables.service - IPv4 firewall with iptables
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
   Active: inactive (dead)

root@alex:/home/alex #systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

root@alex:/home/alex #systemctl stop firewalld
root@alex:/home/alex #systemctl mask firewalld
Created symlink from /etc/systemd/system/firewalld.service to /dev/null.


如下如果不做的话, minikube dashboard 会返回:unexpected response code: 503

root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -P INPUT ACCEPT
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -F
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -X
root@alex:/home/alex/base/DevOps/minikube # iptables -NX
root@alex:/home/alex/base/DevOps/minikube/nginx #iptables -Z
root@alex:/home/alex/base/DevOps/minikube/nginx #systemctl restart iptables.service

add aliyun repo

-bash-4.2# cat  /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

install

[alex@abc]$sudo yum install kubectl kubeadm

先尝试原版minikube: https://github.com/kubernetes/minikube/releases

root@alex:/home/alex/abc #./minikube.sh 
......
E1019 18:07:03.540155    8671 start.go:254] Error updating cluster:  downloading binaries: downloading kubeadm: Error downloading kubeadm v1.10.0: failed to download: failed to download to temp file: failed to copy contents: read tcp 192.168.0.101:40246->216.58.197.112:443: read: connection reset by peer
================================================================================
由于无法指定mirror, 原版的minikube无法下载需要的软件,所以没法继续.

改用aliyun 版本的minikube

[alex@abc]$curl -Lo minikube http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v0.30.0/minikube-linux-amd64

碰到了kubelet 状态异常的问题.

troubleshooting

root@alex:/home/alex/abc #systemctl status kubelet -l

root@alex:/home/alex/abc #journalctl -xefu kubelet 

kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

原因是kubelet与docker使用的 cgroupfs不同,首先尝试修改kubelet的配置
root@alex:/etc/systemd/system/kubelet.service.d #vi 10-kubeadm.conf
  --cgroup-driver 修改为  =systemd

问题还是发生,因为发现kubeadm会restart kubelet

[alex@~]$systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Fri 2018-10-19 22:22:04 CST; 6s ago
     Docs: https://kubernetes.io/docs/
  Process: 5645 ExecStart=/usr/bin/kubelet --cluster-dns=10.96.0.10 --authorization-mode=Webhook --client-ca-file=/var/lib/minikube/certs/ca.crt --cadvisor-port=0 --fail-swap-on=false --pod-manifest-path=/etc/kubernetes/manifests --hostname-override=minikube --allow-privileged=true --cluster-domain=cluster.local --cgroup-driver=cgroupfs --kubeconfig=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf (code=exited, status=255)
 Main PID: 5645 (code=exited, status=255)

只能修改 docker 的cgroup-driver配置,修改很成功

$ vim /lib/systemd/system/docker.service 将 --exec-opt native.cgroupdriver=systemd 修改为：--exec-opt native.cgroupdriver=cgroupfs

start again

root@alex:/home/alex/abc #./minikube.sh 
.......
WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS
	The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks

Loading cached images from config file.
检查一下状态
root@alex:/home/alex/abc #minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.0.101

可以看到minikube下载/运行了很多images
[alex@base]$docker images

[alex@base]$docker ps -a
CONTAINER ID        IMAGE                                                                                                                                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
e87305073a6e        registry.cn-hangzhou.aliyuncs.com/google_containers/coredns@sha256:a905b50f8ecddf67cd1ad92ef95f8547aea9fd1970359e00c16aa0ab1d517baa                      "/coredns -conf /e..."   28 minutes ago      Up 28 minutes                           k8s_coredns_coredns-5bfd87b64b-8l4sp_kube-system_e1f8077c-d3f8-11e8-a152-00e06f69c642_0
2f70bdb9a83b        registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64@sha256:1d2e1229a918f4bc38b5a3f9f5f11302b3e71f8397b492afac7f273a0008776a   "/dashboard --inse..."   28 minutes ago      Up 28 minutes                           k8s_kubernetes-dashboard_kubernetes-dashboard-866c7586d-bctvw_kube-system_e16384c4-d3f8-11e8-a152-00e06f69c642_0
b089eda4be39        registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a          "/storage-provisioner"   28 minutes ago      Up 28 minutes                           k8s_storage-provisioner_storage-provisioner_kube-system_e16e436b-d3f8-11e8-a152-00e06f69c642_0
a556ade3370c        registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64@sha256:a08b7f5ce38f736da9aca5d9df8ea5e356a01a51ad9ffd70027aba62a941fdfb             "/usr/local/bin/ku..."   28 minutes ago      Up 28 minutes                           k8s_kube-proxy_kube-proxy-5ftfb_kube-system_df7c6342-d3f8-11e8-a152-00e06f69c642_0
e4c6b1a09dac        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 28 minutes ago      Up 28 minutes                           k8s_POD_coredns-5bfd87b64b-8l4sp_kube-system_e1f8077c-d3f8-11e8-a152-00e06f69c642_0
3bb9a286eeab        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 28 minutes ago      Up 28 minutes                           k8s_POD_storage-provisioner_kube-system_e16e436b-d3f8-11e8-a152-00e06f69c642_0
92fa0b8466f8        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 28 minutes ago      Up 28 minutes                           k8s_POD_kubernetes-dashboard-866c7586d-bctvw_kube-system_e16384c4-d3f8-11e8-a152-00e06f69c642_0
18a2204c0dec        6f7f2dc7fab5                                                                                                                                             "/sidecar --v=2 --..."   29 minutes ago      Up 29 minutes                           k8s_sidecar_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
22a7b1f75236        c2ce1ffb51ed                                                                                                                                             "/dnsmasq-nanny -v..."   29 minutes ago      Up 29 minutes                           k8s_dnsmasq_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
49d935f03e80        80cc5ea4b547                                                                                                                                             "/kube-dns --domai..."   29 minutes ago      Up 29 minutes                           k8s_kubedns_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
a9e52f3944a2        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_kube-dns-b4bd9576-xcgnw_kube-system_df7d2299-d3f8-11e8-a152-00e06f69c642_0
8acee2605b73        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_kube-proxy-5ftfb_kube-system_df7c6342-d3f8-11e8-a152-00e06f69c642_0
6729fbb00727        52920ad46f5b                                                                                                                                             "etcd --listen-cli..."   29 minutes ago      Up 29 minutes                           k8s_etcd_etcd-minikube_kube-system_297504360d343dfdf5aeec497ed68de6_0
44d59e849826        704ba848e69a                                                                                                                                             "kube-scheduler --..."   29 minutes ago      Up 29 minutes                           k8s_kube-scheduler_kube-scheduler-minikube_kube-system_6842c2514f06b2c32c75c69be9c6618a_0
98a224697f05        af20925d51a3                                                                                                                                             "kube-apiserver --..."   29 minutes ago      Up 29 minutes                           k8s_kube-apiserver_kube-apiserver-minikube_kube-system_c0f55fe15a4c62b7baa15edc14baa3a2_0
6f219881f2a6        ad86dbed1555                                                                                                                                             "kube-controller-m..."   29 minutes ago      Up 29 minutes                           k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_5a3aced2c0bbcdcc542d4e179bbb3593_0
8c8bbdad6172        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_etcd-minikube_kube-system_297504360d343dfdf5aeec497ed68de6_0
06dc3a7abda2        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_kube-scheduler-minikube_kube-system_6842c2514f06b2c32c75c69be9c6618a_0
a8449fda08f8        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_kube-apiserver-minikube_kube-system_c0f55fe15a4c62b7baa15edc14baa3a2_0
2adaa8ff3da6        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 29 minutes ago      Up 29 minutes                           k8s_POD_kube-controller-manager-minikube_kube-system_5a3aced2c0bbcdcc542d4e179bbb3593_0
bcc84c2a05d7        9c16409588eb                                                                                                                                             "/opt/kube-addons.sh"    30 minutes ago      Up 30 minutes                           k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_e7f9b63d8d56a24739db3329dc512f82_0
6ec73f69b30b        k8s.gcr.io/pause-amd64:3.1                                                                                                                               "/pause"                 30 minutes ago      Up 30 minutes                           k8s_POD_kube-addon-manager-minikube_kube-system_e7f9b63d8d56a24739db3329dc512f82_0
[alex@base]$


检查网络的话可以看到 none driver
[alex@base]$docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
ed6663bfdf39        bridge              bridge              local
7c26b74fdab4        host                host                local
b4cb21683a90        none                null                local

you can use the --extra-config flag on the minikube start command.

minikube start --extra-config=apiserver.v=10 --extra-config=kubelet.max-pods=100

实际是传给 The kubeadm bootstrapper 

[命令行]尝试创建一个test deployment:hello

root@alex:/home/alex/base #kubectl run hello --image=docker.io/mongo   

dashboard 可以看到deployment/pods/Replica Sets 增加了 hello 相关内容

[命令行]创建service

root@alex:/home/alex/base # kubectl expose deployment hello --port=80 --target-port=8000

部署mongodb

kubectl run mongodb  --image=docker.io/mongo  --env="p='27017:27017'" --env="v='/home/alex/mongodb:/data'" 
kubectl expose deployment mongodb --port=27017 --target-port=27017

从网页上找到 service 对应的 ip 地址: 10.109.21.194

测试mongodb端口,成功

用python 访问mongodb数据库成功.

minikube addon: kubernetes/minikube

root@minikube:/home/alex/base/DevOps/minikube #minikube addons list
- addon-manager: enabled
- coredns: enabled
- dashboard: enabled
.............

用dashboard deploy 一个app(2 pods)

对应到两个container

[alex@~]$docker ps 
CONTAINER ID        IMAGE                                                                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
db5f927cf0e3        docker.io/mongo@sha256:c4e8225e68348b18283cf5c523f99122426fe1e15d883104e24c61bedb4b2ca7   "docker-entrypoint..."   10 minutes ago      Up 10 minutes                           k8s_mongodba_mongodba-6d5d6ddf64-r6f69_default_8e227c01-d5d2-11e8-a0d4-00e06f69c642_0
e0506d12d49e        docker.io/mongo@sha256:c4e8225e68348b18283cf5c523f99122426fe1e15d883104e24c61bedb4b2ca7   "docker-entrypoint..."   10 minutes ago      Up 10 minutes                           k8s_mongodba_mongodba-6d5d6ddf64-jw4fv_default_8e23d7e3-d5d2-11e8-a0d4-00e06f69c642_0

内部只是以 tcp port 来区分.

尝试用deploymet-scale 到3个pods 或一个pods

按 kubernetes/minikube 用 yaml 语法创建volume

attach to the pod:

[alex@~]$kubectl proxy --port=8080
then:

[alex@minikube]$kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
mongodba-6d5d6ddf64-jw4fv   1/1     Running   0          16h

可能因为minikube 是root身份start的,如下命令也只能以root 运行.
root@minikube:/home/alex/base/DevOps/minikube #kubectl exec -it mongodba-6d5d6ddf64-jw4fv bash

some query

root@minikube:/home/alex/base/DevOps/minikube #kubectl config view

root@minikube:/home/alex/base/DevOps/minikube #kubectl cluster-info

查namesapce
root@minikube:/home/alex/base/DevOps/minikube #kubectl get po,svc -n kube-system

point the ‘docker’ command to your Minikube’s Docker daemon 失败

root@minikube:/home/alex/base/DevOps/minikube #(minikube docker-env)
'none' driver does not support 'minikube docker-env' command

enable heapster:

root@alex:/home/alex/base/DevOps/minikube #minikube addons enable heapster
heapster was successfully enabled

然后dashboard 就可以看到如下load 统计了:

stop minikube cluster:

root@alex:/home/alex/base/DevOps #minikube stop
Stopping local Kubernetes cluster...
Machine stopped.

docker⽤户过渡到kubectl命令⾏指南

$ docker run -d --restart=always -e DOMAIN=cluster --name nginxapp -p 80:80 nginx

kubectl

$ kubectl run --image=nginx nginx-app --port=80 --env="DOMAIN=cluster"

$ kubectl expose deployment nginx-app --port=80 --name=nginx-http

对应docker exec

$ kubectl exec -ti nginx-app-5jyvm -- /bin/sh

最终宝典:

https://github.com/kubernetes/minikube/tree/master/docs