k8s集群终于跑正常了

redis和docker这两个词语会自动被修改为首字母大写并链接到知识库，所以在这里先写一遍，后面就不会被改写了。

0、具体操作见 （vmware中搭建k8s），virtulbox也是相同的流程。

1、学习k8s，读的是这本书《KUBERNETES权威指南  从DOCKET到KURBERNETES实践全接触.pdf》

2、这边书刚开始讲的是在单机上搭建一个k8s的hello world，用的是centos7.

    于是我也在virtulbox中安装了centos7，并成功运行hello world。

3、然后，开始尝试集群了。在网上找了个教程，比较简单，很快就成功了，kubectl get nodes能看到各node了。

    但是，应用跑的不正常，有的节点可以访问，有的节点不可以。而且从不同节点访问，查到的数据不相同，似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。   

     接着，集群坏了，k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时，用的是链接式拷贝，是不是原始的虚机安装了其它软件，导致k8s集群启动不了。

4、删掉重来，用完全拷贝的方式建立虚机。

     一切正常，但是在启动redis-master-controller.yaml时，docker中无法建立容器，

     用kubectl describe pod redis-master命令排查，发现下拉不了镜像。其实这些镜像已经存在于docker中了（搭建单机k8s时，自动pull的）。只是镜像名字前面多加了docker.io/  我修改了redis-master-controller.yaml中镜像的名字，仍然pull失败。奇怪。

5、今天定位了网络问题（见virtualbox虚拟机无法上网），网络搞通后，hello world终于正常运行了。

之前的一些疑问，有答案了

1、随便访问哪个node的ip（比如http://192.168.56.251:30001/,http://192.168.56.252:30001/）(http://192.168.56.250:30001/是不行的)，都可以访问到服务的（proxy自动转的）

2、在内部时，访问真实端口也是可以的。

[root@centm ~]# kubectl get svc
NAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
frontend       10.254.218.57    <nodes>       80/TCP     5m
kubernetes     10.254.0.1       <none>        443/TCP    15d
redis-master   10.254.142.174   <none>        6379/TCP   8m
redis-slave    10.254.201.123   <none>        6379/TCP   6m

curl 10.254.218.57:80     可以通。

ping  10.254.218.57 不通

3、node中多了一个127.0.0.1 ，不知道为什么

[root@centm ~]# kubectl get nodes
NAME        STATUS     AGE
127.0.0.1   NotReady   15d
cents1      Ready      1d
cents2      Ready      1d
[root@centm ~]# ps -ef|grep kube
kube       578     1  0 Jan20 ?        00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube       588     1  0 Jan20 ?        00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube      2079     1  0 Jan20 ?        00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
4、clusterip是一个虚ip，实际是iptables中的几个转发规则。

[root@cents2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 82058sec preferred_lft 82058sec
    inet6 fe80::b171:84d0:5173:de63/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7a:2414/64 scope link
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff
    inet 172.16.80.0/16 scope global flannel.1
       valid_lft forever preferred_lft forever
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff
    inet 172.16.80.1/24 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:72ff:fe2f:1fae/64 scope link
       valid_lft forever preferred_lft forever
7: vethc56c1d4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::90c8:3dff:fe3f:b949/64 scope link
       valid_lft forever preferred_lft forever
9: vethf961994@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::d4be:4bff:fe6e:2681/64 scope link
       valid_lft forever preferred_lft forever
11: vethe4cd28e@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
       valid_lft forever preferred_lft forever

[root@cents2 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*filter
:INPUT ACCEPT [27:4324]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [25:2821]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-SERVICES - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Sun Jan 22 00:41:01 2017
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2:119]
:POSTROUTING ACCEPT [2:119]
:DOCKER - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]
:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]
:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]
:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]
:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]
:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]
:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80
-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:80
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:6379
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:80
-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
COMMIT
# Completed on Sun Jan 22 00:41:01 2017

----------------------------------------------------

尝试了本地卷

[root@centm ~]# cat redis-master-controller_with_volume.yaml
apiVersion: v1
kind: ReplicationController  
metadata:                               
  labels:edis-master  
    name: redis-master
spec:
  replicas: 1
  selector:
    name: redis-master
  template:    
    metadata:
      labels:
        name: redis-master
    spec:
        volumes:
        - name: "gf-dir1"
          hostPath:
            path: "/tmp"

        containers:
        - name: master          
          ports:: docker.io/kubeguide/redis-master:latest   
          - containerPort : 6379
          volumeMounts:
          - name: "gf-dir1"
            mountPath: "/gf1"