Istio: Traffic Management for your Microservices Practices

Istio: Traffic Management for your Microservices  
https://github.com/IBM/microservices-traffic-management-using-istio?cm_mc_uid=13613521364015024112758&cm_mc_sid_50200000=1503073248#3-traffic-flow-management-using-istio-pilot---modify-service-routes

Install kubetcl
https://kubernetes.io/docs/tasks/tools/install-kubectl/
Kubernetes 安装
http://wiki.ling2.cn/index.php/Kubernetes_%E5%AE%89%E8%A3%85

Install Lstio CLI >>>>>>
https://github.com/IBM/microservices-traffic-management-using-istio/blob/master/GETTING_STARTED.md

Install kubectl binary via curl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Install with Homebrew on macOS
brew install kubectl
kubectl version

https://stackoverflow.com/questions/37921222/kubectl-connection-to-server-was-refused
 
HaideMacBook-Pro:am-reporting-microservice$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

After install the Minikube component, the error message has changed below:

HaideMacBook-Pro:.kube$ kubectl version -s https://192.168.99.101:8443

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T17:03:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: Bad Request

Root Cause:
HaideMacBook-Pro:.kube$ sudo kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-07-26T00:12:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}


minikube >>>>>>>>>>
https://github.com/kubernetes/minikube/blob/v0.21.0/README.md

What is Minikube?
Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.

HaideMacBook-Pro:~$ minikube start
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Downloading Minikube ISO
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

HaideMacBook-Pro:.kube$ pwd
/Users/<home-user>/.kube
HaideMacBook-Pro:.kube$ cat config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/<home-user>/.minikube/ca.crt
    server: https://192.168.99.101:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /Users/<home-user>/.minikube/apiserver.crt
    client-key: /Users/<home-user>/.minikube/apiserver.key

minikube ip
minikube logs
Error:
Aug 24 12:29:42 minikube localkube[3460]: E0824 12:29:42.292211    3460 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)" failed: rpc error: code = 2 desc = unable to pull sandbox image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: {"message":"Get https://gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout"}

Solution:
docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0

docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/echoserver:1.4
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/echoserver:1.4

Error:
Aug 24 16:23:22 minikube localkube[4096]: E0824 16:23:22.173003    4096 kuberuntime_manager.go:843] PodSandboxStatus of sandbox "00a8841d5906013fb342a1d0d284d25e51ce692f60edbb167de4686e3dc7e71e" for pod "kube-addon-manager-minikube_kube-system(c654b2f084cf26941c334a2c3d6db53d)" error: rpc error: code = 2 desc = Error: No such container: 00a8841d5906013fb342a1d0d284d25e51ce692f60edbb167de4686e3dc7e71e
Aug 24 16:23:22 minikube localkube[4096]: E0824 16:23:22.173175    4096 generic.go:241] PLEG: Ignoring events for pod kube-addon-manager-minikube/kube-system: rpc error: code = 2 desc = Error: No such container: 00a8841d5906013fb342a1d0d284d25e51ce692f60edbb167de4686e3dc7e71e


>>>>>> Practice minikube article
https://kubernetes.io/docs/getting-started-guides/minikube/

HaideMacBook-Pro:.kube$ kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
error: failed to discover supported resources: Get https://192.168.99.101:8443/api: Bad Request

HaideMacBook-Pro:.kube$ kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
error: failed to discover supported resources: Get https://192.168.99.101:8443/api: read tcp 192.168.31.241:53959->60.19.29.21:8080: read: operation timed out


sudo kubectl get pods
HaideMacBook-Pro:SourceCodes liuhaidl$ sudo kubectl get pods
NAME                             READY     STATUS              RESTARTS   AGE
hello-minikube-180744149-vg3qq   1/1       Running             1          28m
istio-egress-664558847-7dtlh     0/1       ContainerCreating   0          6m
istio-ingress-1292229662-22pzj   0/1       ContainerCreating   0          6m
istio-mixer-2555573127-lt0p6     0/1       ContainerCreating   0          6m
istio-pilot-4143248751-0v0q9     0/2       ContainerCreating   0          6m

HaideMacBook-Pro:SourceCodes liuhaidl$ sudo kubectl get deployments
NAME             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-minikube   1         1         1            1           28m
istio-egress     1         1         1            0           7m
istio-ingress    1         1         1            0           7m
istio-mixer      1         1         1            0           7m
istio-pilot      1         1         1            0           7m

aideMacBook-Pro:SourceCodes$ sudo kubectl describe pod istio-egress-664558847-7dtlh
Name:        istio-egress-664558847-7dtlh
Namespace:    default
Node:        minikube/192.168.99.101
Start Time:    Fri, 25 Aug 2017 01:12:12 +0800
Labels:        istio=egress
        pod-template-hash=664558847
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"istio-egress-664558847","uid":"5e8a0ed8-88ef-11e7-9f88-080027490...
Status:        Pending
IP:        
Created By:    ReplicaSet/istio-egress-664558847
Controlled By:    ReplicaSet/istio-egress-664558847
Containers:
  proxy:
    Container ID:    
    Image:        docker.io/istio/proxy_debug:0.1.6
    Image ID:        
    Port:        <none>
    Args:
      proxy
      egress
      -v
      2
    State:        Waiting
      Reason:        ContainerCreating
    Ready:        False
    Restart Count:    0
    Environment:
      POD_NAMESPACE:    default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-10tld (ro)
Conditions:
  Type        Status
  Initialized     True
  Ready     False
  PodScheduled     True
Volumes:
  default-token-10tld:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-10tld
    Optional:    false
QoS Class:    BestEffort
Node-Selectors:    <none>
Tolerations:    <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath        Type        Reason            Message
  ---------    --------    -----    ----            -------------        --------    ------            -------
  8m        8m        1    default-scheduler                Normal        Scheduled        Successfully assigned istio-egress-664558847-7dtlh to minikube
  8m        8m        1    kubelet, minikube                Normal        SuccessfulMountVolume    MountVolume.SetUp succeeded for volume "default-token-10tld"
  8m        8m        1    kubelet, minikube    spec.containers{proxy}    Normal        Pulling            pulling image "docker.io/istio/proxy_debug:0.1.6"

istio-egress-664558847-7dtlh     0/1       ContainerCreating   0          6m
istio-ingress-1292229662-22pzj   0/1       ContainerCreating   0          6m
istio-mixer-2555573127-lt0p6     0/1       ContainerCreating   0          6m
istio-pilot-4143248751-0v0q9     0/2       ContainerCreating   0          6m

docker.io/istio/pilot:0.1.6
docker.io/istio/proxy_debug:0.1.6
docker.io/istio/mixer:0.1.6

Sample >>>>>>
docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/echoserver:1.4
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/echoserver:1.4

$ docker pull docker.io/istio/pilot:0.1.6
0.1.6: Pulling from istio/pilot
f192b9e1e0ee: Downloading [===========>                                       ]  10.8 MB/47.78 MB
48f388fded4e: Download complete

>>>>>> Based on the above testing, we can know we can pull the docker image from docker.io without any problem. However we cannot access
the gcr.io website to pull the docker image(only very slow)!!!!!!

mikibue ssh >>>>>>
docker images
istio/proxy_debug                                                 0.1.6               5623de9317ff        292 years ago       825 MB
istio/mixer                                                       0.1.6               1f4a2ce90af6        292 years ago       158.9 MB
istio/pilot                                                       0.1.6               e0c24bd68c04        292 years ago       144.4 MB

HaideMacBook-Pro:SourceCodes$ sudo kubectl get pods
Password:
NAME                             READY     STATUS    RESTARTS   AGE
hello-minikube-180744149-vg3qq   1/1       Running   1          8h
istio-egress-664558847-7dtlh     1/1       Running   0          8h
istio-ingress-1292229662-22pzj   1/1       Running   0          8h
istio-mixer-2555573127-lt0p6     1/1       Running   0          8h
istio-pilot-4143248751-0v0q9     2/2       Running   0          8h

HaideMacBook-Pro:SourceCodes$ sudo kubectl apply -f istio/samples/apps/bookinfo/bookinfo.yaml
Password:
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created

HaideMacBook-Pro:SourceCodes$ sudo kubectl get deployments
NAME             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
details-v1       1         1         1            0           1m
hello-minikube   1         1         1            1           8h
istio-egress     1         1         1            1           8h
istio-ingress    1         1         1            1           8h
istio-mixer      1         1         1            1           8h
istio-pilot      1         1         1            1           8h
productpage-v1   1         1         1            0           1m
ratings-v1       1         1         1            0           1m
reviews-v1       1         1         1            0           1m
reviews-v2       1         1         1            0           1m
reviews-v3       1         1         1            0           1m

HaideMacBook-Pro:SourceCodes$ sudo kubectl get pods
NAME                             READY     STATUS              RESTARTS   AGE
details-v1-3041109565-dnm6l      0/1       ImagePullBackOff    0          2m
hello-minikube-180744149-vg3qq   1/1       Running             1          8h
istio-egress-664558847-7dtlh     1/1       Running             0          8h
istio-ingress-1292229662-22pzj   1/1       Running             0          8h
istio-mixer-2555573127-lt0p6     1/1       Running             0          8h
istio-pilot-4143248751-0v0q9     2/2       Running             0          8h
productpage-v1-924765411-95t55   0/1       ContainerCreating   0          2m
ratings-v1-3236500269-rl3l8      0/1       ContainerCreating   0          2m
reviews-v1-1515829678-sh6kw      0/1       ContainerCreating   0          2m
reviews-v2-3456803578-r4jvl      0/1       ContainerCreating   0          2m
reviews-v3-1174163554-s5wrm      0/1       ContainerCreating   0          2m

minikube ssh
docker pull istio/examples-bookinfo-productpage-v1 = docker.io/istio/examples-bookinfo-productpage-v1 <<< This is very slow for image to download
docker pull registry.cn-hangzhou.aliyuncs.com/examples-bookinfo-productpage-v1

HTTP_PROXY=http://<proxy-dns>:8080

$ ps -ef|grep docker
root      3291     1  0 02:53 ?        00:00:12 dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.0.0.0/24

sudo systemctl restart docker
ExecStart=
ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.0.0.0/24

$ docker search examples-bookinfo-productpage-v1
Error response from daemon: Get https://index.docker.io/v1/search?q=examples-bookinfo-productpage-v1&n=25: http: error connecting to proxy http://<proxy-dns>:8080: dial tcp 60.19.29.21:8080: i/o timeout
           <proxy-dns>:8080

HaideMacBook-Pro:~$ ping proxy.emea.ibm.com //ping in the host machine
PING proxy.emea.ibm.com (9.139.246.81): 56 data bytes
64 bytes from 9.139.246.81: icmp_seq=0 ttl=246 time=305.314 ms
64 bytes from 9.139.246.81: icmp_seq=1 ttl=246 time=324.450 ms
64 bytes from 9.139.246.81: icmp_seq=2 ttl=246 time=266.806 ms

HaideMacBook-Pro:~$ ping 9.139.246.81 //ping in the host machine
PING 9.139.246.81 (9.139.246.81): 56 data bytes
64 bytes from 9.139.246.81: icmp_seq=0 ttl=246 time=282.739 ms
64 bytes from 9.139.246.81: icmp_seq=1 ttl=246 time=302.533 ms
64 bytes from 9.139.246.81: icmp_seq=2 ttl=246 time=267.423 ms

$ ping 9.139.246.81 //ping in the minikube VM
PING 9.139.246.81 (9.139.246.81): 56 data bytes
64 bytes from 9.139.246.81: seq=0 ttl=63 time=269.231 ms
64 bytes from 9.139.246.81: seq=1 ttl=63 time=279.411 ms
64 bytes from 9.139.246.81: seq=2 ttl=63 time=302.871 ms

$ ping proxy.emea.ibm.com //ping based on dns in the minikube VM
PING proxy.emea.ibm.com (60.19.29.21): 56 data bytes


Solution for the minikube VM >>>>>>
Refer this CSDN article for solution >>>
http://blog.csdn.net/chang_harry/article/details/52116305

1. sudo vi /etc/systemd/system/docker.service.d/10-machine.conf
    #Environment=HTTP_PROXY=http://proxy.emea.ibm.com:8080
    Environment=HTTP_PROXY=http://9.139.246.81:8080
   //Flush changes:
   2. sudo systemctl daemon-reload
  //Restart Docker:
   3. sudo systemctl restart docker
  //Try the following command to see if the proxy works for us or not
   4. docker pull istio/examples-bookinfo-productpage-v1

sudo vi /etc/hosts
9.139.246.81    proxy.emea.ibm.com

ping -c 1  proxy.emea.ibm.com

$ ping -c 1  proxy.emea.ibm.com
PING proxy.emea.ibm.com (9.139.246.81): 56 data bytes

64 bytes from 9.139.246.81: seq=0 ttl=63 time=266.323 ms
--- proxy.emea.ibm.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 266.323/266.323/266.323 ms

istio/examples-bookinfo-reviews-v3                                latest              41857f14cdf2        4 months ago        444.8 MB
istio/examples-bookinfo-reviews-v2                                latest              a2e4ca83561b        4 months ago        444.8 MB
istio/examples-bookinfo-reviews-v1                                latest              1011aea1c7c0        4 months ago        444.8 MB
istio/examples-bookinfo-productpage-v1                            latest              a5dc661788bf        4 months ago        691.3 MB
istio/examples-bookinfo-ratings-v1                                latest              5ffcd281499b        6 months ago        653.3 MB
istio/examples-bookinfo-details-v1                                latest              e919c1705bae        6 months ago        731.1 MB

Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath            Type        Reason            Message
  ---------    --------    -----    ----            -------------            --------    ------            -------
  23m        23m        1    default-scheduler                    Normal        Scheduled        Successfully assigned productpage-v1-924765411-2s6g9 to minikube
  23m        23m        1    kubelet, minikube                    Normal        SuccessfulMountVolume    MountVolume.SetUp succeeded for volume "default-token-10tld"
  23m        23m        1    kubelet, minikube    spec.containers{productpage}    Normal        Pulling            pulling image "istio/examples-bookinfo-productpage-v1"
  15m        15m        1    kubelet, minikube    spec.containers{productpage}    Normal        Pulled            Successfully pulled image "istio/examples-bookinfo-productpage-v1"
  15m        15m        1    kubelet, minikube    spec.containers{productpage}    Normal        Created            Created container
  15m        15m        1    kubelet, minikube    spec.containers{productpage}    Normal        Started            Started container

2017-08-28 >>>>>>
sudo kubectl get services
productpage     10.0.0.190   <nodes>       9080:31667/TCP                2m

Issue >>>
/dev/fd/63: bad file descriptor
Solution >>>
sudo chown -Rf liuhaidl:staff .kube/
HaideMacBook-Pro:SourceCodes liuhaidl$ ls -lrt /Users/liuhaidl/.kube/schema/v1.7.0/api/v1/schema.json
-rw-------  2 liuhaidl  staff  747926  8 25 01:12 /Users/liuhaidl/.kube/schema/v1.7.0/api/v1/schema.json

Access your application >>>
How to get the vm ip address:
HaideMacBook-Pro:SourceCodes liuhaidl$ minikube ip
192.168.99.100

How to get the licensing port:
HaideMacBook-Pro:SourceCodes liuhaidl$ kubectl get services|grep ingress
istio-ingress   10.0.0.166   <pending>     80:31843/TCP,443:31177/TCP    3d

The final access URL for product main page:
http://192.168.99.100:31843/productpage

Grafana GUI:
http://192.168.99.100:30678/dashboard/db/istio-dashboard
HaideMacBook-Pro:SourceCodes liuhaidl$ kubectl get svc | grep grafana
grafana         10.0.0.26    <pending>     3000:30678/TCP                18s

 for i in {1..5}; do echo -n .; curl -s http://192.168.99.100:31843/productpage > /dev/null; done

Zipkin GUI:
kubectl get svc | grep zipkin
HaideMacBook-Pro:SourceCodes liuhaidl$ kubectl get svc | grep zipkin
zipkin          10.0.0.209   <pending>     9411:32227/TCP                39m
http://192.168.99.100:32227

Issue:
sometime you cannot login productpage
Solution:
1. minikube ssh
2. sudo systemctl restart docker

Issue:
sometime the grafana and zipkin cannot log the message
Solution:
  kubectl delete -f istio/samples/apps/bookinfo/bookinfo.yaml
  kubectl apply -f <(istioctl kube-inject -f istio/samples/apps/bookinfo/bookinfo.yaml)

  istioctl delete -f istio/samples/apps/bookinfo/route-rule-all-v1.yaml
  istioctl create -f istio/samples/apps/bookinfo/route-rule-all-v1.yaml
  istioctl delete -f istio/samples/apps/bookinfo/route-rule-reviews-test-v2.yaml
  istioctl create -f istio/samples/apps/bookinfo/route-rule-reviews-test-v2.yaml
  istioctl replace -f istio/samples/apps/bookinfo/route-rule-reviews-50-v3.yaml

  //We don’t need to remove the prometheus component
  kubectl delete -f istio/install/kubernetes/addons/prometheus.yaml
  kubectl apply -f istio/install/kubernetes/addons/prometheus.yaml

  //We don’t need to remove the grafana component
  kubectl delete -f istio/install/kubernetes/addons/grafana.yaml
  kubectl apply -f istio/install/kubernetes/addons/grafana.yaml
 
 //The following are the key steps we will fix the grafana and zipkin cannot log the message
 istioctl mixer rule create global ratings.default.svc.cluster.local -f istio/samples/apps/bookinfo/mixer-rule-ratings-denial.yaml
 istioctl mixer rule get reviews.default.svc.cluster.local reviews.default.svc.cluster.local
 istioctl mixer rule create reviews.default.svc.cluster.local reviews.default.svc.cluster.local -f demo/new-metrics-rule.yaml