Vitess Operator for minikube

Vitess Operator for minikube

Vitess是什么
Vitess是一个用于部署，扩展和管理大型开源数据库实例集群的数据库解决方案，通俗点说，就是在云环境下对大型数据库及数据库集群进行分库分表并提供可靠性处理的方案,详情可参见Vitess官方文档。
Vitess目前支持数据库类型如下：
• Mysql(5.6-8.0)；
• Percona(5.6-8.0)；
• MariaDB(10.0-10.3)
Vitess在github开源，是CNCF少有的毕业项目之一,其项目地址Vitess Github。

minikube环境配置

基础环境配置

# 关闭swap
swapoff -a
注释/etc/fstab里面swap的配置
# 关闭selinux
setenforce 0
将/etc/sysconfig/selinux里面的SELINUX=enforcing修改为SELINUX=disabled
# 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
#修改/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
# 添加hostname
编辑/etc/hosts，添加本机的hostname

docker安装

# step 1: 安装必要的一些系统工具
yum install -y yum-utils device-mapper-persistent-data lvm2

# Step 2: 添加软件源信息
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo


# Step 3: 更新并安装 Docker-CE

yum makecache fast
# Step 3.2 : 安装指定版本的Docker-CE: (VERSION 例如上面的 17.03.0.ce.1-1.el7.centos)
yum -y --setopt=obsoletes=0 install docker-ce-18.06.1.ce-3.el7

# Step 4:设置daemon为systemd
# 此操作是解决k8s init时的一个warning，
# https://kubernetes.io/docs/setup/production-environment/container-runtimes/
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
 {
 "registry-mirrors": ["https://otvjuhk6.mirror.aliyuncs.com"]
 }
EOF

# Step 5: 开启Docker服务
systemctl enable docker && systemctl start docker

安装minikube

参考https://juejin.cn/post/6844903894871457799

curl -Lo minikube http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.2.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

minikube start --vm-driver=none
😄  minikube v1.2.0 on linux (amd64)
✅  using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
🔥  Creating none VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
🐳  Configuring environment for Kubernetes v1.15.0 on Docker 18.06.1-ce
🚜  Pulling images ...
🚀  Launching Kubernetes ...
🤹  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
👉  https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may
⚠️  need to relocate them. For example, to overwrite your own settings:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Verifying: apiserver proxy etcd
 scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"
💡  For best results, install kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/
#注释： 如果一次不成功，需要根据提示修改对应内容，并执行 minikube delete后再次执行minikube start

安装命令行工具kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubectl-1.18.6 

检查集群状态

[root@10-10-30-127 ~]# kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
coredns-6967fb4995-bgnkb           1/1     Running   0          3m
coredns-6967fb4995-xvpx6           1/1     Running   0          3m
etcd-minikube                      1/1     Running   0          2m9s
kube-addon-manager-minikube        1/1     Running   0          3m8s
kube-apiserver-minikube            1/1     Running   0          109s
kube-controller-manager-minikube   1/1     Running   0          2m1s
kube-proxy-7k22t                   1/1     Running   0          3m1s
kube-scheduler-minikube            1/1     Running   0          115s
storage-provisioner                1/1     Running   0          3m

[root@10-10-30-127 ~]# kubectl get nodes
NAME       STATUS   ROLES    AGE     VERSION
minikube   Ready    master   3m27s   v1.15.0

安装vitess

安装方式，使用kubernetes vitess operator安装

参考官方文档

Vitess Operator for Kubernetes

下载v8.0.0版本

wget https://github.com/vitessio/vitess/archive/v8.0.0.tar.gz

解压并查看文件

[root@10-10-30-181 opt]# ll
总用量 15776
-rw-r--r--.  1 root root 16148564 12月 12 14:56 v8.0.0.tar.gz
drwxrwxr-x. 20 root root     4096 10月 27 21:56 vitess-8.0.0
[root@10-10-30-181 opt]# cd vitess-8.0.0/
[root@10-10-30-181 vitess-8.0.0]# ll
总用量 240
-rw-rw-r--.  1 root root   832 10月 27 21:56 ADOPTERS.md
-rwxrwxr-x.  1 root root  8012 10月 27 21:56 bootstrap.sh
-rwxrwxr-x.  1 root root  1751 10月 27 21:56 build.env
-rw-rw-r--.  1 root root   139 10月 27 21:56 CODE_OF_CONDUCT.md
drwxrwxr-x.  6 root root   134 10月 27 21:56 config
-rw-rw-r--.  1 root root   711 10月 27 21:56 CONTRIBUTING.md
drwxrwxr-x.  3 root root    18 10月 27 21:56 data
-rw-rw-r--.  1 root root  1421 10月 27 21:56 DCO
-rw-rw-r--.  1 root root  1478 10月 27 21:56 dev.env
drwxrwxr-x.  4 root root  4096 10月 27 21:56 doc
drwxrwxr-x. 11 root root   170 10月 27 21:56 docker
lrwxrwxrwx.  1 root root    22 10月 27 21:56 Dockerfile -> docker/base/Dockerfile
drwxrwxr-x. 11 root root   152 10月 27 21:56 examples
drwxrwxr-x. 37 root root  4096 10月 27 21:56 go
-rw-rw-r--.  1 root root  5610 10月 27 21:56 go.mod
-rw-rw-r--.  1 root root 87572 10月 27 21:56 go.sum
-rw-rw-r--.  1 root root 10748 10月 27 21:56 GOVERNANCE.md
-rw-rw-r--.  1 root root  1508 10月 27 21:56 GUIDING_PRINCIPLES.md
drwxrwxr-x.  3 root root    55 10月 27 21:56 helm
drwxrwxr-x.  6 root root   150 10月 27 21:56 java
-rw-rw-r--.  1 root root 11358 10月 27 21:56 LICENSE
-rw-rw-r--.  1 root root  1882 10月 27 21:56 MAINTAINERS.md
-rw-rw-r--.  1 root root 13506 10月 27 21:56 Makefile
drwxrwxr-x.  3 root root    56 10月 27 21:56 misc
drwxrwxr-x.  2 root root  4096 10月 27 21:56 proto
-rw-rw-r--.  1 root root  2942 10月 27 21:56 README.md
drwxrwxr-x.  3 root root    17 10月 27 21:56 resources
-rw-rw-r--.  1 root root  5566 10月 27 21:56 SECURITY.md
-rw-rw-r--.  1 root root   421 10月 27 21:56 sonar-project.properties
drwxrwxr-x.  4 root root    54 10月 27 21:56 support
drwxrwxr-x.  4 root root  4096 10月 27 21:56 test
-rwxrwxr-x.  1 root root 21424 10月 27 21:56 test.go
drwxrwxr-x.  3 root root  4096 10月 27 21:56 tools
-rw-rw-r--.  1 root root  2273 10月 27 21:56 Vagrantfile
drwxrwxr-x.  3 root root    65 10月 27 21:56 vagrant-scripts
drwxrwxr-x.  6 root root   184 10月 27 21:56 vitess-mixin
drwxrwxr-x.  4 root root    41 10月 27 21:56 web

新建11个pv

方法如下：

for i in 1 2 3 4 5 6 7 8 9 10; do echo $i;mkdir -p /root/ch/vitess/pv/pv$i; done;
chmod 777  /root/ch/vitess/pv/*
ll /root/ch/vitess/pv
总用量 0
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv1
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv10
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv2
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv3
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv4
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv5
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv6
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv7
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv8
drwxrwxrwx. 2 root root 6 12月 12 15:27 pv9
###############################################
cat <<EOF > pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-hostpath-1                  ##对应修改
  labels:
    type: local
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/root/ch/vitess/pv/pv{1-11}" ##对应修改
EOF
################################################
kubectl apply -f pv.yaml
################################################
kubectl get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-hostpath-1    10Gi       RWO            Retain           Available                                   2m27s
pv-hostpath-10   10Gi       RWO            Retain           Available                                   24s
pv-hostpath-11   10Gi       RWO            Retain           Available                                   3s
pv-hostpath-2    10Gi       RWO            Retain           Available                                   2m12s
pv-hostpath-3    10Gi       RWO            Retain           Available                                   119s
pv-hostpath-4    10Gi       RWO            Retain           Available                                   105s
pv-hostpath-5    10Gi       RWO            Retain           Available                                   93s
pv-hostpath-6    10Gi       RWO            Retain           Available                                   74s
pv-hostpath-7    10Gi       RWO            Retain           Available                                   61s
pv-hostpath-8    10Gi       RWO            Retain           Available                                   49s
pv-hostpath-9    10Gi       RWO            Retain           Available                                   36s

安装operator

cd /opt/vitess-8.0.0/examples/operator
kubectl apply -f operator.yaml
customresourcedefinition.apiextensions.k8s.io/etcdlockservers.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackups.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackupstorages.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesscells.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessclusters.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesskeyspaces.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessshards.planetscale.com created
serviceaccount/vitess-operator created
role.rbac.authorization.k8s.io/vitess-operator created
rolebinding.rbac.authorization.k8s.io/vitess-operator created
priorityclass.scheduling.k8s.io/vitess created
priorityclass.scheduling.k8s.io/vitess-operator-control-plane created
deployment.apps/vitess-operator created

创建vitess集群

kubectl apply -f 101_initial_cluster.yaml
vitesscluster.planetscale.com/example created
secret/example-cluster-config created
##前面已经修改docker仓库为阿里仓库，由于拉取镜像时间比较久，请耐心等待
##如果有pod一直创建失败，可通过命令 kubectl describe pod podname 查看原因
创建成功后查看资源情况
kubectl get pods,svc
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/example-etcd-faf13de3-1                          1/1     Running   0          7m8s
pod/example-etcd-faf13de3-2                          1/1     Running   0          7m8s
pod/example-etcd-faf13de3-3                          1/1     Running   0          7m8s
pod/example-vttablet-zone1-2469782763-bfadd780       3/3     Running   1          7m8s
pod/example-vttablet-zone1-2548885007-46a852d0       3/3     Running   1          7m8s
pod/example-zone1-vtctld-1d4dcad0-c4586f7c8-5qwwp    1/1     Running   4          4m7s
pod/example-zone1-vtgate-bc6cde92-64bfdcfb57-bppvq   1/1     Running   0          71s
pod/vitess-operator-69b8854f64-f85qf                 1/1     Running   0          8m57s

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
service/example-etcd-faf13de3-client    ClusterIP   10.105.246.1     <none>        2379/TCP                       7m8s
service/example-etcd-faf13de3-peer      ClusterIP   None             <none>        2380/TCP                       7m8s
service/example-vtctld-625ee430         ClusterIP   10.96.58.149     <none>        15000/TCP,15999/TCP            7m8s
service/example-vtgate-ae7df4b6         ClusterIP   10.102.110.39    <none>        15000/TCP,15999/TCP,3306/TCP   7m8s
service/example-vttablet-08646a59       ClusterIP   None             <none>        15000/TCP,15999/TCP,9104/TCP   7m8s
service/example-zone1-vtgate-bc6cde92   ClusterIP   10.103.237.233   <none>        15000/TCP,15999/TCP,3306/TCP   7m8s
service/kubernetes                      ClusterIP   10.96.0.1        <none>        443/TCP                        25m


kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                STORAGECLASS   REASON   AGE
pv-hostpath-1                              10Gi       RWO            Retain           Available                                                                                13m
pv-hostpath-10                             10Gi       RWO            Retain           Available                                                                                11m
pv-hostpath-11                             10Gi       RWO            Retain           Available                                                                                10m
pv-hostpath-2                              10Gi       RWO            Retain           Available                                                                                13m
pv-hostpath-3                              10Gi       RWO            Retain           Available                                                                                12m
pv-hostpath-4                              10Gi       RWO            Retain           Available                                                                                12m
pv-hostpath-5                              10Gi       RWO            Retain           Available                                                                                12m
pv-hostpath-6                              10Gi       RWO            Retain           Available                                                                                12m
pv-hostpath-7                              10Gi       RWO            Retain           Available                                                                                11m
pv-hostpath-8                              10Gi       RWO            Retain           Available                                                                                11m
pv-hostpath-9                              10Gi       RWO            Retain           Available                                                                                11m
pvc-01d7a24e-d6c6-4092-8b0d-9bd11d0c852f   10Gi       RWO            Delete           Bound       default/example-vttablet-zone1-2548885007-46a852d0   standard                8m
pvc-804be832-3447-40db-8676-9c38a4ac4a5e   1Gi        RWO            Delete           Bound       default/example-etcd-faf13de3-1                      standard                8m3s
pvc-96f1083f-9999-47a9-8d59-27db18839d80   1Gi        RWO            Delete           Bound       default/example-etcd-faf13de3-2                      standard                8m2s
pvc-eedd49c4-9b0d-425f-abfc-4326c6f87f09   1Gi        RWO            Delete           Bound       default/example-etcd-faf13de3-3                      standard                8m3s
pvc-f1e372d4-7067-4a80-b5d6-b575828d9a16   10Gi       RWO            Delete           Bound       default/example-vttablet-zone1-2469782763-bfadd780   standard                7m59s

从vitess创建完成的资源信息可见，vitess集群部署时会有2个vttablet的实例，和3个etcd的实例，且这五个实例各自单独挂载一份local pv

创建别名

alias vtctlclient="vtctlclient -server=$(kubectl get svc  | grep example-vtctld | awk '{print $3}'):15999"
alias mysql="mysql -h $(kubectl get svc | grep example-zone1 | awk '{print $3}') -P 3306 -u user"

vtctlclient ApplySchema -sql="$(cat /opt/vitess-8.0.0/examples/operator/create_commerce_schema.sql)" commerce
vtctlclient ApplyVSchema -vschema="$(cat /opt/vitess-8.0.0/examples/operator/vschema_commerce_initial.json)" commerce
New VSchema object:
{
  "tables": {
    "corder": {

    },
    "customer": {

    },
    "product": {

    }
  }
}
If this is not what you expected, check the input data (as JSON parsing will skip unexpected fields).
#vtctlclient命令需要生成，下载方法见官方文档

拆分键空间(垂直分库)

将commerce的keyspace中的三张表进行拆分，使product单独一个keyspace,而customer和corder在另一个名为customer的keyspace.

初始化commerce的keyspace

cd /opt/vitess-8.0.0/examples/common
[root@10-10-30-127 common]# ll
总用量 20
-rw-rw-r--. 1 root root 794 10月 27 21:56 insert_commerce_data.sql
-rw-rw-r--. 1 root root 166 10月 27 21:56 select_commerce_data.sql
-rw-rw-r--. 1 root root 125 10月 27 21:56 select_customer0_data.sql
-rw-rw-r--. 1 root root 129 10月 27 21:56 select_customer-80_data.sql
-rw-rw-r--. 1 root root 129 10月 27 21:56 select_customer80-_data.sql
(0 -80 80-)为不通的分片规则，16进制，具体含义情参考官方文档
修改键空间 0为-
vim select_commerce_data.sql
\! echo 'Using commerce/0'
use commerce/0;
\! echo 'Customer'
select * from customer;
\! echo 'Product'
select * from product;
\! echo 'COrder'
select * from corder;
修改为
\! echo 'Using commerce/-'
use commerce/-;
\! echo 'Customer'
select * from customer;
\! echo 'Product'
select * from product;
\! echo 'COrder'
select * from corder;

mysql --table < select_commerce_data.sql
Using commerce/-
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           4 | dan@domain.com     |
|           5 | eve@domain.com     |
+-------------+--------------------+
Product
+----------+-------------+-------+
| sku      | description | price |
+----------+-------------+-------+
| SKU-1001 | Monitor     |   100 |
| SKU-1002 | Keyboard    |    30 |
+----------+-------------+-------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        4 |           4 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

查看vitess的tablets信息，可知当前运行两个tablets实例：

echo "show vitess_tablets;" | mysql --table
+-------+----------+-------+------------+---------+------------------+-------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname    |
+-------+----------+-------+------------+---------+------------------+-------------+
| zone1 | commerce | -     | MASTER     | SERVING | zone1-2469782763 | 172.17.0.11 |
| zone1 | commerce | -     | REPLICA    | SERVING | zone1-2548885007 | 172.17.0.9  |
+-------+----------+-------+------------+---------+------------------+-------------+