Ansible自动化部署配置实践（实现TDEngine集群的自动化部署）

一、说明

Ansible是一款基于Python的自动化运维，用于发布、管理和编排计算机系统的工具、而且可以支持批量系统配置、批量部署运维、批量运行命令等功能。

这里以TDEngine的集群部署为例， 讲解如何通过Ansible实现自动化部署。

二、Ansible的安装

1. 先安装 CentOS 7 的 EPEL repository

   [root@localhost ~]$ yum install -y epel-release
   

   确保yum数据源安装成功。

2. 安装ansible

   [root@localhost ~]$ sudo yum install -y ansible
   

3. 创建秘钥

   ssh-keygen -t rsa
   

4. 配置host信息

   这里的host是指需要通过ansible自动部署管理的主机

   [root@localhost ~]$ vi /home/mirson/ansible/hosts.ini
   

   配置信息如下：

   [servers]
   10.16.131.42 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass=archforce
   10.16.131.43 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass=archforce
   
   [all:vars]
   superuser = root
   username = mirson
   userpasswd = mirson
   script_dir = "/hone/{{ username }}/ASF"
   tmp_dir = "/home/{{ username }}/ansible/"
   

   servers标签是存放部署的主机信息，ansible_ssh_user和ansible_ssh_pass是远程的用户与密码， 如果ssh互信已经配置，那么这里的密码可以不用填写。

   all标签存放的是全局变量，在下面的初始化脚本中需使用， superuser为超级权限用户， username为普通操作用户。

5. 初始化系统信息

   [root@localhost ~]$ ansible-playbook -i /home/mirson/ansible/hosts.ini create_users.yml 
   

   执行create_users.yml配置， 该配置是创建普通用户， 建立SSH互信并设置sudo权限。

   create_users.yml详细配置：

   - hosts: all
     tasks:
       # 创建普通用户
       - name: create user
         user:
           name = {{ username }}
           shell = /bin/bash
           createhome = yes
       
       # 设置用户密码	
       - name: set user passwd
         shell: echo "{{ userpasswd }}" | passwd --stdin "{{ username }}"
         when: userpasswd != ''
       
       # 设置SSH互信
       - name: set authorized key
         authorized_key:
           user: "{{ username }}"
           key: "{{ lookup('file', '/home/{{ username }}/.ssh/id_rsa.pub') }}"
           state: present   
        
       # 配置sudo不需要密码（普通用户）
       - name: update sudoers file for username
         lineinfile:
           dest: /etc/sudoers
           insertafter: EOF
           line: "{{ username }} ALL=(ALL) NOPASSWD: ALL*
           regexp: '*{{ username }}.*NOPASSWD"
           state: present
           
       # 配置sudo不需要密码（超级用户）
       - name: update sudoers file for superuser
         lineinfile:
           dest: /etc/sudoers
           insertafter: EOF
           line: "{{ superuser }} ALL=(ALL) ALL”
           regexp: "*{{ superuser }}"
           state: present
       
       # 去掉只有拥有tty的用户才能使用sudo
       - name: update sudoers file add "Defaults
         lineinfile:
           dest: /etc/sudoers
           insertafter: EOF
           line: 'Defaults !requiretty*'
           regexp: '*Defaults.*!requiretty’
           state: present
   		
   
   

   最后执行初始化系统命令：

   ansible-playbook -i hosts.init init.yml
   

   init.yml配置：

    - hosts:all
      roles:
        - {role:init, cmd:'super_user', become_user:"{{ superuser }}", become:yes}
        - {role:init, cmd:'normal_user’, become_user:"{{ username }}", become:yes}
   

三、实现TDEngine集群自动化部署

1. 修改ansible的配置文件

   [root@localhost ~]$ vi ansible.cfg
   

   配置inventory的文件路径

   [defaults]
   inventory=inventory.ini
   log_path=log/ansible.log
   host_key_checking=False
   fork= 5
   gathering=smart
   
   

   inventory.init配置

   [all:vars]
   superuser=root
   username = mirson
   script_dir = "/home/{{ username }}/ASF"
   tmp_dir = "/home/{{ username }}/ansible/"
   
   [tdengine_server]
   10.16.161.186:22 fqdn=taosi role-master
   10.16.161.187:22 fqdn=taos2 role=dnode
   10.16.161.188:22 fqdn=taos3 role=dnode
     
   [tdengine_server:vars]
   master_fqdn=taos1
   tdengine_port=6030
   # 如果需支持多个目录存储， 开启此配置， 多个目录以逗号分隔
   # tdengine_data_dirs="/data/tdengine/datal, /data/tdengine/data2"
   
   

2. TDengine的配置

   1）安装脚本配置（install_tdengine.yml）：

   # 创建ansible临时目录
   - name: Create ansible tmp dir
     file:
       path: "{{ tmp_dir }}"
       state: directory
       owner: "{{ username }} "
       group: "{{ username }}"
   
   # 创建tdengine安装目录    
   - name: Create TDengine install dir
     file:
       path: "{{ tdengine_path }}"
       state: directory
       owner: "{{ username }}"
       group: "{{ username }} "
     become_user: "{{ superuser }} "
     become: yes
   
   # 解压Tdengine 安装包  
   - name:Extract TDengine
       unarchive:
       src: "{{ tdengine_version }}-Linux-x64.tar.gz”
       dest: "{{ tdengine_path }} "
       creates: "{{ tdengine_path }}/{{ tdengine_version }}.tar.gz”
     become_user: "{{ superuser }} "
     become: yes
   
   # 创建Tdengine 的日志目录  
   - name:Create Logdir
       file:
       path: "{{ tdengine_path }} /log”
       owner: "{{ username }}"
       group: "{{ username }} "
       recurse:yes
       state:directory
     become_user: "{{ superuser }}"
     become: yes
   
   # 创建Tdengine 的数据目录  
   - name:Create Datadir
       file:
       path: "{{ tdengine_path }}/data"
       owner: "{{ username }} "
       group: "{{ username }} "
       recurse:yes
       state:directory
     become_user: "{{ superuser }}"
     become: yes
   
   
   # 创建多个数据存储目录（如果配置则生效）
   - name:Create multidata directory
     file:
       path: "{{ item }} "
       owner: "{{ username }} "
       group: "{{ username }}"
       recurse:yes
       state:directory
     register:exec_result
     when:tdengine_data dirs is defined
     with_items:
       - "{{ tdengine_data_dirs.split(',')  }} "
     become_user: "{{ superuser }} "
     become: yes
   
   # 创建临时目录
   - name:Create tmpdir
     file:
       path: "{{ tdengine_path }}/tmp"
       owner: "{{ username }}"
       group: "{{ username }}"
       recurse:yes
       state:directory
     become_user: "{{ superuser }} "
     become: yes
   
   # 添加HOST映射（多个节点都需加入）
   - name:Add Hosts
     lineinfile:
       path:/etc/hosts
       state:present
       line: "{{ item }}  {{ hostvars[item].fqdn }} "
     become_user: "{{ superuser} "
     become: yes
     with_items:
       - "{{ groups['tdengine_server'] }}"
   
   # 更新安装脚本（将交互的选项改为静默,避免自动化时阻塞）
   - name:update Install.sh
     lineinfile:
       dest: "{{ tdengine_path }}/{{ tdengine_version }}/install.sh"
       line:'{{ item }} ="{{ master_fqdn }} "'
     regexp:'read{{ item }}'
     state:present
     with_items:
       - "firstEp"
       - "emailAddr"
     become_user: "{{ superuser }} "
     become: yes
   
   # 安装Tdengine服务
   - name:Install TDengine
     shell: "cd {{ tdengine_path }}/{{ tdengine_version }} &&./install.sh"
     register:install_result
     become_user: "{{ superuser }} "
     become: yes
   
   # 修改Tdengine的配置文件
   - name:Modify TDengine conf
     template:
       src:taos.cfg.j2
       dest: "/etc/taos/taos.cfg"
       owner: "{{ username }} "
       group: "{{ username }} "
     become_user: "{{ superuser }} "
     become: yes
   
   # 输出执行结果
   - name:print result
     debug:
       msg: "{{ install_result }} "
   
   
   

   通过when判断，可以决定是否执行此步骤，比如tdengine_data_dirs如果配置， 则创建对应的目录，可以通过split拆分为多个目录循环执行；

   如何读取inventory中配置的tdengine_server变量？可以通过groups方式获取， 通过hostvars获取配置的具体属性信息。

   2）启动脚本配置（start_tdengine.yml）

   # 启动tdengine服务
   - name: start tdengine
     command: "systemctl start taosd"
     register: tdengine result
     become_user: "{{ superuser}} "
     become: yes
   
   # 创建集群节点
   - name: create dnodes
     command: taos -s " create node '{{ hostvars[item].fqdn }}: {{ tdengine_port }}' "
     register: tdengine result
     when: role == "master"
     with_items: 
       - " {{ groups['tdengine_server'] }} "
     become_user: "{{ superuser }} "
     become: yes
   
   # 打印执行结果
   - name: print tdengine msg
     debug: 
       msg: " {{ tdengine_result }} "
   
   

   创建集群节点， 只要主节点执行即可，所以需加上when条件判断，通过with_items的groups获取所有节点信息， 逐个进行创建。

   3）卸载集群配置（uninstall_tdengine.yml）

   # 检查TDengine服务是否安装
   - name: Check TDengine
     command: "taos--version"
     register: check result
     ignore_errors: True
     become_user: "{{ superuser }} "
     become: yes
   
   # 卸载tdengine服务
   - name: Uninstall tdengine
     shell: "cd {{ tdengine_path }}/{{ tdengine_version }} && rmtaos"
     register: tdengine_result
     when: checkresult.rc==0
     become_user: "{{ superuser }} "
     become: yes
   
   # 清除tdengine的遗留数据, 包含配置文件
   - name: Clear tdengine data
     file: 
       path: "{{ item }} "
       state: absent
     become_user: "{{ superuser }} "
     become: yes
     with_items: 
      - "/etc/taos/taos.cfg"
      - "{{ tdengine_path }} ”
   
   # 清除tdengine的遗留数据（如果配置了多个目录也做清除）
   - name: Clear tdengine multi data
     file: 
       path: "{{ item }} "
       state: absent
     become_user: "{{ superuser }} "
     become: yes
     when: tdengine_data_dirsisdefined
     with_items: 
       - "{{ tdengine_data_dirs.split(',') }} ”
   
   # 打印执行结果
   - name: print tdengine msg
     debug: 
       msg: "{{ tdengine_result }} "
   
   

   首先会检查tdengine服务是否存在， 如果不存在， 执行会报错， 所以在卸载tdengine服务， 会通过when判断checkresult.rc==0，服务如果正常，才进行卸载处理， 因为这里是调用tdengine内置的rmtaos卸载命令，并需确保服务是正常安装。

3. 整个配置目录结构
   

   说明：

   • files: 存放的是TDengine安装包

   • tasks: 存放的任务执行脚本

   • templates: 存放的是配置文件

     taos.cfg.j2配置文件，每个节点的配置不一样， 这里通过变量配置，按照jinja2模板语法 ：

     #first fully qualified domainname(FQDN) for TD engine system
     firstEp  {{ master_fqdn }}:{{ tdengine_port }}
     #local fully qualified domainname(FQDN)
     fqdn  {{ fqdn }}
     #first port number for the connection ( 12 continuous UDP/TCP portnumber are used)
     serverPort  {{ tdengine_port }}
     #logfile'sdirectory
     logDir  {{ tdengine_port }}/log
     #datafile'sdirectory
     {% if tdengine_data_dirs is defined %}
       {% for tdengine_data_dir in tdengine_data_dirs.split(',') %}
     dataDir  {{ tdengine_data_dir }}
       {% endfor %}
     {% else %}
     dataDir  {{ tdengine_path }}/data
     {% endif %}
     # temporary file's directory
     tempDir {{ tdengine_path }}/tmp
     #numberofthreads per CPU core
     numOfThreadsPerCore 2.0
     #numberofthreads to commit cache data
     numOfCommitThreads 40
     #theproportionof total CPU cores available for query processing
     #2.0:thequerythreads will be set to double of the CPU cores.
     #1.0:allCPUcores are available for query processing[default] .
     #0.5:onlyhalfof the CPU cores are available for query.
     #0.0:onlyonecore available.
     ratioOfQueryCores  2.0
     #thelast_row/first/lastaggregator will not change the original columnname in the result fields
     keepColumnName  1
     #numberofmanagement nodes in the system
     numOfMnodes  2
     #enable/disableloadbalancing
     balance  0
     #numberofseconds allowed for ad node to be offline， for cluster only
     offlineThreshold  86400
     #maxnumberof v groups per db， 0 means configured automatically
     maxVgroupsPerDb 30
     #numberofcache blocks per vnode
     blocks  60
     #thenumberof acknowledgments required for successful data writing
     quorum  2
     #max length of an SQL
     maxSQLLength  1048576
     #system timezone
     timezone  Asia/Shanghai(CST, +0800)
     #system locale
     locale enUS.UTF-8
     # default system charset
     charset UTF-8
     # max number of connections allowed in dnode
     maxShellConns  50000
     # max number of connections allowed in client
     maxConnections  50000
     # enable/disable system monitor
     monitor  1
     wallevel  1 
     

   • vars: 存放的是变量信息

     main.yml配置：

     cmd: ''
     tdengine_version: "tdengine-server-2.2.2.0"
     tdengine_path: "/home/{{ username }}/{{ tdengine_version }} "
     tdengine_config:
       ip: "{{ ansible_host } } "
     

4. 执行自动化部署
   整体配置完成后， 执行对应命令进行操作：
   ① 安装TDEngine

   ansible-playbook tdengine.xml -e cmd=install
   

   ② 启动TDEngine

   ansible-playbook tdengine.xml -e cmd=start
   

   ③ 卸载TDEngine

   ansible-playbook tdengine.xml -e cmd=uninstall