Ansible 服务部署-部署 ELK 集群服务

引用博文：https://blog.csdn.net/qq_22648091/article/details/114140816
侵权联删

Ansible 服务部署-部署 ELK 集群服务

​ 创建三台 云主机分别命名为 elk01、elk02 和 elk03，Ansible 主机可以使用上一题的环境。要求 Ansible 节点编写剧本，执 行 Ansible 剧本可以在这三个节点部署 ELK 集群服务（在/root 目录下创建 install_elk 目录作 为 ansible 工作目录，部署的入口文件命名为 install_elk.yaml）。具体要求为三个节点均安装 Elasticserach 服务并配置为 Elasticserach 集群；kibana 安装在第一个节点；Logstash 安装在 第二个节点。

ELK需要的二进制包,如下是下载链接
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/kibana/kibana-7.17.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.0-linux-x86_64.tar.gz

​ 地址规划：

角色	主机名	IP地址
ansible母机	ansible	10.18.4.46
ela01 && kibana	elk-1	10.18.4.201
ela02 && logstash	elk-2	10.18.4.202
ela03	elk-3	10.18.4.203

​ 基础准备

#域名解析（all node）
cat >> /etc/hosts <<EOF
10.18.4.46 ansible
10.18.4.201 elk-1 
10.18.4.202 elk-2 
10.18.4.203 elk-3 
EOF

#####centos7.9 node#####
hostnamectl set-hostname elk-1
hostnamectl set-hostname elk-2
hostnamectl set-hostname elk-3
####Ansible node#####
mkdir /root/install_elk
touch /root/install_elk/install_elk.yaml
mkdir -p /root/install_elk/roles/{ela,kib,log}/{files,handlers,tasks,templates,vars}
#在ansible节点与三台节点建立免密钥通信
ssh-keygen
ssh-copy-id elk-1
ssh-copy-id elk-2
ssh-copy-id elk-3
#ansible主机清单
cat >>  /etc/ansible/hosts <<EOF
[ela]
elk-1 node_name=elk-1
elk-2 node_name=elk-2
elk-3 node_name=elk-3
[kib]
elk-1
[log]
elk-2
EOF

项目目录结构

主入口文件

​ /root/install_elk/install_elk.yaml

cat > /root/install_elk/install_elk.yaml <<EOF
---
- hosts: ela
  gather_facts: no
  remote_user: root
  roles: 
  - ela

- hosts: kib 
  gather_facts: no
  remote_user: root
  roles:
  - kib 
  
- hosts: log
  gather_facts: no
  remote_user: root
  roles:
  - log
EOF

部署ela集群

​ /root/install_elk/roles/ela/tasks/main.yaml入口文件

cat > /root/install_elk/roles/ela/tasks/main.yaml <<EOF
- name: 安装Java环境
  unarchive:
    src: jdk-8u144-linux-x64.tar.gz  
    dest: /opt
    list_files: yes

- name: 添加Java环境变量
  shell: echo 'export JAVA_HOME=/opt/jdk1.8.0_144' >> /etc/profile && echo 'export PATH=$PATH:/opt/jdk1.8.0_144/bin' >> /etc/profile

- name: 生效环境变量
  shell: source /etc/profile

- name: 创建用户
  user:
    name: ela

- name: 传输本地软件包到远程主机并且解压到指定目录
  unarchive:
    src: elasticsearch-7.17.0-linux-x86_64.tar.gz
    dest: /opt
    owner: ela
    group: ela
    list_files: yes
  register: ret

- name: 创建软链接
  file:
    src: /opt/{{ ret.files.0 | regex_replace('/.*') }}
    dest: /opt/elasticsearch
    state: link

- name: 传输配置文件
  template:
    src: elasticsearch.j2
    dest: /opt/elasticsearch/config/elasticsearch.yml
    owner: ela
    group: ela
    
- name: 传输系统配置文件
  copy:
    src: limits.conf
    dest: /etc/security/limits.conf

- name: 传输系统配置文件
  copy:
    src: sysctl.conf
    dest: /etc/sysctl.conf

- name: 加载 /etc/sysctl.conf文件,使内核参数生效
  shell: sysctl -p

- name: 启动服务
  # 使用ela用户执行此命令
  become: yes
  become_user: ela
  command:
    # argv 是一个列表，存放了需要执行的命令及其参数
    # 一行一个
    argv:
      - nohup
      - /opt/elasticsearch/bin/elasticsearch
      - -d
EOF

​ /root/install_elk/roles/ela/templates/elasticsearch.j2，elasticsearch的主配置文件模板

cat > /root/install_elk/roles/ela/templates/elasticsearch.j2 <<EOF
cluster.name: elk
node.name: {{ node_name }}
node.data: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
   - 10.18.4.201
   - 10.18.4.202
   - 10.18.4.203
cluster.initial_master_nodes: ["elk-1","elk-2","elk-3"]
EOF

​ /root/install_elk/roles/ela/files/limits.conf，系统文件句柄数配置文件

cat > /root/install_elk/roles/ela/files/limits.conf <<EOF
* soft nofile 65535
* hard nofile 65535
* soft nproc  4096
* hard nproc 4096
EOF

​ /root/install_elk/roles/ela/files/sysctl.conf，系统的内核配置文件

只有 # elasticsearch 下面的配置内容是必须的，其他可选

cat > /root/install_elk/roles/ela/files/sysctl.conf <<EOF
vm.max_map_count=655360
EOF

​ 验证：

部署kibana

​ /root/install_elk/roles/kib/tasks/main.yaml任务入口文件

cat > /root/install_elk/roles/kib/tasks/main.yaml <<EOF
- name: 传输本地软件包到远程主机并且解压到指定目录
  unarchive:
    src: kibana-7.17.0-linux-x86_64.tar.gz
    dest: /opt
    owner: ela
    group: ela
    list_files: yes
  register: ret

- name: 创建软链接
  file:
    src: /opt/{{ ret.files.0 | regex_replace('/.*') }}
    dest: /opt/kibana
    state: link

- name: 创建日志与PID存放目录
  shell: mkdir -p /var/log/kibana /run/kibana

- name: 给如上目录设置权限
  shell: chown -R ela:ela /var/log/kibana /run/kibana

- name: 传输配置文件
  copy:
    src: kibana.yml
    dest: /opt/kibana/config/kibana.yml
    
- name: 传输服务管理文件
  template:
    src: kibana.service.j2
    dest: /etc/systemd/system/kibana.service

- name: 启动服务
  systemd:
    name: kibana
    state: started
    daemon_reload: yes
EOF

​ /root/install_elk/roles/kib/files/kibana.ymlKibana 程序的配置文件

cat > /root/install_elk/roles/kib/files/kibana.yml <<EOF
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.18.4.201:9200"]
i18n.locale: "zh-CN"
pid.file: /run/kibana/kibana.pid
logging.dest: /var/log/kibana/kibana.log
EOF

​ /root/install_elk/roles/kib/templates/kibana.service.j2 ，Kibana 程序管理模板文件

cat > /root/install_elk/roles/kib/templates/kibana.service.j2 <<EOF
[Unit]
Description=Kibana
Documentation=https://www.elastic.co
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=ela
Group=ela
ExecStart=/opt/kibana/bin/kibana
ExecStop=/bin/pkill -F "/run/kibana/kibana.pid"
Restart=on-failure
RestartSec=3
StartLimitBurst=3
StartLimitInterval=60
WorkingDirectory=/opt/kibana
StandardOutput=journal
StandardError=inherit
[Install]
WantedBy=multi-user.target
EOF

​ 验证：

部署Logstash

​ /root/install_elk/roles/log/tasks/main.yaml任务入口文件

cat > /root/install_elk/roles/log/tasks/main.yaml <<EOF
- name: 创建日志目录
  file:
    path: /var/log/logstash
    state: directory

- name: 服务日志目录权限
  shell: chown -R ela:ela /var/log/logstash

- name: 传输本地软件包到远程主机并且解压到指定目录
  unarchive:
    src: logstash-7.17.0-linux-x86_64.tar.gz
    dest: /opt
    list_files: yes
  register: ret

- name: 创建软链接
  file:
    src: /opt/{{ ret.files.0 | regex_replace('/.*') }}
    dest: /opt/logstash
    state: link

- name: 传输配置文件
  template:
    src: logstash.yml
    dest: /opt/logstash/config/logstash.yml

- name: 传输管道配置文件
  copy:
    src: logstash.conf
    dest: /opt/logstash/config/logstash-sample.conf
    
- name: 传输系统服务文件
  template:
    src: logstash.service.j2
    dest: /etc/systemd/system/logstash.service

- name: 启动 logstash
  systemd:
    name: logstash
    state: started
    daemon_reload: yes

- name: restart logstash
  systemd:
    name: logstash
    state: restarted
    daemon_reload: yes

#- name: 启动服务
#  become: yes
#  become_user: ela
#  shell: sh /opt/logstash/bin/logstash -f /opt/logstash/config/logstash-sample.conf 
EOF

​ /root/install_elk/roles/log/templates/logstash.yml，Logstash 程序主配置文件

cat > /root/install_elk/roles/log/templates/logstash.yml <<EOF
http.host: "0.0.0.0"
path.logs: /var/log/logstash/
EOF

​ /root/install_elk/roles/log/files/logstash.conf，Logstash 程序管道配置文件

cat > /root/install_elk/roles/log/files/logstash.conf <<EOF
#将本地的/var/log/yum.log内日志标准输入
input {
  file {
    path => "/var/log/yum.log"
    type => "yum_log"
    start_position => "beginning"
  }
}

#标准输出到elasticsearch中
output {
  elasticsearch {
    hosts =>  ["10.18.4.201:9200","10.18.4.202:9200","10.18.4.203:9200"] 
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}
EOF

​ /root/install_elk/roles/log/templates/logstash.service.j2，Logstash 程序进程管理模板文件

cat > /root/install_elk/roles/log/templates/logstash.service.j2 <<EOF
Unit]
Description=logstash

[Service]
Type=simple
ExecStart=/opt/logstash/bin/logstash "-f" "/opt/logstash/config/*.conf"
Restart=always
WorkingDirectory=/
LimitNOFILE=65535
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target
EOF

​ 验证：

​ 在任务栏中再到管理 -》堆栈管理。

​ 创建一个索引模式，选择所有的索引。

​ 测试安装一个软件来生成数据。

[root@elk-2 ~]#yum -y install httpd

​ 来到discover中检索数据。