Centos单机部署Elasticsearch、JDK、Zookeeper、Kafka、Neo4j、Redis、MySQL、MinIO、Flink、Vector

配置需求

  • 操作系统:CentOS7.X
  • 内存:128GB
  • CPU:32核*2
  • 硬盘:10TB

第三方软件

安装步骤

环境准备

常用工具安装

sudo yum install vim
sudo yum install net-tools
sudo yum install unzip

新建用户

adduser test

设置密码

passwd test

切换test用户

su test

新建目录结构

mkdir ~/software
mkdir ~/soar
mkdir ~/data

配置Host

sudo vim /etc/hosts

增加如下内容

127.0.0.1 soar01

约定目录结构

将所有的软件、数据放到用户家目录下:

  • ~/software:存放所有第三放软件

  • ~/data:存放所有数据

    • ~/data/es-data:Elasticsearch数据目录
    • ~/data/kafka-data:Kafka数据目录
    • ~/data/zk-data:Zookeeper数据目录
    • ~/data/minio-data:MinIO数据目录
    • ~/data/rsyslog-data:rsyslog数据目录
  • ~/soar:存放所有Spring Boot项目和Flink Job项目

    • ~/soar/soar-admin:soar-admin项目

      • ~/soar/soar-admin/bin:启动脚本
      • ~/soar/soar-admin/config:配置文件
    • ~/soar/soar-persistent:soar-persistent项目

JDK安装

下载JDK11

https://download.java.net/openjdk/jdk11/ri/openjdk-11+28_linux-x64_bin.tar.gz

下载JDK17

https://download.oracle.com/java/17/latest/jdk-17_linux-x64_bin.tar.gz

移动到~/software目录

mv openjdk-11+28_linux-x64_bin.tar.gz /home/test/software/

解压

cd /home/test/software
tar zxvf openjdk-11+28_linux-x64_bin.tar.gz

配置JAVA_HOME环境变量

sudo vim /etc/profile

在文件最后添加如下行

export JAVA_HOME=/home/test/software/jdk-11
export PATH=$JAVA_HOME/bin:$PATH

环境变量生效

source  /etc/profile

Zookeeper单机部署

下载

https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

移动到~/software目录

mv apache-zookeeper-3.7.0-bin.tar.gz /home/test/software/

解压

cd /home/test/software/
tar zxvf apache-zookeeper-3.7.0-bin.tar.gz

修改配置

cd apache-zookeeper-3.7.0-bin
cp conf/zoo_sample.cfg conf/zoo.cfg
vim conf/zoo.cfg

修改如下配置

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/test/data/zoo-data
clientPort=2181
server.1=soar01:2888:3888

新建数据存储目录

mkdir -p /home/test/data/zoo-data

创建myid

echo 1 > /home/test/data/zoo-data/myid

启动

bin/zkServer.sh start
bin/zkServer.sh status

Kafka单机部署

下载

https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz

移动到~/software目录

mv kafka_2.13-2.8.1.tgz /home/test/software/

解压

cd /home/test/software/
tar zxvf kafka_2.13-2.8.1.tgz

修改配置

cd kafka_2.13-2.8.1
vim config/server.properties

修改如下配置

broker.id=1
host.name=soar01
log.dirs=/home/test/data/kafka-data
log.retention.hours=1
zookeeper.connect=soar01:2181/kafka

创建数据目录

mkdir -p /home/test/data/kafka-data

启动

bin/kafka-server-start.sh -daemon config/server.properties

Elasticsearch单机部署

下载

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.1-linux-x86_64.tar.gz

移动到~/software目录

mv elasticsearch-7.15.1-linux-x86_64.tar.gz /home/test/software/

解压

cd /home/test/software/
tar zxvf elasticsearch-7.15.1-linux-x86_64.tar.gz

修改ES_JAVA_HOME环境变量

vim ~/.bashrc

在最后添加

export ES_JAVA_HOME=/home/test/software/elasticsearch-7.15.1/jdk

环境变量生效

source ~/.bashrc

修改配置

cd elasticsearch-7.15.1
vim config/elasticsearch.yml
cluster.name: soar-es
node.name: soar01
path.data: /home/test/data/elastic-data
network.host: soar01
discovery.seed_hosts: ["soar01"]
cluster.initial_master_nodes: ["soar01"]
​
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

创建数据目录

mkdir -p /home/test/data/elastic-data

系统配置修改

  • max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

每个进程最大同时打开文件数太小,修改/etc/security/limits.conf文件,增加配置,用户退出后重新登录生效

vim /etc/security/limits.conf
*               soft    nofile          65536
*               hard    nofile          65536
  • max number of threads [3818] for user [es] is too low, increase to at least [4096]

问题同上,最大线程个数太低。修改配置文件/etc/security/limits.conf,增加配置

*               soft    nproc           4096
*               hard    nproc           4096

可通过命令查看

ulimit -Hu
ulimit -Su
  • max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

修改/etc/sysctl.conf文件,增加配置

vim /etc/sysctl.conf
vm.max_map_count = 262144

执行命令sysctl -p生效

sysctl -p

生成证书

bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""

启动

bin/elasticsearch -d

设置密码

bin/elasticsearch-setup-passwords auto

这将会为不同的内部堆栈用户生成随机密码。或者,您也可以跳过 auto 参数,改为通过 interactive 参数手动定义密码。请记录这些密码,我们很快就会再次用到这些密码。

​
Changed password for user apm_system
PASSWORD apm_system = IRyTBui8lHx3DIrAddaN
​
Changed password for user kibana_system
PASSWORD kibana_system = kWje2eiWafU11WOwBUPr
​
Changed password for user kibana
PASSWORD kibana = kWje2eiWafU11WOwBUPr
​
Changed password for user logstash_system
PASSWORD logstash_system = nz5TL5iJonlRBJJto1zn
​
Changed password for user beats_system
PASSWORD beats_system = 04eczUb33BMz2BtLKwsC
​
Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = 7rKANkqHq1KNzUdZjwFj
​
Changed password for user elastic
PASSWORD elastic = HEj0u3t5gKNsLPE41gsK

安装Neo4j

解压

cd ~/soar/software
tar zxvf neo4j-community-4.4.4-unix.tar.gz

修改配置

cd neo4j-community-4.4.4
vim conf/neo4j.conf

修改配置为

dbms.directories.data为Neo4j数据目录配置为家目录下的soar/soar-data/neo4j-data目录。

dbms.directories.data=/zkqa/soar/soar-data/neo4j-data

dbms.memory.heap.initial_size=8g
dbms.memory.heap.max_size=8g

dbms.default_listen_address=0.0.0.0

启动

bin/neo4j start

修改密码

通过浏览器访问

http://soar02:7474

默认用户名:neo4j
默认密码:neo4j

修改密码

lsjdfl*lkjldf32VBN

Redis单机部署

下载

https://download.redis.io/releases/redis-6.2.6.tar.gz
cd redis-6.2.5
make
​
vim redis.conf
bind soar01
requirepass 4klsadfIE832.sda
daemonize yes

MySQL单机部署

下载

https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.27-1.el7.x86_64.rpm-bundle.tar

解压

tar xvf mysql-8.0.27-1.el7.x86_64.rpm-bundle.tar

安装

sudo rpm -ivh *.rpm --force --nodeps

初始化

mysqld  --initialize
ALTER USER 'root'@'localhost' IDENTIFIED BY 'slmyq^8002IST';

MinIO单机部署

下载

https://dl.min.io/server/minio/release/linux-amd64/minio

修改环境变量

export MINIO_ACCESS_KEY=minioadmin
export MINIO_SECRET_KEY=sdlf#ljfjdasl.@

启动

minio server minio-data/ &

Flink部署

下载

https://www.apache.org/dyn/closer.lua/flink/flink-1.14.0/flink-1.14.0-bin-scala_2.12.tgz

修改配置

vim flink-conf.yaml
taskmanager.host: localhost

Vector部署

下载

https://packages.timber.io/vector/0.17.3/vector-0.17.3-x86_64-unknown-linux-gnu.tar.gz

配置文件

  • dns.toml(dns日志推送配置)
data_dir = ".dns"
​
[sources.file]
type = "file"
include = ["/data/data-logs/dns/**"]
read_from = "beginning"
remove_after_secs = 60
max_line_bytes = 3097152
​
[sinks.kafka]
bootstrap_servers = "soar01:9092,soar02:9092,soar03:9092"
group_id = "soar01-dns"
topic = "dns-log"
type = "kafka"
inputs = ["file"]
encoding.codec = "text"
  • dpi.toml(dpi日志推送配置)
data_dir = ".dpi"
​
[sources.file]
type = "file"
include = ["/data/test/data-logs/dpi/**"]
read_from = "beginning"
remove_after_secs = 60
max_line_bytes = 3097152
​
[sinks.kafka]
bootstrap_servers = "soar01:9092,soar02:9092,soar03:9092"
group_id = "soar01-dpi"
topic = "dpi-log"
type = "kafka"
inputs = ["file"]
encoding.codec = "text"

启动

nohup vector -c dns.toml &
nohup vector -c dpi.toml &

网络策略

开放端口

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.181" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.182" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.183" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.184" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.185" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.186" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.187" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.188" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.189" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.190" port protocol="tcp" port="9092" accept"
​
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.173.6.234" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.173.6.235" port protocol="tcp" port="9092" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.173.100.101" port protocol="tcp" port="9092" accept"
​
firewall-cmd --reload

开放端口

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.10.2.120" port protocol="tcp" port="9300" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.10.2.121" port protocol="tcp" port="9300" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.10.2.122" port protocol="tcp" port="9300" accept"
firewall-cmd --zone=public --add-port=9200/tcp --permanent
firewall-cmd --reload
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.184" port protocol="tcp" port="6379" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.185" port protocol="tcp" port="6379" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.171.80.186" port protocol="tcp" port="6379" accept"
​
firewall-cmd --reload
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.174.220.107" port protocol="tcp" port="8383" accept"
firewall-cmd --reload
​
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.174.220.106" port protocol="tcp" port="8383" accept"
firewall-cmd --reload
firewall-cmd --zone=public --add-port=8081/tcp --permanent
firewall-cmd --reload
firewall-cmd --zone=public --add-port=7474/tcp --permanent
firewall-cmd --reload
firewall-cmd --zone=public --add-port=7687/tcp --permanent
firewall-cmd --reload
​
firewall-cmd --zone=public --remove-port=8081/tcp --permanent
firewall-cmd --reload
firewall-cmd --zone=public --remove-port=7474/tcp --permanent
firewall-cmd --reload
firewall-cmd --zone=public --remove-port=7687/tcp --permanent
firewall-cmd --reload
​
firewall-cmd --zone=public --add-port=514/udp --permanent
firewall-cmd --zone=public --add-port=514/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --zone=public --add-port=22/tcp --permanent
firewall-cmd --reload
​
firewall-cmd --zone=public --remove-port=9200/tcp --permanent
firewall-cmd --reload
flink run formatter-test-http-alert-0.0.1.jar -bootstrap.servers 10.211.88.68:9092
flink run formatter-nsfocus-waf-0.0.1.jar -bootstrap.servers 10.211.88.68:9092
​
flink run accuracy-mark-0.0.1.jar -bootstrap.servers 10.211.88.68:9092
flink run control-mark-0.0.1.jar -bootstrap.servers 10.211.88.68:9092
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.174.220.107" port protocol="tcp" port="8081" accept"
firewall-cmd --reload
​
​
firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="10.174.220.107" port protocol="tcp" port="8081" accept"
firewall-cmd --reload

日志接收

rsyslog配置

日志存放目录结构

/home/test/data/data-logs/dns/
/home/test/data/data-logs/dpi/

Docker部署

Docker安装使用root,客户端访问使用普通用户

配置IPv4 forwarding

vim /etc/sysctl.conf

配置转发

net.ipv4.ip_forward=1

重启服务,让配置生效

systemctl restart network

查看是否成功,如果返回为“net.ipv4.ip_forward = 1”则表示成功

sysctl net.ipv4.ip_forward

重启docker服务

service docker restart 

Spring Boot项目部署

Spring Boot项目部署

部署目录结构

- springboot-test
  - bin
    - startup.sh
    - shutdown.sh
  - config
    - application.properties
  - logs
    - info/info.log
    - warn/warn.log
    - error/error.log
    - trace/trace.log
  - springboot-test.jar

启动

cd soar-persistence
bin/startup.sh

startup.sh

#!/bin/bash

JAVA_HOME=/root/software/jdk11
basepath=$(cd `dirname $0`; pwd)
cd $basepath
cd ..

nohup $JAVA_HOME/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -jar *.jar > /dev/null 2>&1 &

exit

停止

cd soar-persistence
bin/shutdown.sh

shutdown.sh

#!/bin/sh

basepath=$(cd `dirname $0`; pwd)
cd $basepath
cd ..

NAME=`ls *.jar`
if [ -z "$NAME" ]; then
    echo "NAME is empty"
    exit 0
fi
ID=`ps -ef | grep "$NAME" | grep -v "$0" | grep -v "grep" | awk '{print $2}'`
echo "Stopping $NAME..."
for id in $ID
do
kill -9 $id
echo "$NAME Stopped!"
done
  • 1
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

一只小蜗牛呀

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值