部署日志集中管理服务器

1、项目要求

部署一台日志集中管理服务器，用于收集远程主机上的SSH服务日志，并对日志进行按日替换，保存10天的归档数据，10天以外的归档数据自动删除。

2、日志服务系统

2-1、rsyslog日志服务

在 centos7 中，系统日志消息由两个服务负责处理：systemd-journald和rsyslog

/var/log目录由 rsyslog 维护，里面存放一些特定系统和服务的日志文件。

日志文件	作用
/var/log/messages	大部分系统日志消息(不包括：安全和身份验证的消息日志、邮件服务器相关的消息日志、)
/var/log/secure	安全和身份验证相关的消息和登录失败的日志文件。（如ssh远程登录失败）
/var/log/maillog	与邮件服务器相关的消息日志文件
/var/log/cron	与定期执行任务相关的日志文件
/var/log/boot.log	与系统启动相关的消息记录
/var/log/dmesg	与系统启动相关的消息记录
/var/log/wtmp	记录每个用户的登录次数和持续时间等信息，可用last命令查看登录成功的记录，可用 -f 动态查看
/var/log/btmp	查看登录系统失败的或者暴力破解系统的用户，一般小于1M，用lastb命令查看日志，可以使用防火墙拒绝该IP地址的ssh请求

系统日志级别：

日志类名| 分类作用 | 级别（低→高）| 优先级 | 严重性
-------- | -------|-----|-----|-----|-----|
daemon |后台进程相关的| local7| debug |信息对开发人员调试应用程序有用，在操作过程中无用
kem| 内核产生的信息| local6 |info |正常的操作信息，可以收集报告，测量吞吐量等
lpr| 打印系统产生的 |local5 |notice |注意，正常但重要的事件
authpriv| 安全认证| local4 |warning| 警告，提示如果不采取行动，将会发生错误。比如文件系统使用 90%
cron |定时相关| local3 |err| 错误，阻止某个模块或程序的功能不能正常使用
mail| 邮件相关 |local2| crit| 关键的错误，已经影响了整个系统或软件不能正常工作的信息
syslog| 日志服务自身的| local1| alert |警报，需要立刻修改的信息
news| 新闻系统| local0 |emerg| 紧急，内核崩溃等严重信息

rsyslog服务配置文件
通过该配置文件/etc/rsyslog.conf，可以看到各类日志及其日志文件存放位置

root@master ~]# cat /etc/rsyslog.conf 
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514       是否允许514端口接收使用UDP协议转发过来的日志

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514     是否允许514端口接收使用TCP协议转发过来的日志


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
# 类型   日志存放位置
#kern.*                                                 /dev/console    

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog   -表示先存在内存，存到一定量再一次性写到硬盘中


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

2-2、自定义ssh服务的日志类型和存储位置

在rsyslog服务的配置文件中添加ssh的日志级别和日志存放位置

[root@localhost ~]# vim /etc/rsyslog.conf 
 74 local0.*                /var/log/sshd.log

修改sshd的配置文件，设置日志级别与rsyslog中的日志级别相同

[root@localhost ~]# vim /etc/ssh/sshd_config
 32 SyslogFacility local0

2-3、重启服务查看日志

重启rsyslog、sshd服务

[root@localhost etc]# systemctl restart rsyslog.service 
[root@localhost etc]# systemctl restart sshd

看sshd服务的日志存储位置是否有日志产生

[root@localhost ~]# ll /var/log/sshd.log 
-rw-------. 1 root root 147 Jul 21 17:08 /var/log/sshd.log
[root@localhost ~]# cat /var/log/sshd.log 
Jul 21 17:08:44 localhost sshd[25697]: Server listening on 0.0.0.0 port 22.
Jul 21 17:08:44 localhost sshd[25697]: Server listening on :: port 22.

2-3、添加安全权限

添加安全权限防止日志被误删

[root@localhost ~]# chattr +a /var/log/sshd.log
[root@localhost ~]# lsattr /var/log/sshd.log 
-----a---------- /var/log/sshd.log

3、日志切割、日志轮替(logrotate)

3-1、logrotate

linux下的日志分为动态增长和静态增长的。
logrotate支持按时间和大小来自动切割,以防止日志文件太大。
logrotate(轮替、轮循、轮转)：当日志达到某个特定的大小或时间,我们将日志按大小、按时间切割,之前的日志(归档日志、历史日志)保留一个备份,再创建一个同名的文件保存新的日志。
/etc/logrotate.d/ 存放指定服务日志切割规则的配置文件

[root@master ~]# cat /etc/logrotate.conf 
# see "man logrotate" for details
# 全局配置日志切割规则
# rotate log files weekly
weekly    # 每周切割一次

# keep 4 weeks worth of backlogs
rotate 4  # 保留至今的4份历史数据，切割走的数据就是历史数据

# create new (empty) log files after rotating old ones
create   # 指定权限和所有者、所属主

# use date as a suffix of the rotated file
dateext   # 以日期为文件后缀名

# uncomment this if you want your log files compressed
#compress     # 切割后的历史数据，使用gzip压缩

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {    # 单独配置日志切割规则
    monthly
    create 0664 root utmp
	minsize 1M   # 文件达到这个值就进行切割
    rotate 1
}

/var/log/btmp {   # 单独配置日志切割规则
    missingok   # 文件丢失不报错
    monthly     # 每月切割一次
    create 0600 root utmp  # 设置文件的权限和所有者、所属组
    rotate 1   
}

# system-specific logs may be also be configured here.

/var/lib/logrotate/logrotate.status
默认记录logrotate上次轮换日志文件的时间

[root@localhost ~]# cat /var/lib/logrotate/logrotate.status 
logrotate state -- version 2
"/var/log/yum.log" 2020-7-19-12:0:0
"/var/log/cups/page_log" 2020-7-19-12:0:0
"/var/log/cups/error_log" 2020-7-19-12:0:0
"/var/log/sshd.log" 2020-7-21-16:0:0
"/var/log/boot.log" 2020-7-21-12:18:3
"/var/log/sssd/*.log" 2020-7-19-12:0:0
"/var/log/glusterfs/*.log" 2020-7-19-12:0:0
"/var/log/cups/access_log" 2020-7-19-12:0:0
"/var/log/httpd/error_log" 2020-7-19-12:0:0
"/var/log/chrony/*.log" 2020-7-19-12:0:0
"/var/log/wtmp" 2020-7-19-12:0:0
"/var/log/spooler" 2020-7-19-12:0:0
"/var/log/btmp" 2020-7-19-12:0:0
"/var/log/iscsiuio.log" 2020-7-19-12:0:0
"/var/log/maillog" 2020-7-19-12:0:0
"/var/log/libvirt/libvirtd.log" 2020-7-19-12:0:0
"/var/log/libvirt/qemu/*.log" 2020-7-19-12:0:0
"/var/log/wpa_supplicant.log" 2020-7-19-12:0:0
"/var/log/secure" 2020-7-19-12:0:0
"/var/log/numad.log" 2020-7-19-12:0:0
"/var/log/ppp/connect-errors" 2020-7-19-12:0:0
"/var/log/messages" 2020-7-19-12:0:0
"/var/log/httpd/access_log" 2020-7-19-12:0:0
"/var/account/pacct" 2020-7-19-12:0:0
"/var/log/cron" 2020-7-19-12:0:0

3-2、编辑ssh服务日志轮替规则

[root@localhost ~]# vim /etc/logrotate.d/sshd 
/var/log/sshd.log {
 missingok
 daily
 create 0600 root root
 rotate 10
}

重启rsyslog服务

[root@localhost ~]# systemctl restart rsyslog.service 

3-2-1、加载文件试运行

试运行结果提示不需要轮询

[root@localhost ~]# chattr -a /var/log/sshd.log
[root@localhost ~]# logrotate -d /etc/logrotate.d/sshd 
reading config file /etc/logrotate.d/sshd
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/sshd.log  after 1 days (10 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/sshd.log
  log does not need rotating (log has been rotated at 2020-7-21 16:0, that is not day ago yet)

强制轮询

[root@localhost ~]# logrotate -vf /etc/logrotate.d/sshd 
reading config file /etc/logrotate.d/sshd
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/sshd.log  forced from command line (10 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/sshd.log
  log needs rotating
rotating log /var/log/sshd.log, log->rotateCount is 10
dateext suffix '-20200721'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/sshd.log.10 to /var/log/sshd.log.11 (rotatecount 10, logstart 1, i 10), 
old log /var/log/sshd.log.10 does not exist
renaming /var/log/sshd.log.9 to /var/log/sshd.log.10 (rotatecount 10, logstart 1, i 9), 
old log /var/log/sshd.log.9 does not exist
renaming /var/log/sshd.log.8 to /var/log/sshd.log.9 (rotatecount 10, logstart 1, i 8), 
old log /var/log/sshd.log.8 does not exist
renaming /var/log/sshd.log.7 to /var/log/sshd.log.8 (rotatecount 10, logstart 1, i 7), 
old log /var/log/sshd.log.7 does not exist
renaming /var/log/sshd.log.6 to /var/log/sshd.log.7 (rotatecount 10, logstart 1, i 6), 
old log /var/log/sshd.log.6 does not exist
renaming /var/log/sshd.log.5 to /var/log/sshd.log.6 (rotatecount 10, logstart 1, i 5), 
old log /var/log/sshd.log.5 does not exist
renaming /var/log/sshd.log.4 to /var/log/sshd.log.5 (rotatecount 10, logstart 1, i 4), 
old log /var/log/sshd.log.4 does not exist
renaming /var/log/sshd.log.3 to /var/log/sshd.log.4 (rotatecount 10, logstart 1, i 3), 
old log /var/log/sshd.log.3 does not exist
renaming /var/log/sshd.log.2 to /var/log/sshd.log.3 (rotatecount 10, logstart 1, i 2), 
old log /var/log/sshd.log.2 does not exist
renaming /var/log/sshd.log.1 to /var/log/sshd.log.2 (rotatecount 10, logstart 1, i 1), 
old log /var/log/sshd.log.1 does not exist
renaming /var/log/sshd.log.0 to /var/log/sshd.log.1 (rotatecount 10, logstart 1, i 0), 
old log /var/log/sshd.log.0 does not exist
log /var/log/sshd.log.11 doesn't exist -- won't try to dispose of it
fscreate context set to system_u:object_r:var_log_t:s0
renaming /var/log/sshd.log to /var/log/sshd.log.1
creating new /var/log/sshd.log mode = 0600 uid = 0 gid = 0
set default create context

4、搭建日志收集服务器

4-1、简介

日志收集服务器：服务器作为日志接收端，客户端作为日志发送端，所有的客户端上的日志都通过514/tcp端口号发送到服务器上进行管理。
运行原理： 服务端开放514端口，允许客户端通过该端口将指定的日志远程传输到服务端的/var/log/messages文件中。

服务端IP：172.20.10.7

客户端IP：172.20.10.9

4-2、服务端修改配置文件

服务端配置，使得rsyslog服务支持使用tcp协议传输日志
tcp协议收集日志-可靠完整
udp协议收集日志-速度快-不保证数据完整
编辑rsyslog的配置文件，启用tcp协议收集日志

[root@localhost ~]# vim /etc/rsyslog.conf 
19 $ModLoad imtcp
20 $InputTCPServerRun 514

重启服务后查看运行状态

[root@localhost ~]# systemctl restart rsyslog.service 
[root@localhost ~]# netstat -antup | grep 514
udp        0      0 0.0.0.0:514             0.0.0.0:*                           46755/rsyslogd      
udp6       0      0 :::514                  :::*                                46755/rsyslogd      

防火墙开放514/tcp端口

[root@localhost ~]# firewall-cmd --permanent --add-port=514/tcp
success
[root@localhost ~]# firewall-cmd --reload 
success

4-3、客户端修改配置

客户端配置，收集客户端上的所有类型和等级的日志，并发送到指定的日志服务器。

[root@localhost ~]# vim /etc/rsyslog.conf 
90 local0.* @@172.20.10.7:514



#服务器使用 udp 协议，客户端使用的配置文件中这一行只能有一个@
# *.* @172.20.10.7:514
# 服务器使用 tcp 协议，客户端使用的配置文件中这一行必须有两个@@
# *.* @@172.20.10.7:514

重启rsyslog服务

[root@localhost ~]# systemctl restart rsyslog.service

4-4、验证服务器接受情况

用服务器ssh登录客户端查看服务器有无接收日志

[root@localhost ~]# ssh 172.20.10.9
root@172.20.10.9's password: 
Last login: Tue Jul 21 18:20:56 2020 from 172.20.10.7
[root@localhost ~]# exit
logout
Connection to 172.20.10.9 closed.

查看日志

[root@localhost ~]# tail -f /var/log/messages

Jul 21 18:30:01 localhost systemd: Started Session 45 of user root.
Jul 21 18:30:13 localhost sshd[28696]: Accepted password for root from 172.20.10.7 port 43692 ssh2
Jul 21 18:30:18 localhost sshd[28696]: Received disconnect from 172.20.10.7 port 43692:11: disconnected by user
Jul 21 18:30:18 localhost sshd[28696]: Disconnected from 172.20.10.7 port 43692