rsyslog无法发送日志到server端问题定位

最新推荐文章于 2023-11-23 14:40:51 发布

chenwr2018

最新推荐文章于 2023-11-23 14:40:51 发布

阅读量5.9k

点赞数 1

分类专栏：问题集文章标签： rsyslog rsyslogd syslog

本文链接：https://blog.csdn.net/chenwr2018/article/details/121191971

版权

问题集专栏收录该内容

7 篇文章 1 订阅

订阅专栏

问题描述

网络正常的情况下，代码端中使用openlog无法正常将日志发往syslog服务器。即使使用logger也无法正常发送。

环境

ARM设备充当client端。

pc虚拟机Ubuntu充当server端。

验证方式

通过wireshark抓包设备端发送的syslog协议报文。

rsyslog.conf与rsyslog.service网上有很多描述，这里就不展开赘述了。

解决办法

修改rsyslog.conf配置文件

添加sysSock.Name="/dev/log"

修订内容如下：

module(load=“imuxsock” sysSock.Name="/dev/log")

其他配置均不做修改。注意在配置文件尾端添加连接服务器IP和端口，目前我走的是UDP。

添加内容如下：

# ### end of the forwarding rule ###
*.*  @192.168.1.6:514

设备端通过logger 命令发送测试命令【将内容写入syslog，也可以使用logger -p “local1.info” “hello”，来发送自定义级别日志】
在这里插入图片描述

wireshark收到对应的包，说明已经通了。

在这里插入图片描述

根本原因是system启动的systemd-journald与rsyslog sock有冲突，导致rsyslog套接字无法正常发送数据。

解题思路

顺便记录下整个解决过程的思路历程。

手动执行rsyslogd，添加开启debug参数。

rsyslogd -dn

查看对应日志

由于我手动启动多次测试概率性能成功，（但是由系统开机自启动服务则一直失败）因此比较了成功和失败的日志差异。发现成功的日志代码走到这里。

8878.085509240:ffffa6ab7100: ../threads.c: set thread name to 'in:imuxsock'
8878.085627960:imuxsock.c     : imuxsock.c: --------imuxsock calling poll() on 1 fds
signaling new internal message via SIGTTOU: 'rsyslogd fully started up and initialized - begin actual processing [v8.2004.0 try https://www.rsyslog.com/e/0 ]'
8878.085852400:imuxsock.c     : imuxsock.c: Message from UNIX socket: #4, size 131
8878.085876680:imuxsock.c     : datetime.c: ParseTIMESTAMP3339: invalid year: 0, pszTS: 'o'
8878.260699160:imuxsock.c     : wtp.c: main Q:Reg: started with state 0, num workers now 1
8878.260770680:ffffa66b6100: debug.c: thread created, tid 2755, name 'rs:main Q:Reg'
8878.260844000:main Q:Reg/w0  : wti.c: wti 0x2b6d2a0: worker starting
8878.260863600:imuxsock.c     : wtp.c: main Q:Reg: new worker finished initialization with state 3, num workers now 1
8878.260880120:imuxsock.c     : main Q: queue.c: EnqueueMsg advised worker start
8878.260894760:imuxsock.c     : imuxsock.c: --------imuxsock calling poll() on 1 fds
8878.260916120:main Q:Reg/w0  : queue.c: DeleteProcessedBatch: we deleted 0 objects and enqueued 0 objects
8878.474334680:main Q:Reg/w0  : queue.c: rger: deleteBatchFromQStore, nElem 0
8878.474353400:main Q:Reg/w0  : queue.c: doDeleteBatch: delete batch from store, new sizes: log 1, phys 1
8878.474365520:main Q:Reg/w0  : main Q: queue.c: entry deleted, size now log 0, phys 1 entries
8878.474377440:main Q:Reg/w0  : main Q: queue.c: dequeued 1 consumable elements, szlog 0 sz phys 1
8878.474401120:main Q:Reg/w0  : ruleset.c: processBATCH: batch of 1 elements must be processed
8878.474413320:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 0: <46>Nov  4 04:07:58 rsyslogd: [origin software="rsyslogd" swVersion="8.2004.0" x-pid="2668" x-info="https://www.rsyslog.com"] start

锁定关键字imuxsock，对应rsyslog.conf配置文件中的imuxsock模块。

查看官网文档

RSyslog Documentation - rsyslog

syslog

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-773JMcKk-1636270428232)(image/rsyslog无法发送server端问题定位/image-20211107150157885.png)]

找到相应问题的描述，提到systemd确实有相似的现存问题。

通过systemctl status来查看启动rsyslogd状态

root@firewall:~# systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-11-07 05:50:55 UTC; 4min 22s ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 411 (rsyslogd)
      Tasks: 4 (limit: 2066)
     Memory: 4.0M
     CGroup: /system.slice/rsyslog.service
             └─411 /usr/sbin/rsyslogd -n -iNONE

Nov 07 05:52:52 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2004.0 try https://www.rsyslog.com/e/2359 ]
Nov 07 05:52:52 firewall rsyslogd[411]: omfwd/udp: socket 7: sendto() error: Network is unreachable [v8.2004.0 try https://www.rsyslog.com/e/2354 ]
Nov 07 05:52:52 firewall rsyslogd[411]: omfwd: socket 7: error 101 sending via udp: Network is unreachable [v8.2004.0 try https://www.rsyslog.com/e/2354 ]
Nov 07 05:52:52 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. >
Nov 07 05:52:52 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2004.0 try https://www.rsyslog.com/e/2359 ]
Nov 07 05:52:52 firewall rsyslogd[411]: omfwd/udp: socket 7: sendto() error: Network is unreachable [v8.2004.0 try https://www.rsyslog.com/e/2354 ]
Nov 07 05:52:52 firewall rsyslogd[411]: omfwd: socket 7: error 101 sending via udp: Network is unreachable [v8.2004.0 try https://www.rsyslog.com/e/2354 ]
Nov 07 05:52:52 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. >
Nov 07 05:52:52 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' suspended (module 'builtin:omfwd'), next retry is Sun Nov  7 05:53:22 2021, retry nbr 0. There should be messages before>
Nov 07 05:53:23 firewall rsyslogd[411]: action 'action-7-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2004.0 try https://www.rsyslog.com/e/2359 ]

最新版本依然存在这个问题的信息

root@firewall:/var/log# systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-11-07 03:05:07 UTC; 5min ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 411 (rsyslogd)
      Tasks: 4 (limit: 2066)
     Memory: 1.5M
     CGroup: /system.slice/rsyslog.service
             └─411 /usr/sbin/rsyslogd -n

Nov 07 03:05:07 firewall systemd[1]: Starting System Logging Service...
Nov 07 03:05:07 firewall rsyslogd[411]: error during parsing file /etc/rsyslog.conf, on or before line 15: syntax error on token '"immark' [v8.2110.0 try https://www.rsyslog.com/e/2207 ]
Nov 07 03:05:07 firewall rsyslogd[411]: could not interpret master config file '/etc/rsyslog.conf'. [v8.2110.0 try https://www.rsyslog.com/e/2207 ]
Nov 07 03:05:07 firewall rsyslogd[411]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2110.0]
Nov 07 03:05:07 firewall rsyslogd[411]: [origin software="rsyslogd" swVersion="8.2110.0" x-pid="411" x-info="https://www.rsyslog.com"] start
Nov 07 03:05:07 firewall systemd[1]: Started System Logging Service.

imuxsock: Acquired UNIX socket ‘/run/systemd/journal/syslog’ (fd 3) from systemd.

明确是sockket问题。

解决办法，根据官网的说明。

在rsyslog.conf模块中指定sysSock.Name="/dev/log"

补充说明

改socket会影响journalctl打印syslog日志。原因是journalctl默认socket是/dev/log。
rsyslog的日志源有两种，一种是从/dev/log中获取，一种是从/run/systemd/journal/syslog中获取，默认是/run/systemd/journal/syslog。

最终解决方案是：
修改/etc/systemd/journald.conf，开启ForwardToSyslog（控制journal产生的日志是否转发给rsyslog），并将rsyslog的socket改为默认。保证两者不冲突。

弯路记录

X86平台是能够正常发送，比较版本之后发现X86版本与ARM不一致，于是准备移植新版本rsyslog。花费很多时间在.mk和configure上。

后面发现网上有个平台，可以提供各个版本的编译程序，可以直接下载使用验证。减少自己移植浪费不必要的时间，大前提只是为了验证是否是因为版本引起的问题。

State of openSUSE_Leap_15.1_ARM for home:rainergerhards:branches:home:rgerhards / rsyslog - openSUSE Build Service
不过也是很神奇，后面我升级这个版本问题也解决了。不过它直接把我的系统日志和kernel日志全部关闭重定向到串口。后面也没去深究了。
rsyslog-8.2110.0-lp151.65.1.aarch64.rpm版本测试成功的配置文件。确实配置项更多，看起来更高级。

rsyslog.service

[Unit]
Description=System Logging Service
Requires=syslog.socket
Requires=var-run.mount
After=var-run.mount
Conflicts=syslog-ng.service syslogd.service
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/

[Service]
Type=notify
Environment=RSYSLOGD_PARAMS=
EnvironmentFile=-/etc/sysconfig/syslog
ExecStartPre=/usr/sbin/rsyslog-service-prepare
ExecStart=/usr/sbin/rsyslogd -n -iNONE $RSYSLOGD_PARAMS
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
Restart=on-abort

[Install]
WantedBy=multi-user.target
Alias=syslog.service

rsyslog.conf

##
## === When you're using remote logging, enable on-disk queues ===
## === in rsyslog.d/remote.conf. When neccesary also set the   ===
## === SYSLOG_REQUIRES_NETWORK=yes in /etc/sysconfig/syslog,   ===
## === e.g. when rsyslog has to receive on a specific IP only. ===
##
## Note, that when the MYSQL, PGSQL, GSSAPI, GnuTLS or SNMP modules
## (provided in separate rsyslog-module-* packages) are enabled, the
## configuration can't be used on a system with /usr on a remote
## filesystem, except on newer systems where initrd mounts /usr.
## [The modules are linked against libraries installed bellow of
##  /usr thus also installed in /usr/lib*/rsyslog because of this.]
##

#
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance
# and report them at http://bugzilla.novell.com/
#

# since rsyslog v3: load input modules
# If you do not load inputs, nothing happens!

# provides --MARK-- message capability (every 1 hour)
$ModLoad immark.so
$MarkMessagePeriod      3600

# provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock.so

# reduce dupplicate log messages (last message repeated n times)
$RepeatedMsgReduction   on

# kernel logging (may be also provided by /sbin/klogd)
# see also http://www.rsyslog.com/doc-imklog.html.
$ModLoad imklog.so
# set log level 1 (same as in /etc/sysconfig/syslog).
$klogConsoleLogLevel    1

# Use rsyslog native, rfc5424 conform log format as default
# ($ActionFileDefaultTemplate RSYSLOG_FileFormat).
#
# To change a single file to use obsolete BSD syslog format
# (rfc 3164, no high-precision timestamps), set the variable
# bellow or append ";RSYSLOG_FileFormat" to the filename.
# See
#   http://www.rsyslog.com/doc/rsyslog_conf_templates.html
# for more informations.
#
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Include config generated by /etc/init.d/syslog script
# using the SYSLOGD_ADDITIONAL_SOCKET* variables in the
# /etc/sysconfig/syslog file.
#
$IncludeConfig /run/rsyslog/additional-log-sockets.conf

#
# Include config files, that the admin provided? :
#
$IncludeConfig /etc/rsyslog.d/*.conf


###
# print most important on tty10 and on the xconsole pipe
#
if      ( \
            /* kernel up to warning except of firewall  */ \
            ($syslogfacility-text == 'kern')      and      \
            ($syslogseverity <= 4 /* warning */ ) and not  \
            ($msg contains 'IN=' and $msg contains 'OUT=') \
        ) or ( \
            /* up to errors except of facility authpriv */ \
            ($syslogseverity <= 3 /* errors  */ ) and not  \
            ($syslogfacility-text == 'authpriv')           \
        ) \
then {
        /dev/tty10
        |/dev/xconsole
}


# Emergency messages to everyone logged on (wall)
*.emerg                                  :omusrmsg:*

# enable this, if you want that root is informed
# immediately, e.g. of logins
#*.alert                                root


#
# firewall messages into separate file and stop their further processing
#
if      ($syslogfacility-text == 'kern') and \
        ($msg contains 'IN=' and $msg contains 'OUT=') \
then {
        -/var/log/firewall
        stop
}


#
# acpid messages into separate file and stop their further processing
#
# => all acpid messages for debuging (uncomment if needed):
#if     ($programname == 'acpid' or $syslogtag == '[acpid]:') then \
#       -/var/log/acpid
#
# => up to notice (skip info and debug)
if      ($programname == 'acpid' or $syslogtag == '[acpid]:') and \
        ($syslogseverity <= 5 /* notice */) \
then {
        -/var/log/acpid
        stop
}


#
# NetworkManager into separate file and stop their further processing
#
if      ($programname == 'NetworkManager') or \
        ($programname startswith 'nm-') \
then {
        -/var/log/NetworkManager
        stop
}


#
# email-messages
#
mail.*                                  -/var/log/mail
mail.info                               -/var/log/mail.info
mail.warning                            -/var/log/mail.warn
mail.err                                 /var/log/mail.err


#
# news-messages
#
news.crit                               -/var/log/news/news.crit
news.err                                -/var/log/news/news.err
news.notice                             -/var/log/news/news.notice
# enable this, if you want to keep all news messages
# in one file
#news.*                                 -/var/log/news.all


#
# Warnings in one file
#
*.=warning;*.=err                       -/var/log/warn
*.crit                                   /var/log/warn


#
# the rest in one file
#
*.*;mail.none;news.none                 -/var/log/messages


#
# enable this, if you want to keep all messages
# in one file
#*.*                                    -/var/log/allmessages


#
# Some foreign boot scripts require local7
#
local0.*;local1.*                       -/var/log/localmessages
local2.*;local3.*                       -/var/log/localmessages
local4.*;local5.*                       -/var/log/localmessages
local6.*;local7.*                       -/var/log/localmessages

###
*.*  @192.168.1.6

在这里插入图片描述

该平台跟公司惯用的每日编译平台大同小异，需要注册账号，不过这个平台挺齐全的。

在这里插入图片描述
挺方便下载源码和镜像文件。

chenwr2018

关注

1
点赞
踩
10

收藏

觉得还不错? 一键收藏
0
评论
rsyslog无法发送日志到server端问题定位

问题描述网络正常的情况下，代码端中使用openlog无法正常将日志发往syslog服务器。即使使用logger也无法正常发送。环境ARM设备充当client端。pc虚拟机Ubuntu充当server端。验证方式通过wireshark抓包设备端发送的syslog协议报文。rsyslog.conf与rsyslog.service网上有很多描述，这里就不展开赘述了。解决办法修改rsyslog.conf配置文件添加sysSock.Name="/dev/log"修订内容如下：module(lo
复制链接

扫一扫