https://www.loggly.com/ultimate-guide/centralizing-with-syslog/
The syslog protocol is often useful for sending logs to external locations such log management systems or legacy systems. However, since syslog is text-based and the systemd journal is in binary format, there is no simple one-to-one relationship between these two.
Using ForwardToSyslog
Syslog is still part of Linux distributions that incorporate systemd natively, and as we saw before, journal events can be forwarded to syslog with a simple configuration change. That means journal events can be sent over syslog, provided we have:
- THE SYSTEMD-JOURNALD SERVICE RUNNING IN THE SOURCE MACHINE
- THE RSYSLOGD DAEMON RUNNING IN THE SOURCE MACHINE
- THE FORWARDTOSYSLOG PARAMETER SET TO “YES” IN /ETC/SYSTEMD/JOURNALD.CONF
- THE MAXLEVELSYSLOG PARAMETER SET TO “DEBUG” IN /ETC/SYSTEMD/JOURNALD.CONF
Using imjournal
Also, rsyslog has an input module that can import structured journal data into the systemd journal. This module is called imjournal. .There is another import module called imuxsock that imports journal data into the syslog. The difference between the two is that imuxsock imports unstructured journal data, whereas imjournal import structured event messages.
The first few lines of /etc/rsyslog.conf file of a vanilla CentOS 7 installation shows the module is already included:
More information about the module can be found here.
Reviewing Syslog Output
To see how journal events are recorded in syslog, the following command was run in an Ubuntu 15.04 system:
|
1
|
$
journalctl
-
b
-
p
err
|
This showed three error messages from the current boot:
To check if these messages also exist in syslog, the following query looks at the /var/log/syslog file for the last error (shown in bold in the output block above):
|
1
|
$
cat
/
var
/
log
/
syslog
|
grep
Failed
at
step
PAM
spawning
/
lib
/
systemd
/
systemd
:
Operation
not
permitted
|
The output below shows the message does exist in syslog (with the same time stamp):
In a CentOS 7 system, running the journalctl -b -p err command showed us these messages from the current boot:
|
1
2
3
|
--
Logs
begin
at
Tue
2015
-
07
-
07
00
:
35
:
53
EDT
,
end
at
Tue
2015
-
07
-
07
02
:
13
:
01
EDT
.
--
Jul
07
00
:
35
:
55
centos
-
7
-
x64
audispd
[
274
]
:
No
plugins
found
,
exiting
Jul
07
00
:
36
:
43
test
-
centos
systemd
[
1
]
:
Failed
to
start
Crash
recovery
kernel
arming
.
|
Running a query against the /var/log/messages file (syslog) for the first error showed it existed there as well:
Note that the unit name related to the error comes after the date time stamp and the system name field. In this case, it’s the audispd service that encountered an error. The actual error message is then printed.
Although auto redirection of systemd journal messages to syslog seems like a cool feature at first, there are no one-to-one relationships between the fields captured by two. As a result, certain fields from a journal event will be found in its corresponding syslog message, while others won’t be there.
Output in JSON
Journals have rich structured data stored in different fields. For sending journal events to external data consumers that do not understand its native binary format, the data can first be exported to JSON and then transmitted. This is advantageous because JSON format has much wider support from common analysis tools.
Let’s consider the command journalctl -b -p err that we ran in our test Ubuntu system.
One of the output lines was:
|
1
|
Jul
07
19
:
17
:
01
test
-
ubuntu1504
CRON
[
797
]
:
Authentication
token
is
no
longer
valid
;
new
one
required
|
Looking at the same error message when we use the –output=json-pretty switch, we have the following output:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
{
"__CURSOR"
:
"s=06ed88764fb443ef940994ff9f77fc8f;i=4c8;b=2d171eeb6505401db1802a62ba43190b;m=1a82272e;t=51a51383bc6dc;x=44114702d932b028"
,
"__REALTIME_TIMESTAMP"
:
"1436311021668060"
,
"__MONOTONIC_TIMESTAMP"
:
"444737326"
,
"_BOOT_ID"
:
"2d171eeb6505401db1802a62ba43190b"
,
"_UID"
:
"0"
,
"_GID"
:
"0"
,
"_SYSTEMD_SLICE"
:
"system.slice"
,
"_MACHINE_ID"
:
"6312944ca8d9f189228c76ab557a9109"
,
"PRIORITY"
:
"3"
,
"_CAP_EFFECTIVE"
:
"3fffffffff"
,
"_TRANSPORT"
:
"syslog"
,
"SYSLOG_FACILITY"
:
"9"
,
"_COMM"
:
"cron"
,
"_SYSTEMD_CGROUP"
:
"/system.slice/cron.service"
,
"_SYSTEMD_UNIT"
:
"cron.service"
,
"_HOSTNAME"
:
"test-ubuntu1504"
,
"SYSLOG_IDENTIFIER"
:
"CRON"
,
"SYSLOG_PID"
:
"797"
,
"_PID"
:
"797"
,
"MESSAGE"
:
"Authentication token is no longer valid; new one required"
,
"_SOURCE_REALTIME_TIMESTAMP"
:
"1436311021667315"
}
|
The complete list of journal event fields can be found in this man page, but looking at this JSON output we see that only some of the fields map to a syslog message:
- PRIORITY: THIS FIELD CAN HAVE A VALUE BETWEEN 0 (“EMERG”) AND 7 (“DEBUG”). THIS IS COMPATIBLE WITH SYSLOG’S CONCEPT OF PRIORITY, ALTHOUGH SYSLOG PRIORITIES ARE CALCULATED DIFFERENTLY. THIS IS MORE ANALOGOUS TO SYSLOG’S SEVERITY FIELD.
- SYSLOG_FACILITY: THIS MAPS TO SYSLOG’S FACILITY FIELD.
- SYSLOG_IDENTIFIER: THIS IS THE SYSLOG IDENTIFIER STRING.
- SYSLOG_PID: THIS IS CLIENT PROCESS ID.
- _PID: THIS IS THE ID OF THE PROCESS THAT GENERATED THE JOURNAL MESSAGE. IN THE JSON BLOCK ABOVE, WE CAN SEE SEE THE SYSLOG_PID AND THE _PID FIELDS ARE BOTH SHOWING THE SAME INFORMATION.
- _COMM: THIS IS THE NAME OF THE PROCESS THAT GENERATED THE MESSAGE. THIS AND OTHER JOURNAL FIELDS LIKE _EXE OR _CMDLINE ARE EQUIVALENT TO THE APP-NAME FIELD IN A SYSLOG HEADER.
- _SOURCE_REALTIME_TIMESTAMP: THIS MAPS TO THE TIMESTAMP FIELD FROM A SYSLOG MESSAGE HEADER.
- MESSAGE: THIS IS THE ACTUAL MESSAGE. IT MAPS DIRECTLY TO THE MSG FIELD OF A SYSLOG ENTRY.
- _HOSTNAME: THIS IS THE NAME OF THE ORIGINATING HOST. THIS MAPS TO THE HOSTNAME FIELD OF A SYSLOG MESSAGE HEADER, ALTHOUGH THE SYSLOG MESSAGE HOSTNAME CAN HAVE SOME EXTRA DETAILS.
And then there are journal fields that do not map to syslog. These include:
- _SYSTEMD_UNIT: THIS IS RELEVANT IF THE MESSAGE IS COMING FROM A RESOURCE UNIT THAT SUPPORTS SYSTEMD NATIVELY. THIS IS PROBABLY THE MOST OUTSTANDING PIECE OF INFORMATION ABSENT FROM THE SYSLOG. HOWEVER, IT’S TO BE EXPECTED BECAUSE THE ORIGIN OF SYSLOG MESSAGE FORMAT EXISTED LONG BEFORE SYSTEMD CAME ALONG. ALSO, AS WE CAN SEE, THE VALUE OF _SYSTEMD_UNIT (CRON.SERVICE) CORRESPONDS TO THE _COMM (AND IN OTHER CASES _EXE) FIELD VALUE (CRON) AND SINCE _COMM/_EXE FIELDS ARE EQUIVALENT TO THE APP-NAME IN SYSLOG, WE ARE NOT LOSING THE APPLICATION OR SERVICE NAME THAT LOGGED THE MESSAGE.
- _SELINUX_CONTEXT: THIS IS THE SELINUX SECURITY CONTEXT OF THE PROCESS THAT LOGGED THE JOURNAL EVENT. THIS INFORMATION IS ABSENT IN A SYSLOG HEADER.
- CODE_FILE, CODE_LINE, CODE_FUNC: THESE FIELDS SHOW RELEVANT INFORMATION FROM THE PROGRAM CODE FILE THAT’S THROWING AN ERROR OR INFORMATION MESSAGE.
- _TRANSPORT: THIS FIELD SHOWS HOW THE EVENT RECORD WAS LOGGED IN SYSTEMD JOURNAL. POSSIBLE VALUES CAN BE: DRIVER (FOR INTERNALLY GENERATED MESSAGES), SYSLOG (MESSAGES COMING THROUGH THE SYSLOG SOCKET), JOURNAL (MESSAGES NATIVELY TRAPPED BY THE JOURNAL DAEMON), KERNEL (FOR KERNEL MESSAGES), AND STDOUT (FOR MESSAGES COMING FROM AN APP OR SERVICE’S STANDARD OUTPUT).
Other Methods
For those who want to bypass syslog altogether, they can have a look at works already done by talented developers around the world.
For example, several people have written services that forward journals to a log management solution called Loggly. The GitHub project loggly.service from Jose-Luis Rivas creates a custom systemd service that continuously exports events from the journal and sends them to Loggly.
Another GitHub project is journald-forwarder from uswitch.com which uses a slightly more involved method.
7721
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
