[OpenBMC] 从代码谈 Redfish 的Log 机制 (LogService/EventLog)

和往常一样，OpenBMC中官方对Redfish 的Log 机制已经有很详细的介绍，所以这篇是结合一些额外的资料来看整个LogService 的架构和原理 (*如果之后OpenBMC架构有变，这边不会再更新)

docs/redfish-logging-in-bmcweb.md at master · openbmc/docs · GitHubhttps://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md在Redfish 还没问世之前，BMC的世界中会将有问题的事件纪录成System Event Log (SEL)，SEL 在IPMI中是用16 Byte来表示，IPMItool 等工具会将其转换为可读性高的文字，而在Redfish 协议中，系统事件的纪录(开关机，温度异常等)是由LogServices来负责监控和配置这些事件，OpenBMC将这些系统事件称Event Log  (LogServices 不只有系统事件，还有包含audit log, crashdump 等，这篇文章只会介绍EventLog)

Journal - Systemd日志

Systemd has its own logging system called the journal; running a separate logging daemon is not required  systemd/Journal - ArchWiki (archlinux.org)

Journal是systemd本身自帶的日誌系統，它的log分類是根據Syslog協定，透過 "journalctl"指令可以讀取系統日誌，透過設定檔 /etc/systemd/journald.conf 可以更改Journal 設定

//透过 "journalctl"指令可以读取系统日志
> journalctl
Jun 25 08:41:09 intel-obmc bmcweb[370]: pam_succeed_if(webserver:auth): requirement "user ingroup redfish" was met by user "root"

Rsyslog是一个基于Syslog 的开源软件程序，可在设定档 /etc/rsyslog.conf 中，根据Journal不同的fields 来将不同的logs存放在相对应的档案和设定输出格式。

//加上-o json可以将所有filed用json格式印出来，可以看到"PRIORITY" 这样就算一个filed，我们也可以自己定义想要的filed，filed的数量并没有规定，名字可以自定义的

> journalctl  -o json 
{
    "_SOURCE_REALTIME_TIMESTAMP": "1624610486273832",
    "PRIORITY": "6",
    "_BOOT_ID": "7f34e1dd99bd4d82add8c4f627da6d2c",
    "_SYSTEMD_INVOCATION_ID": "3fb0bea4b7ef4285bda15cde5c400b56",
    "SYSLOG_FACILITY": "10",
    "MESSAGE": "pam_succeed_if(webserver:auth): requirement \"user ingroup redfish\" was met by user \"root\"",
    "__MONOTONIC_TIMESTAMP": "105559270358",
    "_CMDLINE": "/usr/bin/bmcweb",
    "SYSLOG_IDENTIFIER": "bmcweb",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "__CURSOR": "s=7ece329c23f3447497b13bfd083c9ab9;i=534;b=7f34e1dd99bd4d82add8c4f627da6d2c;m=1893d2a3d6;t=5c5931aeec754;x=a43f04e856ee1bf",
    "_HOSTNAME": "intel-obmc",
    "_SYSTEMD_SLICE": "system.slice",
    "_COMM": "bmcweb",
    "_GID": "0",
    "_SYSTEMD_UNIT": "bmcweb.service",
    "_UID": "0",
    "SYSLOG_TIMESTAMP": "Jun 25 08:41:26 ",
    "_TRANSPORT": "syslog",
    "_MACHINE_ID": "0c1eaebf149d400b851c027c0a218f06",
    "_EXE": "/usr/bin/bmcweb",
    "__REALTIME_TIMESTAMP": "1624610486273876",
    "_SYSTEMD_CGROUP": "/system.slice/bmcweb.service",
    "_PID": "370"
}

有journal log和filed的概念后，我们来看openbmc中的Rsyslog怎么来分类log，底下是openbmc 中/etc/rsyslog.conf的部分内容

openbmc/rsyslog.conf at master · openbmc/openbmc · GitHubhttps://github.com/openbmc/openbmc/blob/master/meta-intel-openbmc/meta-common/recipes-extended/rsyslog/rsyslog/rsyslog.conf

# Template for Redfish messages
# "<timestamp> <MessageId>,<MessageArgs>"
template(name="RedfishTemplate" type="list") {
    property(name="timereported" dateFormat="rfc3339")
    constant(value=" ")
    property(name="$!REDFISH_MESSAGE_ID")
    constant(value=",")
    property(name="$!REDFISH_MESSAGE_ARGS")
    constant(value="\n")
}

# If the journal entry has a Redfish MessageId, save as a Redfish event
if ($!REDFISH_MESSAGE_ID != "") then {
   action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}

分类步骤

1. 判斷log有無REDFISH_MESSAGE_ID這個filed

# If the journal entry has a Redfish MessageId, save as a Redfish event
if ($!REDFISH_MESSAGE_ID != "") then {
   action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}

2. 用redfishtemplate格式存入/var/log/redfish中

template(name="RedfishTemplate" type="list") {
    property(name="timereported" dateFormat="rfc3339")
    constant(value=" ")
    property(name="!REDFISHMESSAGEID")constant(value=",")property(name="!REDFISH_MESSAGE_ARGS")
    constant(value="\n")
}

3. redfish会去/var/log/redfish中捞资料 (GET: /redfish/v1/Systems/system/LogServices/EventLog/Entries/)

// redfish-core\lib\log_services.hpp
static bool
    getRedfishLogFiles(std::vector<std::filesystem::path>& redfishLogFiles)
{
    static const std::filesystem::path redfishLogDir = "/var/log";
    static const std::string redfishLogFilename = "redfish";
    ...
}

inline void requestRoutesJournalEventLogEntryCollection(App& app)
{
    BMCWEB_ROUTE(app,
                 "/redfish/v1/Systems/system/LogServices/EventLog/Entries/")
        .privileges(redfish::privileges::getLogEntryCollection)
        .methods(boost::beast::http::verb::get)(
           ...
                getRedfishLogFiles(redfishLogFiles);
           ...)
}

新增redfish log可以透过sd_journal_send ，例如

(REDFISH_MESSAGE_ID是必要的，REDFISH_MESSAGE_ARGS则依情况而定)

sd_journal_send("MESSAGE=%s", "journal text", "PRIORITY=%i", <LOG_LEVEL>,
                "REDFISH_MESSAGE_ID=%s",
                "ResourceEvent.1.0.ResourceErrorThresholdExceeded",
                "REDFISH_MESSAGE_ARGS=%s,%d", "Property Name",
                propertyValue, NULL);

Message Registry 

REDFISH_MESSAGE_ID 和 REDFISH_MESSAGE_ARGS要填的内容就要参考Message registry

我们先来看一个registry的范例 [GET] /redfish/v1/Registries/OpenBMC/OpenBMC

bmcweb/openbmc_message_registry.hpp at master · openbmc/bmcweb · GitHub

{
    "@Redfish.Copyright": "Copyright 2018 OpenBMC. All rights reserved.",
    "@odata.type": "#MessageRegistry.v1_4_0.MessageRegistry",
    "Description": "This registry defines the base messages for OpenBMC.",
    "Id": "OpenBMC.0.1.0",
    "Language": "en",
    "Messages": {
        "SystemPowerOnFailed": {
            "Description": "Indicates that the system failed to power on.",
            "Message": "System Power-On Failed.",
            "MessageSeverity": "Critical",
            "NumberOfArgs": 0,
            "Resolution": "None.",
            "Severity": "Critical"
        }},
        "VoltageRegulatorOverheated": {
            "Description": "Indicates that the specified voltage regulator 
                             overheated.",
            "Message": "%1 Voltage Regulator Overheated.",
            "MessageSeverity": "Critical",
            "NumberOfArgs": 1,
            "ParamTypes": ["string"],
            "Resolution": "None.",
            "Severity": "Critical"
        }
    },
    "Name": "OpenBMC Message Registry",
    "OwningEntity": "OpenBMC",
    "RegistryPrefix": "OpenBMC",
    "RegistryVersion": "0.1.0"
}

Message registry 是Redfish的一个Resource，中文翻译叫"注册表"，简单来说就是Redfish 会将它想传递的讯息(例如EventLog)先注册在Message registry中，在runtime的时候会透过MessageId (REDFISH_MESSAGE_ID)来将相对应的讯息输出。

Redfish spec对MessageId的定义

 一个MessageId组成对应registry如下

如果想要记个 VoltageRegulatorOverheated的Event Log，

• REDFISH_MESSAGE_ID = OpenBMC.0.1.VoltageRegulatorOverheated
• REDFISH_MESSAGE_ARGS可以带"CPU 0"

程式码写法如下

sd_journal_send("MESSAGE=%s", "journal text", "PRIORITY=%i", <LOG_LEVEL>,
                "REDFISH_MESSAGE_ID=%s",
                "OpenBMC.0.1.VoltageRegulatorOverheated",
                "REDFISH_MESSAGE_ARGS=%s", "CPU 0", NULL);

> cat /var/log/redfish
    1970-01-01T00:00:15.077204+00:00 OpenBMC.0.1.VoltageRegulatorOverheated,CPU 0

Redfish 会透过REDFISH_MESSAGE_ID 将Event log翻译成可读性高的文字讯息，其中log的一些property是可以和刚刚上面提的message prefix相对应的

{
        "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/15_2",
        "@odata.type": "#LogEntry.v1_4_0.LogEntry",
        "Created": "1970-01-01T00:00:15+00:00",
        "EntryType": "Event",
        "Id": "15_2",
        "Message": "CPU 0 Voltage Regulator Overheated.",
        "MessageArgs": [
            "CPU 0"
        ],
        "MessageId": "OpenBMC.0.1.VoltageRegulatorOverheated",
        "Name": "System Event Log Entry",
        "Severity": "Critical"
}

目前openBMC支援的prefix有Base, TaskEvents, ResourceEvent, OpenBMC四种，可以在bmcweb/redfish-core/include/registries at master · openbmc/bmcweb · GitHub 或透过GET /redfish/v1/Registries 看到

最后，LogService的架构如下

可以透过 /redfish/v1/Systems/system/LogServices/EventLog/Entries 取得Event Log