Loki 数据源Kafaka 消息过滤

原创已于 2023-05-11 09:27:30 修改 · 686 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java

于 2023-05-09 11:36:44 首次发布

本文介绍了如何使用Grafana+Loki+Vector进行Kafka消息筛选，通过示例展示了如何过滤指定字段，并进行了优化，最终以表格形式在Grafana中展示。操作步骤包括在Loki中添加数据源、配置Query、Transform提取字段，以及在Table Panel中组织和调整列。

此文基于 Granfana+Loki+Vector +Kafka 筛选过滤指定字段

添加Query

在Granfana 中配置 Data Source ->Loki --> Add

# 1. 在Granfana 中添加loki 过滤条件
# 2. 过滤 ruleID,rcSysId,logDate,logTime,securityExchange,threshold,stockName 等字段值

{key="mag0506"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"66103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``

原始kafka 数据：

{
    "currentIndex":0,
    "header":{
        "headerMap":{
            "35":"381166"
        }
    },
    "mapList":[
        {
            "10007":"1"
        },
        {
            "37":"4",
            "38":"35000",
            "44":"660000",
            "48":"603900",
            "1301":"1",
            "11001":"建发合诚",
            "11436":"20230508",
            "28000":"20230508",
            "28001":"1683537179648",
            "28005":"1",
            "28101":"1538922182",
            "43220":"成交额:[260000,2945300],成交量:[26000,45000],价格涨跌幅:[0.0126,0.0398387],市场成交量占比:[0.2026,1.28571]",
            "64101":"99003",
            "64102":"1101",
            "64103":"99003-1753个证券-包含上证50证券和6个账户（一个账户与9904重复）",
            "65098":"1",
            "66002":"5",
            "66003":"1",
            "66004":"0",
            "66005":"2",
            "91001":"830035",
            "93020":"1"
        }
    ]
}

将以上kafka 原始消息数据，筛选过滤出指定字段，语句如下：

{key="mag0506"}|= ""|json| line_format "{{.message}}" | json

mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``

优化以上筛选过滤条件：