fluentd 简介，日志收集并导入BigQuery

nvd11

于 2024-09-06 03:15:34 发布

阅读量753

点赞数 3

分类专栏： Cloud spring Etl 文章标签： spring boot

本文链接：https://blog.csdn.net/nvd11/article/details/141947564

版权

Cloud 同时被 3 个专栏收录

24 篇文章 0 订阅

订阅专栏

spring

12 篇文章 0 订阅

订阅专栏

Etl

4 篇文章 0 订阅

订阅专栏

日志收集的工具有很多种

例如 Splunk ，很多大公司都在使用，但是个人使用的话并不合适，主要是需要license的… 钱是1个大问题

另1个常见开源的解决方案是ELK, 但是搭建和学习成本高，如果只是为了日志收集并不值。

对于k8s 方案，还有1个开源选择，就是fluentd，本文的主题。

Fluentd 的简介

Fluentd 是一个开源的数据收集器，旨在实现日志数据的统一收集、处理和转发。它支持多种数据源和数据格式，并具有灵活性和可扩展性，使得在大规模分布式系统中处理日志数据变得更加容易。

以下是 Fluentd 的一些关键特点和优势：

多源数据收集：Fluentd 支持从各种数据源收集数据，包括日志文件、系统日志、应用程序日志、传感器数据等。
数据格式转换：Fluentd 可以将不同格式的数据转换为统一的格式，使其更容易进行处理和分析。
数据过滤：Fluentd 允许用户定义过滤规则，以便在数据传输过程中对数据进行处理和过滤，只传递感兴趣的数据。
数据输出：Fluentd 支持将处理后的数据输出到各种目的地，如文件、数据库、消息队列、存储服务等。
插件生态系统：Fluentd 拥有丰富的插件生态系统，用户可以根据自己的需求选择合适的插件来扩展 Fluentd 的功能。
轻量级和高性能：Fluentd 的设计追求简单和高效，具有低资源消耗和高性能的特点。

总的来说，Fluentd 是一个功能强大且灵活的数据收集工具，适用于各种场景下的日志数据管理和分析需求。

使用Flunetd 把日志导入到另1个文件

我们首先present 1个简单的例子，虽然实际上这个场景较少。

但是如果1个service 有多个instances, 如果把多个instance 的日志导入到同1个文件实际上也可以方便日志查阅。

安装fluentd

安装fluentd 的方法有很多种
具体请情况参考
https://docs.fluentd.org/installation

fluentd 是由Ruby 语言编写的，所以很可能还要配置一些Ruby的library.

但是现在都是容器的年代了，我们有更好的选择，选择容器镜像启动，并不需要安装

编写fluentd.conf

我们只需要准备下面文件：

gateman@MoreFine-S500:~/projects/github/fluentd-home$ pwd
/home/gateman/projects/github/fluentd-home
gateman@MoreFine-S500:~/projects/github/fluentd-home$ ls -l
total 28
-rw-rw-r-- 1 gateman gateman   490  9月  3 01:23 Dockerfile
-rw-rw-r-- 1 gateman gateman  1174  9月  3 03:03 fluent.conf
-rw-rw-r-- 1 gateman gateman 11357  9月  2 21:52 LICENSE

LICENSE 文件可以忽略，实际上我们只需要3个东西

1个docker 运行环境
1个dockerfile
1个fluentd.conf

我们首先编写fluent.conf

<source>
    @type tail
    path /app/logs/springboot.log
    pos_file /app/logs/springboot.log.pos
    tag springboot.logs 
    format none
</source>

<match **>
    @type file
    path /var/log/fluent/springboot.log
    <buffer>
        @type file
        path "/var/log/fluent/springboot.log"
        timekey 2s  # 指定每两秒刷新一次缓冲区
        timekey_wait 2s  # 等待时间
    </buffer>
</match>

fluent 的配置格式很奇怪（其实也支持yaml），编写起来相对简单，当然也可以写得很复杂。
无法包括数据从哪里来，格式转换， filtering ，数据输出到哪里

官方文档：
https://docs.fluentd.org/configuration/config-file

在这个入门例子，我们尽量精简，只配置数据从哪里来，输出到哪里

解释

#  fluentd 配置文件
#
#  <source>  指明日志来源
# @Type tail 表示日志从tail 命令的输出
#  path      指明日志文件的路径
#  pos_file  指明日志文件的偏移量文件的路径， 就是告诉fluent 日志文件被读到了哪里， 就算fluentd 下次启动就能找到上次读到的位置
#  tag       指明日志的tag
#  format    指明日志的格式
#
#  <match>   指明日志的处理规则
#  @type     指明日志处理的类型
#  path      指明处理后的日志文件的路径

值得注意的是这里tag 并没有起作用，因为下面match 的匹配是 ** 所有source

如果有多个数据源(source) ，要分别output 到不同的地方，我们可以用<match “tagvalue”> 来指定对应的source

而且最好加上，就算不加buffer, fluentd 默认还是会配置buffer for file output

编写DockerFile

Docker file 相对简单，关键无非是把 fluent.conf 复制在容器内

# 使用 Fluentd 官方的镜像作为基础镜像
FROM fluentd:latest
USER root
RUN mkdir -p /app/logs/

# 安装 fluent-plugin-bigquery 插件
RUN gem install fluent-plugin-bigquery

# 复制 Fluentd 配置文件到镜像中
COPY fluent.conf /fluentd/etc/fluent.conf

# 定义 CMD 指令用于启动 Fluentd 并指定配置文件
CMD ["fluentd", "-c", "/fluentd/etc/fluent.conf"]

构建镜像

注意这里使用了代理，虽然国内还是能连接docker hub
但是 Ruby 的library仓库是连不上的

git pull && docker build --build-arg http_proxy=http://10.0.1.223:7890 --build-arg https_proxy=http://10.0.1.223:7890 -t gateman/fluentd:1.0.0 .
Already up to date.
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  199.2kB
Step 1/6 : FROM fluentd:latest
 ---> 414115680fc4
Step 2/6 : USER root
 ---> Using cache
 ---> 48b0e0d075ee
Step 3/6 : RUN mkdir -p /app/logs/
 ---> Using cache
 ---> 99d16129060a
Step 4/6 : RUN gem install fluent-plugin-bigquery
 ---> Using cache
 ---> 0e9ce3cc963b
Step 5/6 : COPY fluent.conf /fluentd/etc/fluent.conf
 ---> 06f66a9dd16f
Step 6/6 : CMD ["fluentd", "-c", "/fluentd/etc/fluent.conf"]
 ---> Running in a1832bf42150
Removing intermediate container a1832bf42150
 ---> ef6f02d5299d
Successfully built ef6f02d5299d
Successfully tagged gateman/fluentd:1.0.0

启动容器

gateman@DESKTOP-UIU9RFJ:~/projects/fluentd-home$ docker run -d -e http_proxy=http://10.0.1.223:7890 -e https_proxy=http://10.0.1.223:7890 -v /app/logs:/app/logs --name fluentd2 gateman/fluentd:1.0.0 
b50d398a139a9e92f8d9ff669e6f758cce9f472c497a73ca63b466b850943e00
gateman@DESKTOP-UIU9RFJ:~/projects/fluentd-home$ docker logs fluentd2
fluentd -c /fluentd/etc/fluent.conf
2024-09-05 18:41:42 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-05 18:41:42 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-05 18:41:42 +0000 [info]: gem 'fluentd' version '1.17.1'
2024-09-05 18:41:42 +0000 [info]: gem 'fluent-plugin-bigquery' version '3.1.0'
2024-09-05 18:41:42 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2024-09-05 18:41:42 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/app/logs/springboot.log"
    pos_file "/app/logs/springboot.log.pos"
    tag "springboot.logs"
    format none
    <parse>
      @type none
      unmatched_lines 
    </parse>
  </source>
  <match **>
    @type file
    path "/var/log/fluent/springboot.log"
    <buffer>
      @type "file"
      path "/var/log/fluent/springboot.log"
      timekey 2s
      timekey_wait 2s
    </buffer>
  </match>
</ROOT>
2024-09-05 18:41:42 +0000 [info]: starting fluentd-1.17.1 pid=7 ruby="3.2.4"
2024-09-05 18:41:42 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2024-09-05 18:41:43 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-09-05 18:41:43 +0000 [info]: adding match pattern="**" type="file"
2024-09-05 18:41:43 +0000 [info]: adding source type="tail"
2024-09-05 18:41:43 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2024-09-05 18:41:43 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2024-09-05 18:41:43 +0000 [info]: #0 following tail of /app/logs/springboot.log
2024-09-05 18:41:43 +0000 [info]: #0 fluentd worker is now running worker=0

注意启动时，我吧主机的/app/logs matched 到容器内的/app/logs 文件夹，则这个fluentd 就可以从主机的/app/logs里读取配置文件

启动1个springboot service 测试

在主机上，我们启动1个springboot service, 并把日志输出到 fluentd 定义的/app/logs/springboot.log

gateman@DESKTOP-UIU9RFJ:~/projects/demo_cloud_order$ cd target
gateman@DESKTOP-UIU9RFJ:~/projects/demo_cloud_order/target$ java -jar demo_cloud_order-1.0.2.jar --spring.profiles.active=dev --logging.file.name=/app/logs/springboot.log

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::      (v3.1.11-SNAPSHOT)

2024-09-06T02:44:44.802+08:00  INFO 19701 --- [           main] c.h.clouduser.DemoCloudOrderApplication  : Starting DemoCloudOrderApplication v1.0.2 using Java 17.0.12 with PID 19701 (/home/gateman/projects/demo_cloud_order/target/demo_cloud_order-1.0.2.jar started by gateman in /home/gateman/projects/demo_cloud_order/target)
2024-09-06T02:44:44.815+08:00  INFO 19701 --- [           main] c.h.clouduser.DemoCloudOrderApplication  : The following 1 profile is active: "dev"
2024-09-06T02:44:46.017+08:00  INFO 19701 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2024-09-06T02:44:46.087+08:00  INFO 19701 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 59 ms. Found 1 JPA repository interface.
2024-09-06T02:44:46.436+08:00  INFO 19701 --- [           main] ptablePropertiesBeanFactoryPostProcessor : Post-processing PropertySource instances
2024-09-06T02:44:46.437+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource
2024-09-06T02:44:46.440+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource commandLineArgs [org.springframework.core.env.SimpleCommandLinePropertySource] to EncryptableEnumerablePropertySourceWrapper
2024-09-06T02:44:46.440+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2024-09-06T02:44:46.441+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletContextInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2024-09-06T02:44:46.441+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper
2024-09-06T02:44:46.441+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableSystemEnvironmentPropertySourceWrapper
2024-09-06T02:44:46.442+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper
2024-09-06T02:44:46.442+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper
2024-09-06T02:44:46.443+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Config resource 'class path resource [application.yml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper
2024-09-06T02:44:46.652+08:00  INFO 19701 --- [           main] c.u.j.filter.DefaultLazyPropertyFilter   : Property Filter custom Bean not found with name 'encryptablePropertyFilter'. Initializing Default Property Filter
2024-09-06T02:44:46.666+08:00  INFO 19701 --- [           main] c.u.j.r.DefaultLazyPropertyResolver      : Property Resolver custom Bean not found with name 'encryptablePropertyResolver'. Initializing Default Property Resolver
2024-09-06T02:44:46.669+08:00  INFO 19701 --- [           main] c.u.j.d.DefaultLazyPropertyDetector      : Property Detector custom Bean not found with name 'encryptablePropertyDetector'. Initializing Default Property Detector
2024-09-06T02:44:46.918+08:00  INFO 19701 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2024-09-06T02:44:46.930+08:00  INFO 19701 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-09-06T02:44:46.931+08:00  INFO 19701 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.20]
2024-09-06T02:44:47.025+08:00  INFO 19701 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-09-06T02:44:47.026+08:00  INFO 19701 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2140 ms
2024-09-06T02:44:47.230+08:00  INFO 19701 --- [           main] c.u.j.encryptor.DefaultLazyEncryptor     : String Encryptor custom Bean not found with name 'jasyptStringEncryptor'. Initializing Default String Encryptor
2024-09-06T02:44:47.247+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.key-obtention-iterations, using default value: 1000
2024-09-06T02:44:47.247+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.pool-size, using default value: 1
2024-09-06T02:44:47.248+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.provider-name, using default value: null
2024-09-06T02:44:47.248+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.provider-class-name, using default value: null
2024-09-06T02:44:47.249+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.salt-generator-classname, using default value: org.jasypt.salt.RandomSaltGenerator
2024-09-06T02:44:47.250+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.iv-generator-classname, using default value: org.jasypt.iv.RandomIvGenerator
2024-09-06T02:44:47.251+08:00  INFO 19701 --- [           main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.string-output-type, using default value: base64
2024-09-06T02:44:47.447+08:00  INFO 19701 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2024-09-06T02:44:47.544+08:00  INFO 19701 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.2.24.Final
2024-09-06T02:44:47.547+08:00  INFO 19701 --- [           main] org.hibernate.cfg.Environment            : HHH000406: Using bytecode reflection optimizer
2024-09-06T02:44:47.701+08:00  INFO 19701 --- [           main] o.h.e.boot.internal.EnversServiceImpl    : Envers integration enabled? : true
2024-09-06T02:44:47.930+08:00  INFO 19701 --- [           main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer
2024-09-06T02:44:47.960+08:00  INFO 19701 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2024-09-06T02:44:51.093+08:00  INFO 19701 --- [           main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@15fb7a32
2024-09-06T02:44:51.095+08:00  INFO 19701 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2024-09-06T02:44:51.507+08:00  WARN 19701 --- [           main] org.hibernate.orm.deprecation            : HHH90000025: MySQL8Dialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)
2024-09-06T02:44:51.508+08:00  WARN 19701 --- [           main] org.hibernate.orm.deprecation            : HHH90000026: MySQL8Dialect has been deprecated; use org.hibernate.dialect.MySQLDialect instead
2024-09-06T02:44:51.739+08:00  INFO 19701 --- [           main] o.h.e.c.i.m.AuditMetadataGenerator       : Adding properties for entity: com.home.clouduser.entities.Order
2024-09-06T02:44:52.561+08:00  INFO 19701 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)
2024-09-06T02:44:55.300+08:00  INFO 19701 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2024-09-06T02:44:55.314+08:00  INFO 19701 --- [           main] com.home.clouduser.configs.MainConfigs   : MainConfigs: getHostName ...
2024-09-06T02:44:55.720+08:00  INFO 19701 --- [           main] c.home.clouduser.configs.MyInitializer   : Application started...
2024-09-06T02:44:55.720+08:00  INFO 19701 --- [           main] c.home.clouduser.configs.MyInitializer   : Setting proxy for dev environment...
2024-09-06T02:44:55.720+08:00  INFO 19701 --- [           main] c.home.clouduser.configs.MyInitializer   : https.proxyHost: 10.0.1.223
2024-09-06T02:44:55.721+08:00  INFO 19701 --- [           main] c.home.clouduser.configs.MyInitializer   : https.proxyPort: 7890
2024-09-06T02:44:55.767+08:00  WARN 19701 --- [           main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2024-09-06T02:44:56.369+08:00  INFO 19701 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 4 endpoint(s) beneath base path '/actuator'
2024-09-06T02:44:56.464+08:00  INFO 19701 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2024-09-06T02:44:56.466+08:00  INFO 19701 --- [           main] u.j.c.RefreshScopeRefreshedEventListener : Refreshing cached encryptable property sources on ServletWebServerInitializedEvent
2024-09-06T02:44:56.467+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source commandLineArgs refreshed
2024-09-06T02:44:56.467+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source systemProperties refreshed
2024-09-06T02:44:56.468+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source systemEnvironment refreshed
2024-09-06T02:44:56.468+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source random refreshed
2024-09-06T02:44:56.468+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/' refreshed
2024-09-06T02:44:56.468+08:00  INFO 19701 --- [           main] CachingDelegateEncryptablePropertySource : Property Source Config resource 'class path resource [application.yml]' via location 'optional:classpath:/' refreshed
2024-09-06T02:44:56.468+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource
2024-09-06T02:44:56.469+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2024-09-06T02:44:56.469+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource servletContextInitParams [org.springframework.web.context.support.ServletContextPropertySource] to EncryptableEnumerablePropertySourceWrapper
2024-09-06T02:44:56.469+08:00  INFO 19701 --- [           main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Management Server [org.springframework.boot.actuate.autoconfigure.web.server.ManagementContextAutoConfiguration$SameManagementContextConfiguration$1] to EncryptablePropertySourceWrapper
2024-09-06T02:44:56.488+08:00  INFO 19701 --- [           main] c.h.clouduser.DemoCloudOrderApplication  : Started DemoCloudOrderApplication in 12.329 seconds (process running for 12.891)

进入容器检查输出文件

当然我们也可以吧输出文件的path 映射出来，但是这个例子没有

teman@DESKTOP-UIU9RFJ:~/projects/fluentd-home$ docker exec -it fluentd2 /bin/sh
/ # cd /var/log
/var/log # ls
fluent
/var/log # cd fluent/
/var/log/fluent # ls
springboot.log                       springboot.log.20240905184444_0.log  springboot.log.20240905184448_0.log  springboot.log.20240905184452_0.log  springboot.log.20240905184456_0.log
springboot.log.20240905184142_0.log  springboot.log.20240905184446_0.log  springboot.log.20240905184450_0.log  springboot.log.20240905184454_0.log
/var/log/fluent # cat springboot.log
cat: read error: Is a directory
/var/log/fluent # cd springboot.log/
/var/log/fluent/springboot.log # ls
/var/log/fluent/springboot.log # cd ..
/var/log/fluent # ls
springboot.log                       springboot.log.20240905184444_0.log  springboot.log.20240905184448_0.log  springboot.log.20240905184452_0.log  springboot.log.20240905184456_0.log
springboot.log.20240905184142_0.log  springboot.log.20240905184446_0.log  springboot.log.20240905184450_0.log  springboot.log.20240905184454_0.log
/var/log/fluent # cat springboot.log.2024*
2024-09-05T18:41:43+00:00	fluent.info	{"pid":16,"ppid":7,"worker":0,"message":"starting fluentd worker pid=16 ppid=7 worker=0"}
2024-09-05T18:41:43+00:00	fluent.info	{"message":"following tail of /app/logs/springboot.log"}
2024-09-05T18:41:43+00:00	fluent.info	{"worker":0,"message":"fluentd worker is now running worker=0"}
2024-09-05T18:44:45+00:00	fluent.info	{"message":"detected rotation of /app/logs/springboot.log; waiting 5 seconds"}
2024-09-05T18:44:45+00:00	fluent.info	{"message":"following tail of /app/logs/springboot.log"}
2024-09-05T18:44:45+00:00	springboot.logs	{"message":"2024-09-06T02:44:44.802+08:00  INFO 19701 --- [main] c.h.clouduser.DemoCloudOrderApplication  : Starting DemoCloudOrderApplication v1.0.2 using Java 17.0.12 with PID 19701 (/home/gateman/projects/demo_cloud_order/target/demo_cloud_order-1.0.2.jar started by gateman in /home/gateman/projects/demo_cloud_order/target)"}
2024-09-05T18:44:45+00:00	springboot.logs	{"message":"2024-09-06T02:44:44.815+08:00  INFO 19701 --- [main] c.h.clouduser.DemoCloudOrderApplication  : The following 1 profile is active: \"dev\""}
2024-09-05T18:44:46+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.017+08:00  INFO 19701 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode."}
2024-09-05T18:44:46+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.087+08:00  INFO 19701 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 59 ms. Found 1 JPA repository interface."}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.436+08:00  INFO 19701 --- [main] ptablePropertiesBeanFactoryPostProcessor : Post-processing PropertySource instances"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.437+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.440+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource commandLineArgs [org.springframework.core.env.SimpleCommandLinePropertySource] to EncryptableEnumerablePropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.440+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.441+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletContextInitParams [class org.springframework.core.env.PropertySource$StubPropertySource"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.441+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.441+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableSystemEnvironmentPropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.442+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.442+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.443+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Config resource 'class path resource [application.yml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.652+08:00  INFO 19701 --- [main] c.u.j.filter.DefaultLazyPropertyFilter   : Property Filter custom Bean not found with name 'encryptablePropertyFilter'. Initializing Default Property Filter"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.666+08:00  INFO 19701 --- [main] c.u.j.r.DefaultLazyPropertyResolver      : Property Resolver custom Bean not found with name 'encryptablePropertyResolver'. Initializing Default Property Resolver"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.669+08:00  INFO 19701 --- [main] c.u.j.d.DefaultLazyPropertyDetector      : Property Detector custom Bean not found with name 'encryptablePropertyDetector'. Initializing Default Property Detector"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.918+08:00  INFO 19701 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.930+08:00  INFO 19701 --- [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:46.931+08:00  INFO 19701 --- [main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.20]"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.025+08:00  INFO 19701 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.026+08:00  INFO 19701 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2140 ms"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.230+08:00  INFO 19701 --- [main] c.u.j.encryptor.DefaultLazyEncryptor     : String Encryptor custom Bean not found with name 'jasyptStringEncryptor'. Initializing Default String Encryptor"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.247+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.key-obtention-iterations, using default value: 1000"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.247+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.pool-size, using default value: 1"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.248+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.provider-name, using default value: null"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.248+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.provider-class-name, using default value: null"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.249+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.salt-generator-classname, using default value: org.jasypt.salt.RandomSaltGenerator"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.250+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.iv-generator-classname, using default value: org.jasypt.iv.RandomIvGenerator"}
2024-09-05T18:44:47+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.251+08:00  INFO 19701 --- [main] c.u.j.c.StringEncryptorBuilder           : Encryptor config not found for property jasypt.encryptor.string-output-type, using default value: base64"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.447+08:00  INFO 19701 --- [main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.544+08:00  INFO 19701 --- [main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.2.24.Final"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.547+08:00  INFO 19701 --- [main] org.hibernate.cfg.Environment            : HHH000406: Using bytecode reflection optimizer"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.701+08:00  INFO 19701 --- [main] o.h.e.boot.internal.EnversServiceImpl    : Envers integration enabled? : true"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.930+08:00  INFO 19701 --- [main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer"}
2024-09-05T18:44:48+00:00	springboot.logs	{"message":"2024-09-06T02:44:47.960+08:00  INFO 19701 --- [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting..."}
2024-09-05T18:44:51+00:00	springboot.logs	{"message":"2024-09-06T02:44:51.093+08:00  INFO 19701 --- [main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@15fb7a32"}
2024-09-05T18:44:51+00:00	springboot.logs	{"message":"2024-09-06T02:44:51.095+08:00  INFO 19701 --- [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed."}
2024-09-05T18:44:52+00:00	springboot.logs	{"message":"2024-09-06T02:44:51.507+08:00  WARN 19701 --- [main] org.hibernate.orm.deprecation            : HHH90000025: MySQL8Dialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)"}
2024-09-05T18:44:52+00:00	springboot.logs	{"message":"2024-09-06T02:44:51.508+08:00  WARN 19701 --- [main] org.hibernate.orm.deprecation            : HHH90000026: MySQL8Dialect has been deprecated; use org.hibernate.dialect.MySQLDialect instead"}
2024-09-05T18:44:52+00:00	springboot.logs	{"message":"2024-09-06T02:44:51.739+08:00  INFO 19701 --- [main] o.h.e.c.i.m.AuditMetadataGenerator       : Adding properties for entity: com.home.clouduser.entities.Order"}
2024-09-05T18:44:53+00:00	springboot.logs	{"message":"2024-09-06T02:44:52.561+08:00  INFO 19701 --- [main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)"}
2024-09-05T18:44:55+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.300+08:00  INFO 19701 --- [main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'"}
2024-09-05T18:44:55+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.314+08:00  INFO 19701 --- [main] com.home.clouduser.configs.MainConfigs   : MainConfigs: getHostName ..."}
2024-09-05T18:44:56+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.720+08:00  INFO 19701 --- [main] c.home.clouduser.configs.MyInitializer   : Application started..."}
2024-09-05T18:44:56+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.720+08:00  INFO 19701 --- [main] c.home.clouduser.configs.MyInitializer   : Setting proxy for dev environment..."}
2024-09-05T18:44:56+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.720+08:00  INFO 19701 --- [main] c.home.clouduser.configs.MyInitializer   : https.proxyHost: 10.0.1.223"}
2024-09-05T18:44:56+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.721+08:00  INFO 19701 --- [main] c.home.clouduser.configs.MyInitializer   : https.proxyPort: 7890"}
2024-09-05T18:44:56+00:00	springboot.logs	{"message":"2024-09-06T02:44:55.767+08:00  WARN 19701 --- [main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.369+08:00  INFO 19701 --- [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 4 endpoint(s) beneath base path '/actuator'"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.464+08:00  INFO 19701 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.466+08:00  INFO 19701 --- [main] u.j.c.RefreshScopeRefreshedEventListener : Refreshing cached encryptable property sources on ServletWebServerInitializedEvent"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.467+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source commandLineArgs refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.467+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source systemProperties refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.468+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source systemEnvironment refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.468+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source random refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.468+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/' refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.468+08:00  INFO 19701 --- [main] CachingDelegateEncryptablePropertySource : Property Source Config resource 'class path resource [application.yml]' via location 'optional:classpath:/' refreshed"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.468+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.469+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.469+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource servletContextInitParams [org.springframework.web.context.support.ServletContextPropertySource] to EncryptableEnumerablePropertySourceWrapper"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.469+08:00  INFO 19701 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource Management Server [org.springframework.boot.actuate.autoconfigure.web.server.ManagementContextAutoConfiguration$SameManagementContextConfiguration$1] to EncryptablePropertySourceWrapper"}
2024-09-05T18:44:57+00:00	springboot.logs	{"message":"2024-09-06T02:44:56.488+08:00  INFO 19701 --- [main] c.h.clouduser.DemoCloudOrderApplication  : Started DemoCloudOrderApplication in 12.329 seconds (process running for 12.891)"}

注意这里每两秒生成了1个文件，而springboot.log 本身是1个folder, 里面会存放缓存文件。
测试ok

使用Flunetd 把日志导出到biqquery

以前之所以会使用ELK, 是为因为ES 具有强大的查询性能。

而使用GCP 后， bigquery 同样提供相同的大数据查询能力。

而fluentd 本身是不支持把数据到处到bigquery的， good news是有人开发了fluented的bigquery 插件
github 地址：
https://github.com/fluent-plugins-nursery/fluent-plugin-bigquery#options

编写fluentd.conf

<source>
    @type tail
    path /app/logs/springboot.log
    pos_file /app/logs/springboot.log.pos
    tag springboot.logs 
    format none
</source>

# https://github.com/fluent-plugins-nursery/fluent-plugin-bigquery#options
<match **>
  @type bigquery_insert
  
  <buffer>
    flush_mode immediate
  </buffer>
  
  #auth_method compute_engine
  auth_method json_key
  json_key /app/gcp-key/fluentd-ingress-jason-hsbc.json
  # private_key_passphrase notasecret # default

  project jason-hsbc
  dataset LOGS
  auto_create_table true
  table springboot_log

  <inject>
    time_key timestamp
    # time_type unixtime_millis # out of range
    time_type string
    time_format %Y-%m-%d %H:%M:%S.%L
  </inject>

  schema [
    {
      "name": "timestamp",
      "type": "TIMESTAMP"
    },
    {
      "name": "project_id",
      "type": "STRING"
    },
     {
      "name": "dataset",
      "type": "STRING"
    },
     {
      "name": "table",
      "type": "STRING"
    },
    {
      "name": "worker",
      "type": "STRING"
    },
     {
      "name": "insert_errors",
      "type": "STRING"
    },
    {
      "name": "message",
      "type": "STRING"
    }
  ]
</match>

注意这里source 是没有改动的

改动的只是output 配置
重点：

@type bigquery_insert 表示我们使用bigquery 的streaming insert

insert data over streaming inserts
plugin type is bigquery_insert
for continuous real-time insertions
https://developers.google.com/bigquery/streaming-data-into-bigquery#usecases

load data
plugin type is bigquery_load
for data loading as batch jobs, for big amount of data
https://developers.google.com/bigquery/loading-data-into-bigquery

使用自定义1个field, 记录插入时间
我们使用json_key 作为权限验证方法， bigquery的验证方法还有很多种
There are four methods supported to fetch access token for the service account.

Public-Private key pair of GCP(Google Cloud Platform)'s service account
JSON key of GCP(Google Cloud Platform)'s service account
Predefined access token (Compute Engine only)
Google application default credentials (http://goo.gl/IUuyuX)

由于我这个测试没用gcp的云服务器，就用普通的service account json file
我们需要指定gpc project id 和 bigquery 的dataset name
我们需要定义存放log的表的结构
正常来讲，我们应该修改springboot 配置让日志输出为json格式，这样我们就可以吧日志的各个部分存放到bigquery 不同列中
但是这个例子并没有，这样fluentd 会把日志整行输出到table的 message field 中，所以message field 是必须的

修改docker file

# 使用 Fluentd 官方的镜像作为基础镜像
FROM fluentd:latest
USER root
RUN mkdir -p /app/logs/

RUN mkdir /app/gcp-key/

# 安装 fluent-plugin-bigquery 插件
RUN gem install fluent-plugin-bigquery

# 复制 Fluentd 配置文件到镜像中
COPY fluent.conf /fluentd/etc/fluent.conf

COPY fluentd-ingress-jason-hsbc.json /app/gcp-key/fluentd-ingress-jason-hsbc.json

# 定义 CMD 指令用于启动 Fluentd 并指定配置文件
CMD ["fluentd", "-c", "/fluentd/etc/fluent.conf"]

重点是把用于访问bigquery 的json key也放入容器内

启动容器测试

gateman@DESKTOP-UIU9RFJ:~/projects/fluentd-home$ docker logs fluentd3
fluentd -c /fluentd/etc/fluent.conf
2024-09-05 19:06:04 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-05 19:06:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-05 19:06:04 +0000 [info]: gem 'fluentd' version '1.17.1'
2024-09-05 19:06:04 +0000 [info]: gem 'fluent-plugin-bigquery' version '3.1.0'
2024-09-05 19:06:04 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2024-09-05 19:06:04 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/app/logs/springboot.log"
    pos_file "/app/logs/springboot.log.pos"
    tag "springboot.logs"
    format none
    <parse>
      @type none
      unmatched_lines 
    </parse>
  </source>
  <match **>
    @type bigquery_insert
    auth_method json_key
    json_key xxxxxx
    project "jason-hsbc"
    dataset "LOGS"
    auto_create_table true
    table "springboot_log"
    schema [{"name":"timestamp","type":"TIMESTAMP"},{"name":"project_id","type":"STRING"},{"name":"dataset","type":"STRING"},{"name":"table","type":"STRING"},{"name":"worker","type":"STRING"},{"name":"insert_errors","type":"STRING"},{"name":"message","type":"STRING"}]
    <buffer>
      flush_mode immediate
    </buffer>
    <inject>
      time_key "timestamp"
      time_type string
      time_format "%Y-%m-%d %H:%M:%S.%L"
    </inject>
  </match>
</ROOT>
2024-09-05 19:06:04 +0000 [info]: starting fluentd-1.17.1 pid=8 ruby="3.2.4"
2024-09-05 19:06:04 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2024-09-05 19:06:05 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-09-05 19:06:05 +0000 [info]: adding match pattern="**" type="bigquery_insert"
2024-09-05 19:06:05 +0000 [info]: adding source type="tail"
2024-09-05 19:06:05 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2024-09-05 19:06:05 +0000 [info]: #0 starting fluentd worker pid=18 ppid=8 worker=0
2024-09-05 19:06:05 +0000 [info]: #0 following tail of /app/logs/springboot.log
2024-09-05 19:06:05 +0000 [info]: #0 fluentd worker is now running worker=0
2024-09-05 19:06:06 +0000 [warn]: #0 insert errors project_id="jason-hsbc" dataset="LOGS" table="springboot_log" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007f1696b6a540 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007f1696b690a0 @debug_info=\"\", @location=\"ppid\", @message=\"no such field: ppid.\", @reason=\"invalid\">], @index=0>]"

注意这里有个warning, fluentd 不能把一些meta 信息（并不是from 日志文件）输出到bq table, 原因是没一偶ppid这个field, 忽略

启动springboot service

gateman@DESKTOP-UIU9RFJ:~/projects/demo_cloud_order/target$ java -jar demo_cloud_order-1.0.2.jar --spring.profiles.active=dev --logging.file.name=/app/logs/springboot.log

查询bigquery table

测试成功！

gateman@MoreFine-S500:/app/config$ bq query --nouse_legacy_sql 'SELECT timestamp, message FROM `LOGS.springboot_log` where timestamp > TIMESTAMP("2024-09-05")'
WARNING: This command is using service account impersonation. All API calls will be executed as [terraform@jason-hsbc.iam.gserviceaccount.com].
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|      timestamp      |                                                                                                                                                                              message                                                                                                                                                                              |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2024-09-05 17:12:09 | fluentd worker is now stopping worker=0                                                                                                                                                                                                                                                                                                                           |
| 2024-09-05 19:06:05 | following tail of /app/logs/springboot.log                                                                                                                                                                                                                                                                                                                        |
| 2024-09-05 19:06:05 | fluentd worker is now running worker=0                                                                                                                                                                                                                                                                                                                            |
| 2024-09-05 19:06:06 | insert errors project_id="jason-hsbc" dataset="LOGS" table="springboot_log" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007f1696b6a540 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007f1696b690a0 @debug_info=\"\", @location=\"ppid\", @message=\"no such field: ppid.\", @reason=\"invalid\">], @index=0>]" |
| 2024-09-05 19:07:52 | 2024-09-06T03:07:52.338+08:00  INFO 21005 --- [main] c.h.clouduser.DemoCloudOrderApplication  : Starting DemoCloudOrderApplication v1.0.2 using Java 17.0.12 with PID 21005 (/home/gateman/projects/demo_cloud_order/target/demo_cloud_order-1.0.2.jar started by gateman in /home/gateman/projects/demo_cloud_order/target)                                      |
| 2024-09-05 19:07:52 | 2024-09-06T03:07:52.341+08:00  INFO 21005 --- [main] c.h.clouduser.DemoCloudOrderApplication  : The following 1 profile is active: "dev"                                                                                                                                                                                                                          |
| 2024-09-05 19:07:53 | 2024-09-06T03:07:53.543+08:00  INFO 21005 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.                                                                                                                                                                                                       |
| 2024-09-05 19:07:53 | 2024-09-06T03:07:53.620+08:00  INFO 21005 --- [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 47 ms. Found 1 JPA repository interface.                                                                                                                                                                              |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.952+08:00  INFO 21005 --- [main] ptablePropertiesBeanFactoryPostProcessor : Post-processing PropertySource instances                                                                                                                                                                                                                          |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.953+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource                                                                                                              |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.956+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource commandLineArgs [org.springframework.core.env.SimpleCommandLinePropertySource] to EncryptableEnumerablePropertySourceWrapper                                                                                                            |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.956+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource                                                                                                                                             |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.957+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Skipping PropertySource servletContextInitParams [class org.springframework.core.env.PropertySource$StubPropertySource                                                                                                                                            |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.957+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper                                                                                                                         |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.958+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableSystemEnvironmentPropertySourceWrapper                                |
| 2024-09-05 19:07:54 | 2024-09-06T03:07:53.958+08:00  INFO 21005 --- [main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper                                                                                                                                     |

nvd11

关注

3
点赞
踩
7

收藏

觉得还不错? 一键收藏
打赏
0
评论
fluentd 简介，日志收集并导入BigQuery

Fluentd 是一个开源的数据收集器，旨在实现日志数据的统一收集、处理和转发。它支持多种数据源和数据格式，并具有灵活性和可扩展性，使得在大规模分布式系统中处理日志数据变得更加容易。以下是 Fluentd 的一些关键特点和优势：多源数据收集：Fluentd 支持从各种数据源收集数据，包括日志文件、系统日志、应用程序日志、传感器数据等。数据格式转换：Fluentd 可以将不同格式的数据转换为统一的格式，使其更容易进行处理和分析。
复制链接

扫一扫