java logstash 自定义插件

最新推荐文章于 2024-07-15 15:04:09 发布

qq_28248953

最新推荐文章于 2024-07-15 15:04:09 发布

阅读量503

点赞数 1

文章标签： java

本文链接：https://blog.csdn.net/qq_28248953/article/details/111592597

版权

java logstash 自定义插件

编译源码

下载gradle 配置gradle-6.6\init.d 为国内镜像地址

allprojects {
    repositories {
        maven { url 'file:///C:/Java/maven_repository'}
        mavenLocal()
        maven { name "Alibaba" ; url "https://maven.aliyun.com/repository/public" }
        maven { name "Bstek" ; url "http://nexus.bsdn.org/content/groups/public/" }
        mavenCentral()
    }

    buildscript { 
        repositories { 
            maven { name "Alibaba" ; url 'https://maven.aliyun.com/repository/public' }
            maven { name "Bstek" ; url 'http://nexus.bsdn.org/content/groups/public/' }
            maven { name "M2" ; url 'https://plugins.gradle.org/m2/' }
        }
    }
}

git clone --branch v7.2.0 --single-branch https://github.com/elastic/logstash.git v7.2.0为对应版本，目前7 版本以下不能使用，编译出来少rubyUtils.gradle
编译源码 gradlew.bat assemble，这儿要花一个多小时，下依赖比较久

成功编译后会生成 ./logstash/logstash-core/build/libs/logstash-core-7.2.0.jar，这个jar 后面依赖要使用

下载dome
- git clone https://github.com/logstash-plugins/logstash-filter-java_filter_example.git
- 在项目中添加一个gradle.properties，内容为编译后logstash 路径
```
LOGSTASH_CORE_PATH=D:\\logstash6.8.6\\logstash\\logstash-core  
```
修改代码（以input 为例）
- 修改build.gradle 镜像地址
- 防止报以下错误，修改build.gradle task.register 任务

No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency$TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values: [[task ‘:downloadAndInstallJRuby’, task ‘:removeObsoleteJars’, …]]
Possible solutions: getAt(java.lang.String), getAt(java.lang.String), getAt(int), head(), putAt(java.lang.String, java.lang.Object), wait()

 ```
     修改前：
     tasks.register("gem"){
          dependsOn [downloadAndInstallJRuby,removeObsoleteJars,vendor,generateRubySupportFiles]
          doLast {
            buildGem(projectDir, buildDir, pluginInfo.pluginFullName() + ".gemspec")
          }
      }
      
      修改后：
     tasks.register("gem"){
         dependsOn downloadAndInstallJRuby
         dependsOn removeObsoleteJars
         dependsOn vendor
         dependsOn generateRubySupportFiles
         doLast {
             buildGem(projectDir, buildDir, pluginInfo.pluginFullName() + ".gemspec")
         }
     }
 ```

编写类实现Input 接口，实现方法

package org.logstashplugins;

import co.elastic.logstash.api.*;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import org.logstashplugins.utils.ArtemisPostUtil;

import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.function.Consumer;

/**
 * @program: logstash-input-java_input_example-master
 * @description:  http 自定义插件
 * @author: HeFaQiang
 * @create: 2020-12-21 16:09
 **/
@LogstashPlugin(name="http_plus")
public class HttpPlus implements Input {
        private final CountDownLatch done = new CountDownLatch(1);
        private volatile boolean stopped=false;

        private String id;
        private String host;
        private String url;
        private String key;
        private String secret;

        public static final PluginConfigSpec<String> HOST_CONFIG =
                PluginConfigSpec.stringSetting("host", "",false,true);

        public static final PluginConfigSpec<String> URL_CONFIG =
                PluginConfigSpec.stringSetting("url", "",false,true);

        public static final PluginConfigSpec<String> KEY_CONFIG =
                PluginConfigSpec.stringSetting("key", "",false,true);


        public static final PluginConfigSpec<String> SECRET_CONFIG =
                PluginConfigSpec.stringSetting("secret", "",false,true);


        public HttpPlus(String id, Configuration config, Context context) {
                this.id = id;
                host =config.get(HOST_CONFIG);
                url=config.get(URL_CONFIG);
                key=config.get(KEY_CONFIG);
                secret=config.get(SECRET_CONFIG);

        }
        @Override
        public void start(Consumer<Map<String, Object>> consumer) {
                try {
                        ArtemisPostUtil artemisPostUtil=new ArtemisPostUtil(host,key,secret);
                        if(!stopped){
                                String countResult = ArtemisPostUtil.callPostApi("1", "1", url);
                                int count = JSONObject.parseObject(countResult).getJSONObject("data").getInteger("total");
                                int end = count / 1000 + 2;

                                for (int i = 1; i < end; i++) {
                                        //每次返回1000条数据
                                        String result = ArtemisPostUtil.callPostApi(i + "", "1000", url);
                                        JSONObject object = JSONObject.parseObject(result);

                                        JSONArray jsonArray = object.getJSONObject("data").getJSONArray("list");
                                        for (int j = 0; j < jsonArray.size(); j++) { //1000条数据
                                                JSONObject jsonObject = jsonArray.getJSONObject(j);
                                                if(jsonObject!=null&& jsonObject.size()!=0){
                                                        consumer.accept((Map)jsonObject);
                                                }
                                        }
                                }

                        }
                } finally {
                       stopped=true;
                       done.countDown();
                }
        }

        @Override
        public void stop() {
                stopped = true;
        }

        @Override
        public void awaitStop() throws InterruptedException {
                done.await();
        }

        @Override
        public Collection<PluginConfigSpec<?>> configSchema() {
                return Arrays.asList(HOST_CONFIG, KEY_CONFIG, URL_CONFIG, SECRET_CONFIG);
        }

        @Override
        public String getId() {
                return this.id;
        }
}

@LogstashPlugin(name=“elasticsearch_plus”) name 是以后使用logstash的模块名，一定要与类名一样，用下划线间隔，自定义的属性名，不能使用大写，识别不了，如host,index
- 备注：发送event的，只能等执行了stop 后，consumer 才能接受到数据

打包
- 修改build.gradle 修改为自己定义的类及name
```
pluginInfo.pluginClass     = "HttpPlus"
pluginInfo.pluginName      = "http_plus"
```
- gradlew gem 第一次打包比较久（30-60分钟），要下载ruby 等一些依赖
```
后面打包，注释build.gradle 这行，就不用每次去下载ruby,大概只用花30s 
// dependsOn downloadAndInstallJRuby
```
- 打包完成后,生成如logstash-input-http_plus-1.0.1.gem 以.gem 结尾的文件
安装插件
- 登录到logstash 所在服务器 cd $LOGSTASH_HOME
- 修改Gemfile，把镜像修改为国内镜像
```
source "https://gems.ruby-china.com" 
```

安装插件，第一次比较久，大概30m

bin/logstash-plugin install --no-verify --local /home/logstash/logstash-6.8.6/logstash-input-http_plus-1.0.1.gem 以.gem

6.运行

编写conf

input{
    http_plus{
	      host => ""
		  url => "" 
		  key => ""
          secert => ""		  
	}
}

output {

    stdout {

        # JSON格式输出

        codec => json_lines

    }
}

运行 bin/logstash --java-execution -f /home/logstash/logstash-6.8.6/java_input.conf

7.离线安装

导出插件
bin/logstash-plugin prepare-offline-pack --overwrite --output logstash-output-jdbc.zip logstash-output-jdbc
安装插件
bin/logstash-plugin install file:///home/logstash/logstash-6.8.6/logstash-output-jdbc.zip

8.源码和代码下载

链接 https://pan.baidu.com/s/1eldt68JKaJHDwzbLVPDUTg
提取码 qvak

qq_28248953

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
java logstash 自定义插件

java logstash 自定义插件编译源码下载gradle 配置gradle-6.6\init.d 为国内镜像地址allprojects { repositories { maven { url 'file:///C:/Java/maven_repository'} mavenLocal() maven { name "Alibaba" ; url "https://maven.aliyun.com/repository/pu
复制链接

扫一扫