场景：在bug关联固件的时候将bug的数据放到固件的数据下，可以根据固件数据下是否包含bug数据查询出已关联和未关联的数据。
ES文档结构
目录
1.must查询此bug关联的固件
{
"query":{
"nested":{
"path":"s25_kernel_check",
"query":{
"bool":{
"must":[
{
"match":{
"s25_kernel_check.cve_id":"value"
}
},
{
"match":{
"s25_kernel_check.exploit-db":"value"
}
}
]
}
}
}
}
}同时满足must下的条件才可以被返回
java代码
BoolQueryBuilder must = QueryBuilders.boolQuery();
if (cvesById.getCveId()!=null){
must.must(QueryBuilders.matchQuery("s25_kernel_check.cve_id", cvesById.getCveId()));
}
if (cvesById.getExploitDb()!=null){
must.must(QueryBuilders.matchQuery("s25_kernel_check.exploit-db", cvesById.getExploitDb()));
}
HashMap<String, Object> map2 = new HashMap<>();
NestedQueryBuilder s25_kernel_check = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
SearchQuery queryBuilder = new NativeSearchQueryBuilder()
.withQuery(s25_kernel_check)
.withFields("firmware_id")
.build();
AggregatedPage<Cvesdd> page = template.queryForPage(queryBuilder, Cvesdd.class, new SearchResultMapper() {
@Override
public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
map2.put("total", searchResponse.getHits().totalHits);
List<T> list = new ArrayList<>();
return new AggregatedPageImpl<T>(list);
}
});
Integer total= new Integer(String.valueOf((Long) map2.get("total")));
if (total == 0){
total = 10;
}
NestedQueryBuilder new_s25_kernel_check = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
SearchQuery new_queryBuilder = new NativeSearchQueryBuilder()
.withQuery(new_s25_kernel_check)
.withFields("firmware_id")
.withPageable(PageRequest.of(0,total))
.build();
List<Cvesdd> list = template.queryForList(new_queryBuilder, Cvesdd.class);2.mustNot查询此bug未关联的固件
{
"size": 200,
"query": {
"bool": {
"must_not": [
{
"nested": {
"path": "s25_kernel_check",
"query": {
"bool": {
"must": [
{
"match": {
"s25_kernel_check.cve_id": "value"
}
},
{
"match": {
"s25_kernel_check.exploit-db": "value"
}
}
]
}
}
}
}
]
}
}
}
排除同时满足must下条件的数据并返回
java代码
HashMap<String, Object> totalMap = new HashMap<>();
BoolQueryBuilder must = QueryBuilders.boolQuery();
if (cvesById.getCveId()!=null){
must.must(QueryBuilders.matchQuery("s25_kernel_check.cve_id", cvesById.getCveId()));
}
if (cvesById.getExploitDb()!=null){
must.must(QueryBuilders.matchQuery("s25_kernel_check.exploit-db", cvesById.getExploitDb()));
}
NestedQueryBuilder nestedQuery = QueryBuilders.nestedQuery("s25_kernel_check", must,ScoreMode.Max);
BoolQueryBuilder newBool = QueryBuilders.boolQuery();
newBool.mustNot(nestedQuery);
SearchQuery queryBuilder = new NativeSearchQueryBuilder()
.withQuery(newBool)
.withFields("firmware_id")
.build();
AggregatedPage<Cvesdd> page = template.queryForPage(queryBuilder, Cvesdd.class, new SearchResultMapper() {
@Override
public <T> AggregatedPage<T> mapResults(SearchResponse searchResponse, Class<T> aClass, Pageable pageable) {
totalMap.put("total", searchResponse.getHits().totalHits);
List<T> list = new ArrayList<>();
return new AggregatedPageImpl<T>(list);
}
});
Integer total= new Integer(String.valueOf((Long) totalMap.get("total")));
SearchQuery new_queryBuilder = new NativeSearchQueryBuilder()
.withQuery(newBool)
.withFields("firmware_id")
.withPageable(PageRequest.of(0,total))
.build();
List<Cvesdd> list = template.queryForList(new_queryBuilder, Cvesdd.class);查询语句跟代码的对应关系
3.劫后余生
OK，上面是满足了我想要的效果，但是发现在关联后，在es中再增加一条不同任务但是固件已经被某条bug关联过的数据，就会发现某条bug的未关联固件和已关联固件都会出现这个固件，原因是es中其实是存放的任务跟固件的关联关系，又因为业务关系，会存在多条相同固件但是不同任务的数据，关联之后创建的新任务已有固件数据，其中并未包含某条bug的数据，所以才会导致这样，那么ok，在查询未关联的逻辑之前，先查询一下已经关联的数据，不管它在哪个任务，只要所有的固件数据，有一条满足包含此条bug数据，那么我们就认为此条固件已经关联了这个bug了
4.闲来无事must_not跟must下的多条件关系不一样
还尝试了排除嵌套查询满足或者id存在集合中的数据，gpt说这俩是and的关系，但是怎么测都是or的关系，现在es有三条id等于137的数据，其中有两条是满足嵌套查询的条件，如果是and那么应该是给我返回只是id等于137不满足嵌套查询的数据，但是并没有。。
但是如果将must_not改成must下面的这个两个就成了and的关系，就会返回同时满足这两个条件的数据
就很奇怪！离谱它妈给离谱开门，离谱到家了~
那么我们来看看gpt对疑问的回答
4.1 es查询
{
"size": 200,
"query": {
"bool": {
"must_not": [
{
"terms": {
"firmware_id": [
137
]
}
},
{
"nested": {
"path": "s25_kernel_check",
"query": {
"bool": {
"must": [
{
"match": {
"s25_kernel_check.cve_id": "CVE-2010-3848,CVE-2010-3850,CVE-2010-4073"
}
},
{
"match": {
"s25_kernel_check.exploit-db": "17787"
}
}
]
}
}
}
}
]
}
}
}4.2 java代码
最上面所提到的关联操作
//查询es中要关联的固件数据
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
boolQuery.must(QueryBuilders.matchQuery("firmware_id",ids[i]));
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(boolQuery)
//指定索引
.withIndices("information_result")
//指定type
.withTypes("fulldata")
.build();
List<?> results = template.query(searchQuery, response -> {
List<Object> list = new ArrayList<>();
for (SearchHit hit : response.getHits()) {
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
sourceAsMap.put("id",hit.getId());
list.add(sourceAsMap);
}
return list;
});
//构建批量插入的集合
List<IndexQuery> queries = new ArrayList<>();
for (int i1 = 0; i1 < results.size(); i1++) {
Map oldData = (HashMap) results.get(i1);
ArrayList s25_kernel_check1 =new ArrayList();
boolean key1 = oldData.containsKey("s25_kernel_check");
if (key1){
s25_kernel_check1 = (ArrayList) oldData.get("s25_kernel_check");
}
//添加的bug信息
HashMap<String, Object> stringObjectHashMap = new HashMap<>();
//.....添加本条新bug数据
s25_kernel_check1.add(stringObjectHashMap);
oldData.put("s25_kernel_check",s25_kernel_check1);
//构建插入数据
IndexQuery indexQuery = new IndexQuery();
//指定id,覆盖原有数据
indexQuery.setId(oldData.get("id").toString());
oldData.remove("id");
JSONObject newData=new JSONObject(oldData);
//插入数据
indexQuery.setSource(newData.toString());
//索引
indexQuery.setIndexName("information_result");
//类型
indexQuery.setType("fulldata");
//3.添加到queries
queries.add(indexQuery);
}
//4.添加数据
template.bulkIndex(queries);
queries.clear();
1649
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
