elasticsearch导入json数据

最新推荐文章于 2024-04-02 10:25:23 发布

马大头

最新推荐文章于 2024-04-02 10:25:23 发布

阅读量1w

点赞数 3

分类专栏： ElasticSearch

ElasticSearch 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

导入json数据到es

每条记录必须有一条对应的index声明。json文件应符合Bulk API要求
json文件中每条数据前需包含index信息（index/type/id...）

curl -H 'Content-Type: application/x-ndjson'  -s -XPOST localhost:9200/_bulk --data-binary @accounts.json

{"index":{"_index":"index2","_type":"type2","_id":0}}
{"age":10,"name":"jim"}
    {"index":{"_index":"index2","_type":"type2","_id":1}}
{"age":16,"name":"tom"}

若导入文件中为同一个index（/index/type），则可在url中声明默认index（/index/type），则json文件中无须声明index和type，同时，支持部分记录显示声明index和type。

curl -H 'Content-Type: application/x-ndjson'  -s -XPOST localhost:9200/index1/_bulk --data-binary @accounts.json #URL中声明默认index
curl -H 'Content-Type: application/x-ndjson'  -s -XPOST localhost:9200/index1/type1/_bulk --data-binary @accounts.json #URL中声明默认index/type

{"index":{}
{"age":6,"name":"bob"}
{"index":{"_id":"2"}}
{"age":10,"name":"jim"}
{"index":{"_id":"6"}}
{"index":{"_type":"type2","_id":1}}
{"age":16,"name":"tom"}
{"index":{"_index":"index2","_type":"type3","_id":1}}
{"age":20,"name":"lucy"}

可直接导入符合格式要求的json，无须事先创建mapping，es可根据json数据自动创建。
也可事先声明mapping，，进行自定义设置
curl命令导入示例

导入shapefile到es

查阅众多资料后，理论上有如下几种方式：

使用GDAL直接将shapefile导入ES，失败
使用GDAL（或ArcMap）将shapefile导成json文件，使用curl命令导入json文件
说明：生成的json文件格式不符合Bulk API要求，需处理后方能导入
使用Arcpy编写脚本，将shapefile导出成符合Bulk要求的json，再使用curl命令导入（或脚本实现）
使用ArcMap导出geojson（或使用Arcpy），使用python（java）解析json文件，使用Bulk API编写脚本导入ES

其中需注意的问题为：

使用GDAL或ArcMap导出的geojson格式和Bulk API要求的数据格式不同
数据需符合Geo-shape datatype或Geo-point datatype方能使用curl命令导入

综上，最终选择使用第4种方法解决，使用GDAL将shapefile导出成geojson文件，再使用python elasticsearch bulk API编写脚本，解析geojson并导入ES。

1. shapefile to es（失败）

GDAL for ES Driver提供shapefile直接导入ES的方法，但导入时报错

ogr2ogr -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

#ERROR 1: HTTP error code : 405
#ERROR 8: Could not connect to server
#ElasticSearch driver failed to create http://localhost:9200/

2. 通过json文件中转（需处理json文件格式）

另外一种方式为使用ogr2ogr工具将shapefile转换为geojson文件，再将geojson文件导入ES。

2.1 shapefile to geojson

How to convert and import Arc Shapefile from Zillow into an elastic search database?
Ask

ogr2ogr -f GeoJSON  map1.geojson map.shp

生成的json文件格式如下，不包含index信息，Bulk API无法直接导入。

{
"type": "FeatureCollection",
                                                                                
"features": [
{ "type": "Feature", "properties": { "ID_0": 45, "NAME_1": "Anhui" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 119.632111, 31.139344000000108 ], [ 119.644439000000148, 31.115657 ], [ 119.624672, 31.084624000000133 ] ] ] } } 
{ "type": "Feature", "properties": { "ID_0": 45, "NAME_1": "Beijing" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 117.379737, 40.226871 ], [ 117.382896, 40.208718000000147 ], [ 117.369484, 40.190997 ] ] ] } }
]
}

2.2 create index mapping

Indexing Geo Shapes
可针对geo字段和重点字段进行mapping，其余字段导入时自动生成。

PUT /gis1
{
  "mappings": {
    "province1": {
       "properties": {
          "location": {
             "type": "geo_shape"
          },          
          "ID_0": {
             "type": "text"
          },
            "NAME_1": {
             "type": "text"
          }
       }
    }
  }
}

2.3 import json

curl -H 'Content-Type: application/x-ndjson' -XPOST 'http://localhost:9200/gis1/province1/_bulk?pretty' --data-binary @map1.geojson

#报错："type":"json_e_o_f_exception","reason":"Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@1ffa375d; line: 1, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@1ffa375d; line: 1, column: 3]"

需将数据改造成如下格式后，再使用bulk命令导入。

啥内容都不该,就是ctrl + i格式化成如下格式就好了, 可能ES对格式有要求吧!!!

{"index":{"_id":"1"}}
{ "type": "Feature", "properties": { "ID_0": 45, "NAME_1": "Zhejiang" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 121.187362, 27.86903100000012 ], [ 121.190140000000156, 27.847919 ], [ 121.156249, 27.823749000000134 ]] ] ] } }
{"index":{"_id":"2"}}
{ "type": "Feature", "properties": { "ID_0": 46, "NAME_1": "Jiangsu" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 121.187362, 27.86903100000012 ], [ 121.190140000000156, 27.847919 ], [ 121.156249, 27.823749000000134 ]] ] ] } }

4. python脚本导入GDAL生成的geojson文件

# rasa 20180212
# 使用ES python api插入geojson面数据（map.geojson）

import json
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk


def set_mapping(es, index_name="content_engine", doc_type_name="en"):
    my_mapping = {
        doc_type_name: {
            "properties": {
                "location": {
                    "type": "geo_shape"
                },
                "ID_0": {
                    "type": "text"
                },
                "NAME_1": {
                    "type": "text"
                }
            }
        }
    }

    # ignore 404 and 400
    es.indices.delete(index=index_name, ignore=[400, 404])
    print("delete_index")

    # ignore 400 cause by IndexAlreadyExistsException when creating an index
    create_index = es.indices.create(index=index_name, ignore=400)
    mapping_index = es.indices.put_mapping(index=index_name, doc_type=doc_type_name, body=my_mapping)
    if create_index["acknowledged"] is not True or mapping_index["acknowledged"] is not True:
        print("Index creation failed...")


def set_data(es, input_file, index_name, doc_type_name="en"):
    with open(input_file, 'r') as f:
        data = json.load(f)

    features = data["features"]

    ACTIONS = []
    i = 0
    count = 0

    for feature in features:
        action = {}

        if (feature["geometry"]["type"] == "Polygon"):  # 判断geometry类型为polygon
            action = {
                "_index": index_name,
                "_type": doc_type_name,
                "_source": {
                    "ID_0": feature["properties"]["ID_0"],
                    "NAME_1": feature["properties"]["NAME_1"],
                    "location": {
                        "type": "polygon",
                        "coordinates": feature["geometry"]["coordinates"]
                    }
                }
            }
        else:  # geometry类型为multipolygon
            action = {
                "_index": index_name,
                "_type": doc_type_name,
                "_source": {
                    "ID_0": feature["properties"]["ID_0"],
                    "NAME_1": feature["properties"]["NAME_1"],
                    "location": {
                        "type": "multipolygon",
                        "coordinates": feature["geometry"]["coordinates"]
                    }
                }
            }

        i += 1
        print("prepare insert:  %s" % feature["properties"]["NAME_1"])
        print("type:  %s" % feature["geometry"]["type"])
        ACTIONS.append(action)
        if (i == 5):
            success, _ = bulk(es, ACTIONS, index=index_name, raise_on_error=True)
            count += success
            i = 0
            ACTIONS = []
            print("insert %s lines" % count)

    success, _ = bulk(es, ACTIONS, index=index_name, raise_on_error=True)
    count += success
    print("insert %s lines" % count)


if __name__ == '__main__':
    # es = Elasticsearch(hosts=["127.0.0.1:9200"], http_auth=('elastic','changeme'),timeout=5000)
    es = Elasticsearch(hosts=["127.0.0.1:9200"], timeout=5000)
    set_mapping(es, "gis6", "province")

    # geojson文件为ogr2ogr生成格式

    # set_data(es, "./data/map-fujian.geojson", "gis6", "province")  # multipolygon
    # set_data(es, "./data/map-anhui.geojson", "gis6", "province") # polygon
    set_data(es, "./data/map-full.geojson", "gis6", "province")  # polygon & multipolygon
    # set_data(es, "./data/map.geojson", "gis6", "province")       # polygon

问题

1.使用python bulk导入multipolygon时报错：Invalid LinearRing found. Found a single coordinate when expecting a coordinate array

sourcecode 638行

原因：

同一个geojson文件中存在存在类型为polygon和multipolygon的feature
设定mapping为polygon，插入multipolygon数据会报错：
invalid number of points in LinearRing (found [1] - must be >= [4])；
设定mapping为multipolygon，插入polygon数据会报错：
Invalid LinearRing found. Found a single coordinate when expecting a coordinate array

解决方法：
判断feature的类型，创建的mapping不同，ES支持同一个type中同时存储polygon/multipolygon

作者：myparamita
链接：https://www.jianshu.com/p/f3d13a9d8c06
来源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。

马大头

关注

3
点赞
踩
10

收藏

觉得还不错? 一键收藏
0
评论
elasticsearch导入json数据

导入json数据到es每条记录必须有一条对应的index声明。json文件应符合Bulk API要求 json文件中每条数据前需包含index信息（index/type/id...）curl -H 'Content-Type: application/x-ndjson' -s -XPOST localhost:9200/_bulk --data-binary @accounts.jso...
复制链接

扫一扫