ElasticSearch 2 (5) - Document APIs

ElasticSearch 2.1.1 (5) - Document APIs

This section describes the following CRUD APIs:

Single document APIs

Index API

Query:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'  

Result:

{
    "_shards" : {
        "total" : 10,
        "failed" : 0,
        "successful" : 10
    },
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 1,
    "created" : true
}

• total -

  Indicates to how many shard copies (primary and replica shards) the index operation should be executed on.

• successful -

  Indicates the number of shard copies the index operation succeeded on.

• failures -

  An array that contains replication related errors in the case an index operation failed on a replica shard.

Replica shards may not all be started when an indexing operation successfully returns (by default, a quorum is required). In that case, total will be equal to the total shards based on the index replica settings and successful will be equal to the number of shards started (primary plus replicas). As there were no failures, the failed will be 0.

Automatic Index Creation

• Automatic index creation disable

    action.auto_create_index    =>      false

• Automatic mapping(type) creation disable

    index.mapper.dynamic        =>      false   

• Index creation black/white list

    action.auto_create_index    =>      +aaa*,-bbb*,+ccc*,-*

  • +: allowed
  • -: disallowed

Versioning

• Optimistic concurrency control

Transactional read-then-update. It is recommended to set preference to *_primary*

curl -XPUT 'localhost:9200/twitter/tweet/1?version=2' -d '{
    "message" : "elasticsearch now has versioning support, double cool!"
}'    

NOTE: versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, then the operation is executed without any version checks.

• Version types

  • internal

    only index the document if the given version is identical to the version of the stored document.

  • external or external_gt

    only index the document if the given version is strictly higher than the version of the stored document or if there is no existing document. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.

  • external_gte

    only index the document if the given version is equal or higher than the version of the stored document. If there is no existing document the operation will succeed as well. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.
    

  • force

    correcting errors

Operation Type

• op_type

    $ curl -XPUT 'http://localhost:9200/twitter/tweet/1?op_type=create' -d '{
        "user" : "kimchy",
        "post_date" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
    }'  

• create

    $ curl -XPUT 'http://localhost:9200/twitter/tweet/1/_create' -d '{
        "user" : "kimchy",
        "post_date" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
    }'        

Automatic ID Generation

POST used instead of PUT (op_type will automatically be set to create)

$ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

automatic ID generation (6a8ca01c-7896-48e9-81cc-9f70661fcb32)

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32",
    "_version" : 1,
    "created" : true
}    

Routing

• default

  Hash of the document’s id value

• explicit control

  The value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter

    $ curl -XPOST 'http://localhost:9200/twitter/tweet?routing=kimchy' -d '{
        "user" : "kimchy",
        "post_date" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
    }'

  When setting up explicit mapping, the **_routing** field can be optionally used to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the **_routing** mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted.

Parents & Children

Child document index by parent(Automatically)

$ curl -XPUT localhost:9200/blogs/blog_tag/1122?parent=1111 -d '{
    "tag" : "something"
}'

Timestamp (Deprecated in 2.0.0-beta2.)

Use Date

TTL (time to live) (Deprecated in 2.0.0-beta2.)

Future

Distributed

The index operation is directed to the primary shard based on its route (see the Routing section above) and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.

Primary shard => Replicas
    

Write Consistency

• Quorum

  (>replicas/2+1) of active shards are available

• action.write_consistency

  • one
  • quorum
  • all

• behavior

  • node-by-node
  • per-operation

• sync replication

  The index operation only returns after all active shards within the replication group have indexed the document

Refresh

• To refresh the shard (not the whole index)
• True - poor performance
• GetAPI - realtime (doesn't require refresh)

Noop Updates

• detect_noop (version)

  • true: compare document content
  • false: ignore document content

• no hard and fast rule

Timeout

• default - 1 min

• explicit

    $ curl -XPUT 'http://localhost:9200/twitter/tweet/1?timeout=5m' -d '{
        "user" : "kimchy",
        "post_date" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
    }'

Get API

Get:

curl -XGET 'http://localhost:9200/twitter/tweet/1'

Result:

{
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 1,
    "found": true,
    "_source" : {
        "user" : "kimchy",
        "postDate" : "2009-11-15T14:12:12",
        "message" : "trying out Elasticsearch"
    }
}  

Check Exists:

curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1'

Result:

HTTP/1.1 404 Not Found
es.resource.type: index_expression
es.resource.id: twitter
es.index: twitter
Content-Type: text/plain; charset=UTF-8
Content-Length: 0    

Realtime

• To disable

    action.get.realtime => false

• fields (Good Practice)

  • BECAUSE: At least for a period of time, basically, until the next flush
  • THEREFORE: Assume fields will be loaded from source when using realtime GET

Optional Type

• _type: optional

• _all: fetch first cross types

Source filtering

• Default: open

• To disable
  • use fields

  • **_source** false

      curl -XGET 'http://localhost:9200/twitter/tweet/1?_source=false'

• Large document
  • _source_include
    
  • _source_exclude
• Parameter
  • list
  • wildcards

• Example

    curl -XGET 'http://localhost:9200/twitter/tweet/1?_source_include=*.id&_source_exclude=entities'

• Short notation(_source_include)

    curl -XGET 'http://localhost:9200/twitter/tweet/1?_source=*.id,retweeted'

Fields

• Example

  curl -XGET 'http://localhost:9200/twitter/tweet/1?fields=title,content'

• Backward compatibility

  If the requested fields are not stored, they will be fetched from the _source

  Be replaced by source filtering

• Field values

  • Document (always array)

  • Meta (never array)

    _routing

    _parent

• Leaf/Object

  • Leaf success
  • Object fail

Generated fields

• No refresh occurred between indexing and refresh

  GET will access the transaction log to fetch the document

• Some fields are generated ONLY when indexing

  • default

      error

  • ignore

      ignore_errors_on_generated_fields=true

Getting the _source directly

• Direct

    curl -XGET 'http://localhost:9200/twitter/tweet/1/_source'

• Source filtering

    curl -XGET 'http://localhost:9200/twitter/tweet/1/_source?_source_include=*.id&_source_exclude=entities'

• Existence

    curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1/_source'

Routing

Get:

    curl -XGET 'http://localhost:9200/twitter/tweet/1?routing=kimchy'
    

Error:

{"error":{"root_cause":[
    {"type":"index_not_found_exception",
    "reason":"no such index",
    "resource.type":"index_expression",
    "resource.id":"twitter",
    "index":"twitter"}],
"type":"index_not_found_exception",
"reason":"no such index",
"resource.type":"index_expression",
"resource.id":"twitter",
"index":"twitter"},"status":404}
    

Preference

Controls a preference of which shard replicas to execute the get request on

• Default

  Random

• Preference

  • _primary

    The operation will go and be executed only on the primary shards.

  • _local

    The operation will prefer to be executed on a local allocated shard if possible.

• Custom(string) value

  A custom value will be used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session id, or the user name.

Refresh

It may cause a heavy load on the system (and slows down indexing)

Distributed

1. The get operation gets hashed into a specific shard id.

2. It then gets redirected to one of the replicas within that shard id and returns the result.

The replicas are the primary shard and its replicas within that shard id group. This means that the more replicas we will have, the better GET scaling we will have.

Versioning support

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

Delete API

Delete:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1'

Result:

{
    "_shards" : {
        "total" : 10,
        "failed" : 0,
        "successful" : 10
    },
    "found" : true,
    "_index" : "twitter",
    "_type" : "tweet",
    "_id" : "1",
    "_version" : 2
}

Versioning

Each document indexed is versioned. When deleting a document, the version can be specified to make sure the relevant document we are trying to delete is actually being deleted and it has not changed in the meantime. Every write operation executed on a document, deletes included, causes its version to be incremented.

Routing

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1?routing=kimchy'

Note, issuing a delete without the correct routing, will cause the document to not be deleted.

Many times, the routing value is not known when deleting a document. For those cases, when specifying the **_routing** mapping as required, and no routing value is specified, the delete will be broadcast automatically to all shards.

Parent

Note that deleting a parent document does not automatically delete its children.

• delete all (parent_type#parent_id)

  delete-by-query plugin

Automatic index creation

• Automatically creates an index if it has not been created before

• Automatically creates a dynamic type mapping for the specific type if it has not been created before

Distributed

Write Consistency

Refresh

Timeout

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/1?timeout=5m'

Update API

curl -XPUT localhost:9200/test/type1/1 -d '{
    "counter" : 1,
    "tags" : ["red"]
}'

Scripted updates

• Increment the counter

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : {
            "inline": "ctx._source.counter += count",
            "params" : {
                "count" : 4
            }
        }
    }'

• Add a tag (no duplication check)

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : {
            "inline": "ctx._source.tags += tag",
            "params" : {
                "tag" : "blue"
            }
        }
    }'

• ctx map

  • _source
  • _index
  • _type
  • _id
  • _version
  • _routing
  • _parent
  • _timestamp
  • _ttl

• Add field

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : "ctx._source.name_of_new_field = \"value_of_new_field\""
    }'

• Remove field

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : "ctx._source.remove(\"name_of_field\")"
    }'

• Condition

  delete or noop

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : {
            "inline": "ctx._source.tags.contains(tag) ? ctx.op = \"delete\" : ctx.op = \"none\"",
            "params" : {
                "tag" : "blue"
            }
        }
    }'

Updates with a partial document

• Merge

  Simple recursive merge, inner merging of objects, replacing core "keys/values" and arrays

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "doc" : {
            "name" : "new_name"
        }
    }'

  script > doc

Detecting noop updates

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
    "doc" : {
        "name" : "new_name"
    },
    "detect_noop": false
}'

Upserts

• upsert

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "script" : {
            "inline": "ctx._source.counter += count",
            "params" : {
                "count" : 4
            }
        },
        "upsert" : {
            "counter" : 1
        }
    }'

• scripted_upsert

    curl -XPOST 'localhost:9200/sessions/session/dh3sgudg8gsrgl/_update' -d '{
        "scripted_upsert":true,
        "script" : {
            "id": "my_web_session_summariser",
            "params" : {
                "pageViewEvent" : {
                    "url":"foo.com/bar",
                    "response":404,
                    "time":"2014-01-01 12:32"
                }
            }
        },
        "upsert" : {}
    }'

• doc_as_upsert

    curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
        "doc" : {
            "name" : "new_name"
        },
        "doc_as_upsert" : true
    }'

Parameters

• retry_on_conflict

  In between the get and indexing phases of the update, it is possible that another process might have already updated the same document. By default, the update will fail with a version conflict exception. The retry_on_conflict parameter controls how many times to retry the update before finally throwing an exception.

• routing

  Routing is used to route the update request to the right shard and sets the routing for the upsert request if the document being updated doesn’t exist. Can’t be used to update the routing of an existing document.

• parent

  Parent is used to route the update request to the right shard and sets the parent for the upsert request if the document being updated doesn’t exist. Can’t be used to update the parent of an existing document.

• timeout

  Timeout waiting for a shard to become available.

• consistency

  The write consistency of the index/delete operation.

• refresh

  Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately.

• fields

  Return the relevant fields from the updated document. Specify _source to return the full updated source.

• version & version_type

  The update API uses the Elasticsearch’s versioning support internally to make sure the document doesn’t change during the update. You can use the version parameter to specify that the document should only be updated if it’s version matches the one specified. By setting version type to force you can force the new version of the document after update (use with care! with force there is no guarantee the document didn’t change).Version types external & external_gte are not supported.

Multi-document APIs

• mget

    curl 'localhost:9200/_mget' -d '{
        "docs" : [
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "1"
            },
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "2"
            }
        ]
    }'

• against index

    curl 'localhost:9200/test/_mget' -d '{
        "docs" : [
            {
                "_type" : "type",
                "_id" : "1"
            },
            {
                "_type" : "type",
                "_id" : "2"
            }
        ]
    }'

• against type

    curl 'localhost:9200/test/type/_mget' -d '{
        "docs" : [
            {
                "_id" : "1"
            },
            {
                "_id" : "2"
            }
        ]
    }'

• ids

    curl 'localhost:9200/test/type/_mget' -d '{
        "ids" : ["1", "2"]
    }'

Multi Get API

Optional Type

• same document twice

    curl 'localhost:9200/test/_mget' -d '{
        "ids" : ["1", "1"]
    }'

• explicit

    GET /test/_mget/
    {
      "docs" : [
            {
                "_type":"typeA",
                "_id" : "1"
            },
            {
                "_type":"typeB",
                "_id" : "1"
            }
        ]
    }

Source filtering

_source, _source_include & _source_exclude

    curl 'localhost:9200/_mget' -d '{
        "docs" : [
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "1",
                "_source" : false
            },
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "2",
                "_source" : ["field3", "field4"]
            },
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "3",
                "_source" : {
                    "include": ["user"],
                    "exclude": ["user.location"]
                }
            }
        ]
    }'

Fields

• example

    curl 'localhost:9200/_mget' -d '{
        "docs" : [
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "1",
                "fields" : ["field1", "field2"]
            },
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "2",
                "fields" : ["field3", "field4"]
            }
        ]
    }'

• specify default

    curl 'localhost:9200/test/type/_mget?fields=field1,field2' -d '{
        "docs" : [
            {
                "_id" : "1" 
            },
            {
                "_id" : "2",
                "fields" : ["field3", "field4"] 
            }
        ]
    }'

  result:

    "_id" : "1"  => returns field1 and field2
  
    "_id" : "2"  => returns field3 and field4 

Generated fields

Fields are generated only when indexing.

Routing

• example

    curl 'localhost:9200/_mget?routing=key1' -d '{
        "docs" : [
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "1",
                "_routing" : "key2"
            },
            {
                "_index" : "test",
                "_type" : "type",
                "_id" : "2"
            }
        ]
    }'

  result:

    test/type/1 => key2
    test/type/2 => key1

Security

URL-based access control

Bulk API

The bulk API makes it possible to perform many index/delete operations in a single API call

• Increase the indexing speed
• Client support for bulk requests
  • Perl
  • Python

• Endpoint */_bulk*

    action_and_meta_data\n
    optional_source\n
    action_and_meta_data\n
    optional_source\n
    ....
    action_and_meta_data\n
    optional_source\n           

• Actions

    index
    create
    delete
    update

• Curl

  • text (--data-binary)

  • document (-d)

    The latter doesn't preserve newlines

      $ cat requests
      { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
      { "field1" : "value1" }
      $ curl -s -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
      {"took":7,"items":[{"create":{"_index":"test","_type":"type1","_id":"1","_version":1}}]}        

• Example

    { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
    { "field1" : "value1" }
    { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
    { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
    { "field1" : "value3" }
    { "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
    { "doc" : {"field2" : "value2"} }

• Above

  • merge

  • endpoints(use default unless explicit)

    /_bulk

    /{index}/_bulk

    {index}/{type}/_bulk

  • only action_meta_data is parsed on the receiving node side (fast)
    

  • response - large JSON structure

  • number of actions - should be optimized per workload

  • HTTP API - no chunks (slow down)

Versioning

Routing

Parent

Timestamp

Deprecated in 2.0.0-beta2.

TTL

Deprecated in 2.0.0-beta2.

Write Consistency

Refresh

Update

• Action inline **_retry_on_conflict**
• Supports
  • doc (partial document)
  • upsert
  • doc_as_upsert
  • script
  • params (for script)
  • lang (for script)
  • fields
    

• Example

    { "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
    { "doc" : {"field" : "value"} }
    { "update" : { "_id" : "0", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
    { "script" : { "inline": "ctx._source.counter += param1", "lang" : "js", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
    { "update" : {"_id" : "2", "_type" : "type1", "_index" : "index1", "_retry_on_conflict" : 3} }
    { "doc" : {"field" : "value"}, "doc_as_upsert" : true }
    { "update" : {"_id" : "3", "_type" : "type1", "_index" : "index1", "fields" : ["_source"]} }
    { "doc" : {"field" : "value"} }
    { "update" : {"_id" : "4", "_type" : "type1", "_index" : "index1"} }
    { "doc" : {"field" : "value"}, "fields": ["_source"]}

Security

Term Vectors

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html

Multi termvectors API

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-multi-termvectors.html

Reference

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html