memcached binary protocol

最新推荐文章于 2021-04-10 11:41:12 发布

xlight2023

最新推荐文章于 2021-04-10 11:41:12 发布

阅读量1.8k

点赞数

分类专栏：缓存技术

缓存技术专栏收录该内容

1 篇文章 0 订阅

订阅专栏

Introduction

Memcache is a high performance key-value cache. It is intentionally a dumb cache, optimized for speed only. Applications using memcache should not rely on it for data -- a persistent database with guaranteed reliability is strongly recommended -- but applications can run much faster when cached data is available in memcache.

Memcache was originally written to make LiveJournal [LJ] faster. It now powers all of the fastest web sites that you love.

Conventions Used In This Document

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BinaryProtocolRevamped#Normative_References .

Packet Structure

General format of a packet:

Response header

Header fields description

Magic

Magic number identifying the package (See BinaryProtocolRevamped#Magic_Byte)

Opcode

Command code (See BinaryProtocolRevamped#Command_opcodes)

Key length

Length in bytes of the text key that follows the command extras

Status

Status of the response (non-zero on error) (See BinaryProtocolRevamped#Response_Status)

Extras length

Length in bytes of the command extras

Data type

Reserved for future use (See BinaryProtocolRevamped#Data_Type)

vbucket id

The virtual bucket for this command

Total body length

Length in bytes of extra + key + value

Opaque

Will be copied back to you in the response

CAS

Data version check

Defined Values

Magic Byte

Magic byte / version. For each version of the protocol, we'll use a different request/response value pair. This is useful for protocol analyzers to distinguish the nature of the packet from the direction which it is moving. Note, it is common to run a memcached instance on a host that also runs an application server. Such a host will both send and receive memcache packets.

The version should hopefully correspond only to different meanings of the command byte. In an ideal world, we will not change the header format. As reserved bytes are given defined meaning, the protocol version / magic byte values should be incremented.

Traffic analysis tools are encouraged to identify memcache packets and provide detailed interpretation if the magic bytes are recognized and otherwise to provide a generic breakdown of the packet. Note, that the key and value positions can always be identified even if the magic byte or command opcode are not recognized.

Response Status

Possible values of this two-byte field:

Command Opcodes

Possible values of the one-byte field. See BinaryProtocolRevamped#Commands for more information about a given command.

Commands marked with is not currently fixed, but this is the current proposal for 1.6.

As a convention all of the commands ending with "Q" for Quiet. A quiet version of a command will omit responses that are considered uninteresting. Whether a given response is interesting is dependent upon the command. See the descriptions of the set commands (Section 4.2) and set commands (Section 4.3) for examples of commands that include quiet variants.

Data Types

Possible values of the one-byte field:

Commands

Introduction

All communication is initiated by a request from the client, and the server will respond to each request with zero or multiple packets for each request. If the status code of a response packet is non-nil, the body of the packet will contain a textual error message. If the status code is nil, the command opcode will define the layout of the body of the message.

Example

The following figure illustrates the packet layout for a packet with an error message.

Get, Get Quietly, Get Key, Get Key Quietly

Request:

MUST NOT have extras.
MUST have key.
MUST NOT have value.

Response (if found):

MUST have extras.
MAY have key.
MAY have value.

Extra data for the get commands:

The get command gets a single key. The getq command will not send a response on a cache miss. Getk and getkq differs from get and getq by adding the key into the response packet.

Clients should implement multi-get (still important for reducing network roundtrips!) as n pipelined requests, the first n-1 being getq/getkq, the last being a regular get/getk. That way you're guaranteed to get a response, and you know when the server's done. You can also do the naive thing and send n pipelined get/getks, but then you could potentially get back a lot of "NOT_FOUND" error code packets. Alternatively, you can send 'n' getq/getkqs, followed by a 'noop' command.

Example

To request the data associated with the key "Hello" the following fields must be specified in the packet.

If the item exist on the server the following packet is returned, otherwise a packet with status code != 0 will be returned (see Introduction (Section 4.1))

The response packet for a getk and getkq request differs from get(q) in that the key is present:

Set, Add, Replace

Request:

MUST have extras.
MUST have key.
MAY have value.

Extra data for set/add/replace:

Response:

MUST have CAS
MUST NOT have extras
MUST NOT have key
MUST NOT have value

Constraints:

If the Data Version Check (CAS) is nonzero, the requested operation MUST only succeed if the item exists and has a CAS value identical to the provided value.
Add MUST fail if the item already exist.
Replace MUST fail if the item doesn't exist.
Set should store the data unconditionally if the item exists or not.

Quiet mutations only return responses on failure. Success is considered the general case and is suppressed when in quiet mode, but errors should not be allowed to go unnoticed.

Example

The following figure shows an add-command for with key = "Hello", value = "World", flags = 0xdeadbeef and expiry: in two hours.

The result of the operation is signaled through the status code. If the command succeeds, the CAS value for the item is returned in the CAS-field of the packet:

Delete

Request:

MUST NOT have extras.
MUST have key.
MUST NOT have value.

Response:

MUST NOT have extras
MUST NOT have key
MUST NOT have value

Delete the item with the specific key.

Example

The following figure shows a delete message for the item "Hello".

The result of the operation is signaled through the status code.

Increment, Decrement

Request:

MUST have extras.
MUST have key.
MUST NOT have value.

Extra data for incr/decr:

Response:

MUST NOT have extras.
MUST NOT have key.
MUST have value.

These commands will either add or remove the specified amount to the requested counter. If you want to set the value of the counter with add/set/replace, the objects data must be the ascii representation of the value and not the byte values of a 64 bit integer.

If the counter does not exist, one of two things may happen:

If the expiration value is all one-bits (0xffffffff), the operation will fail with NOT_FOUND.
For all other expiration values, the operation will succeed by seeding the value for this key with the provided initial value to expire with the provided expiration time. The flags will be set to zero.

Decrementing a counter will never result in a "negative value" (or cause the counter to "wrap"). instead the counter is set to 0. Incrementing the counter may cause the counter to wrap.

Example

The following figure shows an incr-command for key = "counter", delta = 0x01, initial = 0x00 which expires in two hours.

If the key doesn't exist, the server will respond with the initial value. If not the incremented value will be returned. Let's assume that the key didn't exist, so the initial value is returned:

quit

Request:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Response:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Close the connection to the server.

Example

The response-packet contains no extra data, and the result of the operation is signaled through the status code. The server will then close the connection.

Flush

Request:

MAY have extras.
MUST NOT have key.
MUST NOT have value.

Extra data for flush:

Response:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Flush the items in the cache now or some time in the future as specified by the expiration field. See the documentation of the textual protocol for the full description on how to specify the expiration time.

Example

To flush the cache (delete all items) in two hours, the set the following values in the request

The result of the operation is signaled through the status code.

noop

Request:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Response:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Used as a keep alive.

Example

Noop request:

The result of the operation is signaled through the status code.

version

Request:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Response:

MUST NOT have extras.
MUST NOT have key.
MUST have value.

Request the server version.

The server responds with a packet containing the version string in the body with the following format: "x.y.z"

Example

Request:

Response:

Append, Prepend

Request:

MUST NOT have extras.
MUST have key.
MUST have value.

Response:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.
MUST have CAS

These commands will either append or prepend the specified value to the requested key.

Example

The following example appends '!' to the 'Hello' key.

The result of the operation is signaled through the status code.

Stat

Request:

MUST NOT have extras.
MAY have key.
MUST NOT have value.

Response:

MUST NOT have extras.
MAY have key.
MAY have value.

Request server statistics. Without a key specified the server will respond with a "default" set of statistics information. Each piece of statistical information is returned in its own packet (key contains the name of the statistical item and the body contains the value in ASCII format). The sequence of return packets is terminated with a packet that contains no key and no value.

Example

The following example requests all statistics from the server

The server will send each value in a separate packet with an "empty" packet (no key / no value) to terminate the sequence. Each of the response packets look like the following example:

Verbosity

Request:

MUST have extras.
MUST NOT have key.
MUST NOT have value.

Extra data for flush:

Response:

MUST NOT have extras.
MUST NOT have key.
MUST NOT have value.

Set the verbosity level of the server. This may cause your memcached server to generate more or less output.

Example

To set the verbosity level to two, set the following values in the request

@todo add me

The result of the operation is signaled through the status code.

Touch, GAT and GATQ

Request:

MUST have extras.
MUST have key.
MUST NOT have value.

Extra data for set/add/replace:

Touch is used to set a new expiration time for an existing item. GAT (Get and touch) and GATQ will return the value for the object if it is present in the cache.

Example

@todo add me

Get/Set/Del VBucket

@todo add me

TAP Connect

@todo add me

TAP Mutation / Delete / Flush

@todo add me

TAP Opaque

@todo add me

TAP VBucket set

@todo add me

TAP Checkpoint start/stop

@todo add me

Security Considerations

Memcache has no authentication or security layers whatsoever. It is RECOMMENDED that memcache be deployed strictly on closed, protected, back-end networks within a single data center, within a single cluster of servers, or even on a single host, providing shared caching for multiple applications. Memcache MUST NOT be made available on a public network.