coded_stream.h

coded_stream.h

#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io

This file contains the CodedInputStream and CodedOutputStream classes, which wrap a ZeroCopyInputStream or ZeroCopyOutputStream, respectively, and allow you to read or write individual pieces of data in various formats.

In particular, these implement the varint encoding for integers, a simple variable-length encoding in which smaller numbers take fewer bytes.

Typically these classes will only be used internally by the protocol buffer library in order to encode and decode protocol buffers. Clients of the library only need to know about this class if they wish to write custom message parsing or serialization procedures.

CodedOutputStream example:

 Write some data to "myfile".  First we write a 4-byte "magic number"
 to identify the file type, then write a length-delimited string.  The
 string is composed of a varint giving the length followed by the raw
 bytes.
int fd = open("myfile", O_WRONLY);
ZeroCopyOutputStream* raw_output = new FileOutputStream(fd);
CodedOutputStream* coded_output = new CodedOutputStream(raw_output);

int magic_number = 1234;
char text[[]] = "Hello world!";
coded_output->WriteLittleEndian32(magic_number);
coded_output->WriteVarint32(strlen(text));
coded_output->WriteRaw(text, strlen(text));

delete coded_output;
delete raw_output;
close(fd);

CodedInputStream example:

 Read a file created by the above code.
int fd = open("myfile", O_RDONLY);
ZeroCopyInputStream* raw_input = new FileInputStream(fd);
CodedInputStream coded_input = new CodedInputStream(raw_input);

coded_input->ReadLittleEndian32(&magic_number);
if (magic_number != 1234) {
  cerr << "File not in expected format." << endl;
  return;
}

uint32 size;
coded_input->ReadVarint32(&size);

char* text = new char[[]size + 1];
coded_input->ReadRaw(buffer, size);
text[[]size] = '\0';

delete coded_input;
delete raw_input;
close(fd);

cout << "Text is: " << text << endl;
delete [[]] text;

For those who are interested, varint encoding is defined as follows:

The encoding operates on unsigned integers of up to 64 bits in length. Each byte of the encoded value has the format:

• bits 0-6: Seven bits of the number being encoded.
• bit 7: Zero if this is the last byte in the encoding (in which case all remaining bits of the number are zero) or 1 if more bytes follow. The first byte contains the least-significant 7 bits of the number, the second byte (if present) contains the next-least-significant 7 bits, and so on. So, the binary number 1011000101011 would be encoded in two bytes as "10101011 00101100".

In theory, varint could be used to encode integers of any length. However, for practicality we set a limit at 64 bits. The maximum encoded length of a number is thus 10 bytes.

Classes in this file
CodedInputStream Class which reads and decodes binary data which is composed of varint- encoded integers and fixed-width pieces.
CodedOutputStream Class which encodes and writes binary data which is composed of varint- encoded integers and fixed-width pieces.
CodedOutputStream::StaticVarintSize32 Compile-time equivalent of  VarintSize32().

class CodedInputStream

#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io

Class which reads and decodes binary data which is composed of varint- encoded integers and fixed-width pieces.

Wraps a ZeroCopyInputStream. Most users will not need to deal with CodedInputStream.

Most methods of CodedInputStream that return a bool return false if an underlying I/O error occurs or if the data is malformed. Once such a failure occurs, the CodedInputStream is broken and is no longer useful.

Members
explicit	CodedInputStream(ZeroCopyInputStream * input) Create a  CodedInputStream that reads from the given  ZeroCopyInputStream.
explicit	CodedInputStream(const uint8 * buffer, int size) Create a  CodedInputStream that reads from the given flat array.  more...
	~CodedInputStream() Destroy the  CodedInputStream and position the underlying  ZeroCopyInputStream at the first unread byte.  more...
bool	IsFlat() const Return true if this  CodedInputStream reads from a flat array instead of a  ZeroCopyInputStream.
bool	Skip(int count) Skips a number of bytes.  more...
bool	GetDirectBufferPointer(const void ** data, int * size) Sets *data to point directly at the unread part of the  CodedInputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.  more...
void	GetDirectBufferPointerInline(const void ** data, int * size) Like GetDirectBufferPointer, but this method is inlined, and does not attempt to Refresh() if the buffer is currently empty.
bool	ReadRaw(void * buffer, int size) Read raw bytes, copying them into the given buffer.
bool	ReadString(string * buffer, int size) Like ReadRaw, but reads into a string.  more...
bool	InternalReadStringInline(string * buffer, int size) Like the above, with inlined optimizations.  more...
bool	ReadLittleEndian32(uint32 * value) Read a 32-bit little-endian integer.
bool	ReadLittleEndian64(uint64 * value) Read a 64-bit little-endian integer.
bool	ReadVarint32(uint32 * value) Read an unsigned integer with Varint encoding, truncating to 32 bits.  more...
bool	ReadVarint64(uint64 * value) Read an unsigned integer with Varint encoding.
uint32	ReadTag() Read a tag.  more...
bool	ExpectTag(uint32 expected) Usually returns true if calling  ReadVarint32() now would produce the given value.  more...
bool	ExpectAtEnd() Usually returns true if no more bytes can be read.  more...
bool	LastTagWas(uint32 expected) If the last call to  ReadTag() returned the given value, returns true.  more...
bool	ConsumedEntireMessage() When parsing message (but NOT a group), this method must be called immediately after MergeFromCodedStream() returns (if it returns true) to further verify that the message ended in a legitimate way.  more...
void	SetTotalBytesLimit(int total_bytes_limit, int warning_threshold) Sets the maximum number of bytes that this  CodedInputStream will read before refusing to continue.  more...
void	SetRecursionLimit(int limit) Sets the maximum recursion depth. The default is 100.
bool	IncrementRecursionDepth() Increments the current recursion depth.  more...
void	DecrementRecursionDepth() Decrements the recursion depth.
void	SetExtensionRegistry(const DescriptorPool * pool, MessageFactory * factory) Set the pool used to look up extensions.  more...
constDescriptorPool*	GetExtensionPool() Get the  DescriptorPool set via  SetExtensionRegistry(), or NULL if no pool has been provided.
MessageFactory*	GetExtensionFactory() Get the  MessageFactory set via  SetExtensionRegistry(), or NULL if no factory has been provided.
static const uint8 *	ReadLittleEndian32FromArray(const uint8 * buffer, uint32 * value) These methods read from an externally provided buffer.  more...
static const uint8 *	ReadLittleEndian64FromArray(const uint8 * buffer, uint64 * value) Read a 64-bit little-endian integer.  more...
static const uint8 *	ExpectTagFromArray(const uint8 * buffer, uint32 expected) Like above, except this reads from the specified buffer.  more...
Limits Limits are used when parsing length-delimited embedded messages. After the message's length is read,  PushLimit() is used to prevent the CodedInputStream from reading beyond that length. Once the embedded message has been parsed,  PopLimit() is called to undo the limit.
typedef	int Limit Opaque type used with  PushLimit() and  PopLimit().  more...
Limit	PushLimit(int byte_limit) Places a limit on the number of bytes that the stream may read, starting from the current position.  more...
void	PopLimit(Limit limit) Pops the last limit pushed by  PushLimit().  more...
int	BytesUntilLimit() const Returns the number of bytes left until the nearest limit on the stack is hit, or -1 if no limits are in place.
int	CurrentPosition() const Returns current position relative to the beginning of the input stream.

---

explicit CodedInputStream::CodedInputStream(
        const uint8 * buffer,
        int size)

Create a CodedInputStream that reads from the given flat array.

This is faster than using an ArrayInputStream. PushLimit(size) is implied by this constructor.

---

CodedInputStream::~CodedInputStream()

Destroy the CodedInputStream and position the underlying ZeroCopyInputStream at the first unread byte.

If an error occurred while reading (causing a method to return false), then the exact position of the input stream may be anywhere between the last value that was read successfully and the stream's byte limit.

---

bool CodedInputStream::Skip(
        int count)

Skips a number of bytes.

Returns false if an underlying read error occurs.

---

bool CodedInputStream::GetDirectBufferPointer(
        const void ** data,
        int * size)

Sets *data to point directly at the unread part of the CodedInputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.

This will always either produce a non-empty buffer or return false. If the caller consumes any of this data, it should then call Skip() to skip over the consumed bytes. This may be useful for implementing external fast parsing routines for types of data not covered by the CodedInputStreaminterface.

---

bool CodedInputStream::ReadString(
        string * buffer,
        int size)

Like ReadRaw, but reads into a string.

Implementation Note: ReadString() grows the string gradually as it reads in the data, rather than allocating the entire requested size upfront. This prevents denial-of-service attacks in which a client could claim that a string is going to be MAX_INT bytes long in order to crash the server because it can't allocate this much space at once.

---

bool CodedInputStream::InternalReadStringInline(
        string * buffer,
        int size)

Like the above, with inlined optimizations.

This should only be used by the protobuf implementation.

---

bool CodedInputStream::ReadVarint32(
        uint32 * value)

Read an unsigned integer with Varint encoding, truncating to 32 bits.

Reading a 32-bit value is equivalent to reading a 64-bit one and casting it to uint32, but may be more efficient.

---

uint32 CodedInputStream::ReadTag()

Read a tag.

This calls ReadVarint32() and returns the result, or returns zero (which is not a valid tag) if ReadVarint32() fails. Also, it updates the last tag value, which can be checked with LastTagWas(). Always inline because this is only called in once place per parse loop but it is called for every iteration of said loop, so it should be fast. GCC doesn't want to inline this by default.

---

bool CodedInputStream::ExpectTag(
        uint32 expected)

Usually returns true if calling ReadVarint32() now would produce the given value.

Will always return false if ReadVarint32() would not return the given value. If ExpectTag() returns true, it also advances past the varint. For best performance, use a compile-time constant as the parameter. Always inline because this collapses to a small number of instructions when given a constant parameter, but GCC doesn't want to inline by default.

---

bool CodedInputStream::ExpectAtEnd()

Usually returns true if no more bytes can be read.

Always returns false if more bytes can be read. If ExpectAtEnd() returns true, a subsequent call to LastTagWas() will act as if ReadTag() had been called and returned zero, and ConsumedEntireMessage() will return true.

---

bool CodedInputStream::LastTagWas(
        uint32 expected)

If the last call to ReadTag() returned the given value, returns true.

Otherwise, returns false;

This is needed because parsers for some types of embedded messages (with field type TYPE_GROUP) don't actually know that they've reached the end of a message until they see an ENDGROUP tag, which was actually part of the enclosing message. The enclosing message would like to check that tag to make sure it had the right number, so it calls LastTagWas() on return from the embedded parser to check.

---

bool CodedInputStream::ConsumedEntireMessage()

When parsing message (but NOT a group), this method must be called immediately after MergeFromCodedStream() returns (if it returns true) to further verify that the message ended in a legitimate way.

For example, this verifies that parsing did not end on an end-group tag. It also checks for some cases where, due to optimizations, MergeFromCodedStream() can incorrectly return true.

---

void CodedInputStream::SetTotalBytesLimit(
        int total_bytes_limit,
        int warning_threshold)

Sets the maximum number of bytes that this CodedInputStream will read before refusing to continue.

To prevent integer overflows in the protocol buffers implementation, as well as to prevent servers from allocating enormous amounts of memory to hold parsed messages, the maximum message length should be limited to the shortest length that will not harm usability. The theoretical shortest message that could cause integer overflows is 512MB. The default limit is 64MB. Apps should set shorter limits if possible. If warning_threshold is not -1, a warning will be printed to stderr after warning_threshold bytes are read. For backwards compatibility all negative values get squached to -1, as other negative values might have special internal meanings. An error will always be printed to stderr if the limit is reached.

This is unrelated to PushLimit()/PopLimit().

Hint: If you are reading this because your program is printing a warning about dangerously large protocol messages, you may be confused about what to do next. The best option is to change your design such that excessively large messages are not necessary. For example, try to design file formats to consist of many small messages rather than a single large one. If this is infeasible, you will need to increase the limit. Chances are, though, that your code never constructs a CodedInputStream on which the limit can be set. You probably parse messages by calling things likeMessage::ParseFromString(). In this case, you will need to change your code to instead construct some sort of ZeroCopyInputStream (e.g. anArrayInputStream), construct a CodedInputStream around that, then call Message::ParseFromCodedStream() instead. Then you can adjust the limit. Yes, it's more work, but you're doing something unusual.

---

bool CodedInputStream::IncrementRecursionDepth()

Increments the current recursion depth.

Returns true if the depth is under the limit, false if it has gone over.

---

void CodedInputStream::SetExtensionRegistry(
        const DescriptorPool * pool,
        MessageFactory * factory)

Set the pool used to look up extensions.

Most users do not need to call this as the correct pool will be chosen automatically.

WARNING: It is very easy to misuse this. Carefully read the requirements below. Do not use this unless you are sure you need it. Almost no one does.

Let's say you are parsing a message into message object m, and you want to take advantage of SetExtensionRegistry(). You must follow these requirements:

The given DescriptorPool must contain m->GetDescriptor(). It is not sufficient for it to simply contain a descriptor that has the same name and content -- it must be the *exact object*. In other words:

assert(pool->FindMessageTypeByName(m->GetDescriptor()->full_name()) ==
       m->GetDescriptor());

There are two ways to satisfy this requirement: 1) Use m->GetDescriptor()->pool() as the pool. This is generally useless

 because this is the pool that would be used anyway if you didn't call
 SetExtensionRegistry() at all.

2) Use a DescriptorPool which has m->GetDescriptor()->pool() as an

 "underlay".  Read the documentation for DescriptorPool for more
 information about underlays.

You must also provide a MessageFactory. This factory will be used to construct Message objects representing extensions. The factory's GetPrototype() MUST return non-NULL for any Descriptor which can be found through the provided pool.

If the provided factory might return instances of protocol-compiler- generated (i.e. compiled-in) types, or if the outer message object m is a generated type, then the given factory MUST have this property: If GetPrototype() is given a Descriptor which resides inDescriptorPool::generated_pool(), the factory MUST return the same prototype which MessageFactory::generated_factory() would return. That is, given a descriptor for a generated type, the factory must return an instance of the generated class (NOT DynamicMessage). However, when given a descriptor for a type that is NOT in generated_pool, the factory is free to return any implementation.

The reason for this requirement is that generated sub-objects may be accessed via the standard (non-reflection) extension accessor methods, and these methods will down-cast the object to the generated class type. If the object is not actually of that type, the results would be undefined. On the other hand, if an extension is not compiled in, then there is no way the code could end up accessing it via the standard accessors -- the only way to access the extension is via reflection. When using reflection, DynamicMessage and generated messages are indistinguishable, so it's fine if these objects are represented using DynamicMessage.

Using DynamicMessageFactory on which you have called SetDelegateToGeneratedFactory(true) should be sufficient to satisfy the above requirement.

If either pool or factory is NULL, both must be NULL.

Note that this feature is ignored when parsing "lite" messages as they do not have descriptors.

---

static const uint8 * CodedInputStream::ReadLittleEndian32FromArray(
        const uint8 * buffer,
        uint32 * value)

These methods read from an externally provided buffer.

static

The caller is responsible for ensuring that the buffer has sufficient space. Read a 32-bit little-endian integer.

---

static const uint8 * CodedInputStream::ReadLittleEndian64FromArray(
        const uint8 * buffer,
        uint64 * value)

Read a 64-bit little-endian integer.

static

---

static const uint8 * CodedInputStream::ExpectTagFromArray(
        const uint8 * buffer,
        uint32 expected)

Like above, except this reads from the specified buffer.

The caller is responsible for ensuring that the buffer is large enough to read a varint of the expected size. For best performance, use a compile-time constant as the expected tag parameter.

Returns a pointer beyond the expected tag if it was found, or NULL if it was not.

---

typedef CodedInputStream::Limit

Opaque type used with PushLimit() and PopLimit().

Do not modify values of this type yourself. The only reason that this isn't a struct with private internals is for efficiency.

---

Limit CodedInputStream::PushLimit(
        int byte_limit)

Places a limit on the number of bytes that the stream may read, starting from the current position.

Once the stream hits this limit, it will act like the end of the input has been reached until PopLimit() is called.

As the names imply, the stream conceptually has a stack of limits. The shortest limit on the stack is always enforced, even if it is not the top limit.

The value returned by PushLimit() is opaque to the caller, and must be passed unchanged to the corresponding call to PopLimit().

---

void CodedInputStream::PopLimit(
        Limit limit)

Pops the last limit pushed by PushLimit().

The input must be the value returned by that call to PushLimit().

class CodedOutputStream

#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io

Class which encodes and writes binary data which is composed of varint- encoded integers and fixed-width pieces.

Wraps a ZeroCopyOutputStream. Most users will not need to deal with CodedOutputStream.

Most methods of CodedOutputStream which return a bool return false if an underlying I/O error occurs. Once such a failure occurs, theCodedOutputStream is broken and is no longer useful. The Write* methods do not return the stream status, but will invalidate the stream if an error occurs. The client can probe HadError() to determine the status.

Note that every method of CodedOutputStream which writes some data has a corresponding static "ToArray" version. These versions write directly to the provided buffer, returning a pointer past the last written byte. They require that the buffer has sufficient capacity for the encoded data. This allows an optimization where we check if an output stream has enough space for an entire message before we start writing and, if there is, we call only the ToArray methods to avoid doing bound checks for each individual value. i.e., in the example above:

CodedOutputStream coded_output = new CodedOutputStream(raw_output);
int magic_number = 1234;
char text[[]] = "Hello world!";

int coded_size = sizeof(magic_number) +
                 CodedOutputStream::VarintSize32(strlen(text)) +
                 strlen(text);

uint8* buffer =
    coded_output->GetDirectBufferForNBytesAndAdvance(coded_size);
if (buffer != NULL) {
 The output stream has enough space in the buffer: write directly to
 the array.
  buffer = CodedOutputStream::WriteLittleEndian32ToArray(magic_number,
                                                         buffer);
  buffer = CodedOutputStream::WriteVarint32ToArray(strlen(text), buffer);
  buffer = CodedOutputStream::WriteRawToArray(text, strlen(text), buffer);
} else {
 Make bound-checked writes, which will ask the underlying stream for
 more space as needed.
  coded_output->WriteLittleEndian32(magic_number);
  coded_output->WriteVarint32(strlen(text));
  coded_output->WriteRaw(text, strlen(text));
}

delete coded_output;

Members
explicit	CodedOutputStream(ZeroCopyOutputStream * output) Create an  CodedOutputStream that writes to the given  ZeroCopyOutputStream.
	~CodedOutputStream() Destroy the  CodedOutputStream and position the underlying  ZeroCopyOutputStream immediately after the last byte written.
bool	Skip(int count) Skips a number of bytes, leaving the bytes unmodified in the underlying buffer.  more...
bool	GetDirectBufferPointer(void ** data, int * size) Sets *data to point directly at the unwritten part of the  CodedOutputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.  more...
uint8 *	GetDirectBufferForNBytesAndAdvance(int size) If there are at least "size" bytes available in the current buffer, returns a pointer directly into the buffer and advances over these bytes.  more...
void	WriteRaw(const void * buffer, int size) Write raw bytes, copying them from the given buffer.
void	WriteString(const string & str) Equivalent to WriteRaw(str.data(), str.size()).
void	WriteLittleEndian32(uint32 value) Write a 32-bit little-endian integer.
void	WriteLittleEndian64(uint64 value) Write a 64-bit little-endian integer.
void	WriteVarint32(uint32 value) Write an unsigned integer with Varint encoding.  more...
void	WriteVarint64(uint64 value) Write an unsigned integer with Varint encoding.
void	WriteVarint32SignExtended(int32 value) Equivalent to  WriteVarint32() except when the value is negative, in which case it must be sign-extended to a full 10 bytes.
void	WriteTag(uint32 value) This is identical to  WriteVarint32(), but optimized for writing tags.  more...
int	ByteCount() const Returns the total number of bytes written since this object was created.
bool	HadError() const Returns true if there was an underlying I/O error since this object was created.
static uint8 *	WriteRawToArray(const void * buffer, int size, uint8 * target) Like  WriteRaw() but writing directly to the target array.  more...
static uint8 *	WriteStringToArray(const string & str, uint8 * target) Like  WriteString() but writing directly to the target array.
static uint8 *	WriteLittleEndian32ToArray(uint32 value, uint8 * target) Like  WriteLittleEndian32() but writing directly to the target array.
static uint8 *	WriteLittleEndian64ToArray(uint64 value, uint8 * target) Like  WriteLittleEndian64() but writing directly to the target array.
static uint8 *	WriteVarint32ToArray(uint32 value, uint8 * target) Like  WriteVarint32() but writing directly to the target array.
static uint8 *	WriteVarint64ToArray(uint64 value, uint8 * target) Like  WriteVarint64() but writing directly to the target array.
static uint8 *	WriteVarint32SignExtendedToArray(int32 value, uint8 * target) Like  WriteVarint32SignExtended() but writing directly to the target array.
static uint8 *	WriteTagToArray(uint32 value, uint8 * target) Like  WriteTag() but writing directly to the target array.
static int	VarintSize32(uint32 value) Returns the number of bytes needed to encode the given value as a varint.
static int	VarintSize64(uint64 value) Returns the number of bytes needed to encode the given value as a varint.
static int	VarintSize32SignExtended(int32 value) If negative, 10 bytes. Otheriwse, same as  VarintSize32().

---

bool CodedOutputStream::Skip(
        int count)

Skips a number of bytes, leaving the bytes unmodified in the underlying buffer.

Returns false if an underlying write error occurs. This is mainly useful with GetDirectBufferPointer().

---

bool CodedOutputStream::GetDirectBufferPointer(
        void ** data,
        int * size)

Sets *data to point directly at the unwritten part of the CodedOutputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.

This will always either produce a non-empty buffer or return false. If the caller writes any data to this buffer, it should then call Skip() to skip over the consumed bytes. This may be useful for implementing external fast serialization routines for types of data not covered by theCodedOutputStream interface.

---

uint8 * CodedOutputStream::GetDirectBufferForNBytesAndAdvance(
        int size)

If there are at least "size" bytes available in the current buffer, returns a pointer directly into the buffer and advances over these bytes.

The caller may then write directly into this buffer (e.g. using the ToArray static methods) rather than go through CodedOutputStream. If there are not enough bytes available, returns NULL. The return pointer is invalidated as soon as any other non-const method of CodedOutputStream is called.

---

void CodedOutputStream::WriteVarint32(
        uint32 value)

Write an unsigned integer with Varint encoding.

Writing a 32-bit value is equivalent to casting it to uint64 and writing it as a 64-bit value, but may be more efficient.

---

void CodedOutputStream::WriteTag(
        uint32 value)

This is identical to WriteVarint32(), but optimized for writing tags.

In particular, if the input is a compile-time constant, this method compiles down to a couple instructions. Always inline because otherwise the aformentioned optimization can't work, but GCC by default doesn't want to inline this.

---

static uint8 * CodedOutputStream::WriteRawToArray(
        const void * buffer,
        int size,
        uint8 * target)

Like WriteRaw() but writing directly to the target array.

This is _not_ inlined, as the compiler often optimizes memcpy into inline copy loops. Since this gets called by every field with string or bytes type, inlining may lead to a significant amount of code bloat, with only a minor performance gain.

template struct CodedOutputStream::StaticVarintSize32

#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io

template <typename Value>

Compile-time equivalent of VarintSize32().

Members
const int	value = (Value < (1 << 7)) ? 1 : (Value < (1 << 14)) ? 2 : (Value < (1 << 21)) ? 3 : (Value < (1 << 28)) ? 4 : 5