Avro C++

Namespaces | Classes | Typedefs | Enumerations | Functions | Variables
avro Namespace Reference

A bunch of templates and specializations for encoding and decoding specific types. More...

## Namespaces
| concepts
| The concept classes are used to simplify NodeImpl.

## Classes
class | ArraySchema
class | BoolSchema
class | BytesSchema
struct | codec_traits
| Codec_traits tells avro how to encode and decode an object of given type. More...
struct | codec_traits< avro::null >
| codec_traits for Avro null. More...
struct | codec_traits< bool >
| codec_traits for Avro boolean. More...
struct | codec_traits< double >
| codec_traits for Avro double. More...
struct | codec_traits< float >
| codec_traits for Avro float. More...
struct | codec_traits< GenericDatum >
| Specialization of codec_traits for GenericDatum. More...
struct | codec_traits< int32_t >
| codec_traits for Avro int. More...
struct | codec_traits< int64_t >
| codec_traits for Avro long. More...
struct | codec_traits< std::array< uint8_t, N > >
| codec_traits for Avro fixed. More...
struct | codec_traits< std::conditional< avro::is_not_defined< bool_codec_traits >::value, std::vector< bool >::const_reference, void >::type >
struct | codec_traits< std::map< std::string, T > >
| codec_traits for Avro maps. More...
struct | codec_traits< std::pair< ValidSchema, GenericDatum > >
| Specialization of codec_traits for Generic datum along with its schema. More...
struct | codec_traits< std::string >
| codec_traits for Avro string. More...
struct | codec_traits< std::vector< T > >
| codec_traits for Avro arrays. More...
struct | codec_traits< std::vector< uint8_t > >
| codec_traits for Avro bytes. More...
class | CompoundLayout
class | CustomAttributes
class | DataFileReader
| Reads the contents of data file one after another. More...
class | DataFileReaderBase
| The type independent portion of reader. More...
class | DataFileWriter
| An Avro datafile that can store objects of type T. More...
class | DataFileWriterBase
| Type-independent portion of DataFileWriter. More...
class | Decoder
| Decoder is an interface implemented by every decoder capable of decoding Avro data. More...
class | DoubleSchema
class | Encoder
| The abstract base class for all Avro encoders. More...
class | EnumSchema
class | Exception
| Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More...
class | FixedSchema
class | FloatSchema
class | GenericArray
| The generic container for Avro arrays. More...
class | GenericContainer
| The base class for all generic type for containers. More...
class | GenericDatum
| Generic datum which can hold any Avro type. More...
class | GenericEnum
| Generic container for Avro enum. More...
class | GenericFixed
| Generic container for Avro fixed. More...
class | GenericMap
| The generic container for Avro maps. More...
class | GenericReader
| A utility class to read generic datum from decoders. More...
class | GenericRecord
| The generic container for Avro records. More...
class | GenericUnion
| Generic container for unions. More...
class | GenericWriter
| A utility class to write generic datum to encoders. More...
class | InputStream
| A no-copy input stream. More...
class | IntSchema
struct | is_defined
| Check if a `T` is a complete type i.e. More...
struct | is_not_defined
| Similar to is_defined, but used to check if T is not defined. More...
struct | is_promotable
struct | is_serializable
| Define an is_serializable trait for types we can serialize natively. More...
class | Layout
class | LogicalType
class | LongSchema
class | MapSchema
class | Name
class | Node
| Node is the building block for parse trees. More...
class | NodeArray
class | NodeEnum
class | NodeFixed
class | NodeImpl
| Implementation details for Node. More...
class | NodeMap
class | NodePrimitive
class | NodeRecord
class | NodeSymbolic
class | NodeUnion
struct | Null
| define a type to represent Avro Null in template functions More...
class | NullSchema
class | NullValidator
class | OutputStream
| A no-copy output stream. More...
class | Parser
| Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More...
class | PrimitiveLayout
class | ReaderImpl
| Parses from an avro encoding to the requested type. More...
class | RecordSchema
class | Resolver
class | ResolverSchema
class | ResolvingDecoder
| ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fields within a record. More...
class | ResolvingReader
class | Schema
| The root Schema object is a base class. Nobody constructs this class directly. More...
class | SeekableInputStream
| An InputStream which also supports seeking to a specific offset. More...
class | Serializer
| Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More...
struct | StreamReader
| A convenience class for reading from an InputStream. More...
struct | StreamWriter
| A convenience class to write data into an OutputStream. More...
class | StringSchema
class | SymbolicSchema
struct | type_to_avro
class | UnionSchema
class | Validator
| This class is used by both the ValidatingSerializer and ValidationParser objects. More...
class | ValidSchema
| A ValidSchema is basically a non-mutable Schema that has passed some minimum of sanity checks. More...
class | WriterImpl
| Class for writing avro data to a stream. More...

## Typedefs
typedef std::array< uint8_t, SyncSize > | DataFileSync
| The sync value.
using | DecoderPtr = std::shared_ptr< Decoder >
| Shared pointer to Decoder.
using | ResolvingDecoderPtr = std::shared_ptr< ResolvingDecoder >
| Shared pointer to ResolvingDecoder.
using | EncoderPtr = std::shared_ptr< Encoder >
| Shared pointer to Encoder.
using | NodePtr = std::shared_ptr< Node >
using | NoName = concepts::NoAttribute< Name >
using | HasName = concepts::SingleAttribute< Name >
using | HasDoc = concepts::SingleAttribute< std::string >
using | NoLeaves = concepts::NoAttribute< NodePtr >
using | SingleLeaf = concepts::SingleAttribute< NodePtr >
using | MultiLeaves = concepts::MultiAttribute< NodePtr >
using | NoLeafNames = concepts::NoAttribute< std::string >
using | LeafNames = concepts::MultiAttribute< std::string >
using | MultiAttributes = concepts::MultiAttribute< CustomAttributes >
using | NoAttributes = concepts::NoAttribute< CustomAttributes >
using | NoSize = concepts::NoAttribute< int >
using | HasSize = concepts::SingleAttribute< int >
using | NodeImplPrimitive = NodeImpl< NoName, NoLeaves, NoLeafNames, MultiAttributes, NoSize >
using | NodeImplSymbolic = NodeImpl< HasName, NoLeaves, NoLeafNames, NoAttributes, NoSize >
using | NodeImplRecord = NodeImpl< HasName, MultiLeaves, LeafNames, MultiAttributes, NoSize >
using | NodeImplEnum = NodeImpl< HasName, NoLeaves, LeafNames, NoAttributes, NoSize >
using | NodeImplArray = NodeImpl< NoName, SingleLeaf, NoLeafNames, NoAttributes, NoSize >
using | NodeImplMap = NodeImpl< NoName, MultiLeaves, NoLeafNames, NoAttributes, NoSize >
using | NodeImplUnion = NodeImpl< NoName, MultiLeaves, NoLeafNames, NoAttributes, NoSize >
using | NodeImplFixed = NodeImpl< HasName, NoLeaves, NoLeafNames, NoAttributes, HasSize >
using | Reader = ReaderImpl< NullValidator >
using | ValidatingReader = ReaderImpl< Validator >
typedef boost::blank | null
typedef codec_traits< std::vector< bool >::const_reference > | bool_codec_traits
typedef std::unique_ptr< InputStream > | InputStreamPtr
typedef std::unique_ptr< SeekableInputStream > | SeekableInputStreamPtr
typedef std::unique_ptr< OutputStream > | OutputStreamPtr
using | Writer = WriterImpl< NullValidator >
using | ValidatingWriter = WriterImpl< Validator >

## Enumerations
enum | Codec { NULL_CODEC, DEFLATE_CODEC }
| Specify type of compression to use when writing data files.
enum | SchemaResolution { RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT, RESOLVE_PROMOTABLE_TO_DOUBLE }
enum | Type { AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG, AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL, AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP, AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES, AVRO_UNKNOWN = -1 }
| The "type" for the schema. More...

## Functions
template<typename Reader , typename T >
void | parse (Reader &p, T &val)
| The main parse entry point function. More...
template<typename T >
void | parse (ResolvingReader &p, T &val)
template<typename Reader , typename T >
void | parse (Reader &p, T &val, const std::false_type &)
| Type trait should be set to is_serializable in otherwise force the compiler to complain.
template<typename Reader , typename T >
void | translatingParse (Reader &p, T &val, const std::false_type &)
template<typename Reader , typename T >
void | parse (Reader &p, T &val, const std::true_type &)
| The remainder of the file includes default implementations for serializable types.
template<typename Reader >
void | parse (Reader &p, std::vector< uint8_t > &val, const std::true_type &)
template<typename T >
void | translatingParse (ResolvingReader &p, T &val, const std::true_type &)
template<typename Writer , typename T >
void | serialize (Writer &s, const T &val)
| The main serializer entry point function. More...
template<typename Writer , typename T >
void | serialize (Writer &s, const T &val, const std::false_type &)
| Type trait should be set to is_serializable in otherwise force the compiler to complain.
template<typename Writer , typename T >
void | serialize (Writer &s, T val, const std::true_type &)
| The remainder of the file includes default implementations for serializable types.
template<typename Writer >
void | serialize (Writer &s, const std::vector< uint8_t > &val, const std::true_type &)
AVRO_DECL void | compileJsonSchema (std::istream &is, ValidSchema &schema)
| Given a stream containing a JSON schema, compiles the schema to a ValidSchema object. More...
AVRO_DECL bool | compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error)
| Non-throwing version of compileJsonSchema. More...
AVRO_DECL ValidSchema | compileJsonSchemaFromStream (InputStream &is)
AVRO_DECL ValidSchema | compileJsonSchemaFromMemory (const uint8_t *input, size_t len)
AVRO_DECL ValidSchema | compileJsonSchemaFromString (const char *input)
AVRO_DECL ValidSchema | compileJsonSchemaFromString (const std::string &input)
AVRO_DECL ValidSchema | compileJsonSchemaFromFile (const char *filename)
AVRO_DECL DecoderPtr | binaryDecoder ()
| Returns an decoder that can decode binary Avro standard.
AVRO_DECL DecoderPtr | validatingDecoder (const ValidSchema &schema, const DecoderPtr &base)
| Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema.
AVRO_DECL DecoderPtr | jsonDecoder (const ValidSchema &schema)
| Returns an decoder that can decode Avro standard for JSON.
AVRO_DECL ResolvingDecoderPtr | resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base)
| Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema. More...
AVRO_DECL EncoderPtr | binaryEncoder ()
| Returns an encoder that can encode binary Avro standard.
AVRO_DECL EncoderPtr | validatingEncoder (const ValidSchema &schema, const EncoderPtr &base)
| Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema.
AVRO_DECL EncoderPtr | jsonEncoder (const ValidSchema &schema)
| Returns an encoder that encodes Avro standard for JSON.
AVRO_DECL EncoderPtr | jsonPrettyEncoder (const ValidSchema &schema)
| Returns an encoder that encodes Avro standard for pretty printed JSON.
std::ostream & | operator<< (std::ostream &os, const Name &n)
NodePtr | resolveSymbol (const NodePtr &node)
template<typename T >
std::string | intToHex (T i)
Type | nextType (Parser< ValidatingReader > &p)
bool | currentRecordName (Parser< ValidatingReader > &p, std::string &name)
bool | nextFieldName (Parser< ValidatingReader > &p, std::string &name)
std::unique_ptr< Resolver > | constructResolver (const ValidSchema &writerSchema, const ValidSchema &readerSchema, const Layout &readerLayout)
template<typename T >
void | encode (Encoder &e, const T &t)
| Generic encoder function that makes use of the codec_traits.
template<typename T >
void | decode (Decoder &d, T &t)
| Generic decoder function that makes use of the codec_traits.
AVRO_DECL OutputStreamPtr | memoryOutputStream (size_t chunkSize=4 *1024)
| Returns a new OutputStream, which grows in memory chunks of specified size.
AVRO_DECL InputStreamPtr | memoryInputStream (const uint8_t *data, size_t len)
| Returns a new InputStream, with the data from the given byte array. More...
AVRO_DECL InputStreamPtr | memoryInputStream (const OutputStream &source)
| Returns a new InputStream with the contents written into an OutputStream. More...
AVRO_DECL std::shared_ptr< std::vector< uint8_t > > | snapshot (const OutputStream &source)
| Returns the contents written so far into the output stream, which should be a memory output stream. More...
AVRO_DECL OutputStreamPtr | fileOutputStream (const char *filename, size_t bufferSize=8 *1024)
| Returns a new OutputStream whose contents would be stored in a file. More...
AVRO_DECL InputStreamPtr | fileInputStream (const char *filename, size_t bufferSize=8 *1024)
| Returns a new InputStream whose contents come from the given file. More...
AVRO_DECL SeekableInputStreamPtr | fileSeekableInputStream (const char *filename, size_t bufferSize=8 *1024)
AVRO_DECL OutputStreamPtr | ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024)
| Returns a new OutputStream whose contents will be sent to the given std::ostream. More...
AVRO_DECL InputStreamPtr | istreamInputStream (std::istream &in, size_t bufferSize=8 *1024)
| Returns a new InputStream whose contents come from the given std::istream. More...
AVRO_DECL InputStreamPtr | nonSeekableIstreamInputStream (std::istream &is, size_t bufferSize=8 *1024)
| Returns a new InputStream whose contents come from the given std::istream. More...
void | copy (InputStream &in, OutputStream &out)
| A convenience function to copy all the contents of an input stream into an output stream.
constexpr bool | isPrimitive (Type t) noexcept
| Returns true if and only if the given type is a primitive. More...
constexpr bool | isCompound (Type t) noexcept
| Returns true if and only if the given type is a non primitive valid type. More...
constexpr bool | isAvroType (Type t) noexcept
| Returns true if and only if the given type is a valid avro type.
constexpr bool | isAvroTypeOrPseudoType (Type t) noexcept
| Returns true if and only if the given type is within the valid range of enumeration.
const AVRO_DECL std::string & | toString (Type type) noexcept
| Converts the given type into a string. More...
AVRO_DECL std::ostream & | operator<< (std::ostream &os, avro::Type type)
| Writes a string form of the given type into the given ostream.
std::ostream & | operator<< (std::ostream &os, const Null &null)
| Writes schema for null `null` type to `os`. More...
constexpr AVRO_DECL uint64_t | encodeZigzag64 (int64_t input) noexcept
constexpr AVRO_DECL int64_t | decodeZigzag64 (uint64_t input) noexcept
constexpr AVRO_DECL uint32_t | encodeZigzag32 (int32_t input) noexcept
constexpr AVRO_DECL int32_t | decodeZigzag32 (uint32_t input) noexcept
AVRO_DECL size_t | encodeInt32 (int32_t input, std::array< uint8_t, 5 > &output) noexcept
AVRO_DECL size_t | encodeInt64 (int64_t input, std::array< uint8_t, 10 > &output) noexcept

## Variables
class AVRO_DECL | InputStream
class AVRO_DECL | ValidSchema
| This class is used to implement an avro spec parser using a flex/bison compiler. More...
const int | SyncSize = 16
class AVRO_DECL | Schema

## Detailed Description

A bunch of templates and specializations for encoding and decoding specific types.

Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for arbitrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.

Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.

## Enumeration Type Documentation

## ◆ SchemaResolution

enum avro::SchemaResolution

Enumerator
RESOLVE_NO_MATCH | The schemas definitely do not match.
RESOLVE_MATCH | The schemas match at a cursory level.
    For records and enums, this means the name is the same, but it does not
    necessarily mean that every symbol or field is an exact match.
RESOLVE_PROMOTABLE_TO_LONG | For primitives, the matching may occur if the type is promotable. This means that the writer matches reader if the writer's type is promoted the specified type.

## ◆ Type

enum avro::Type

The "type" for the schema.

Enumerator
AVRO_STRING | String
AVRO_BYTES | Sequence of variable length bytes data
AVRO_INT | 32-bit integer
AVRO_LONG | 64-bit integer
AVRO_FLOAT | Floating point number
AVRO_DOUBLE | Double precision floating point number
AVRO_BOOL | Boolean value
AVRO_NULL | Null
AVRO_RECORD | Record, a sequence of fields
AVRO_ENUM | Enumeration
AVRO_ARRAY | Homogeneous array of some specific type
AVRO_MAP | Homogeneous map from string to some specific type
AVRO_UNION | Union of one or more types
AVRO_FIXED | Fixed number of bytes
AVRO_NUM_TYPES | Marker
AVRO_SYMBOLIC | User internally to avoid circular references.
AVRO_UNKNOWN | Used internally.

## Function Documentation

## ◆ compileJsonSchema() [1/2]

AVRO_DECL void avro::compileJsonSchema | ( | std::istream & | is,
|  | ValidSchema & | schema
| ) |  |

Given a stream containing a JSON schema, compiles the schema to a ValidSchema object.

Throws if the schema cannot be compiled to a valid schema

## ◆ compileJsonSchema() [2/2]

AVRO_DECL bool avro::compileJsonSchema | ( | std::istream & | is,
|  | ValidSchema & | schema,
|  | std::string & | error
| ) |  |

Non-throwing version of compileJsonSchema.

Returns
  True if no error, false if error (with the error string set)

## ◆ fileInputStream()

AVRO_DECL InputStreamPtr avro::fileInputStream | ( | const char * | filename,
|  | size_t | bufferSize = `8 *1024`
| ) |  |

Returns a new InputStream whose contents come from the given file.

Data is read in chunks of given buffer size.

## ◆ fileOutputStream()

AVRO_DECL OutputStreamPtr avro::fileOutputStream | ( | const char * | filename,
|  | size_t | bufferSize = `8 *1024`
| ) |  |

Returns a new OutputStream whose contents would be stored in a file.

Data is written in chunks of given buffer size.

If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.

## ◆ isCompound()

constexpr bool avro::isCompound | ( | Type | t | ) |
| inline constexpr noexcept

Returns true if and only if the given type is a non primitive valid type.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_NUM_TYPES, and AVRO_RECORD.

## ◆ isPrimitive()

constexpr bool avro::isPrimitive | ( | Type | t | ) |
| inline constexpr noexcept

Returns true if and only if the given type is a primitive.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_RECORD, and AVRO_STRING.

## ◆ istreamInputStream()

AVRO_DECL InputStreamPtr avro::istreamInputStream | ( | std::istream & | in,
|  | size_t | bufferSize = `8 *1024`
| ) |  |

Returns a new InputStream whose contents come from the given std::istream.

The std::istream object should outlive the returned InputStream.

## ◆ memoryInputStream() [1/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream | ( | const OutputStream & | source | ) |

Returns a new InputStream with the contents written into an OutputStream.

The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new InputStream are the snapshot of the output stream. One can construct any number of memory input stream from a single memory output stream.

## ◆ memoryInputStream() [2/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream | ( | const uint8_t * | data,
|  | size_t | len
| ) |  |

Returns a new InputStream, with the data from the given byte array.

It does not copy the data, the byte array should remain valid until the InputStream is used.

## ◆ nonSeekableIstreamInputStream()

AVRO_DECL InputStreamPtr avro::nonSeekableIstreamInputStream | ( | std::istream & | is,
|  | size_t | bufferSize = `8 *1024`
| ) |  |

Returns a new InputStream whose contents come from the given std::istream.

Use this instead of istreamInputStream if the istream does not support seekg (e.g. compressed streams). The returned InputStream would read off bytes instead of seeking. Of, course it has a performance penalty when reading instead of seeking; So, use this only when seekg does not work. The std::istream object should outlive the returned InputStream.

## ◆ operator<<()

std::ostream& avro::operator<< | ( | std::ostream & | os,
|  | const Null & | null
| ) |  |

Writes schema for null `null` type to `os`.

Parameters

  os | The ostream to write to.
  null | The value to be written.

## ◆ ostreamOutputStream()

AVRO_DECL OutputStreamPtr avro::ostreamOutputStream | ( | std::ostream & | os,
|  | size_t | bufferSize = `8 *1024`
| ) |  |

Returns a new OutputStream whose contents will be sent to the given std::ostream.

The std::ostream object should outlive the returned OutputStream.

## ◆ parse()

template<typename Reader , typename T >

void avro::parse | ( | Reader & | p,
|  | T & | val
| ) |  |

The main parse entry point function.

Takes a parser (either validating or plain) and the object that should receive the parsed data.

## ◆ resolvingDecoder()

AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder | ( | const ValidSchema & | writer,
|  | const ValidSchema & | reader,
|  | const DecoderPtr & | base
| ) |  |

Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.

The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.

## ◆ serialize()

template<typename Writer , typename T >

void avro::serialize | ( | Writer & | s,
|  | const T & | val
| ) |  |

The main serializer entry point function.

Takes a serializer (either validating or plain) and the object that should be serialized.

## ◆ snapshot()

AVRO_DECL std::shared_ptr<std::vector<uint8_t> > avro::snapshot | ( | const OutputStream & | source | ) |

Returns the contents written so far into the output stream, which should be a memory output stream.

That is it must have been returned by a previous call to memoryOutputStream().

## ◆ toString()

const AVRO_DECL std::string& avro::toString | ( | Type | type | ) |
| noexcept

Converts the given type into a string.

Useful for generating messages.

## Variable Documentation

## ◆ ValidSchema

class AVRO_DECL avro::ValidSchema

This class is used to implement an avro spec parser using a flex/bison compiler.

In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.