v2_0_c_overview

msgpack-c C version users guide

This document is under construction.

How to use

Include msgpack.h header file.

#include <msgpack.h>

Link msgpack library. When you use UNIX, the library name is libmsgpackc.a. When you use Windows, the library name is msgpackc.lib for static link and msgpackc_import.lib for dynamic link.

Packing

When you pack your data, use buffers and a packer.

msgpack_sbuffer sbuf; /* buffer */
msgpack_packer pk;    /* packer */

msgpack_sbuffer_init(&sbuf); /* initialize buffer */
msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); /* initialize packer */

Buffers

The buffers store packed byte data. The buffers are used by packer. msgpack-c provides four buffers. sbuffer, fbuffer, zbuffer, and vrefbuffer.

sbuffer is a simple memory buffer. It uses malloc, realloc, and free. fbuffer is a file buffer. It uses FILE structure in C standard library. zbuffer is a compressing buffer using zlib. So packed data is compressed. vrefbuffer is an iovec based buffer. So packed data is stored as iovec.

Initialize, Destroy, and Write

All buffers have at least three functions. Initialize, Destroy, and Write.

Buffer	Initialize	Destroy	Write
sbuffer	msgpack_sbuffer_init	msgpack_sbuffer_destroy	msgpack_sbuffer_write
zbuffer	msgpack_zbuffer_init	msgpack_zbuffer_destroy	msgpack_zbuffer_write
vrefbuffer	msgpack_vrefbuffer_init	msgpack_vrefbuffer_destroy	msgpack_vrefbuffer_write
fbuffer	fopen	fclose	msgpack_fbuffer_write

When you initialize packer, you pass a pointer to the buffer as the second argument, and pass write function pointer as the third argument. Packer calls the write function through the pointer internally.

The buffers have additional buffer specific functions. Please check the source code.

Packer

Packer provides packing mechanism. Packer requires a buffer. In order to use packer, call msgpack_packer_init() as follows:

msgpack_sbuffer sbuf; /* buffer */
msgpack_packer pk;    /* packer */

msgpack_sbuffer_init(&sbuf); /* initialize buffer */
msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); /* initialize packer */

After initializing, you can call the following packing functions:

int msgpack_pack_char(msgpack_packer* pk, char d);

int msgpack_pack_signed_char(msgpack_packer* pk, signed char d);
int msgpack_pack_short(msgpack_packer* pk, short d);
int msgpack_pack_int(msgpack_packer* pk, int d);
int msgpack_pack_long(msgpack_packer* pk, long d);
int msgpack_pack_long_long(msgpack_packer* pk, long long d);
int msgpack_pack_unsigned_char(msgpack_packer* pk, unsigned char d);
int msgpack_pack_unsigned_short(msgpack_packer* pk, unsigned short d);
int msgpack_pack_unsigned_int(msgpack_packer* pk, unsigned int d);
int msgpack_pack_unsigned_long(msgpack_packer* pk, unsigned long d);
int msgpack_pack_unsigned_long_long(msgpack_packer* pk, unsigned long long d);

int msgpack_pack_uint8(msgpack_packer* pk, uint8_t d);
int msgpack_pack_uint16(msgpack_packer* pk, uint16_t d);
int msgpack_pack_uint32(msgpack_packer* pk, uint32_t d);
int msgpack_pack_uint64(msgpack_packer* pk, uint64_t d);
int msgpack_pack_int8(msgpack_packer* pk, int8_t d);
int msgpack_pack_int16(msgpack_packer* pk, int16_t d);
int msgpack_pack_int32(msgpack_packer* pk, int32_t d);
int msgpack_pack_int64(msgpack_packer* pk, int64_t d);

int msgpack_pack_fix_uint8(msgpack_packer* pk, uint8_t d);
int msgpack_pack_fix_uint16(msgpack_packer* pk, uint16_t d);
int msgpack_pack_fix_uint32(msgpack_packer* pk, uint32_t d);
int msgpack_pack_fix_uint64(msgpack_packer* pk, uint64_t d);
int msgpack_pack_fix_int8(msgpack_packer* pk, int8_t d);
int msgpack_pack_fix_int16(msgpack_packer* pk, int16_t d);
int msgpack_pack_fix_int32(msgpack_packer* pk, int32_t d);
int msgpack_pack_fix_int64(msgpack_packer* pk, int64_t d);

int msgpack_pack_float(msgpack_packer* pk, float d);
int msgpack_pack_double(msgpack_packer* pk, double d);

int msgpack_pack_nil(msgpack_packer* pk);
int msgpack_pack_true(msgpack_packer* pk);
int msgpack_pack_false(msgpack_packer* pk);

int msgpack_pack_array(msgpack_packer* pk, size_t n);

int msgpack_pack_map(msgpack_packer* pk, size_t n);

int msgpack_pack_str(msgpack_packer* pk, size_t l);
int msgpack_pack_str_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_v4raw(msgpack_packer* pk, size_t l);
int msgpack_pack_v4raw_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_bin(msgpack_packer* pk, size_t l);
int msgpack_pack_bin_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_ext(msgpack_packer* pk, size_t l, int8_t type);
int msgpack_pack_ext_body(msgpack_packer* pk, const void* b, size_t l);

int msgpack_pack_object(msgpack_packer* pk, msgpack_object d);

Those functions are corresponding to msgpack format

To pack STR, RAW(v4 format), BIN, and EXT, call msgpack_pack_[str|v4raw|bin|ext] to provide a size and then call msgpack_pack_[str|v4raw|bin|ext]_body to provide a payload. To pack ARRAY, call msgpack_pack_array to provide the number of elements and then call any msgpack_pack_* functions the number times. To pack MAP, call msgpack_pack_map to provide the number of elements and then call any msgpack_pack_* functions the number*2 times (key and value).

When you use packer as value (not pointer), you don't need to call destroy function. Be careful that msgpack_packer_free is not a destroy function for a packer as value. It is for msgpack_packer_new. If you want to create a packer on heap, call msgpack_packer_new to create a packer and msgpack_packer_free to destroy(free) the packer.

Unpack

Using unpacker

When you unpack message pack format data, use msgpack_unpacker. When you use msgpack_unpacker on the stack, use msgpack_unpacker_init() and msgpack_unpacker_destroy() as follows:

/* unpacker on the stack */
msgpack_unpacker unp;

/* Initialize the unpacker. 100 means initial buffer size. */
/* It can expand later. */
bool result = msgpack_unpacker_init(&unp, 100);
/* If memory allocation is failed, result is false, else result is true. */
if (result) {
    /* Do unpacking */
}
msgpack_unpacker_destroy(&unp);

When you use msgpack_unpacker on the heap, use msgpack_unpacker_new() and msgpack_unpacker_free() as follows:

/* unpacker on the heap */
msgpack_unpacker* unp = msgpack_unpacker_new(100);
if (unp) {
    /* Do unpacking */
}
msgpack_unpacker_free(unp);

Here is declaration of the functions above:

bool msgpack_unpacker_init(msgpack_unpacker* mpac, size_t initial_buffer_size);
void msgpack_unpacker_destroy(msgpack_unpacker* mpac);

msgpack_unpacker* msgpack_unpacker_new(size_t initial_buffer_size);
void msgpack_unpacker_free(msgpack_unpacker* mpac);

After initializing msgpack_unpacker, do unpacking process. I assume that unp is on the stack in the following codes.

First, check the msgpack_unpacker's buffer capacity. Let's say request_size is the size you need. You can check the buffer size and expand it if the buffer size is not enough.

    if (msgpack_unpacker_buffer_capacity(&unp) < request_size) {
        bool result = msgpack_unpacker_reserve_buffer(&unp, request_size);
        if (!result) {
            /* Memory allocation error. */
        }
    }

Now, msgpack_unpacker has enough buffer size. You can get buffer pointer using msgpack_unpacker_buffer(). And then, write the data to the buffer.

    memcpy(msgpack_unpacker_buffer(&unp), data, request_size);

The number of writing bytes is might not always equal to reqest_size. See https://github.com/msgpack/msgpack-c/blob/c_master/example/lib_buffer_unpack.c.

After writing, you need to notify the number of wrote bytes to msgpack_unpacker using msgpack_unpacker_buffer_consumed().

    msgpack_unpacker_buffer_consumed(&unp, request_size);

Now, msgpack_unpacker is ready to unpack. An unpacked object is stored in msgpack_unpacked. You need to initialize it using msgpack_unpacked_init(). Then, call msgpack_unpacker_next().

    {
        msgpack_unpacked und;
        msgpack_unpack_return ret;
        msgpack_unpacked_init(&und);
        ret = msgpack_unpacker_next(&unp, &und);
        switch(ret) {
        case MSGPACK_UNPACK_SUCCESS:
            {
                msgpack_object obj = und.data;
                /* Extract msgpack_object and use it. */
            }
            break;
        case MSGPACK_UNPACK_CONTINUE:
            /* cheking capacity, reserve buffer, copy additional data to the buffer, */
            /* notify consumed buffer size, then call msgpack_unpacker_next(&unp, &und) again */
            break;
        case MSGPACK_UNPACK_PARSE_ERROR:
            /* Error process */
            break;
        }
    }

msgpack_unpacker_next() returns msgpack_unpacked. The meaning of msgpack_unpacked_return is as follows:

return code	meaning
MSGPACK_UNPACK_SUCCESS	Successfully unpacked
MSGPACK_UNPACK_CONTINUE	msgpack format data is incomplete. You need to provide additional bytes.
MSGPACK_UNPACK_PARSE_ERROR	msgpack format data is invalid.

If msgpack_unpacked_return is MSGPACK_UNPACK_SUCCESS, msgpack_unpacked has unpacked msgpack_object. It is set as the member variable named data.

After unpacked, you need to destroy msgpack_unpacked using msgpack_unpacked_destroy().

    msgpack_unpacked_destroy(&und);

If you pass the same msgpack_unpacked to the next msgpack_unpacker_next(), you don't need to call msgpack_unpacked_destroy() and msgpack_unpacked_init() before calling msgpack_unpacker_next(), because msgpack_unpacker_next() destroys and initializes given msgpack_unpacked internally.

Here is the memory model of msgpack_unpacked:

Using unpack function

msgpack_unpack_return
msgpack_unpack_next(msgpack_unpacked* result,
        const char* data, size_t len, size_t* off);

If msgpack data contains STR, BIN, or EXT, unpacked msgpack_object refer to msgpack data buffer. You need to keep the lifetime of the buffer during you use unpacked msgpack_object. If you don't want to manage the data lifetime, use msgpack_unpacker.

This function is obsolete:

msgpack_unpack_return
msgpack_unpack(const char* data, size_t len, size_t* off,
               msgpack_zone* result_zone, msgpack_object* result);

If msgpack data contains STR, BIN, or EXT, unpacked msgpack_object refer to msgpack data buffer. You need to keep the lifetime of the buffer during you use unpacked msgpack_object. Also you need to keep zone's lifetime. It's pretty complecated, so I recommend to use msgpack_unpack_next() or msgpack_unpacker instead of msgpack_unpack().

msgpack_unpack() and msgpack_unpack_next() return msgpack_unpacked. The meaning of msgpack_unpacked_return is as follows:

return code	meaning
MSGPACK_UNPACK_SUCCESS	Successfully unpacked. No extra bytes.
MSGPACK_UNPACK_EXTRA_BYTES	Successfully unpacked. There are extra bytes.
MSGPACK_UNPACK_CONTINUE	msgpack format data is incomplete. You need to provide additional bytes.
MSGPACK_UNPACK_PARSE_ERROR	msgpack format data is invalid.
MSGPACK_UNPACK_NOMEM_ERROR	Memory allocation failed.

Zone

Zone is a kind of memory pool. Zone is used by unpacker and msgpack_unpack_next(). msgpack_object is allocated on the zone. When you use an unpacker, you don't need to care about zone. Zone is generated and managed by the unpacker.

If you use msgpack_unpack() (obsolete), you need to care about a zone. You need to initialize and destroy the zone.

Here is zone's initialize and destroy functions:

/* On stack */
bool msgpack_zone_init(msgpack_zone* zone, size_t chunk_size);
void msgpack_zone_destroy(msgpack_zone* zone);

/* On heap */
msgpack_zone* msgpack_zone_new(size_t chunk_size);
void msgpack_zone_free(msgpack_zone* zone);

Object

typedef struct msgpack_object {
    msgpack_object_type type;
    msgpack_object_union via;
} msgpack_object;

msgpack_object is an unpacked data. msgpack_object has a member variable that contains type information named type. Types are corresponding to https://github.com/msgpack/msgpack/blob/master/spec.md.

typedef enum {
    MSGPACK_OBJECT_NIL                  = 0x00,
    MSGPACK_OBJECT_BOOLEAN              = 0x01,
    MSGPACK_OBJECT_POSITIVE_INTEGER     = 0x02,
    MSGPACK_OBJECT_NEGATIVE_INTEGER     = 0x03,
    MSGPACK_OBJECT_FLOAT                = 0x04,
#if defined(MSGPACK_USE_LEGACY_NAME_AS_FLOAT)
    MSGPACK_OBJECT_DOUBLE               = MSGPACK_OBJECT_FLOAT, /* obsolete */
#endif /* MSGPACK_USE_LEGACY_NAME_AS_FLOAT */
    MSGPACK_OBJECT_STR                  = 0x05,
    MSGPACK_OBJECT_ARRAY                = 0x06,
    MSGPACK_OBJECT_MAP                  = 0x07,
    MSGPACK_OBJECT_BIN                  = 0x08,
    MSGPACK_OBJECT_EXT                  = 0x09
} msgpack_object_type;

msgpack_object has a union data structue named via. msgpack_object_union is defined as follows:

typedef union {
    bool boolean;
    uint64_t u64;
    int64_t  i64;
#if defined(MSGPACK_USE_LEGACY_NAME_AS_FLOAT)
    double   dec; /* obsolete*/
#endif /* MSGPACK_USE_LEGACY_NAME_AS_FLOAT */
    double   f64;
    msgpack_object_array array;
    msgpack_object_map map;
    msgpack_object_str str;
    msgpack_object_bin bin;
    msgpack_object_ext ext;
} msgpack_object_union;

The union fields corresponding to the type.

msgpack_object_array is defined as follows:

typedef struct {
    uint32_t size;
    struct msgpack_object* ptr;
} msgpack_object_array;

msgpack_object_map is defined as follows:

typedef struct {
    uint32_t size;
    struct msgpack_object_kv* ptr;
} msgpack_object_map;

typedef struct msgpack_object_kv {
    msgpack_object key;
    msgpack_object val;
} msgpack_object_kv;

msgpack_object_str, msgpack_object_bin, and msgpack_object_ext are defined as follows:

typedef struct {
    uint32_t size;
    const char* ptr;
} msgpack_object_str;

typedef struct {
    uint32_t size;
    const char* ptr;
} msgpack_object_bin;

typedef struct {
    int8_t type;
    uint32_t size;
    const char* ptr;
} msgpack_object_ext;