mpack简明教程-编程知识

文章目录

- 摘要
- MessagePack简介
- MPACK的简单使用
- 在定长的buffer存储不定长的数据
- 读取截断的数据

摘要

本文先简单介绍MessagePack的基本概念。

然后，介绍一个MessagePack C API - MPack的通常使用。

接着尝试对MPack截断数据的读取。

注：本文完整代码见仓库。

MessagePack简介

如果你使用过C/C++的json库，那么上手MessagePack是比较容易的。关于C/C++ Json库的使用可见：C++ JSON库的一般使用方法-CSDN博客。

这里，我先说下结论，对于用户层面而言，MessagePack相对于json节省空间，但牺牲了可读性(对于人类而言)。（更多区别见：Why Not Just Use JSON?）

下面我们来看两个示例，了解下MessagPack是如何压缩json内容。这个压缩过程，遵循MessagePack specification(MessagePack规范)

第一个示例，来自MessagePack首页。它将27个字节的JSON内容，压缩到18个字节。

在这里插入图片描述

我们根据MessagePack规范来看下上面是如何压缩的。

# 82的含义
## 对于元素个数不超过15个的map存储，遵循如下格式
## 82 == 1000 0010 表示该结构为map，有两个元素
+--------+~~~~~~~~~~~~~~~~~+
|1000XXXX|   N*2 objects   |
+--------+~~~~~~~~~~~~~~~~~+# A7的含义
## 对于长度不超过31的固定长度字符串存储，遵循如下格式
## A7 == 1010 0111 表示该结构为字符串，有7个字符
+--------+========+
|101XXXXX|  data  |
+--------+========+# C3含义
## false:
+--------+
|  0xc2  |
+--------+
## true:
+--------+
|  0xc3  |
+--------+# A6的含义--略，查询过程同上面A7# 00的含义
## 可以使用7-bit存储的正整数，遵循如下格式
## 00 = 0000 0000
+--------+
|0XXXXXXX|
+--------+

我们再来看一个示例，练习下。这个示例来自： mpack/docs/expect.md at develop · ludocode/mpack

["hello","world!",[1,2,3,4]
]

上面这个json，使用MessagPack编码如下。

93                     # an array containing three elementsa5 68 65 6c 6c 6f    # "hello"a6 77 6f 72 6c 64 21 # "world!"94                   # an array containing four elements01                 # 102                 # 203                 # 304                 # 4

MPACK的简单使用

了解了MessagePack的概念之后，我们看下它的C api的使用。MPACK是MessagePack C语言实现的API之一。mpack的stars没有msgpack-c多。但是mpack对libc的版本没有要求。下面是对mpack的使用教程。

JSON的使用包含两个部分：将json数据写入内存/文件；从已有的json数据中读取内容。

mapck在使用结构上，和json类似，分为写和读。MPack api是mpack的api文档，接口使用的详细介绍见文档。写数据的部分调用Write API。如果没有内存限制，读取的时候使用Node API。如果有内存限制，则使用Reader API+Expect API。(这里说的内存限制是指，程序运行的内存限制，而不是数据存储的内存限制)

这个参考示例不错，在开始之前，可以一读：一个C语言MessagePack库：mpack

下面我们开始写demo。写一个最简单，也是最常用的demo：存储数据的内存无限制时，使用mpack进行数据的写入和读取。示例参考自官方的README:ludocode/mpack: MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]. 这里对这些API进行简单的介绍，详细介绍见官方文档。

mpack_writer_init_growable(mpack_writer_t* writer, char** target_data, size_t* target_size)使用一个会增长的buffer。
在调用mpack_writer_destroy后，将上面target_data指向写入数据的地址。
最后必须调用MPACK_FREE()释放申请的数据。
使用mpack_build_map开始构建一个map，我们此时不知道map中元素的个数。如果知道元素个数，使用mpack_start_map()替代。
使用mpack_build_map后，后面必须跟偶数个数的元素，在结束的时候调用 mpack_complete_map()。
使用Node API读取数据。

#include "mpack.h"
#include <stdio.h>/*
{"name": "3-1","number": 56,"students": [{"name": "zhangsan","score": 76.8},{"name": "lisi","score": 77}]
}
*/mpack_error_t class_information_serialize(char **data, size_t *size) {mpack_writer_t writer;mpack_writer_init_growable(&writer, data, size);mpack_build_map(&writer);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "3-1");mpack_write_cstr(&writer, "number");mpack_write_u8(&writer, 56);mpack_write_cstr(&writer, "students");mpack_build_array(&writer);mpack_start_map(&writer, 2);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "zhangsan");mpack_write_cstr(&writer, "score");mpack_write_float(&writer, 76.8);mpack_finish_map(&writer);mpack_start_map(&writer, 2);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "lisi");mpack_write_cstr(&writer, "score");mpack_write_float(&writer, 77);mpack_finish_map(&writer);mpack_complete_array(&writer);mpack_complete_map(&writer);mpack_error_t ret = mpack_writer_destroy(&writer);return ret;
}mpack_error_t class_information_deserialize(const char *data, size_t length) {mpack_tree_t tree;mpack_tree_init_data(&tree, data, length);mpack_tree_parse(&tree);mpack_node_t root = mpack_tree_root(&tree);const char *name = mpack_node_str(mpack_node_map_cstr(root, "name"));size_t name_len = mpack_node_strlen(mpack_node_map_cstr(root, "name"));uint8_t number = mpack_node_u8(mpack_node_map_cstr(root, "number"));printf("name:%.*s\n", name_len, name);printf("number:%u\n", number);printf("students:\n");mpack_node_t students = mpack_node_map_cstr(root, "students");size_t student_num = mpack_node_array_length(students);for (unsigned int i = 0; i < student_num; i++) {mpack_node_t student = mpack_node_array_at(students, i);const char *name = mpack_node_str(mpack_node_map_cstr(student, "name"));size_t name_len = mpack_node_strlen(mpack_node_map_cstr(student, "name"));float score = mpack_node_float(mpack_node_map_cstr(student, "score"));printf("  name:%.*s\n", name_len, name);printf("  score:%.2f\n", score);}mpack_error_t ret = mpack_tree_destroy(&tree);return ret;
}int main(int argc, char *argv[]) {char *data = NULL;size_t size = 0;class_information_serialize(&data, &size);class_information_deserialize(data, size);MPACK_FREE(data);
}

程序输出如下。

name:3-1
number:56
students:name:zhangsanscore:76.80name:lisiscore:77.00

在定长的buffer存储不定长的数据

工作的时候，会想使用比较奇怪的调用。首先，所有的数据都要存储在一个定长的buffer里面。因为这个buffer是从一个内存池中取出的，所以它的长度是定长的。但是，往里面写入数据的时候，会写多次，长度不一定。

修改上面的示例：现在要写入不定个数的student到数组中；允许截断；

截断的时候发生了什么？截断后还能否读取？

遇到截断，writer会设置错误的标记位。正常编码的时候，这个writer数据不应该在后续进行读取。因为它已经被标记为错误。

但是，总有些头铁的需求，想用发生截断时，已经写入内存的数据。这个从API文档里面是看不出来的，要看mpack的源码。

我先说结论：

从前往后，不断将build中的内容，复制写入buffer。写之前检查空间是否足够，不够则停止写入，并标记错误。
使用 node api 无法读取。因为数据被截断，不合法。

大体结构图如下。

在这里插入图片描述

示例代码如下。

#include "mpack.h"
#include <stdio.h>/*
{"name": "3-1","number": 56,"students": [{"name": "zhangsan","score": 76.8},{"name": "lisi","score": 77}]
}
*/mpack_error_t class_information_serialize(char *data, size_t *size) {mpack_writer_t writer;mpack_writer_init(&writer, data, *size);mpack_build_map(&writer);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "3-1");mpack_write_cstr(&writer, "number");mpack_write_u8(&writer, 56);mpack_write_cstr(&writer, "students");mpack_build_array(&writer);for (unsigned int i = 0; i < 56; i++) {mpack_start_map(&writer, 2);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "zhangsan");mpack_write_cstr(&writer, "score");mpack_write_float(&writer, 76.8);if (mpack_writer_error(&writer) != mpack_ok) {printf("error_%u: %d\n", i, mpack_writer_error(&writer));}mpack_finish_map(&writer);}mpack_complete_array(&writer);mpack_complete_map(&writer);if (mpack_writer_error(&writer) != mpack_ok) {printf("after write all students error: %d\n", mpack_writer_error(&writer));}*size = mpack_writer_buffer_used(&writer);mpack_error_t ret = mpack_writer_destroy(&writer);return ret;
}mpack_error_t class_information_deserialize(const char *data, size_t length) {mpack_tree_t tree;mpack_tree_init_data(&tree, data, length);mpack_tree_parse(&tree);if (mpack_tree_error(&tree) != mpack_ok) {printf("parse tree error: %d\n", mpack_tree_error(&tree));}mpack_node_t root = mpack_tree_root(&tree);const char *name = mpack_node_str(mpack_node_map_cstr(root, "name"));size_t name_len = mpack_node_strlen(mpack_node_map_cstr(root, "name"));uint8_t number = mpack_node_u8(mpack_node_map_cstr(root, "number"));printf("name:%.*s\n", name_len, name);printf("number:%u\n", number);printf("students:\n");mpack_node_t students = mpack_node_map_cstr(root, "students");size_t student_num = mpack_node_array_length(students);for (unsigned int i = 0; i < student_num; i++) {mpack_node_t student = mpack_node_array_at(students, i);const char *name = mpack_node_str(mpack_node_map_cstr(student, "name"));size_t name_len = mpack_node_strlen(mpack_node_map_cstr(student, "name"));float score = mpack_node_float(mpack_node_map_cstr(student, "score"));printf("  name:%.*s\n", name_len, name);printf("  score:%.2f\n", score);}mpack_error_t ret = mpack_tree_destroy(&tree);return ret;
}int main(int argc, char *argv[]) {
#define DATA_BUFFER_SIZE 35char data[DATA_BUFFER_SIZE] = {0};size_t size = DATA_BUFFER_SIZE;class_information_serialize(data, &size);class_information_deserialize(data, size);
}

输出如下。

after write all students error: 6
parse tree error: 3
name:
number:0
students:

读取截断的数据

既然使用node api解析数据会失败。那不要解析，顺序读取，一直读取到异常。

参考自：Using the Expect API。代码如下。

#include "mpack.h"
#include <stdio.h>/*
[{"name": "zhangsan","score": 76.8},{"name": "lisi","score": 77}...
]*/mpack_error_t class_information_serialize(char *data, size_t *size) {mpack_writer_t writer;mpack_writer_init(&writer, data, *size);mpack_build_array(&writer);for (unsigned int i = 0; i < 56; i++) {mpack_build_map(&writer);mpack_write_cstr(&writer, "name");mpack_write_cstr(&writer, "zhangsan");mpack_write_cstr(&writer, "score");mpack_write_float(&writer, 76.8);mpack_complete_map(&writer);}mpack_complete_array(&writer);if (mpack_writer_error(&writer) != mpack_ok) {printf("after write all students error: %d\n", mpack_writer_error(&writer));}*size = mpack_writer_buffer_used(&writer);mpack_error_t ret = mpack_writer_destroy(&writer);return ret;
}mpack_error_t class_information_deserialize(const char *data, size_t length) {mpack_reader_t reader;mpack_reader_init_data(&reader, data, length);uint32_t students_num = mpack_expect_array(&reader);printf("students num: %u\n", students_num);for (unsigned int i = 0; i < students_num; i++) {uint32_t elem_cnt = mpack_expect_map(&reader);mpack_expect_cstr_match(&reader, "name");char name[20];size_t name_len = mpack_expect_str_buf(&reader, name, 20);mpack_expect_cstr_match(&reader, "score");float score = mpack_expect_float(&reader);if (mpack_reader_error(&reader) != mpack_ok) {break;}printf("name:%.*s\n", name_len, name);printf("score:%.2f\n", score);mpack_done_map(&reader);}return mpack_ok;
}int main(int argc, char *argv[]) {
#define DATA_BUFFER_SIZE 100char data[DATA_BUFFER_SIZE] = {0};size_t size = DATA_BUFFER_SIZE;class_information_serialize(data, &size);class_information_deserialize(data, size);
}

输出如下。

after write all students error: 6
students num: 56
name:zhangsan
score:76.80
name:zhangsan
score:76.80
name:zhangsan
score:76.80