To advance the MessagePack data serialization methods, I propose the introduction of predefined extension types. These are designed to improve parsing, amplify storage efficiency, and adapt to real-world applications such as GraphQL result serialization.
Central to this proposal is its adaptability. Depending on client requests, the MessagePack format can be dynamically adjusted. If a client request indicates support for these extensions through an aware reading library, data is serialized utilizing the enhanced format. If not, data is rendered in the traditional manner. An essential highlight here is the consistent presentation: irrespective of the serialization method used, an aware reading library will ensure that the resulting data presented to the API caller is identical, obfuscating the underlying complexities and making the use of extensions transparent.
The proposal is forward compatible to any future changes or extensions of the MessagePack specification. And except for occupying 4 pre-defined type slots does not limit any future changes or extensions of the MessagePack specification.
a) MessagePack Container:
- Definition: An extension type with its content rendered in MessagePack.
- Utility: Ideal for maps or arrays where byte length isn't directly deducible from the header. An aware reading library facilitates efficient on-demand reading, allowing certain sections to be bypassed without parsing nested structures. The transparent replacement of this extension type with its content means that its use remains hidden from the API user.
b) Deduplication Array Container:
- Definition: An extension type encompassing a MessagePack array.
- Utility: Ideally the first element in a MessagePack stream, its content is a MessagePack array, whose elements can be any valid MessagePack type. While this deduplication array enhances efficiency, it remains concealed from the API by an aware reading library, ensuring a seamless user experience.
c) Deduplication Reference:
- Definition: Contains an integer (1, 2, or 4-byte) pointing to an index within the deduplication array.
- Utility: The aware reading library leverages this reference to swiftly access the deduplication array, substituting the current element with the indexed content. The transparent replacement of these references remains hidden from the API user.
d) Pre-defined Map:
- Definition: Commences with a MessagePack-encoded integer targeting a deduplication array index, trailed by a sequence of elements sans an array header.
- Utility: The deduplication array entry specified by the index should be an array, symbolizing the map's keys. The following elements represent the values, with the aware reading library presenting this as a standard map. The end of the list of elements is implicitly given by the fact that it is contained in an extension type with exact byte length. If the number of elements is less than the length of the name array, all trailing map entries have a nil value. The transparent replacement of this extension type with a normal map means that it remains hidden from the API user.
The proposed extensions are particularly effective for serializing GraphQL query results, known for redundant field names and structurally similar maps, especially in list-based queries. Through deduplication, data redundancy is substantially curtailed.
The proposal does not make any changes to the underlying existing MessagePack format to maximize compatibility. It can in principle be implemented as an additional wrapper on the writing and reading size of existing MessagePack libraries (but would be more efficient on the reading side when integrated into the library, and one the writing side with an extended API to allow the API user to pre-define maps and likely duplicated elements and then refer to these while writing, without the library having to find duplicated elements and repeating map structures dynamically).
In wrapping up, this proposal seeks to create a MessagePack format that's efficient, responsive, and mindful of diverse client capabilities. Constructive feedback is sought to further refine and enhance this proposal.
To advance the MessagePack data serialization methods, I propose the introduction of predefined extension types. These are designed to improve parsing, amplify storage efficiency, and adapt to real-world applications such as GraphQL result serialization.
Central to this proposal is its adaptability. Depending on client requests, the MessagePack format can be dynamically adjusted. If a client request indicates support for these extensions through an aware reading library, data is serialized utilizing the enhanced format. If not, data is rendered in the traditional manner. An essential highlight here is the consistent presentation: irrespective of the serialization method used, an aware reading library will ensure that the resulting data presented to the API caller is identical, obfuscating the underlying complexities and making the use of extensions transparent.
The proposal is forward compatible to any future changes or extensions of the MessagePack specification. And except for occupying 4 pre-defined type slots does not limit any future changes or extensions of the MessagePack specification.
a) MessagePack Container:
b) Deduplication Array Container:
c) Deduplication Reference:
d) Pre-defined Map:
The proposed extensions are particularly effective for serializing GraphQL query results, known for redundant field names and structurally similar maps, especially in list-based queries. Through deduplication, data redundancy is substantially curtailed.
The proposal does not make any changes to the underlying existing MessagePack format to maximize compatibility. It can in principle be implemented as an additional wrapper on the writing and reading size of existing MessagePack libraries (but would be more efficient on the reading side when integrated into the library, and one the writing side with an extended API to allow the API user to pre-define maps and likely duplicated elements and then refer to these while writing, without the library having to find duplicated elements and repeating map structures dynamically).
In wrapping up, this proposal seeks to create a MessagePack format that's efficient, responsive, and mindful of diverse client capabilities. Constructive feedback is sought to further refine and enhance this proposal.