TechPrufer
Sign inGet started

Blogs›Datadog›Protobuf Serialization Performance

Protobuf Serialization Performance

Protobuf Serialization Performance

1
posts
2017

This post details the investigation into the performance of Protocol Buffers (protobuf) serialization in Python, particularly in the context of extracting metrics from kube-state-metrics. It covers the basics of protobuf, including defining messages, generating code, and handling streaming multiple messages by prepending message size. The post benchmarks payload sizes for protobuf versus plain text and highlights the performance limitations of the pure Python protobuf implementation, suggesting the use of C++ extensions for significant speed improvements.

2017

Protobuf parsing in Python | Datadog

6/8/2017

This post introduces the concept of Protocol Buffers for data serialization and explores its practical application within Datadog for extracting metrics from Kubernetes via kube-state-metrics. It details the process of defining `.proto` files, using `protoc` to generate Python code, and implementing custom logic for streaming multiple protobuf messages by prepending their sizes using internal `google.protobuf.internal` encoder/decoder functions. The post includes benchmarks comparing payload sizes of protobuf and plain text formats, and discusses the performance implications of the pure Python implementation versus C++ extensions.