TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Data Analysis Tooling

2 posts · Jul 2017 – Mar 2020

What this trail covers

Cloudflare's data analysis tooling has evolved to include command-line utilities for generating ASCII histograms, monitoring real-time rate of change, and summing floating-point numbers. This post details the development and open-sourcing of these tools (mmsum, mmwatch, mmhistogram) to aid engineers in quickly processing and understanding data from various sources, including network traffic and system metrics. The post also explores the performance implications of Bloom filters, highlighting the trade-off between memory access latency and computational cost, and introduces a custom tool 'mmuniq-bloom' to address large-scale deduplication challenges.

All Cloudflare trails
  1. When Bloom filters don't bloomThis post introduces the development and open-sourcing of command-line data analysis tools (mmsum, mmwatch, mmhistogram) for generating ASCII histograms, monitoring real-time rate of change, and summing floating-point numbers. It also details the creation of 'mmuniq-bloom', a custom tool for probabilistic deduplication using Bloom filters, and analyzes its performance bottlenecks, specifically identifying random memory access latency as a significant factor. The post demonstrates the use of profiling tools like strace and perf to diagnose performance issues and discusses the trade-offs between computational cost and memory access in Bloom filter implementations.
  2. Three little tools: mmsum, mmwatch, mmhistogramThis post introduces and open-sources three command-line tools: mmhistogram for generating ASCII histograms of data, mmwatch for monitoring command output and interpreting numerical changes as rates, and mmsum for summing lists of floating-point numbers. Examples demonstrate their use with CSV data, disk usage, and real-time rainfall data.