Iris - Turning observations into actionable insights for enhanced decision making

This post introduces Iris, a new observability platform for ETL jobs, specifically focusing on Spark and Presto. Iris captures critical job metrics at the JVM level, including runtime, CPU/memory utilization, garbage collection, and stage/task execution details. It supports both real-time monitoring via a TIG stack (Telegraf, InfluxDB, Grafana) and offline analytics by pushing metrics to the data lake. The platform's architecture includes a Metrics Collector module that uses Uber's JVM Profiler and sparkMeasure to gather detailed metrics without requiring application code changes. This provides granular insights into resource consumption, enabling better provisioning and cost optimization.