IBM

Analyze your Spark application using explain

May 16, 2024Sunitha Kambhampati
Analyze your Spark application using explain
SummaryAI, from the source post

Introduces the Spark `explain` API and its usage for inspecting parsed logical, analyzed logical, optimized logical, and physical execution plans of Spark SQL queries. Demonstrates how to interpret the output of `explain` and `explain(true)`, including understanding stages, exchanges, and aggregate operations. Explains how to leverage the Spark UI's SQL tab to visualize DAGs and execution plans for running or completed jobs, and how to use the Spark history server. It also shows the effect of disabling `spark.sql.codegen.wholeStage` on the physical plan.

More from the Spark Performance Optimization trail