Analyze your Spark application using explain
May 16, 2024Sunitha Kambhampati
Summary
Introduces the Spark `explain` API and its usage for inspecting parsed logical, analyzed logical, optimized logical, and physical execution plans of Spark SQL queries. Demonstrates how to interpret the output of `explain` and `explain(true)`, including understanding stages, exchanges, and aggregate operations. Explains how to leverage the Spark UI's SQL tab to visualize DAGs and execution plans for running or completed jobs, and how to use the Spark history server. It also shows the effect of disabling `spark.sql.codegen.wholeStage` on the physical plan.