Introducing Pandas UDFs for PySpark

Introduced Pandas UDFs (formerly Vectorized UDFs) for PySpark in Apache Spark 2.3. These UDFs leverage Apache Arrow to enable low-overhead, high-performance UDFs entirely in Python, addressing the serialization and invocation overhead of row-at-a-time UDFs. The post details two types: scalar Pandas UDFs (taking pandas.Series, returning pandas.Series) and grouped map Pandas UDFs (taking pandas.DataFrame, returning pandas.DataFrame). Examples include 'Plus One', 'Cumulative Probability', 'Subtract Mean', and 'Ordinary Least Squares Linear Regression'. Performance comparisons show 3x to over 100x improvement over row-at-a-time UDFs.