Spydra – Ephemeral Hadoop Clusters Using Google Compute Platform

This post introduces Spydra, a new open-sourced project from Spotify designed to automate the lifecycle management of ephemeral Hadoop clusters on Google Compute Platform (GCP) Dataproc. Spydra simplifies job submission, enhances troubleshooting of failed jobs, and supports both ephemeral clusters for single job executions and static, long-living clusters. It is a key component of Spotify's data infrastructure migration to GCP, leveraging experience from managing a 2500-node on-premise Hadoop cluster. Spydra enables dual use of on-premise Hadoop and GCP Dataproc during the migration.