TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

CPU-based PyTorch Training Acceleration

1 post · Jan 2023

What this trail covers

Hugging Face now supports accelerating PyTorch training jobs on clusters of Intel Sapphire Rapids CPUs using Intel oneAPI Collective Communications Library (CCL) and Intel Extension for PyTorch (IPEX). This enables significant speedups for distributed training by leveraging new CPU instructions like AMX, making CPU-based training a more cost-effective and scalable option, especially for smaller datasets and cloud spot instances.

All Hugging Face trails
  1. Accelerating PyTorch Transformers with Intel Sapphire Rapids - part 1This post introduces the acceleration of PyTorch Transformer training jobs on Intel Sapphire Rapids CPU clusters. It details the setup of a cluster on AWS using bare metal instances, installation of necessary libraries (PyTorch, IPEX, CCL, Transformers), and configuration for distributed training. The post demonstrates an 8x speedup for fine-tuning DistilBERT on the SQUAD dataset compared to previous generation CPUs, highlighting the benefits of AMX instructions and distributed training on CPUs.