TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

IPU-Ready Transformers Optimization

8 posts · May 2022 – Feb 2023

What this trail covers

Hugging Face Optimum integrates with Intel IPUs, offering optimized performance for a range of Transformer models across computer vision, NLP, and speech. This partnership expands the hardware acceleration options for users, enabling efficient training and inference on Graphcore's specialized hardware. The latest update includes support for new models and leverages Graphcore's Bow IPU and Poplar SDK for enhanced performance and power efficiency without code changes.

All Hugging Face trails
  1. Accelerating PyTorch Transformers with Intel Sapphire Rapids - part 2This post benchmarks inference performance of popular PyTorch Transformers models on Intel Sapphire Rapids CPUs using Hugging Face Optimum Intel. It demonstrates a 60-65% speedup for single predictions and achieves single-digit prediction latency, comparable to GPUs, by leveraging AMX instructions with bfloat16 and JIT compilation. The post provides detailed setup instructions, benchmarking methodology, and full results for various models and sequence lengths.
  2. Optimum+ONNX Runtime - Easier, Faster training for your Hugging Face modelsThis post details the integration of ONNX Runtime for training within Hugging Face's Optimum library. It introduces the `ORTTrainer` and `ORTTrainingArguments` APIs as replacements for the standard `Trainer` and `TrainingArguments`, enabling users to leverage ONNX Runtime's training optimizations with minimal code changes. Performance benchmarks demonstrate significant training speedups (39%-130%) when using ONNX Runtime with DeepSpeed ZeRO Stage 1 compared to PyTorch baseline. The post also outlines ONNX Runtime's underlying memory and compute optimizations, such as fused Adam optimizer and mixed-precision training, and discusses future plans for optimizing larger models and edge training.
  3. Accelerating PyTorch Transformers with Intel Sapphire Rapids - part 1This post details the process of accelerating PyTorch Transformers training on Intel Sapphire Rapids CPUs using AWS EC2 R7iz instances. It covers setting up a distributed training cluster with Intel oneAPI Collective Communications Library (CCL) and Intel Extension for PyTorch (IPEX), leveraging new CPU instructions like AMX (BF16 and INT8) for accelerated matrix multiplication. The post provides a step-by-step guide for building the cluster, configuring the master node, and launching a distributed training job for question answering on the SQUAD dataset, demonstrating significant speedups compared to previous generations of Intel CPUs.
  4. Accelerate your models with 🤗 Optimum Intel and OpenVINOThis post details inference performance improvements on Intel Sapphire Rapids CPUs using Optimum Intel, demonstrating significant speedups for popular NLP models. It highlights the integration of OpenVINO™ Runtime for optimized inference on Intel hardware, enabling faster execution of models like ViT-Food101.
  5. Pre-Train BERT with Hugging Face Transformers and Habana GaudiThis post details the process of pre-training BERT models on Habana Gaudi accelerators using the Hugging Face Transformers library. It provides a practical guide and code examples for setting up and executing the pre-training process, highlighting the integration of Habana's hardware with the Transformers ecosystem.
  7. Deep Dive: Vision Transformers On Hugging Face Optimum GraphcoreThis post details the fine-tuning of Vision Transformer (ViT) models on Graphcore IPUs using the Hugging Face Optimum library, providing a step-by-step guide and a notebook for training on the ChestX-ray14 dataset. It highlights the suitability of IPUs for ViT models due to their parallel processing capabilities and the ease of use provided by the Optimum library for integrating pre-trained models and datasets.
  8. Convert Transformers to ONNX with Hugging Face OptimumThis post details how to convert Hugging Face Transformers models to the ONNX format using the Hugging Face Optimum library. It explains the benefits of ONNX for deployment and provides three methods for conversion: low-level `torch.onnx.export`, mid-level `transformers.onnx.export`, and high-level `optimum.onnxruntime.ORTModelForXxx.from_pretrained`. It also lists supported architectures and suggests next steps like accelerated inference, quantization, and training with ONNX.
  9. Graphcore and Hugging Face Launch New Lineup of IPU-Ready TransformersThis post announces the expansion of Hugging Face Optimum's support for Graphcore IPUs, adding 10 new Transformer models across computer vision (ViT), NLP (GPT-2, RoBERTa, DeBERTa, BART, LXMERT, T5), and speech (HuBERT, Wav2Vec2). It highlights the seamless integration with Graphcore's latest hardware (Bow IPU) and software (Poplar SDK v2.5), allowing users to leverage IPU performance with minimal code changes. The post also emphasizes the partnership's goal of lowering barriers to AI innovation and provides resources for getting started.