Tokenization Abstraction

Hugging Face has introduced a new, more modular, and simplified tokenization system within the Transformers library. This update aims to improve the developer experience by making tokenization more intuitive and easier to integrate into various ML workflows. The new architecture allows for greater flexibility and maintainability of tokenization components. The v5 release further refines this by introducing a clearer API, improved modularity, and enhanced support for custom tokenizers, making it easier to manage and extend tokenization logic.