Tokenizer Performance Improvements

The tokenizers library has undergone a significant refactor for version 1, focusing on performance improvements to address tokenization becoming a bottleneck in ML workflows. Key changes include a workspace split for modularity, a no-alloc merge working set, the use of bitstreams and SIMD instructions (bitcannon) for faster splitting instead of regex, a rewrite of the merge loop using intrusive linked lists and batch processing, a thread-local word cache for repeated words, and native parallelism allowing multiple threads to encode concurrently. These optimizations aim to significantly reduce tokenization latency and improve scaling.