Case Folding for Code Search

This feature thread tracks the evolution of case folding for code search. Initially, the focus was on basic case-insensitive matching. This post details significant performance optimizations for case folding, particularly for ASCII characters, by removing early exits and optimizing for vectorization. It also covers efficient handling of Unicode case folding and memory management to avoid unnecessary allocations. The new implementation achieves memory bandwidth speeds for ASCII folding by eliminating branches and leveraging vectorization. The open-sourced Rust crate `casefold` implements simple, 1-to-1 folds and avoids heap allocations by mutating input strings in place for ASCII and allocating a worst-case buffer for Unicode when necessary.