Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

This post introduces Quantization-Aware Healing, a novel technique for compressing large language models to 4-bit precision while achieving performance that surpasses the original full-precision model. This builds upon existing 8-bit matrix multiplication integrations by pushing the boundaries of model compression and efficiency.