Semantic IDs: Product Understanding at Scale

What this post added

Developed and implemented Semantic IDs by adapting RQ-VAE with a contrastive loss function that utilizes the hierarchical structure of the product catalog for supervision. This involved designing a hierarchical batch sampling strategy to ensure effective training of the contrastive loss. The system generates multi-level integer codes that represent product similarity, enabling better product understanding and discovery across different categories. Two distinct embedding strategies (ESCI and ESCI+Gemma) were developed to cater to different use cases (precision vs. discovery).