Search indexing optimisation

This post details optimizations to Grab's incremental search data indexing process. It introduces the use of MySQL binary logs (Binlog) for Change Data Capture (CDC) to replace the previous Kafka stream approach. Key optimizations include: 1. Filtering irrelevant stream events by comparing PayloadBefore and PayloadAfter to only process changes relevant to Elasticsearch, reducing DB reads by 90% and filtering 55% of irrelevant events. 2. Performing partial updates in Elasticsearch by using the differences between PayloadBefore and PayloadAfter, reducing DB load by 80% and Elasticsearch load by only sending changed fields. These changes eliminate data loss, reduce coupling with MySQL table structure, and prevent redundant Elasticsearch updates and DB reads.