The evolution of Grab's machine learning feature store

This post details the evolution of Grab's ML feature store from a feature-centric approach (Amphawa) to a feature-table centric design using Amazon Aurora Postgres. Key technical contributions include: 1. Introduction of 'feature tables' for managing schema and data on a per-table basis, enabling contextual retrieval for composite keys and atomic updates for entire collections of features. 2. Design of a new ingestion framework emphasizing atomicity and reproducibility, involving landing feature tables in the data lake (Parquet on S3), registering and deploying collection summaries, pre-ingestion validation, reverse ETL into Aurora Postgres, and post-ingestion warm-up via shadow reading. 3. Leveraging AWS Aurora for PostgreSQL, specifically its distributed storage architecture, independent scaling of reader and writer nodes (using Serverless for writer and Provisioned for readers), and managed service benefits to meet non-functional requirements like isolation of read/write capacity and cost-effectiveness.