Design an Event Analytics Platform

Design an end-to-end data platform for raw product events that supports analytics and downstream reporting.

Event Schema

The platform must handle events with the following schema:

• event_id: Unique identifier for the event.
• event_type: Type of event (e.g., 'page_view', 'button_click').
• user_id: Identifier for the user performing the event.
• user_setting: A nested object containing user-specific settings at the time of the event.
• event_metadata: A nested object containing additional event-specific metadata.

Discussion Points

The design discussion should cover the following aspects:

• The choice between a data lake, a data warehouse, or a hybrid approach, including their respective trade-offs.
• Strategies for event ingestion, specifically evaluating the utility of Kafka.
• When to employ streaming versus batch processing paradigms.
• Mechanisms for scheduling and orchestrating data jobs.
• Design principles for implementing data quality checks.
• Data modeling for warehouse tables, focusing on events, users, and workplaces.
• Methods for preserving historical changes in a user dimension table (e.g., Slowly Changing Dimensions).
• Approaches to model the many-to-many relationship between users and workplaces.
• The trade-offs associated with flattening nested object columns versus retaining semi-structured data.