Homepage Globe Data Pipeline

2 posts · Dec 2020

What this trail covers

This feature thread tracks the evolution of the data pipeline and processing for the GitHub homepage's global community visualization. Initially, the focus was on collecting and displaying real-time data for a 3D globe. This post details the data goals, querying strategies using a data warehouse with Presto and Kafka, surfacing interesting data via repository health scores, and geocoding user-provided locations using Mapbox's API. It also touches on the scheduling of these tasks with Airflow. Subsequent posts in this thread have detailed the rendering of the globe using WebGL and three.js, performance optimizations for perceived speed, and the creative constraints imposed by performance requirements.

All GitHub trails

Dec 2020Visualizing GitHub’s global communityThis post details the data pipeline for the GitHub homepage globe. It covers querying a data warehouse using Presto and Kafka for real-time data like pull requests, using a repository health score model to surface interesting data, and geocoding user locations with Mapbox's forward geocoding API. It also mentions the use of Airflow for scheduling these data processing tasks.
Dec 2020How we built the GitHub globeThis post details the engineering effort behind the rendering of the GitHub globe using WebGL and the three.js library. It explains the scene composition with five layers (halo, globe, Earth's regions, open PR spikes, merged PR arcs), the use of CircleBufferGeometry and InstancedMesh for rendering Earth's regions by mapping circles to a world map PNG, and the visualization of pull requests using CubicBezierCurve3 and TubeBufferGeometry with setDrawRange for animation. It also covers centering the globe based on the device's timezone offset and the use of custom shaders for a halo effect to hide pixelation and reduce moiré patterns.
Advertisement