Sitar-agent: Building a reliable dynamic configuration sidecar at scale

This post introduces the sitar-agent, a Kubernetes sidecar for dynamic configuration delivery. It details the configuration delivery lifecycle, including preloading snapshots from S3 on pod startup and periodic polling for updates. Key design decisions discussed are maintaining the agent as an isolated sidecar container (over cost savings from a main container library) to ensure reliability and multi-language support, optimizing the pull model with server-side caching and token-based database scanning to reduce load, and replacing the legacy Sparkey-backed datastore with RocksDB to address performance and concurrency limitations.