Revolutionizing Computational Chemistry with Data Streaming

What this post added

This post details how Atomic Tessellator, a computational chemistry startup, uses Confluent and Apache Kafka to build a scalable and flexible platform for molecular research. It describes their transition from a batch-based ETL process using RabbitMQ, Celery, and a Django app to a data streaming architecture. The new architecture enables rapid ingestion of complex molecular data from databases and generative models into Confluent, followed by streaming to downstream "evaluation" applications. This facilitates quick and reliable replication of chemical reactions for catalyst discovery. The post highlights business benefits such as avoiding over-provisioning of compute resources, the ability to tweak pipelines mid-workload, and clear visibility into production pipeline status through JMX monitoring. It also mentions plans to explore Apache Flink for building a "pipeline templates" system.