Synthetic Data Generation for Financial AI Research with NVIDIA NeMo | NVIDIA Technical Blog

What this post added

This post details the engineering of an iterative synthetic data generation pipeline for financial AI research. It describes the architecture and implementation of a system that uses NVIDIA NeMo Data Designer for structured generation, NeMo Curator for scalable semantic deduplication (with specific parameters for embedding models, similarity thresholds, and clustering), and NVIDIA Nemotron models served via vLLM (with details on tensor parallelism and concurrent requests). The post elaborates on the iterative process, including category-weighted sampling, filtering malformed outputs, global semantic deduplication against an accumulated corpus, selection of diverse few-shot examples, and dynamic adjustment of category weights. It also provides specific software versions, hardware configurations, and pipeline parameters used for reproducibility, highlighting the engineering challenges of creating diverse and unique datasets at scale.