Generating Test Data for Pinecone

What this post added

Introduces a multi-stage workflow for generating test data for vector databases, focusing on creating realistic datasets for performance and accuracy testing. This includes utilities for generating Parquet files from text datasets, creating embeddings locally using different hardware (CPU, GPU, MPS), and preparing data in a format suitable for Pinecone's bulk import. The workflow addresses challenges like metadata size limits and the need for scalable data generation by splitting the process into distinct stages.