AI Application Performance Measurement
Introducing import from object storage for more efficient data transfer to Pinecone serverless

Introducing import from object storage for more efficient data transfer to Pinecone serverless

9/24/2024 · Ben Esh, Gibbs Cullen

What this post added

Introduced a new capability for bulk importing data directly from object storage (initially S3, with GCS and Azure Blob Storage planned) into Pinecone serverless namespaces. This feature offers up to 6x lower ingest costs compared to traditional upsert operations, handles ingestion of billions of records asynchronously, and supports Parquet source files. It is designed to streamline large-scale data ingestion for POCs, tenant onboarding, model experimentation, and production workload migrations.

Read the original post ↗