Blogs
Elastic logo

Elastic

Engineering blog · elastic.co/blog/category/engineering

57posts0this week0watchers

Feature trails

21
Feature trailActive

Bring Your Own Key (BYOK) for Encryption at Rest

Elastic Cloud now supports Bring Your Own Key (BYOK) for encryption at rest with AWS KMS, Azure Key Vault, and Google Cloud KMS for existing deployments. This allows users to add a customer-managed encryption key to a running deployment in place, without recreating it. The process involves replacing each instance with an encrypted one, moving data with native Elasticsearch shard allocation. Snapshots taken before the transition are not available through the new repository, but a fresh snapshot is taken automatically. Key rotations are picked up automatically, and revoking the key locks the deployment's data. Considerations include running on highly available deployments, understanding snapshot history changes, testing on smaller deployments, and potential data transfer costs.

6 posts · 2024–2026

Timeline

20242026
Feature trailActive

Agentic AI Context Layer

Elastic's platform supports securing AI systems and environments by providing visibility into AI-specific risks, such as prompt injection, model abuse, untrusted model onboarding, agent manipulation, RAG data poisoning, and safety control tampering. This includes Elastic AI Agent, Workflows, LLM security detections, and agentic SOC platform capabilities. The Elastic 9.4 release enhances Workflows with native automation for triage, enrichment, response, and case management, and introduces Agent B. This post emphasizes the need for visibility and context to detect unusual behavior and respond at machine speed in the context of AI agents in banking cybersecurity, highlighting the importance of correlating signals across applications, APIs, identities, endpoints, networks, and cloud environments to distinguish normal automated activity from threats. It also discusses the evolution of the SOC to combine human judgment with automated investigation and response, and the need for governance and transparency in autonomous systems.

21 posts · 2024–2026

Timeline

20242026
Feature trailActive

Compute-Optimized ARM Instances on Elastic Cloud for Azure

Elastic Cloud on Azure now offers compute-optimized ARM hardware profiles, leveraging Microsoft's Cobalt ARM processors. This provides improved price-performance for CPU-bound workloads, particularly in indexing, aggregations, and various search operations. The new profiles utilize the Dpldsv6 VM series with local NVMe SSD storage for enhanced random read performance. This offering is available in select Azure regions and supports migration of existing deployments.

1 post · 2026

Timeline

Feature trailActive

Cross-Cluster Search Performance

Elastic Cloud Serverless now offers cross-project search (CPS), allowing queries across up to 100 linked projects without data movement. This GA release enhances ML/AI capabilities to span linked projects, supports versionless and cloud-agnostic operation across all Elastic solutions (Observability, Security, Search, Vector DB), and provides granular access control and cost reduction. Programmatic access via a new user role is also introduced.

2 posts · 2023–2026

Timeline

20232026
Feature trailActive

MOSA Compliance for Data and Analytics

Elastic's platform supports the DoD's Modular Open Systems Approach (MOSA) by providing a severable data and analytics module that integrates with existing systems. Key aspects include leveraging OpenTelemetry for ingestion, OCSF/STIX/TAXII for security content, REST/JSON APIs for query and management, standard identity protocols (SAML, OIDC, PKI), S3 API for storage snapshots, and Kubernetes orchestration via Elastic Cloud on Kubernetes. The platform's open-source roots (Apache Lucene, AGPL v3). This post details how federal agencies can leverage Elastic to meet CISA's Logging Reference Architecture (LRA) requirements for OMB M-26-14, focusing on schema-first architecture with Elastic Common Schema (ECS), searchable data directly from object storage, policy enforcement points, and AI integration for logging operations and analysis. It emphasizes measurable normalization and data quality views, and the ability to keep data searchable directly from object storage.

2 posts · 2026

Timeline

Feature trailActive

Unified Data Platform for Fraud Detection

Elastic's platform approach, unifying data for AI, enables partners to scale faster and optimize delivery. This contrasts with fragmented portfolios that create integration friction and delay ROI. The platform's unified data layer, single indexing, and query layer reduce costs and expand capabilities, allowing partners to focus on delivering business value and advisory services for AI transformation. This post emphasizes the need for governed retrieval layers for AI agents in the public sector, and the integration of observability and security data for enhanced threat detection and incident response.

3 posts · 2025–2026

Timeline

20252026
Feature trailActive

Unified Financial Services Data Platform

Elastic and Google Cloud enable financial institutions to build comprehensive customer financial profiles by integrating data from mainframes into BigQuery for processing and enrichment, and then into Elastic Cloud for advanced search capabilities. This architecture supports use cases like enhanced customer experience, Customer 360 views, partnership management, cost optimization, and risk reduction through fraud detection. The system allows for data migration and orchestration from mainframes to cloud-based data stores, enabling unified data for AI-powered fraud prevention, security, and compliance.

3 posts · 2023–2026

Timeline

20232026
Feature trailActive

IT/OT Data Convergence for Smart Cities

Elastic's platform enables IT/OT data convergence for smart cities by unifying data from operational technology (OT) and information technology (IT) systems. This allows for correlated analysis, faster incident response, and improved operational insight. The platform uses Elastic Agent and Fleet for consistent data collection and normalization, Elastic Cloud as a central datastore, and Kibana for visualization. It addresses protocol sprawl and vendor lock-in by providing a common data layer and supports geospatial analysis for location-aware insights. Security is enhanced through Zero Trust principles and protocol-aware detections mapped to MITRE ATT&CK for ICS.

1 post · 2026

Timeline

Feature trailActive

ECK Dynamic Namespace and Orchestration Management

ECK 3.5 enhances Elastic Cloud on Kubernetes with dynamic namespace management via label selectors, allowing live onboarding/offboarding without operator restarts. It introduces 'pause orchestration' for controlled maintenance windows, expands mutual TLS across all stack components connecting to Elasticsearch, and improves StackConfigPolicy for declarative Elasticsearch role definitions and reusable policies with dynamic substitution variables.

1 post · 2026

Timeline

Feature trailActive

Columnar Storage and VectorDB Indexing

Elasticsearch 9.5 introduces Columnar Mode for reduced storage and faster analytics, and VectorDB index mode for out-of-the-box vector search optimization. These features enhance data storage efficiency and simplify the implementation of AI and RAG applications. The new columnar metrics engine in Elastic Observability enables public sector IT teams to combine logging, metrics, and traces in one platform, improving uptime and cost efficiency. It offers native Prometheus and PromQL support, agenti

6 posts · 2023–2026

Timeline

20232026
Feature trailActive

Elastic Cloud Serverless Global Expansion

Elastic Cloud Serverless has expanded its global coverage to 33 regions across Google Cloud and Microsoft Azure, enabling users to deploy search, observability, security, and RAG applications with enhanced scalability and independent scaling of search and ingest tiers. This expansion aims to eliminate infrastructure bottlenecks and allow developers to focus on building solutions. Azure Private Link support is now generally available, allowing private network connections between Azure workloads and Elastic Cloud Serverless deployments, enhancing security by keeping traffic within Azure's network backbone. AWS PrivateLink support is also available.

4 posts · 2026

Timeline

Feature trailActive

Observability Pipeline Cost Optimization

Elastic's Observability platform provides end-to-end monitoring for SAP workloads, integrating cloud infrastructure, SAP machine layer metrics (via Google Cloud's Agent for SAP and Elastic Agent), and SAP application layer data (via Kyndryl's custom connector). This unified approach leverages Kibana for visualization and AI-driven insights, aiming to reduce complexity and cost in monitoring SAP environments. The platform now includes enhanced SLO/SLI monitoring capabilities for defining, tracking, and visualizing service level objectives and indicators.

3 posts · 2023–2026

Timeline

20232026
Feature trailActive

Elastic Security and Google Cloud Integration for Security Analytics

Elastic Security integrates with Google Cloud services, including Security Command Center (SCC) and Mandiant threat intelligence, to provide a comprehensive security analytics platform. This integration streamlines data ingestion from various Google Cloud sources via Elastic Agent or Dataflow, enhances threat detection and analysis, and aids in mitigating misconfigurations and compliance issues through Cloud Security Posture Management (CSPM). The solution aims to offer end-to-end security from data ingestion to incident response for hybrid workloads.

4 posts · 2023–2026

Timeline

20232026
Feature trailActive

Space Cybersecurity Monitoring

This post introduces the critical need for enhanced cybersecurity in the space domain, highlighting how space infrastructure has become load-bearing for critical systems like finance and emergency response. It details the unique challenges of securing space missions, which involve multiple interconnected segments (user, ground, link, space) and utilize specialized protocols (CCSDS, SDLS, PUS) that traditional security tools cannot interpret. The post emphasizes the necessity of a common data schema for space cybersecurity monitoring, advocating for an open standard that any operator, constellation, government program, or commercial venture can adopt and contribute to. It also highlights the SPARTA framework as a resource for cataloging the space attack surface and the gap between threat frameworks and operational security due to the lack of readily available detection rules. The post argues that a common schema and shareable detection rules are crucial for building collective defenses in the space cybersecurity domain.

1 post · 2026

Timeline

Feature trailCooling

Vector and Graph Database Comparison

Elasticsearch 9.5 introduces Columnar Mode for reduced storage and faster analytics, and VectorDB index mode for out-of-the-box vector search optimization. These features enhance data storage efficiency and simplify the implementation of AI and RAG applications. The new columnar metrics engine in Elastic Observability enables public sector IT teams to combine logging, metrics, and traces in one platform, improving uptime and cost efficiency. It offers native Prometheus and PromQL support, agenti

4 posts · 2024

Timeline

20242026
Feature trailCooling

Detections as Code (DaC) for Security Rules

Elastic Security enhances its Detections as Code (DaC) capabilities by making the detection-rules repository more accessible and user-friendly for custom rule management. This includes features to minimize merge conflicts when forking the repository, allowing users to specify custom rule directories, configure which Elastic-provided unit tests to run or skip, and manage additional rule settings like exceptions and actions alongside detections. The goal is to provide an end-to-end experience for detection engineers.

3 posts · 2024

Timeline

20242026
Feature trailQuiet since 2024

Logstash Production Data Sampling to Pre-production

This post introduces a method for safely sampling production data into pre-production environments using Logstash and UDP. This approach allows for a subset of production data to be routed to pre-production clusters without risking disruption to the production data flow, offering a lightweight alternative to more complex patterns like the output isolator. The solution leverages Logstash's pipeline-to-pipeline communication with UDP, which operates on a fire-and-forget basis and doesn't require acknowledgement from the receiver. This ensures that even if the pre-production pipeline is blocked, it won't block data flow to the production cluster. This method is particularly useful for organizations with limited resources or operational constraints where maintaining separate end-to-end environments can be impractical.

1 post · 2024

Timeline

20242026
Feature trailQuiet since 2024

Elasticsearch Query Language (ES|QL)

Elasticsearch Query Language (ES|QL) is now generally available, offering a powerful, piped language for data exploration, transformation, and analysis. This release includes advanced capabilities like multi-step analysis, data enrichment via lookup policies, and integration with visualization tools. The platform also enhances security with GA of Bring Your Own Key (BYOK) for encryption at rest, and improves vector search performance with SIMD optimizations and int8 quantization by default.

1 post · 2023–2024

Timeline

20232026
Feature trailQuiet since 2024

Bring Your Own Key (BYOK) for Encryption at Rest

Elastic Cloud now supports Bring Your Own Key (BYOK) for encryption at rest with AWS KMS, Azure Key Vault, and Google Cloud KMS for existing deployments. This allows users to add a customer-managed encryption key to a running deployment in place, without recreating it. The process involves replacing each instance with an encrypted one, moving data with native Elasticsearch shard allocation. Snapshots taken before the transition are not available through the new repository, but a fresh snapshot is taken after the transition. This post details the strategic imperative for BYOK, its security benefits, limitations, and how enterprises can leverage customer-managed keys with AWS KMS, Azure Key Vault, and GCP Cloud KMS for enhanced control and compliance. It also mentions that the subsequent blog post will focus on setting up AWS KMS with Elastic Cloud.

1 post · 2024

Timeline

20242026
Feature trailQuiet since 2023

Kibana Discover Data Table Enhancements

The Kibana Discover feature has undergone significant evolution, focusing on improving the user experience for data exploration and analysis. Key enhancements include a new enhanced data table powered by EuiDataGrid, offering features like resizable columns, adjustable row display (1-20 lines or auto-fit), and improved multi-field sorting. The ability to display images directly in the table by formatting fields as URLs has been added. Full-screen mode and a flyout for viewing selected documents have also been improved. Field statistics have been added to Discover, providing quick insights into a field's top values, cardinality, and distribution. Performance improvements have been made by splitting Elasticsearch requests for faster document visibility. Alerting capabilities have been integrated, allowing users to create search threshold alerts directly from Discover. Future enhancements include the ability to query data without a pre-existing data view.

2 posts · 2022–2023

Timeline

20222026
Feature trailQuiet since 2023

Elasticsearch Cluster Health API

The Elasticsearch Health API provides a comprehensive diagnosis of cluster health, going beyond basic shard status to perform root cause analysis for unhealthy indices and data streams. It introduces modular health indicators for various components like master stability, repository integrity, shard availability, disk space, Index Lifecycle Management (ILM), and Snapshot Lifecycle Management (SLM). The API offers both a high-level status and detailed reporting with actionable resolution steps.

2 posts · 2023

Timeline

20232026