Elastic Feature Trails

Elastic Security continues to evolve its agentic AI capabilities, integrating with AWS services like Amazon Bedrock and GuardDuty. The platform now features Elastic Workflows for automated triage, enrichment, and response, alongside Agent Builder for AI-driven reasoning. New Entity Analytics capabilities enhance context for AI agents, improving accuracy and proactive threat hunting. This builds upon the foundation of providing AI-specific security for AWS infrastructure and AI applications. This post emphasizes the critical role of trusted context, governance, and observability for enterprise-scale Agentic AI in financial services, highlighting the need for unified data, observability as an AI control plane, and evolving security practices to manage AI-driven actions.

Elasticsearch 9.5 introduces Columnar Mode for reduced storage and faster analytics, and VectorDB index mode for out-of-the-box vector search optimization. These features enhance data storage efficiency and simplify the implementation of AI and RAG applications. The new columnar metrics engine in Elastic Observability enables public sector IT teams to combine logging, metrics, and traces in one platform, improving uptime and cost efficiency. It offers native Prometheus and PromQL support, agenti. This post highlights Elastic's continued leadership in observability platforms, emphasizing its high cost-efficiency for logs, metrics, and traces through optimized storage engines (columnar for metrics, full-text for logs/traces). It also details advancements in AI-driven investigations with Elastic Streams for log structuring and Knowledge Indicators, and the use of AI agents with full context retrieval powered by semantic search and context engineering. The platform's commitment to open standards like OpenTelemetry and Prometheus is also reiterated.

Elastic Cloud Serverless has expanded its global coverage to 33 regions across Google Cloud and Microsoft Azure, enabling users to deploy search, observability, security, and RAG applications with enhanced scalability and independent scaling of search and ingest tiers. This expansion aims to eliminate infrastructure bottlenecks and allow developers to focus on building solutions. Azure Private Link support is now generally available, allowing private network connections between Azure workloads and Serverless projects, enhancing security by keeping traffic off the public internet. This complements existing AWS PrivateLink support.

This post emphasizes the need for a robust data foundation to achieve real AI ROI, moving beyond AI activity to measurable outcomes. It highlights the importance of data readiness, observability for AI agents, consolidating data platforms for efficiency and context, and investing in people alongside technology. The unified data platform is presented as a key enabler for efficient AI, reducing query costs and improving outputs by ensuring data relevance and eliminating fragmentation. This aligns with the consolidation of AI efforts onto unified platforms to overcome the limitations of fragmented portfolios and achieve scalable AI deployment. It highlights how a unified platform, like Elastic's, enables partners to bypass integration friction, scale faster, and optimize delivery for AI investments. Examples like PepsiCo and Colsubsidio demonstrate the benefits of unifying data architectures for observability, reducing costs, and improving MTTR.

This post details the integration of Elastic's Search AI Platform with Google Cloud services to enhance security analytics from data ingestion to incident response. It covers threat intelligence integration (Mandiant, VirusTotal), cloud security posture management (CSPM) with Google Cloud Security Command Center (SCC), and seamless security log ingestion using Elastic Agent (with Fleet management) and Google Cloud Dataflow/Logstash. The integration aims to provide a unified solution for securing data and building comprehensive customer financial profiles by enabling data movement from mainframes to Google Cloud, processing and enrichment in BigQuery, and search capabilities through Elastic Cloud. It also addresses mainframe capacity planning with remote dataset transcoding and data transfer via FICON and Interconnect.

This post introduces the critical need for enhanced cybersecurity in the space domain, highlighting how space infrastructure has become load-bearing for critical systems like finance and emergency response. It details the unique challenges of securing space missions, which involve multiple interconnected segments (user, ground, link, space) and utilize specialized protocols (CCSDS, SDLS, PUS) that traditional security tools cannot interpret. The post emphasizes the necessity of a common data schema for space mission telemetry and the integration of threat intelligence frameworks like SPARTA with operational detection rules to enable effective anomaly detection and incident response across these diverse segments. It advocates for a collaborative approach to developing open standards for space cybersecurity to foster collective defense.

This post details the creation of a hybrid geospatial RAG application using Elasticsearch and Amazon Bedrock. It highlights Elasticsearch's capabilities in combining lexical, geospatial, and vector search within a single query. The application leverages Elasticsearch for storing and querying geospatial and vector data, Amazon Bedrock for LLM integration and entity extraction, and AWS Location Service for geocoding. This demonstrates a practical application of advanced search techniques for generative AI. The post also explains the fundamental concepts behind semantic search and vector embeddings, illustrating how text can be represented numerically to enable similarity searches. It contrasts keyword search with semantic search, emphasizing the latter's ability to understand meaning and context. The discussion covers the challenges of LLMs, such as data limitations and hallucinations, and introduces Retrieval Augmented Generation (RAG) as a solution to ground LLM responses in specific data sources. The post outlines the process of turning semantic search into a mathematical problem using vector embeddings and multi-dimensional representations of concepts.

This post introduces a method for safely sampling production data into pre-production environments using Logstash and UDP. This approach allows for a subset of production data to be routed to pre-production clusters without risking disruption to the production data flow, offering a lightweight alternative to more complex patterns like the output isolator. It addresses scenarios with resource constraints where full environment separation is impractical. The solution leverages Logstash's pipeline-to-pipeline communication with UDP's fire-and-forget nature, implemented via a Ruby filter for sampling and a UDP output.

Elastic Cloud now supports Bring Your Own Key (BYOK) for encryption at rest with AWS KMS and Azure Key Vault, allowing users to leverage their own keys to encrypt data and snapshots stored within Elastic Cloud deployments. This enhances data security and control, with features for key rotation and revocation.

The Kibana Discover feature has undergone significant evolution, focusing on improving the user experience for data exploration and analysis. Key enhancements include a new enhanced data table powered by EuiDataGrid, offering features like resizable columns, adjustable row display (1-20 lines or auto-fit), and improved multi-field sorting. The ability to display images directly in the table by formatting fields as URLs has been added. Full-screen mode and a flyout for viewing selected documents are also available. A new 'Links panel' has been introduced to streamline navigation between dashboards, allowing users to organize and link related dashboards, carry over context (filters, queries) between linked dashboards, and save panels to a library for reuse across multiple dashboards.

Elasticsearch Query Language (ES|QL) is a new piped language for data exploration and investigation, simplifying data transformation, enrichment, and exploration. It offers features like auto-complete, in-app documentation, ad-hoc data exploration, aggregations, data transformation, enrichment, and in-line visualization editing. ES|QL can be used for observability and security alerts. The introduction of ES|QL also includes new ES|QL alerting rule types and the ability to create and manage ES|QL enrichment index policies.

The Elasticsearch Health API provides a comprehensive diagnosis of cluster health, going beyond basic shard status to perform root cause analysis for unhealthy indices and data streams. It introduces modular health indicators for various components like master stability, repository integrity, shard availability, disk space, Index Lifecycle Management (ILM), and Snapshot Lifecycle Management (SLM). The API offers both a high-level status and detailed reporting with actionable resolution steps. In 8.9, the ILM Health Indicator is enhanced with new rules to detect ILM policies stuck in specific steps for too long, providing early warnings and actionable insights for performance and stability issues. This indicator will also appear in the Elastic Cloud Deployments console.