Log Management and Observability

4/30/2026

Introduced the WATCH (Weekly Attack Testing for Continuous Health) framework for automated detection testing. WATCH simulates malicious behavior in a staging environment to validate end-to-end detection firing from log source to SOAR alert routing. It leverages GitLab CI/CD for scheduling, test execution, and verification, and integrates with GitLab Duo for AI-assisted test creation by providing a `BaseSecurityTest` abstract class and Duo Agent Skills for defining test creation standards.

4/28/2026

This post details the implementation of a CI/CD observability solution for GitLab self-managed instances. It outlines the challenges of measuring CI/CD performance at scale, proposes a solution architecture involving the gitlab-ci-pipelines-exporter, Prometheus, and Grafana, and provides detailed Kubernetes deployment instructions for these components. It also includes configuration references for the exporter, Prometheus, and Grafana, along with key metrics related to pipeline and job execution.

2/19/2026

Introduced trend tracking, vulnerability age distribution, and risk scoring to the GitLab Security Dashboard. Risk scores are calculated using vulnerability age, EPSS, and KEV scores. This allows for measuring remediation velocity and program effectiveness.

3/26/2025

Introduces the enablement of event-level product usage data collection from GitLab Self-Managed and GitLab Dedicated instances in GitLab 18.0. This new capability will track product usage interactions like CI/CD pipeline initiation, merge request completion, webhook triggers, and issue creation. User identifiers will be pseudonymized, and source code will not be collected. The goal is to provide more granular insights for performance analysis, optimization, trend discovery, and product improvements.

8/19/2024

Introduces the capability to stream pod and container logs directly from the GitLab UI for Kubernetes deployments, enhancing observability and troubleshooting without requiring users to switch tools. Details the steps for configuring a GitLab project with a Kubernetes agent, deploying a sample workload, and accessing the log streaming feature through the GitLab Environments UI.

6/13/2024

Introduces the Beta release of Distributed Tracing as a new capability within GitLab Observability. This feature allows engineers to collect trace data using OpenTelemetry, ingest and query it at scale using a ClickHouse-based solution, and analyze it through a new Trace Explorer UI. The post details the concepts of traces and spans, provides examples of how distributed tracing helps in microservices architectures, and showcases internal use cases for debugging the GitLab Agent for Kubernetes and optimizing build pipeline durations. It also mentions upcoming releases for Metrics and Logging.

1/25/2024

Introduced a feature flag (FF_SCRIPT_SECTIONS) in GitLab 16.7 and GitLab Runner 16.7 that adds collapsible output sections to CI job logs for multi-line commands, specifically for jobs executed in the Bash shell. This improves the debugging experience by allowing users to expand and view all commands within a multi-line script block. The post also demonstrates how to combine this with custom collapsible sections for further log output customization and notes that PowerShell shell support is planned for a future release.

12/14/2023

This post details the stress-testing of the Product Analytics stack by instrumenting GitLab.com, leading to a 17x increase in load and over 20 million events per day. This allowed for validation of scaling strategies, identification and resolution of query performance concerns, improvement of the onboarding experience for the upcoming Beta program, and planning for future improvements towards general availability. The team also explored leveraging AI to make it easier to find and understand Product Analytics data.

3/27/2023

This post introduces the new Product Analytics capability within GitLab. It details the vision for enabling developers to easily add instrumentation to their applications, providing infrastructure to receive and process data, run experiments, and enable consumers of the data to gain insights. The initial focus is on web applications using JavaScript and Ruby on Rails, with plans to expand to other frameworks and tech stacks. The implementation will leverage open-source technologies: Snowplow for instrumentation, ClickHouse for data storage, and Cube.dev for data processing and visualization, with ECharts for charting. Configuration will be managed via files in GitLab projects, allowing for collaboration through merge requests. The post also highlights the commitment to user privacy by allowing users to provide their own Kubernetes clusters for data processing and storage, ensuring data ownership.

10/18/2022

Introduces incident timelines as a new feature within GitLab's incident management capabilities. Incident timelines serve as a single source of truth for key updates and events during an incident, facilitating faster updates via quick actions and enabling retroactive additions for accurate record-keeping. This feature aims to improve incident response efficiency and post-incident review accuracy.

6/27/2022

Introduced Streaming Audit Events to provide real-time visibility into GitLab group and project activities. Events are sent to a user-configurable HTTPS destination. This enables automation workflows, such as responding to changes in merge request approval settings or creating issues based on account additions. Demonstrated integration with Pipedream to simplify the consumption and automation of these audit events, including data transformation and forwarding to other tools or custom code execution.

6/14/2022

This post defines and contrasts observability with traditional monitoring, emphasizing its importance in modern DevOps for holistic system understanding and debugging. It introduces the three fundamental pillars of observability (logs, metrics, tracing) and mentions OpenTelemetry as a standard. The post also highlights the benefits of observability for reducing MTTR, improving UX, and its role in chaos engineering and shifting left in the CI/CD pipeline.

4/28/2022

This post details the technical implementation of a system to analyze Stack Overflow questions related to GitLab. It describes the architecture involving the GitLab DevOps Platform, a Kubernetes cluster, and open-source Python libraries (scikit-learn for TF-IDF, Streamlit for visualization, and Spacy). The process involves pulling data from the StackOverflow API, preprocessing text, calculating TF-IDF, scanning for security vulnerabilities, and deploying the application to Kubernetes. It highlights the use of multi-project pipelines for automation and demonstrates the creation of dashboards to identify common community questions, such as those related to GitLab CI, pipelines, and Docker images.

4/5/2022

This post defines observability as the collection and analysis of data logs, metrics, and traces, and highlights its critical role in cloud-native architectures. It explains how observability provides end-to-end visibility, enables faster issue detection and resolution, and lays the foundation for AI and machine learning. It also positions GitLab's observability capability as foundational to its DevOps platform, emphasizing its open-source nature, reliance on open APIs, and integration into the native workflow.

3/10/2022

This post details how GitLab shares Service Level Indicators (SLIs) across engineering departments to monitor feature and service availability. It explains the use of Grafana dashboards, generated using Grafonnet, to display SLIs separated by feature, and the implementation of multiwindow, multi-burn-rate alerts based on Google's SRE workbook to identify issues with feature-specific error budgets. The goal is to enable development groups to recognize when it's important to prioritize performance and availability work for the features they build.

1/20/2022

This post details the integration of Falco for container host security within GitLab. It outlines the steps to deploy Falco using GitLab-Managed Apps and CI/CD pipelines, including the necessary Helmfile and CI configuration. The post also explains how to define custom Falco rules for detecting specific system behaviors (e.g., file integrity checks) and how to configure alerts for rule violations, providing insights into runtime threats in Kubernetes environments.

1/11/2022

This post emphasizes the importance of comprehensive and continuous software documentation, including leveraging automated processes and observability data to create real-time, ongoing documentation streams. It highlights how logs, scripts, configuration files, issue/alert logs, and observability dashboards contribute to this continuous documentation effort, ultimately improving reliability, security practices, and cloud utilization.

11/18/2020

This post introduces a Python script (`check_docker_hub_limit.py`) and a Prometheus exporter (`docker-hub-limit-exporter`) to monitor Docker Hub rate limits. The script allows users to check the current limit and remaining pull requests, set warning and critical thresholds, and provides performance metrics. The Prometheus exporter exposes these metrics via an HTTP endpoint for scraping by Prometheus, with a demo environment including Grafana for visualization. The post also references existing GitLab blog posts on mitigating Docker Hub rate limits and using the Dependency Proxy.

8/3/2020

This post introduces GitLab's alert and incident management capabilities, focusing on aggregating IT alerts via a generic webhook receiver, triaging alerts in a list view, indicating ownership, assigning alerts, promoting alerts to incidents by creating GitLab issues, and investigating metrics within the alert. It also outlines upcoming features such as embedded logs, linked runbooks, a custom integration builder, and an incident dashboard.

6/25/2020

This post details learnings from shadowing an SRE, emphasizing the interconnectedness of systems, the importance of resource management (e.g., CPU allocation for PostgreSQL vs. Redis), and how SREs use tools like Prometheus and Kibana for monitoring and incident detection. It highlights that incidents are essentially bugs in site infrastructure and that SREs proactively scale services based on resource usage trends. The post also touches on the role of SREs in supporting and developing GitLab infrastructure beyond just incident response.

5/7/2020

This post introduces and details GitLab Metrics, a feature powered by Prometheus for monitoring Kubernetes clusters and custom application metrics. It explains how to use it as a managed application within GitLab or by pointing to an external Prometheus instance. Key capabilities include default and custom metrics, alerts, visualization of deploys on metrics charts, and custom dashboards. The post also highlights GitLab's internal dogfooding of this capability, with the goal of eventually replacing Grafana.

4/29/2020

GitLab has built a Status Page tool for communicating incident status and maintenance times. The tool allows for incident updates to be managed in a single place, balancing public communication with the need for controlled visibility. It supports displaying rich data from GitLab incident descriptions and comments. The backend converts incident issues to JSON via a background job, and the frontend is a stand-alone VueJS application hosted in cloud storage. This enables users to manage incident communications without tool switching and provides a controlled way to inform the public about incidents.

4/13/2020

This post provides a first-hand account of shadowing Site Reliability Engineers (SREs) at GitLab, detailing their daily workflow, tooling, and challenges. It highlights the SRE's role in managing alerts, distinguishing between noise and actionable incidents, and responding to actual incidents. Key takeaways include the importance of streamlined tooling (GitLab issues, Slack, Zoom), the challenge of managing alert noise, the relative infrequency of major incidents, the difficulty of effective monitoring, the critical role of communication (especially in an all-remote, async environment), the love for documenting everything in issues for handover and root cause analysis, the collaborative nature of incident resolution, and the growing field of monitoring with the creation of a Scalability team to curate alerting criteria.

4/3/2020

Introduced GitLab Incident Management as a viable category, aiming to decrease overhead by centralizing incident data and communication within a single DevOps platform. Key capabilities include incident issues for data capture, Slack integration for communication, Zoom linking, embedding of Kubernetes and Grafana metrics, a generic webhook receiver for alerts, and automatic issue closure upon Prometheus recovery alerts. The post outlines manual creation of incident issues via the 'incident' label and automatic creation from alerts using issue templates. It also details embedding metrics and linking action items (related issues/MRs) to incident tickets.

3/3/2020

Introduced Elastic log Stack and Log Explorer in the 12.8 release for log aggregation and searching. Enabled deployment of Elastic stack with Filebeat to Kubernetes clusters to collect and surface logs from across environments in the GitLab UI. Enhanced context preservation when navigating from metric charts to the log explorer. Highlighted the use of Jaeger for distributed tracing.

1/23/2020

This post details the iterative development process for GitLab's error tracking feature. It outlines the product strategy for error tracking, breaking it down into epics for instrumentation, correlation, resolution, and triaging. Specifically, it highlights the development of a 'Create an Issue' button from Sentry error details, which was implemented in three frontend iterations and supported by backend work to associate issues with errors and prevent duplicates. The post emphasizes the benefits of iterative development, including faster shipping of value and increased merge request rates.

12/16/2019

This post details a frontend engineer's experience shadowing an on-call SRE, observing incident response activities. Key takeaways include the utility of change-based alerting, the distinction between alerts and incidents, the diverse toolset used by SREs (PagerDuty, Slack, Grafana, Kibana, Zoom, internal GitLab projects), the potential single point of failure in monitoring GitLab.com with GitLab itself, and feature proposals for multi-user issue editing and more webhooks for issues. The post highlights the complexity of SRE work and the ongoing development of features to support these workflows.

12/16/2019

This post announces the migration of GitLab's observability suite (custom metrics, logging, tracing, and alerting) from proprietary code to the open-source Core product. It outlines the rationale behind this move, emphasizing stewardship, the fundamental nature of observability for developers, and the desire for community input. Specific areas for potential contribution include enhancing custom metrics integration with Prometheus, innovating log tailing capabilities, and deepening the integration of Jaeger for tracing and alerting. The post also touches on closing the DevOps loop by integrating alerting with incident management issues.

8/29/2019

This post details GitLab's vision and roadmap for building out its observability capabilities, focusing on closing the DevOps loop. It highlights the development of new monitoring products for application performance and deployment health, with priorities in metrics, cluster monitoring, and incident management. Key aspects include ease of instrumentation, seamless insight gathering, and proactive response to SLO degradation. The post mentions existing Prometheus integration for Kubernetes metrics, MVC stage development for logging, tracing, and APM, and plans for cluster cost optimization and alerting. It also emphasizes dogfooding the incident management system and extending monitoring capabilities to GitLab self-managed instances.

7/16/2019

This post details the progress and challenges in enabling Elasticsearch for advanced search on GitLab.com. Key contributions include implementing selective indexing with per-project and per-group controls, fixing pre-existing bugs and regressions related to Elasticsearch usage and performance, and improving the administration experience by automating backfill tasks and allowing configuration of index parameters. It also highlights ongoing work to address scaling issues and improve the management of the Elasticsearch schema and search result quality.

11/16/2018

GitLab 11.5 introduces support for Elasticsearch 6.x and removes support for Elasticsearch 5.5 and earlier. This upgrade is necessary due to breaking changes in Elasticsearch 6, specifically the removal of mapping types and the introduction of the 'join' datatype for parent-child relationships. The post details the technical implications of these changes on GitLab's indexing strategy, including the implementation of a custom 'type' field and the use of the 'join' datatype with routing by parent ID. Elasticsearch 5.6 is supported as a transitional version due to its early support for the 'join' datatype. This upgrade requires a reindex of Elasticsearch data.

9/27/2018

This post details GitLab's integration of Prometheus for monitoring and observability, advocating for its use by customers. It describes a specific incident where Prometheus was crucial in debugging a catastrophic failure related to Amazon's EFS filesystem, enabling rapid diagnosis and resolution. The post also highlights the availability of GitLab.com's Prometheus dashboard as a template for self-managed instances and the ongoing training of GitLab's Support team on Prometheus.

4/13/2018

This post details the integration of GitLab's logging framework with the ELK Stack (Elasticsearch, Logstash, Kibana) and Logz.io, enabling centralized log aggregation, analysis, and visualization for monitoring, troubleshooting, and analyzing team activity. It covers configuring Filebeat to ship various GitLab logs (production_json.log, production.log, api_json.log, application.log) to an ELK deployment or Logz.io, and demonstrates basic log analysis and visualization techniques in Kibana.

1/5/2017

This post introduces the initial integration of Prometheus for monitoring GitLab installations. It details the MVP plan to bundle Prometheus with GitLab CE for self-monitoring, including related issues for exposing service metrics, monitoring CI end-to-end, building public monitoring infrastructure, and creating a high-availability federated Prometheus solution. It also outlines future directions for web application monitoring, status monitoring, APM, end-user experience monitoring, user metrics/analytics, and business metrics, emphasizing deep integration into the developer experience. Additionally, it introduces 'deploy monitoring' to track deployment states and provides mockups for a deployboard.

8/18/2015

This post introduces the integration of GitLab Mattermost, an open-source, on-premises messaging app, into the Omnibus packages. This integration simplifies the setup for users requiring on-premises solutions and enhances internal communication capabilities by providing a Slack alternative. The Mattermost integration includes automatic OAuth configuration with the GitLab OAuth provider and is disabled by default to conserve resources. Future work will focus on easier posting from GitLab and GitLab CI to Mattermost, external security audits, and reaching a beta state for Mattermost.

12/8/2014

Introduced UDP log shipping as an alternative to TCP for log transmission in GitLab Enterprise Edition (7.1 and up) Omnibus packages. This provides a non-blocking, fire-and-forget mechanism for faster and more resilient log shipping, preventing log volume issues from affecting the GitLab instance. Configuration is simplified via `/etc/gitlab/gitlab.rb` and leverages `svlogd` and `remote_syslog` for log handling and transmission.