TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

Log Management and Observability

38 posts · Dec 2014 – Jul 2026

What this trail covers

GitLab's observability capability is open-sourced and relies on open APIs like Prometheus and OpenTelemetry. It is built into the GitLab DevOps platform to help users use the capability right away within their native workflow. The vision is to make every GitLab project observable by default, with features that are easy to operate without specialized, expert skills. Teams can connect the dots between ever-changing systems and their impact on user experience. This post introduces Distributed Tracing, which allows engineers to identify the source of application performance issues by analyzing the timing of each operation and any errors as they occur. Traces are composed of spans, representing individual operations, and are particularly valuable for microservices architectures. Data is collected using OpenTelemetry, ingested and retained at scale with ClickHouse and Kubernetes, and analyzed through an advanced user interface including a Trace Explorer with filtering, error highlighting, RED metrics, and a timeline view. GitLab engineers are using Distributed Tracing to debug issues in the GitLab Agent for Kubernetes and optimize build pipeline duration by identifying performance bottlenecks.

All GitLab trails
  1. Green DevOps: Why carbon measurement belongs in your CI/CD pipelineThis post introduces Green DevOps practices by detailing the integration of third-party tools (Eco CI and Carmen) into GitLab CI/CD pipelines for carbon measurement. It explains how Eco CI measures per-job emissions and how Carmen provides deeper insights into infrastructure and application carbon footprints, including operational and embodied carbon. The post provides a concrete example of integrating Carmen into a `.gitlab-ci.yml` file and illustrates practical use cases, such as identifying inefficient dependency installation in CI jobs and decommissioning idle services. It emphasizes the low effort and real payoff of this strategy, linking carbon efficiency to FinOps benefits and preparing for future emissions reporting regulations.
  2. Automate detection testing with GitLab CI/CD and DuoThis post introduces the WATCH (Weekly Attack Testing for Continuous Health) framework, an automated system for testing security detections. WATCH simulates malicious behavior in GitLab's staging environment to validate that detections fire end-to-end through the SIEM and SOAR. It leverages GitLab CI/CD for scheduling, execution, and verification. The framework is designed for ease of use, allowing engineers to write new tests by subclassing `BaseSecurityTest` and focusing on setup, execution, and cleanup. GitLab Duo is highlighted as a tool to accelerate test creation by scaffolding WATCH tests based on prompts and Duo Agent Skills.
  3. How to build CI/CD observability at scaleThis post details the implementation of a containerized observability solution for GitLab CI/CD, combining the open-source gitlab-ci-pipelines-exporter with Prometheus and Grafana. It outlines the challenges of measuring CI/CD performance at scale, defines key metrics (pipeline duration, job success rates, queue times, runner utilization), and presents solution architecture with specific Grafana dashboards (Pipeline Overview, Job Performance, Runner & Infrastructure, Deployment Frequency). It provides Kubernetes deployment instructions for the Pipeline Exporter, Node Exporter, Prometheus, and Grafana, along with network policy configuration and configuration references for exporters, Prometheus, and Grafana data sources. Key metrics like pipeline duration, job status, and artifact size are also detailed.
  4. Track vulnerability remediation with the updated GitLab Security DashboardIntroduced trend tracking, vulnerability age distribution, and risk scoring to the GitLab Security Dashboard. Risk scores are calculated using vulnerability age, EPSS, and KEV scores. This allows for measuring remediation velocity and program effectiveness.
  5. More granular product usage insights for GitLab Self-Managed and DedicatedIntroduces the enablement of event-level product usage data collection from GitLab Self-Managed and GitLab Dedicated instances in GitLab 18.0. This new capability will track product usage interactions like CI/CD pipeline initiation, merge request completion, webhook triggers, and issue creation. User identifiers will be pseudonymized, and source code will not be collected. The goal is to provide more granular insights for performance analysis, optimization, trend discovery, and product improvements.
  7. Improve security auditing with GitLab Operational Container ScanningThis post introduces GitLab's Operational Container Scanning feature, which enables DevSecOps practitioners to run container vulnerability scans against deployed containers in a Kubernetes environment. It details the prerequisites, the process of deploying a sample application, connecting a Kubernetes cluster via the GitLab Agent for Kubernetes, and configuring the scanning cadence and target namespaces through the `config.yaml` file. The results are then viewable in the GitLab UI's 'Secure > Vulnerability report' under the 'Operational vulnerabilities' tab. This extends GitLab's existing container scanning capabilities by providing post-deployment security auditing.
  8. How to stream logs through the GitLab Dashboard for KubernetesIntroduces direct streaming of Kubernetes pod and container logs within the GitLab UI via the GitLab Kubernetes Agent. This feature allows developers to monitor and troubleshoot deployments without switching tools, enhancing GitOps workflows. It details the configuration of the agent, installation on a cluster, deployment of a sample workload, and creation of a GitLab environment to view logs. The post highlights the ability to view logs directly from the 'Environments' page by clicking 'View Logs' on a pod.
  9. Monitor application performance with Distributed TracingThis post announces the Beta release of Distributed Tracing as a new feature within GitLab Observability. It explains what distributed tracing is, how it works (collecting data with OpenTelemetry, ingesting/retaining with ClickHouse/Kubernetes, and analyzing via a new UI), and provides two internal use cases: debugging the GitLab Agent for Kubernetes and optimizing build pipeline duration. It also highlights the integration with OpenTelemetry as a vendor-neutral approach and mentions the underlying storage solution based on the Opstrace acquisition.
  10. Inside the improved CI logs management experience for multi-line commandsIntroduced a feature flag (FF_SCRIPT_SECTIONS) in GitLab 16.7 and GitLab Runner 16.7 that adds collapsible output sections to CI job logs for multi-line commands, specifically for jobs executed in the Bash shell. This improves the debugging experience by allowing users to expand and view all commands within a multi-line script block. The post also demonstrates how to combine this with custom collapsible sections for further log output customization and notes that PowerShell shell support is planned for a future release.
  11. Building GitLab with GitLab: Stress-testing Product AnalyticsThis post details the stress-testing of the Product Analytics stack by instrumenting GitLab.com, resulting in a 17x increase in daily event volume (over 20 million events/day). This allowed for validation of scaling strategies, identification and resolution of query performance concerns, improvement of the onboarding experience for the upcoming Beta program, and planning for future improvements towards general availability. The post also mentions the development of an event framework to simplify instrumentation for GitLab developers and the ongoing work on Customizable Dashboards and Visualization Designer, with exploration into leveraging AI for data discovery.
  12. Product Analytics: A sneak peek at our upcoming featureThis post introduces the new Product Analytics capability within GitLab. It details the vision for enabling developers to easily add instrumentation to their applications, providing infrastructure to receive and process data, run experiments, and enable consumers of the data to gain insights. The initial focus is on web applications using JavaScript and Ruby on Rails, with plans to expand to other frameworks and tech stacks. The implementation will leverage open-source technologies: Snowplow for instrumentation, ClickHouse for data storage, and Cube.dev for data processing and visualization, with ECharts for charting. Configuration will be managed via files in GitLab projects, allowing for collaboration through merge requests. The post also highlights the commitment to user privacy by allowing users to provide their own Kubernetes clusters for data processing and storage, ensuring data ownership.
  13. How to leverage GitLab incident timelinesIntroduces incident timelines as a new feature within GitLab's incident management capabilities. Incident timelines serve as a single source of truth for key updates and events during an incident, with functionality to add items via quick actions or comments, and to post-date events retroactively. The feature aims to improve efficiency during incidents and aid in post-incident reviews by providing a historical record.
  14. How we improved on-call life by reducing pager noiseIntroduced alert grouping in Alertmanager using the 'type' and 'env' labels to consolidate multiple SLI alerts for a single service into one notification. Implemented Alertmanager inhibition rules based on service dependencies defined in the metrics-catalog to prevent cascading alerts from downstream services. This reduced the number of pages for on-call engineers during service degradations and site-wide outages.
  15. Streaming audit events: Connect GitLab to your tech stackThis post introduces GitLab's Streaming Audit Events feature, which sends real-time events to a user-defined HTTPS destination. It highlights how Pipedream can simplify the process of receiving, transforming, and automating actions based on these events. Examples include alerting security teams on sensitive setting changes and using the GitLab API for further automation. The integration is open-source and contributions are welcome.
  16. Observability vs. monitoring in DevOpsThis post elaborates on the distinction between monitoring and observability, highlighting observability's ability to provide a holistic view of application behavior and user experience. It emphasizes the benefits of observability for debugging and reducing MTTR, contrasting it with traditional monitoring. The post introduces the three pillars of observability (logs, metrics, tracing) and mentions OpenTelemetry as a standard. It also discusses the importance of observability in DevOps and for business stakeholders.
  17. Enable SLO-as-Code with Nobl9 and GitLabThis post introduces an integration with Nobl9 to enable SLO-as-Code. It details how to configure GitLab CI/CD variables (CLIENT_ID, CLIENT_SECRET, ACCESS_TOKEN, PROJECT, SLOCTL_YML) and a CI/CD job using the `nobl9/nobl9-ci-template` to automatically push SLO definitions to Nobl9 whenever the referenced SLOCTL_YML file changes. This allows for version-controlled SLO definitions and ensures consistency between the code repository and Nobl9.
  18. How We Built a Stack Overflow Community Questions AnalyzerThis post details the technical implementation of a system to analyze Stack Overflow questions related to GitLab. It describes the architecture involving the GitLab DevOps Platform, a Kubernetes cluster, and open-source Python libraries (scikit-learn for TF-IDF, Streamlit for visualization, and Spacy). The process involves pulling data from the StackOverflow API, preprocessing text, calculating TF-IDF, scanning for security vulnerabilities, and deploying the application to Kubernetes. It highlights the use of multi-project pipelines for automation and demonstrates the creation of dashboards to identify common community questions, such as those related to GitLab CI, pipelines, and Docker images.
  19. Observability's role in cloud-native app developmentThis post defines observability as the collection and analysis of logs, metrics, and traces, highlighting its importance in cloud-native environments beyond traditional monitoring. It explains how observability improves cloud-native tech adoption by providing end-to-end visibility, enabling faster issue identification and resolution, and laying the foundation for AI/ML. The post emphasizes GitLab's vision to make every project observable by default, leveraging open-source APIs like Prometheus and OpenTelemetry, and integrating this capability into the existing DevOps platform for ease of use and to avoid vendor lock-in.
  20. How we share SLIs across engineering departmentsThis post details the implementation of sharing Service Level Indicators (SLIs) across engineering departments at GitLab. It explains how feature-specific availability scores are derived from the same metrics used for overall GitLab.com availability. The post highlights the use of Grafana dashboards with Grafonnet for visualizing error rates separated by feature and the application of multiwindow, multi-burn-rate alerts based on Google's SRE workbook. It emphasizes that by providing feature-specific data, it becomes easier to prioritize performance and availability work, and to surface issues in features with lower traffic that might be missed in broader service aggregations.
  21. Detecting container host anomalies with GitLab and FalcoThis post details the integration of Falco for container host security within GitLab. It outlines the steps to deploy Falco using GitLab-Managed Apps and CI/CD pipelines, including the necessary Helmfile and CI configuration. The post also explains how to define custom Falco rules for detecting specific system behaviors (e.g., file integrity checks) and how to configure alerts for rule violations, providing insights into runtime threats in Kubernetes environments.
  22. Downtime happens, but GitLab Incident Management can helpThis post introduces GitLab's Incident Management capabilities, detailing how it integrates with observability tools (metrics, logs, errors, runbooks, traces) to streamline incident response. It highlights the benefits of co-locating code, CI/CD, and monitoring within a single platform to reduce context switching and improve MTTR. The post outlines manual and automatic incident creation (via PagerDuty webhook), alert management, on-call schedules, escalation policies, and notification mechanisms (email, with plans for Slack, SMS, and phone calls). It emphasizes the goal of reducing overhead and accelerating problem resolution by surfacing alerts and incidents where code is developed.
  23. GitLab's unifiied and integrated monitoring strategiesThis post details how GitLab provides integrated monitoring capabilities for deployments, including an Environment Dashboard, CD pipeline visualization, performance job testing, and rollback functionality. It also explains the use of Feature Flags and Canary deployments for controlled rollouts and user feedback. Finally, it showcases cluster-wide and pod-specific monitoring dashboards for tracking system and application metrics.
  24. How to make Docker Hub rate limit monitoring a breezeThis post introduces a Python script (`check_docker_hub_limit.py`) and a Prometheus exporter (`docker-hub-limit-exporter`) to monitor Docker Hub rate limits. The script allows users to check the current limit and remaining pull requests, set warning and critical thresholds, and provides performance metrics. The Prometheus exporter exposes these metrics via an HTTP endpoint for scraping by Prometheus, with a demo environment including Grafana for visualization. The post also references existing GitLab blog posts on mitigating Docker Hub rate limits and using the Dependency Proxy.
  25. Designing an incident management workflow from scratch and where its usedThis post details the development of dedicated incidents within GitLab as a specialized issue type for managing alerts and incidents. It describes the problem validation and solution validation process, including the use of a Special Interest Group (SIG) for customer feedback. The post explains the evolution from using generic issues for incidents to creating dedicated incidents with a more tailored UI for faster resolution. It also highlights the significant increase in usage of this feature. The post also touches on the upcoming work on on-call management.
  26. How we manage IT Alerts in GitLabThis post introduces GitLab's alert and incident management capabilities, focusing on centralizing IT alerts and incident response within a single interface. It details current features such as aggregating alerts via a generic webhook receiver, list view triaging, ownership indication, alert assignment, promotion to GitLab issues for incidents, and direct investigation of metrics within alerts. It also outlines upcoming features like embedded logs, linked runbooks, a custom integration builder, and an incident dashboard. The post emphasizes the benefit of having these tools integrated within the existing GitLab DevOps lifecycle for faster remediation.
  27. 7 things I’ve learnt while shadowing an SREThis post details learnings from shadowing an SRE, focusing on incident management, resource usage monitoring (Grafana, Kibana), resource allocation tradeoffs (CPU for PostgreSQL vs. Redis), SRE on-call veto power during incidents, interconnectedness of systems requiring debugging and log tracking, primary incident reporting sources (Prometheus alerting), and the SRE role in supporting and developing infrastructure beyond incident response. It highlights the importance of observability tools and understanding system behavior for effective incident management and infrastructure development.
  28. How application performance monitoring metrics helps developersThis post introduces GitLab Metrics, powered by Prometheus, as a component of the broader GitLab Monitoring solution. It details how GitLab Metrics integrates with Kubernetes to provide default metrics (e.g., memory usage, NGINX metrics), custom metrics via promQL queries, alerts, and custom dashboards configurable via YAML. It also explains how to integrate an existing Prometheus instance and how GitLab's infrastructure team dogfoods this capability by providing feedback to improve charting and eventually phase out Grafana.
  29. How we built Status PageThis post details the technical implementation of GitLab's Status Page, a new tool for communicating incident status and maintenance times. It addresses requirements for no tool switching during incidents, controlled visibility of updates, and display of rich data from GitLab incident descriptions. The implementation involved a Spike exercise, evaluation of open-source solutions, and a decision to build a custom implementation. The backend approach involves a background job converting incident issues to JSON and publishing to the Status Page. The frontend is built with VueJS and GitLab UI, and is a stand-alone application published to cloud storage (S3). The Status Page solution ensures that incident issues are converted to JSON and mirrored on the Status Page by a background job.
  30. Shadowing a Site Reliability EngineerThis post details a frontend engineer's experience shadowing Site Reliability Engineers (SREs) at GitLab. It highlights takeaways on tooling (using standard communication tools like issues, Slack, and Zoom), managing alert noise, the relative infrequency of actual incidents, the challenges of monitoring, the importance of communication and collaboration, and the growing field of monitoring with the creation of a Scalability team. It also mentions the SRE team's work on features to help with their workflow and the Ops Section Product Vision.
  31. Understand incident management with GitLabIntroduced GitLab Incident Management as a viable category, aiming to decrease overhead by centralizing incident data and communication within a single DevOps platform. Key capabilities include incident issues for data capture, Slack integration for communication, Zoom linking, embedding of Kubernetes and Grafana metrics, a generic webhook receiver for alerts, and automatic issue closure upon Prometheus recovery alerts. The post outlines manual creation of incident issues via the 'incident' label and automatic creation from alerts using issue templates. It also details embedding metrics and linking action items (related issues/MRs) to incident tickets.
  32. How to integrate operation logs and metrics in GitLabThis post details the introduction of Elastic log Stack and Log Explorer in GitLab 12.8 for log aggregation and interaction. It also reiterates the use of Prometheus for metrics and Jaeger for distributed tracing, emphasizing the goal of providing integrated observability within the DevSecOps platform. The post highlights the importance of monitoring and observability in cloud-native architectures and the move towards open-sourcing observability features.
  33. Why we scoped down to build up error trackingThis post details the iterative development process for GitLab's error tracking feature. It outlines the product strategy for error tracking, breaking it down into epics for instrumentation, correlation, resolution, and triaging. Specifically, it highlights the development of a 'Create an Issue' button from Sentry error details, which was implemented in three frontend iterations and supported by backend work to associate issues with errors and prevent duplicates. The post emphasizes the benefits of iterative development, including faster shipping of value and increased merge request rates.
  34. My week shadowing a GitLab Site Reliability EngineerThis post details a frontend engineer's shadowing experience with an SRE, observing incident response. It highlights the use of various tools (PagerDuty, Slack, Grafana, Kibana, Zoom) for incident management and proposes features for GitLab like multi-user editing on issues and more webhooks for issues to improve incident response and post-mortem workflows. It also touches on the crypto miner incident and performance degradation incident.
  35. We're moving our observability suite to CoreThis post announces the migration of observability features (custom metrics, logging, tracing, alerting) from a proprietary codebase to the open-source Core product. It outlines the rationale behind this move, emphasizing stewardship, the importance of observability for all developers, and the desire for community input. Specific areas for contribution include improving custom metrics integration with Prometheus, enhancing log tailing capabilities, and deepening the integration of Jaeger for tracing and alerting.
  36. How we plan to build more observability tools on GitLab monitoringThis post outlines GitLab's strategy and roadmap for expanding its observability tools. It details the vision for monitoring within the DevOps lifecycle, emphasizing ease of instrumentation, proactive issue resolution, and seamless insight gathering. Key areas of development include enhancing metrics collection (building on Prometheus integration), simplifying Kubernetes cluster monitoring (including memory/CPU usage and cost optimization), and developing a new incident management system based on GitLab issues. The post also highlights the 'dogfooding' approach, where internal teams use these developing tools, and the strategy to expose these capabilities to administrators of self-managed GitLab instances. The core challenge identified is the evolving industry consensus on what constitutes a robust observability suite in a cloud-native world.
  37. Update: The challenge of enabling Elasticsearch on GitLab.comThis post details the ongoing efforts to enable and improve Elasticsearch integration for GitLab.com's advanced search functionality. Key contributions include the implementation of selective indexing (per-project and per-group controls) to manage index size, numerous bug fixes identified through dogfooding and production readiness reviews, and improvements to the administration experience for managing the indexing process. It also outlines future challenges and plans, such as managing the Elasticsearch schema and reducing index size.
  38. GitLab 11.5 adds Elasticsearch 6, removes ES 5.5 supportGitLab 11.5 introduces support for Elasticsearch 6.x and removes support for Elasticsearch 5.5 and earlier. This upgrade is necessary due to breaking changes in Elasticsearch 6, specifically the removal of mapping types and the introduction of the 'join' datatype for parent-child relationships. The post details the technical implications of these changes on GitLab's indexing strategy, including the implementation of a custom 'type' field and the use of the 'join' datatype with routing by parent ID. Elasticsearch 5.6 is supported as a transitional version due to its early support for the 'join' datatype. This upgrade requires a reindex of Elasticsearch data.
  39. Why Prometheus is for everyoneThis post details how GitLab Support and Production teams leveraged Prometheus for debugging critical infrastructure issues, specifically identifying a bottleneck in Amazon's EFS file system impacting Gitaly operations. It highlights the value of Prometheus for real-time metrics, alerting, and faster issue resolution, advocating for its adoption across customer deployments. The post also provides a template for a GitLab.com production-scale dashboard that customers can import.
  40. GitLab monitoring: Setting up Logz.io and ELK stackThis post details the technical steps for integrating GitLab logs with the ELK Stack or Logz.io using Filebeat. It outlines the types of GitLab logs available (production_json.log, production.log, api_json.log, application.log), provides Filebeat configuration examples for shipping these logs to Elasticsearch or Logz.io, and demonstrates basic log analysis and visualization techniques in Kibana, such as querying for failed logins and error responses, and visualizing commits and issues.
  41. How our production team runs the weekly on-call handoverThis post details the automation of the on-call handover process using a custom-built 'on-call robot assistant'. This program pulls data from PagerDuty (incidents), GitLab issues (production backlog), and GitLab Prometheus (key performance metrics) to generate a GitLab issue report. The report includes a review of past incidents, issues with the 'on-call' label, and seven-day graphs of key metrics. The automation simplifies the handover process for a distributed team and makes the reports publicly available.
  42. Prometheus and GitLabThis post outlines the initial vision for integrating Prometheus into GitLab for monitoring both GitLab installations and end-user applications. It details an MVP to bundle Prometheus with GitLab CE for instance monitoring, and a future direction for web application monitoring (throughput, latency, errors, CPU, memory, I/O) initially focused on Kubernetes. It also introduces concepts for integrating monitoring alerts directly into merge requests and a mockup for a monitoring dashboard, as well as 'deploy monitoring' to track the states of deployments across multiple containers.
  43. GitLab Mattermost, an open source on-premises Slack alternativeThis post introduces the integration of GitLab Mattermost, an open-source, on-premises messaging app, into the Omnibus packages. This integration simplifies the setup for users requiring on-premises solutions and enhances internal communication capabilities by providing a Slack alternative. The Mattermost integration includes automatic OAuth configuration with the GitLab OAuth provider and is disabled by default to conserve resources. Future work will focus on easier posting from GitLab and GitLab CI to Mattermost, external security audits, and reaching a beta state for Mattermost.
  44. How you can send your logs ballistically using UDPIntroduced UDP log shipping as an alternative to TCP for sending logs to a separate server. This is achieved by configuring `logging['udp_log_shipping_host']` and `logging['udp_log_shipping_port']` in `/etc/gitlab/gitlab.rb` and running `gitlab-ctl reconfigure`. The implementation uses `svlogd` and `remote_syslog` to handle STDOUT and `.log` files respectively, enabling fire-and-forget log transfer.