TechPrufer
Get started
Microsoft
Microsoft logoMicrosoft · Feature trail

Publish Code Coverage Results Task

20 posts · Jun 2012 – Feb 2024

What this trail covers

Azure Pipelines continues to evolve its CI/CD capabilities, with this post detailing the integration of Codecov for code coverage tracking. This builds upon existing functionality for publishing test results and enables tokenless uploads for public Azure Pipelines projects, simplifying secret management and providing enhanced reporting for code coverage over time. The integration allows for ongoing coverage reports without manual secret management.

All Microsoft trails
  1. JUnit Attachments Support for Publish Test ResultsIntroduced support for file attachments within the Publish Test Results task in Azure Pipelines when using JUnit reports. This is achieved by parsing a convention within the JUnit XML format (`[[ATTACHMENT|/path/to/file.ext]]`) to identify and upload associated files. The post provides an example of configuring Playwright to generate JUnit reports with screenshots and trace files on failure, and how to use the PublishTestResults@2 task to publish these results, making attachments available in the test results view.
  2. New version of Publish Code Coverage Results taskIntroduced a V2 version of the Publish Code Coverage Results task, expanding support for .coverage and .covx formats in addition to Cobertura and JaCoCo. This version is compatible with build quality check task version 2.231.0 and later. The V1 task will be phased out, with warnings to be added in late February 2024 and eventual removal planned for Azure DevOps Server. No automatic conversion or migration scripts will be provided.
  3. Improved Code Coverage Tab ExperienceThis post details improvements to the code coverage tab experience in Azure DevOps. Previously, the tab was only visible after build completion and lacked informative messages for failures or unconfigured states. The new experience shows the Code Coverage tab by default, displaying messages for "Pipeline is queued", "Code coverage in progress", and failure scenarios, providing root cause analysis. It also shows "Code Coverage not configured" when applicable. Additionally, pull requests will now show comments indicating whether code coverage policies have been met.
  4. Improvements to code coverage experience under test automationThis post announces the ability to view '.coverage' format code coverage results directly in the Azure DevOps web portal, bringing parity with other code coverage formats and improving the developer experience. It also introduces improvements to the pull request experience, allowing comments on rejected PRs due to unmet code coverage policies, which enhances collaboration and clarity during code reviews.
  5. Publish Code Coverage Report in Azure DevOps Services pipeline execution summaryThis post details the steps and YAML configuration required to publish code coverage reports in Azure DevOps Services pipelines. It introduces the use of the 'coverlet.msbuild' NuGet package to generate Cobertura-formatted coverage files from .NET Core projects. It also outlines the necessary Azure Pipelines tasks: 'UseDotNet@2' for SDK setup, 'DotNetCoreCLI@2' for building and testing with coverage collection enabled, 'reportgenerator@4' (a third-party task) for processing the coverage results, and 'PublishCodeCoverageResults@1' to display the results in the pipeline summary. The post also highlights limitations, such as the lack of native UI support for code coverage in build logs for .NET Core projects and the inapplicability to non-SDK-style .NET Framework projects.
  7. Uploading to Codecov just got easierThis post introduces the integration of Codecov with Azure Pipelines for code coverage tracking. It details how to replace existing CI configurations (like .travis.yml) with an Azure Pipelines YAML definition to run tests using `coverage.py` and then upload the results to Codecov using their bash uploader. A key contribution is the enablement of tokenless uploads for public Azure Pipelines projects, simplifying secret management for users.
  8. MSTest V2: in-assembly parallel test executionIntroduces in-assembly parallel test execution for MSTest V2, allowing tests within a single assembly (DLL) to run in parallel. This feature can be enabled globally via .runsettings or through assembly-level annotations. It supports class-level and method-level parallelization, with an option for custom plugins. Opt-out mechanisms using `[DoNotParallelize()]` are provided. The feature composes with existing process-parallelization and distributed test execution in Azure Pipelines.
  9. VSTS is now a Symbol ServerIntroduces the VSTS Symbol Server as a public preview, allowing users to publish symbols directly from the 'Index Sources and Publish Symbols' task. It also details how to consume these symbols from Visual Studio and WinDbg.
  10. Visual Studio Test Platform - upcoming changes to data collectorsThis post announces upcoming changes to the Visual Studio Test Platform, specifically focusing on data collectors. It highlights the active development of `vstest.console.exe` from the `Microsoft/vstest` repository, which is cross-platform and open-source, and will eventually replace the older version. This new version introduces enhancements to the data collection infrastructure, necessitating the migration of existing data collectors. A migration guide is provided for developers to follow.
  11. Accelerated Continuous Testing with Test Impact Analysis – Part 4This post details how to extend Azure Pipelines Test Impact Analysis (TIA) to support new scenarios beyond out-of-the-box managed code. It explains how to provide an explicit dependencies-map in an XML file to map test methods to source files, enabling TIA to run tests for other languages (JavaScript, C++), and scenarios where tests and product code run on different machines. An example using MSTest V2 repository is provided, demonstrating how to specify dependencies at the folder level and use test case filters.
  12. Accelerated Continuous Testing with Test Impact Analysis - Part 3This post details the core mechanism of Test Impact Analysis (TIA) by explaining how it collects dynamic dependencies at the file level to determine which tests are affected by code changes. It elaborates on TIA's fallback logic when unknown file types are encountered and provides detailed configuration options for controlling TIA's behavior, including the use of `DisableTestImpactAnalysis`, `TIA_IncludePathFilters` for path-specific triggers, and `TIA_IncludePathFilters` with negation for ignoring specific file types. It also offers methods for validating TIA's effectiveness and encourages its adoption in PR workflows.
  13. Accelerated Continuous Testing with Test Impact Analysis - Part 2This post provides empirical data (graphs) demonstrating the reduction in CI definition and test run times achieved by enabling Test Impact Analysis (TIA). It reiterates the benefits of TIA for accelerating CI/CD pipelines, highlights its support for CI and PR workflows, and details its features: robust test selection (including previously failing and newly added tests), safe fallback to running all tests when reasoning is not possible, and configurable overrides. It also lists caveats for Visual Studio 2015: serial execution for parallel tests and no code coverage collection when TIA is enabled.
  14. Accelerated Continuous Testing with Test Impact Analysis - Part 1This post introduces Test Impact Analysis (TIA) as a feature within Azure Pipelines to accelerate continuous testing. It explains how TIA incrementally validates code commits by automatically selecting and running only relevant tests, thereby reducing test execution time and improving feedback loops. The post details the enabling of TIA via the Visual Studio Test task, including specific configurations for IIS interactions using a .runsettings file. It also outlines the components of robust test selection (existing impacted tests, previously failing tests, newly added tests) and the current scope of TIA support, including limitations like multi-box topology and .NET Core.
  15. Traceability with Continuous TestingIntroduces the ability to link automated tests directly to Requirements and track Requirement quality on Dashboards using a new 'Requirements Quality' widget. This feature works for tests written in any language and using any framework. A new icon on the Test Results page allows associating automated tests with Requirements, and a 'Group By' option on Requirement shows the quality of associated tests and unassociated tests.
  16. Browse Code Coverage reportsEnabled a new Code Coverage tab on the Build summary page in Visual Studio Team Services (VSTS) to allow users to browse HTML reports generated from Jacoco or Cobertura code coverage data. Previously, users had to download the report to view it with full fidelity. Security considerations led to styles being stripped when viewing reports directly in the tab, requiring download for full fidelity.
  17. Remote testing - Distributing tests based on number of machinesIntroduced a new checkbox in the 'Run Functional Tests' task (formerly 'Run Tests using Test Agent') that allows tests to be distributed to remote machines based on the number of machines, rather than solely at the assembly level. This addresses a customer request for improved remote testing distribution.
  18. Test Results in BuildThis post introduces the integration of test results directly into the Azure DevOps build summary page. It provides an aggregated status of total tests, failed tests, pass percentage, and test duration. It breaks down test failures into 'New' and 'Existing' categories to identify regressions. For failed tests, it offers detailed information including error messages, stack traces, and recent outcome trends. It also enables the creation of bugs directly from failed tests, pre-populating them with relevant information. The post mentions upcoming support for XAML builds and the ability to view all test results, including passed tests, with filtering options.
  19. Code Coverage in Microsoft Test Manager – Deep DiveThis post details the introduction of code coverage collection for manual testing of ASP.NET web applications running under IIS in Microsoft Test Manager (MTM) with Visual Studio 2012 Update 1. It explains the simplified configuration process, including the need for a StandardLab Environment and Test Agent installation, and how to specify assemblies for coverage. The post also addresses multi-user testing scenarios on shared IIS servers, explaining that only one code coverage session is active on the web server and how users receive coverage attachments at the end of their runs. A key new feature is the build-level coverage file, which consolidates coverage information from all published test runs against a build, improving decision-making for build quality. The post also notes changes to the warehouse functionality, where function-level coverage data is no longer written to the database, and provides C# code examples for downloading and parsing the build coverage file using the CoverageInfo class. Caveats regarding administrator privileges for other profilers, ignored settings for second users in multi-user scenarios, and avoiding TFS/Test Agent co-installation on the same machine are also highlighted.
  20. Publishing test results through command line test runnerIntroduced TfsPublisher, a logger extension for vstest.console.exe, to publish test results from the command line to TFS builds. Details usage, command-line options (Collection, BuildName, TeamProject, Platform, Flavor, RunTitle), and provides examples for publishing with and without code coverage data. Notes compatibility with Visual Studio 2012 Update 1 and Team Foundation Server 2012.
  21. Code Coverage in Microsoft Test Manager in Visual Studio Update 1This post details the functionality of Code Coverage in Microsoft Test Manager (MTM) as part of Visual Studio 2012 Update 1. It explains how QA professionals can track code coverage by executing Test Cases and Exploratory testing within MTM. The post covers viewing coverage results in build summary logs and within Visual Studio, enabling the feature by creating Test Settings with the Code Coverage data adapter, and configuring settings like symbol file paths and excluded DLLs. It also outlines how to run tests with specific builds and test settings to capture coverage data.
  22. Visual Studio 2012 RC – What’s new in Code CoverageThis post details the "visible" changes to the code coverage tool in Visual Studio 2012 RC. Key improvements include: no .testsettings overheads (dynamic, runtime instrumentation), support in the Test Explorer window for running tests and getting coverage data simultaneously, first-class support for native code (including mixed solutions), a leaner and self-contained .coverage file, support in the new vstest.console.exe test executor with a "/enablecodecoverage" switch, and integration with Team Build via the "Enable Code Coverage" option in build definitions when using the Visual Studio Test Runner. It also notes support for .NET/C++ custom unit test adapters, with a caveat for adapters using runtime magic like .NET profiler interfaces.