TechPrufer
Get started
Atlassian
Atlassian logoAtlassian · Feature trail

Mutation Testing for Test Effectiveness

1 post · Dec 2025 – Mar 2026

What this trail covers

Atlassian is scaling its testing practices by integrating mutation testing, particularly with tools like Pitest, to improve the effectiveness of test suites beyond traditional coverage metrics. This approach aims to identify tests that do not adequately protect against code changes. The Rovo Dev CLI is being leveraged to automate the process of writing new tests to catch these mutants, thereby enhancing code quality and developer velocity. This initiative is being rolled out across backend codebases, with a focus on improving testing for new and changed code by setting mutation coverage threshold gates in pull requests. The Rovo Dev CLI, combined with MCP tools, can parse Pitest reports, identify areas with low mutation coverage, and automatically generate tests to catch surviving mutants, aiming to reach an 80% mutation coverage threshold before raising a pull request. Future plans include expanding this to frontend mutation testing.

All Atlassian trails
  1. Rovo Dev CLI Ralph Wiggums Loop for Large Scale Test RefactoringThis post details the application of the 'Ralph Wiggums' AI loop using the Rovo Dev CLI for large-scale test refactoring. The approach focuses on iteratively processing individual test files, validating success with green tests, and deferring complex cases. Key improvements include identifying target files, refining specs with linked context, and managing lessons learned by summarizing them to prevent context blow-up. The bottleneck of local test execution speed is noted, suggesting batch processing for efficiency. The outcome includes 160 files refactored, with tests running significantly faster.
  2. Automating Mutation Coverage with AI: Our Journey and Key LearningsThis post details the development and adoption of an AI-powered Mutation Coverage AI Assistant, launched via the Rovo Dev CLI. The assistant automates the process of improving mutation coverage by analyzing mutation reports, recommending classes to target, writing tests aligned with project conventions, and validating coverage improvements. Key learnings include the need for package-level analysis for large projects, tightening workflow instructions to prevent LLM errors, providing detailed mutant-specific test-writing guidelines, recognizing that AI cannot fix design issues like bloated classes, and establishing a collaborative process for evolving prompts as code artifacts.
  3. Rovo Dev CLI and Mutation Testing to Write Better TestsThis post details the application of mutation testing using Pitest in Jira's backend codebases to improve test suite effectiveness. It explains the concept of mutation testing, contrasting it with traditional line coverage, and demonstrates how Pitest identifies 'surviving' mutants. The post then introduces the Rovo Dev CLI as a tool to automate the writing of new tests specifically to catch these mutants. It outlines a process involving running Pitest, analyzing mutation coverage data with MCP tools, identifying code with low coverage, and using Rovo Dev CLI to generate and test new tests until an 80% mutation coverage threshold is met. A sample GitHub repository is provided for users to replicate the process. The post also mentions plans to extend mutation testing to frontend codebases.