Show HN: Extreme Mutation Testing to Optimize LLM Agent-Based Mutation Testing
Hey HN, Steven here from CodeIntegrity https://github.com/codeintegrity-ai/mutahunter We’re obsessed with automating software testing, specifically mutation testing, and have been frustrated with its slow adoption despite its proven success. I have a pretty different perspective on mutation testing and have shared my thoughts on its current state - https://www.jungsteven.com/blog/2024-07-03-past-present-future-mutation-testing
Over the past few months, we’ve developed a new mutation testing tool that’s easy to use and compatible with any programming language. We believe LLM-based injections are the future for generating relevant faults and automating the process of analyzing surviving mutants to better fix bugs and improve test suites.
Our approach to mutation testing is as follows: first, use Extreme Mutation Testing to cover all pseudo-tested methods, and then use LLM to inject relevant mutants. LLMs will also specifically guide users on where to improve the test suite or fix the codebase so that developers don’t have to manually go through the surviving mutants. We envision mutation testing to be something like Netflix’s Chaos Monkey, injecting relevant faults and seeing how your system breaks. We are currently developing these features to make mutation testing more seamless and easier during the development process, so that more and more people get used to using mutation testing.
Please give it a try, and we’d appreciate any feedback from you guys. Excited to see how this turns out!
clickable link https://www.jungsteven.com/blog/2024-07-03-past-present-futu...
Every place I have worked required showing that tests worked during code review, break the code, watch the tests fail, unbreak the code, watch tests pass.
If I were going to spend resources on making testing better and my choice was between mutation testing and property testing, I'd rather spend that energy on property testing.
Mutation testing feels like something that the test framework should be doing anyway, maybe not every run, but every time it sees a new test.
If you keep going in this direction, your LLMs will be expert vulnerability injectors, possibly your end goal. All technologies are dual-use.
When I tap and hold on your blog link and tap Search on mobile, the top result ("My Opinion on LLM-based Mutation Testing") has a malformed URL where "mutatino" is used instead of "mutation" and 404s. When corrected, the page loads.