It really turns into a tug-of-war, see my answer below. Despite it being quite good at debugging imo still requires a human in the loop for it not run in circles after a “bug” that is almost never relevant in practice but whose fix complicates the code.
Try asking independent LLMs to find bugs in a code consecutively (fixing the founds bugs in between) and even after the rightfully major bugs are resolved, the independent newer instances will keep suggesting fixes (those which sometimes undoes its previous “fixes”).
That is why you ask for visualizations and test real failure scenarios to see if they are detected properly.
Large language models are not the same as they were 5 years ago. Pumping trillions of dollars to automate software engineering did som what pay off… They are really quite good nowadays despite the amount of hate they still get. Often they can detect sloppy mistakes by human coders too.
Do not trust them blindly but test their results. Or use them only to generate tests on your artisan handwritten code to see if it can detect any mistakes.
That is why you ask the LLM to find bugs and other instances where the first LLM cheated. The checking LLM will try to find abuse with all its might.
It really turns into a tug-of-war, see my answer below. Despite it being quite good at debugging imo still requires a human in the loop for it not run in circles after a “bug” that is almost never relevant in practice but whose fix complicates the code.
Try asking independent LLMs to find bugs in a code consecutively (fixing the founds bugs in between) and even after the rightfully major bugs are resolved, the independent newer instances will keep suggesting fixes (those which sometimes undoes its previous “fixes”).
That is why you ask for visualizations and test real failure scenarios to see if they are detected properly.
Large language models are not the same as they were 5 years ago. Pumping trillions of dollars to automate software engineering did som what pay off… They are really quite good nowadays despite the amount of hate they still get. Often they can detect sloppy mistakes by human coders too.
Do not trust them blindly but test their results. Or use them only to generate tests on your artisan handwritten code to see if it can detect any mistakes.
I agree, opus 4.8 and Kimi k2.7 are my go to models and opus is very very good if given a proper brief and spec cheat. K2.7 is close.