It is a very well known truism that it is harder to debug code than it is to write it. This is, in my experience, doubly true of the convoluted code generated by current frontier LLMs.
Allowing LLM coding at our org basically stole most of this year’s progress from us as every PR made this way still has yet to be merged because the code quality just never reaches anywhere near our minimum requirement.
We are investigating ways to improve this (a style guide for agents etc) but the best step we’ve taken so far is just to ask people to stop using it and see what happens. (Code quality jumped up and PRs started getting merged, though the LLM ones are still languishing and probably will need rewriting from scratch before we can merge them)
That doesn’t change the fact that it’s harder to properly review code than to write code, especially when that code is ten times longer than it needs to be, makes no sensible decisions about how to separate concerns, and might, if you’re lucky, pay lip service to naming things well.
I don’t necessarily think this will always be the case with generated code, but I expect we’ll need a proper breakthrough in the technology before it can write good code (it needs a number of things: actual world models, the ability to go back and edit its output when it realises a mistake – which is crucially different from saying a new thing that contradicts the mistake – and continual learning, to name the three that spring to mind).
Even if it started writing perfect code, I’d be dubious of signing your org’s ability to write code away to a service that you don’t own and has a notable history of intellectual property infringement and an insane disregard for both the law and morality. At the very least we should all be using open weights models.
It is a very well known truism that it is harder to debug code than it is to write it. This is, in my experience, doubly true of the convoluted code generated by current frontier LLMs.
Allowing LLM coding at our org basically stole most of this year’s progress from us as every PR made this way still has yet to be merged because the code quality just never reaches anywhere near our minimum requirement.
We are investigating ways to improve this (a style guide for agents etc) but the best step we’ve taken so far is just to ask people to stop using it and see what happens. (Code quality jumped up and PRs started getting merged, though the LLM ones are still languishing and probably will need rewriting from scratch before we can merge them)
Well, my point is that the review should be before the pr, not after.
That doesn’t change the fact that it’s harder to properly review code than to write code, especially when that code is ten times longer than it needs to be, makes no sensible decisions about how to separate concerns, and might, if you’re lucky, pay lip service to naming things well.
I don’t necessarily think this will always be the case with generated code, but I expect we’ll need a proper breakthrough in the technology before it can write good code (it needs a number of things: actual world models, the ability to go back and edit its output when it realises a mistake – which is crucially different from saying a new thing that contradicts the mistake – and continual learning, to name the three that spring to mind).
Even if it started writing perfect code, I’d be dubious of signing your org’s ability to write code away to a service that you don’t own and has a notable history of intellectual property infringement and an insane disregard for both the law and morality. At the very least we should all be using open weights models.