On answering programming questions: "We found that 52 percent of ChatGPT answers contain misinformation, 77 percent of the answers are more verbose than human answers, and 78 percent of the answers suffer from different degrees of inconsistency to human answers."

To be fair, I do expect AI answers to get better over time, but it's certainly premature to use it as a trusted toolkit for software development today. One might argue that its answers are more like suggestions for an engineer to check and adapt as appropriate, but will they really be used that way?

I think it's more likely that AI agents will be used to build software by people who want to avoid engaging with a real, human engineer, or people who want to cut corners for one reason or another. So I think the warnings are appropriate: LLMs are bad at coding and we shouldn't trust what they say. #AI

[Link]