[Things we learned about LLMs in 2024]

Simon's overview of what happened in the LLM space during 2024 is genuinely excellent. For example, on the environmental impact:

"Companies like Google, Meta, Microsoft and Amazon are all spending billions of dollars rolling out new datacenters, with a very material impact on the electricity grid and the environment. There’s even talk of spinning up new nuclear power stations, but those can take decades.

Is this infrastructure necessary? DeepSeek v3’s $6m training cost and the continued crash in LLM prices might hint that it’s not. But would you want to be the big tech executive that argued NOT to build out this infrastructure only to be proven wrong in a few years’ time?"

His comparison to the railway bubbles of the late 1800s and the UK's railway mania is inspired, and a helpful way to think about what's happening. (I will say that similar claims were made about the crypto space: that the resulting infrastructure would be useful even after the crashes. Is it?)

There's also an important lesson about how the prevalence of slop isn't actually making training LLMs harder, despite frequent claims to the contrary.

The whole piece is very much worth your time.

#AI

[Link]