The remarkable and rapid uptake of ChatGPT and similar large language model (LLM)-based AIs may be driving the biggest increase in demand for computing power since the advent of the internet.
The neural networks that power LLM AI solutions, such as ChatGPT, rely on massively parallel processing. This processing is similar in some respects to the massively parallel graphics processing that was demanded in the past by computer games.
Computer games and complex graphics processing applications drove the evolution of specialized processors capable of handling such parallel workloads—the graphics processing unit (GPU). LLM processing has resulted in an explosion of demand for GPUs. As the producer of the H100 Tensor Core GPU, the most popular GPU for AI workloads, NVIDIA has benefited from this spike in demand.
On a recent podcast, Noam Shazeer, one of the pioneers of modern generative AI, mentioned that NVIDIA is expected to manufacture about 1.5 million H100 GPUs in 2024. Each GPU is capable of about 10^15 floating-point operations per second (FLOPS), which means we are adding about 1.5x10^21 FLOPS to the Earth’s computing technology in a single year. That added capacity is sufficient to provide 250 million FLOPS to every human being!
It’s hard to imagine how that additional computing capacity, added to today’s powerful LLM-based AIs, will impact human productivity. There will likely be enough computing power to allow for each of us to have virtually full-time access to AI models that will finally live up to the promise of personal digital assistants.
However, this massive increase in computing power is not entirely free of consequence. It’s expected that in 2024 the power consumption from AI GPUs will exceed that of many small nations.
AI is expected to lead to many significant breakthroughs in medicine, science, and engineering. Some should help us cope with the impact of climate change through improved weather prediction and natural disaster warning systems. AI systems could also optimize energy grids and improve the utilization of existing resources. It’s not impossible that, in the long term, AI systems will allow for the sort of energy breakthroughs that will enable us to reach a net-zero carbon emissions target.
However, all of these breakthroughs are speculative and lie in the future. In the short term, it looks like we may see an exponential growth in the demand for data center power that hinders efforts to reach carbon-neutral solutions.
Blockchain is often criticized for its carbon footprint. The proof-of-work algorithm is computationally expensive, and Bitcoin mining still represents about half a percent of the world’s electricity consumption. Most other blockchains have moved away from proof-of-work to proof-of-stake and, as a result, have seen a drastic reduction in their carbon footprint. Unfortunately, there is no similar quick fix for LLM-based AI workloads. In the long term, hardware advances—particularly quantum computing—may allow these sorts of AI workloads to be performed at a fraction of the current CPU cost. However, in the short term, AI’s carbon footprint is a serious concern. If AI is to be a net benefit to humanity, ways need to be found to support it by renewable energy sources and to take advantage of less computationally intensive models.