AI researchers have for some time been relying on a fairly simple concept to improve their systems: the more data culled from the internet that they pumped into large language models — the technology behind chatbots — the better those systems performed.
But Hassabis, who oversees Google DeepMind, the company’s primary AI lab, now says that method is running out of steam simply because tech companies are running out of data.
“Everyone in the industry is seeing diminishing returns,” Hassabis said this month in an interview with The New York Times as he prepared to accept a Nobel Prize for his work on AI.
Hassabis is not the only AI expert warning of a slowdown. Interviews with 20 executives and researchers showed a widespread belief that the tech industry is running into a problem many would have thought was unthinkable just a few years ago: They have used up most of the digital text available on the internet.
That problem is starting to surface even as billions of dollars continue to be poured into AI development. On Tuesday, Databricks, an AI data company, said it was closing in on $10 billion in funding — the largest-ever private funding round for a startup. And the biggest companies in tech are signaling that they have no plans to slow down their spending on the giant data centers that run AI systems.
Discover the stories of your interest
Not everyone in the AI world is concerned. Some, including OpenAI CEO Sam Altman, say progress will continue at the same pace, albeit with some twists on old techniques. Dario Amodei, CEO of AI startup Anthropic, and Jensen Huang, CEO of Nvidia, are also bullish. (The Times has sued OpenAI, claiming copyright infringement of news content related to AI systems. OpenAI has denied the claims.)
The roots of the debate trace to 2020 when Jared Kaplan, a theoretical physicist at Johns Hopkins University, published a research paper showing that large language models steadily grew more powerful and lifelike as they analyzed more data.
Researchers called Kaplan’s findings “the Scaling Laws.” Just as students learn more by reading more books, AI systems improved as they ingested increasingly large amounts of digital text culled from the internet, including news articles, chat logs and computer programs. Seeing the raw power of this phenomenon, companies such as OpenAI, Google and Meta raced to get their hands on as much internet data as possible, cutting corners, ignoring corporate policies and even debating whether they should skirt the law, according to an examination this year by the Times.
It was the modern equivalent of Moore’s Law, the oft-quoted maxim coined in the 1960s by Intel co-founder Gordon Moore. He showed that the number of transistors on a silicon chip doubled every two years or so, steadily increasing the power of the world’s computers. Moore’s Law held up for 40 years. But eventually, it started to slow.
The problem is: Neither the Scaling Laws nor Moore’s Law are immutable laws of nature. They’re simply smart observations. One held up for decades. The others may have a much shorter shelf life. Google and Kaplan’s new employer, Anthropic, cannot just throw more text at their AI systems because there is little text left to throw.
“There were extraordinary returns over the last three or four years as the Scaling Laws were getting going,” Hassabis said. “But we are no longer getting the same progress.”
Hassabis said existing techniques would continue to improve AI in some ways. But he said he believed that entirely new ideas were needed to reach the goal that Google and many others were chasing: a machine that could match the power of the human brain.
Ilya Sutskever, who was instrumental in pushing the industry to think big as a researcher at both Google and OpenAI before leaving OpenAI to create a new startup this past spring, made the same point during a speech last week. “We’ve achieved peak data, and there’ll be no more,” he said. “We have to deal with the data that we have. There’s only one internet.”
Hassabis and others are exploring a different approach. They are developing ways for large language models to learn from their own trial and error. By working through various math problems, for instance, language models can learn which methods lead to the right answer and which do not. In essence, the models train on data that they themselves generate. Researchers call this “synthetic data.”
OpenAI recently released a new system called OpenAI o1 that was built this way. But the method only works in areas such as math and computing programming, where there is a firm distinction between right and wrong.
Even in these areas, AI systems have a way of making mistakes and making things up. That can hamper efforts to build AI “agents” that can write their own computer programs and take actions on behalf of internet users, which experts see as one of AI’s most important skills.
Sorting through the wider expanses of human knowledge is even more difficult.
“These methods only work in areas where things are empirically true, like math and science,” said Dylan Patel, chief analyst for research firm SemiAnalysis, who closely follows the rise of AI technologies. “The humanities and the arts, moral and philosophical problems are much more difficult.”
People such as Altman say these new techniques will continue to push the technology ahead. But if progress reaches a plateau, the implications could be far-reaching, even for Nvidia, which has become one of the most valuable companies in the world thanks to the AI boom.
During a call with analysts last month, Huang was asked how the company was helping customers work through a potential slowdown and what the repercussions might be for its business. He said that evidence showed there were still gains being made, but that businesses were also testing new processes and techniques on AI chips.
“As a result of that, the demand for our infrastructure is really great,” Huang said.
Although he is confident about Nvidia’s prospects, some of the company’s biggest customers acknowledge that they must prepare for the possibility that AI will not advance as quickly as expected.
“We have had to grapple with this. Is this thing real or not?” said Rachel Peterson, vice president of data centers at Meta. “It is a great question because of all the dollars that are being thrown into this across the board.”
