What’s coming next for LLMs and AI agents?

“Three or four years ago, we were super excited when our [AI] models could solve eighth-grade math problems,” Jeff Dean, chief scientist, Google DeepMind and Google Research, said during a panel discussion at Nvidia’s GTC developer show last week.

By last year, Google’s Gemini had reached the gold-medal standard at the International Mathematical Olympiad and it has now won a variety of coding contests.

That’s just one example of how quickly AI has upended the world. It’s taken over businesses, warped economies, rapidly changed the job market and careers and, perhaps, could even alter the future of humanity.

Tech industry experts have described AI as more revolutionary than electricity and the internet. (It’s also been called more dangerous than nuclear weapons, because the technology in the wrong hands could wreak havoc.)

After several years of rapid evolution, what’s next for AI? Dean and Nvidia’s chief scientist, Bill Dally, shared some ideas during the panel discussion at GTC. Here’s a rundown on some of what they envision for large language models (LLMs) and agentic AI.

Autonomous models: The arrival earlier this year of OpenClaw provides a glimpse of how AI agents can complete work unsupervised without any human intervention. But the current computing pipeline — including chips, power requirements, communications and cost — are lagging. To empower these kinds of futuristic agents, things need to be faster, Dean said.

Nvidia is working to make agents faster, including the use of technology that allows data to be transferred using optical networking technologies. “We call it the speed of light,” said Dally. 

Free agents evolving on their own: Here’s a scary thought: agents could be creating the next version of themselves, or least creating updated versions that can run the latest large language models (LLMs) and genAI tools.

That’s not happening quite yet, but there are signs it’s coming, Dean said, noting that AI agents can already self-evolve by accepting and dismissing ideas.

There’s history here, too. In 2017, AI researchers came up with the concept of “meta learning,” where AI could search for models best suited for experiments and problem solving. The search parameters at the time were mostly specified in code, but now that can be done with natural language, Dean said.

Natural language interaction makes it easier for agents to find ways to get better, such as finding new information, specific algorithms, or distillation mechanisms. AI can be seen as a performance multiplier that frees researchers to think up new ideas. “It’s a partnership between super-capable researchers and super-capable agents,” Dean said.

More interactive LLMs: As AI technology progresses, LLMs could become more interactive with the real world, actually re-learning and updating themselves in real time and taking actions based on that new knowledge.

Today’s LLMs are basically strapped on a board, streamed through internet data, and then presented to the world, Dean said, with results that are largely predetermined.

But future models will learn on the fly by instantly interleaving physical and digital information. With that information, LLMs will be better able to direct robotic actions and predict answers to questions.

While that is already done in post training, what’s better is interleaving at the pre-training stage. “We sort of have this artificial distinction now…. It seems like that shouldn’t exist for the long term,” Dean said.

Continual-learning models are already emerging without any fixed parameters, with organically growing models advancing, pruning and compressing parameters, Dean said.

The Master agent: Nvidia and Google are already using AI for chip design; the next step is to figure out how to automate the process so chip designers and developers can do other things.

The process might well involve a “master” agent calling on sub-agents that specialize in creating on-chip functions or fixing bugs. Those agents would have to negotiate improvements on chips and iterate if the results aren’t good.

“They’ll have the same kind of meetings we have, but between agents,” Dally said.

Machine speed agents: AI development tools are designed for human speed, but need to perform at machine speed, the panelists said. Because agents reason, decide and act significantly faster than people, human tools like a slow-loading C++ compiler get in the way of quick progress.

“We’re going to need to start to reengineer the tools that these models [use],” Dean said.

That’s already happening for coding tools and document manipulation, which previously extracted information at human speed. Put simply, ”We’ll need new forms of tools,” Dean said.

Security experts highlighted the ability for machine-speed AI agents to tackle cyberattacks. That’s because humans might be too slow at stopping agentic AI-based  attacks.

Better educational tools: The panelists criticized universities that have restricted the use of AI in classrooms. Instead, educators need to lean into AI to accelerate learning, said Dally, who previously was a computer science professor at Stanford University.

Models will soon serve as “amazing” personalized tutors that promote learning concepts efficiently without giving away answers, Dean said. He noted how calculators removed bottlenecks around learning math, helping students quickly move to higher levels.

“Maybe I should quit my day job and go and do it myself,” Dean said.