Chatbots may make learning feel easy — but it’s superficial

When it comes to learning something new, old-fashioned Googling might be the smarter move compared with asking ChatGPT.

Large language models, or LLMs — the artificial intelligence systems that power chatbots like ChatGPT — are increasingly being used as sources of quick answers. But in a new study, people who used a traditional search engine to look up information developed deeper knowledge than those who relied on an AI chatbot, researchers report in the October PNAS Nexus.

“LLMs are fundamentally changing not just how we acquire information but how we develop knowledge,” says Shiri Melumad, a consumer psychology researcher at the University of Pennsylvania. “The more we learn about their effects — both their benefits and risks — the more effectively people can use them, and the better they can be designed.”

Melumad and Jin Ho Yun, a neuroscientist at the University of Pennsylvania, ran a series of experiments comparing what people learn through LLMs versus traditional web searches. Over 10,000 participants across seven experiments were randomly assigned to research different topics — such as how to grow a vegetable garden or how to lead a healthier lifestyle — using either Google or ChatGPT, then write advice for a friend based on what they’d learned. The researchers evaluated how much participants learned from the task and how invested they were in their advice.

Even controlling for the information available — for instance, by using identical sets of facts in simulated interfaces — the pattern held: Knowledge gained from chatbot summaries was shallower compared with knowledge gained from web links. Indicators for “shallow” versus “deep” knowledge were based on participant self-reporting, natural language processing tools and evaluations by independent human judges.

The analysis also found that those who learned via LLMs were less invested in the advice they gave, produced less informative content and were less likely to adopt the advice for themselves compared with those who used web searches. “The same results arose even when participants used a version of ChatGPT that provided optional web links to original sources,” Melumad says. Only about a quarter of the roughly 800 participants in that “ChatGPT with links” experiment were even motivated to click on at least one link.

“While LLMs can reduce the load of having to synthesize information for oneself, this ease comes at the cost of developing deeper knowledge on a topic,” she says. She also adds that more could be done to design search tools that actively encourage users to dig deeper. 

Psychologist Daniel Oppenheimer of Carnegie Mellon University in Pittsburgh says that while this is a good project, he would frame it differently. He thinks it’s more accurate to say that “LLMs reduce motivation for people to do their own thinking,” rather than claiming that people who synthesize information for themselves gain a deeper understanding than those who receive a synthesis from another entity, such as an LLM.

However, he adds that he would hate for people to abandon a useful tool because they think it will universally lead to shallower learning. “Like all learning,” he says, “the effectiveness of the tool depends on how you use it. What this finding is showing is that people don’t naturally use it as well as they might.”