Science & the Public | Science News

Be a Champion for Science

Get your subscription to

Science News when you join.

Science & the Public

Where science and society meet

Science News

Science & the Public

Science & the Public

Online reviews can make over-the-counter drugs look way too effective

‘Evidence-based hearsay’ and personal anecdotes overshadow clinical trial results

person in drug store

TOO-GREAT EXPECTATIONS  Online reviews by users of over-the-counter cholesterol reduction and weight loss products represent select samples that vastly overstate outcomes for most people. The popularity of online reviews of many medical treatments underscores a growing need for well-conducted clinical trials.

Sponsor Message

Here’s one good reason why people often take medications and use health products that don’t live up to expectations or just don’t work — digital word of mouth.

The reviews can be glowing. Take this scuttlebutt about a cholesterol treatment: “I have been using this product for 2 years. Within the first 3 – 4 months my cholesterol was down 30 points. Just got cholesterol tested last week: down from 245 to 196.”  

That’s incredible. It might even be true. But the big picture is alarming: Patients’ online reviews of three over-the-counter drugs — two for lowering cholesterol and one for losing weight — greatly exaggerate how well these substances actually work for most people, says psychologist Mícheál de Barra of the University of Aberdeen in Scotland. Online reviewers portray these medications as working three to six times better than they do in clinical trials that randomly assign drugs or placebos to broad samples of volunteers, he says.

“To learn what works in medicine, you need systematically collected data, usually randomized clinical trials,” de Barra says. “It’s very risky to rely on observations and word of mouth, whether it’s electronic or in person.”

Online reviewers probably aren’t lying, at least not in most cases. Neither are online review readers unrealistically optimistic about health treatments. The problem is that people who benefit most from a medication are especially apt to post their experiences online. Review readers get bamboozled by a wave of positive recommendations that don’t accurately represent how a drug works, or doesn’t work, for people in general.

The duping doesn’t end there. Direct word of mouth about medical treatments, as well as glowing testimonials about health products in advertisements, also traffic in positively skewed information.

In some cases, science does, too. Researchers have recently raised alarms about the “file-drawer problem,” in which studies that find no statistical effects are not published (SN: 5/19/12, p. 26). Published studies reporting positive effects then look unjustifiably bulletproof.

Clinical trials can tilt results in other ways as well. Authors of published trials often fail to preregister details of what they plan to study and how they’ll measure it, enabling data manipulation, selective reporting of results and self-serving interpretations of findings. Clinical trials report patients’ “average” responses to, say, an anticholesterol drug, but do a poor job of predicting which individuals will benefit from that treatment. “Published evidence from randomized trials is already an amalgam of evidence-based medicine and hearsay,” notes Stanford University epidemiologist John Ioannidis.

Online reviews are just old-fashioned word of mouth with a global bullhorn. But they offer one way to quantify the positive bias inherent in so much communication about medical treatments. Patients’ medication reviews that include numerical information, such as cholesterol levels before and after taking an anticholesterol drug, represent what Ioannidis calls “evidence-based hearsay.” Even if patients’ reviews rate treatments with a star system rather than including quantified measures of some kind, those opinions become powerful hearsay evidence for other patients, Ioannidis suspects.

De Barra evaluated 908 online reviews of two cholesterol reduction products, Benecol caramel smart chews and CholestOff, written on or before March 18, 2015. He also assessed 767 reviews of a weight-loss drug, Alli (orlistat), written on or before February 28, 2015.

Benecol reviewers reported a not-too-shabby average cholesterol decline of 45 milligrams per deciliter, versus a small average decline of 14 milligrams per deciliter in nine clinical trials. Respective numbers for CholestOff were 31 milligrams per deciliter and 13 milligrams per deciliter. Alli reviewers reported an average weight loss of about 10 kilograms after taking the drug for three months, versus an average of about two or three kilograms in two clinical trials. That disparity widened slightly after taking Alli for seven months.

Each drug garnered mainly positive reviews, some reporting effects far larger than the average, peppered with a few pans, De Barra reports in the March Social Science & Medicine.

What’s particularly concerning is that these three medications displayed small or negligible effects in clinical trials that probably already inflated how well the drugs work, Ioannidis says. So people searching for help in lowering cholesterol or shedding pounds may instead be getting a massive dose of disappointment from online reviews.

Online reviews can also exaggerate a treatment’s worst effects, whether real or alleged. Evidence-based hearsay may be particularly catchy online and in person when it’s frightening, de Barra suspects.

The anti-vaccination movement provides a potent example, due in part to the success of vaccines. When people face no immediate threat from measles or other infectious diseases, anti-vaccine stories may go viral, so to speak, thanks to misapplied but deeply felt intuitions. Two cognitive scientists, Helena Miton of Central European University in Budapest and Hugo Mercier of the Cognitive Sciences Institute in Bron, France, proposed this scenario in the November 2015 Trends in Cognitive Sciences.

However medical treatments acquire halos or horns, only well-conducted, transparent clinical trials can identify effective medical treatments, such as childhood vaccinations, and their potential side effects, Ioannidis says. But an ominous research trend is gathering steam, he warns. Researchers, professional societies, funders and lobbyists increasingly push to replace clinical trials of randomly selected volunteers with studies of interventions for select samples of people who eat certain foods, take certain drugs or behave in certain ways. The latter investigations may inflate a treatment’s effects even more than online patient reviews do, Ioannidis asserts.

Perhaps a new wave of studies showing how evidence-based hearsay misrepresents many medications will be just what the doctor ordered to resuscitate evidence-based medicine, he muses.

Science & Society

Data-driven crime prediction fails to erase human bias

By Rachel Ehrenberg 10:00am, March 8, 2017
Software programs that predict where crimes will occur don’t eliminate bias; they exacerbate it.
Human Development,, Science & Society,, Health

Transgender children are at greater risk of mental health problems

By Laura Sanders 7:00am, March 1, 2017
The Trump administration has rescinded federal protections for transgender kids in public schools, a move that the American Academy of Pediatrics condemns.
Astronomy,, Technology

Citizen scientists are providing stunning new views of Jupiter

By Ashley Yeager 6:00am, February 17, 2017
A camera aboard NASA’s Juno spacecraft is giving citizen scientists a crack at discovering never-before-seen features of Jupiter.
Science & Society

Trump administration clampdowns on research agencies worry scientists

By Rachel Ehrenberg 4:00pm, January 26, 2017
Mixture of bans on federal research communications create confusion and fear.
Psychology,, Science & Society

You’ve probably been tricked by fake news and don’t know it

By Erika Engelhaupt 6:00am, December 4, 2016
In the fight against falsified facts, the human brain is both the weakest link and our only hope.
Technology,, Science & Society

Obama worried about research funding

By Janet Raloff 11:25am, April 30, 2013
Barack Obama offered yet another argument about why the current federal-budget stalemate is so risky: “[T]he sequester, as it’s known in Washington-speak — it’s hitting our scientific research.” As things now stand, “we could lose a year, two years of scientific research as a practical matter, because of misguided priorities here in this town.”
Technology,, Humans & Society,, Life & Evolution,, Genes & Cells,, Earth & Environment,, Ecology

Antarctic test of novel ice drill poised to begin

By Janet Raloff 12:37am, December 15, 2012
Any day now, a team of 40 scientists and support personnel expects to begin using a warm, high pressure jet of water to bore a 30 centimeter hole through 83 meters of ice. Once it breaks through to the sea below, they’ll have a few days to quickly sample life from water before the hole begins freezing up again. It's just a test. But if all goes well, in a few weeks the team will move 700 miles and bore an even deeper hole to sample for freshwater life that may have been living for eons outside even indirect contact with Earth’s atmosphere.
Humans & Society

This snowbird is really going SOUTH

By Janet Raloff 9:30am, December 6, 2012
Many people of a certain age (like my folks) enjoy flying south to warmer climes when winter weather threatens. I’m also flying south this December — but not to warm up. As a guest of the National Science Foundation, I’ll be checking out summer in the really deep South: Antarctica. Temps expected at certain sites I’m scheduled to visit, such as the South Pole, threaten to surpass the worst that my hometown will encounter in the dead of winter.
Animals,, Humans & Society,, Earth & Environment

Epidemic of skin lesions reported in reef fish

By Janet Raloff 5:58pm, August 1, 2012
A British-Australian research team has just found coral trout living on the south side of the Great Barrier Reef sporting dark skin raised, scablike, brown-black growths. Although the authors believe they’ve stumbled onto an epidemic of melanoma — a type of skin cancer — other experts have their doubts. Strong ones.
Subscribe to RSS - Science & the Public