Maryland: Doctors have many concerns about online crowdsourced ratings which
are intended to make patients better-informed consumers of health care,
but this is a big one: They worry that complainers will be the most
outspoken contributors to rating sites, skewing scores and resulting in a
heckler's veto. A new study from the Robert H. Smith School of Business
at the University of Maryland finds that fear is unwarranted.
Researchers compared the ratings of 1,425 doctors in three metropolitan
areas — Denver, Kansas City and Memphis — on the popular site RateMDs.com against thorough surveys of patient satisfaction conducted by Checkbook.org, a nonprofit consumer research organization. The surveys were designed by the Agency for Healthcare Research and Quality (AHRQ) within the U.S. Department of Health and Human Services.
The study confirmed that there was a correlation between the online
ratings and the more thorough examinations of patient satisfaction. This
suggests that the ratings were representative of a broad spectrum of
the patient population. More surprisingly, physicians who did poorly in
the government evaluations tended to receive fewer online ratings than
those who did well - the opposite of what one would expect if patients
with bad experiences dominated the ratings.
"The concern that ratings aggregation sites will become digital
soapboxes for disgruntled patients appears to be unfounded," wrote Gordon Gao and Ritu Agarwal
of the Smith School, Brad N. Greenwood of Temple University (and a
Smith PhD), and Jeffrey McCullough of the University of Minnesota in the
study. Agarwal and Gao co-direct the Smith School's Center for Health Information and Decision Systems (CHIDS).
In other areas of the economy, unhappy customers tend to be the most
vocal. Why might that not be true in health care? The authors offer
several possible explanations. First, it's conceivable that the patients
of the worst doctors might have less access to the Internet or be less
familiar with online reviews. Second, patients might be worried that if
they leave reviews, health-care providers might retaliate against them
in some way, even if the reviews are anonymous. Finally, customers might
just evaluate health care in a different way than they evaluate
products on Amazon.
The effectiveness of online ratings is a subject of intense interest
that is only increasing: 37 percent of patients have consulted a ratings
website when they sought healthcare. According to the new study, online
star ratings tended to be most helpful for distinguishing doctors in
the middle 50 percent of performance (as measured by the government
surveys). A "hyperbole effect" was evident for doctors in the
highest-performing and lowest-performing quartiles: Their rankings
tended to group together, meaning that small differences in star ratings
had no significance.
One caveat is that the study was limited to an evaluation of patient
satisfaction, as opposed to objective measures of patient outcomes or
protocols doctors followed. A study published in the February 2015 issue
of JAMA Internal Medicine
by Gao and four co-authors found little statistically significant
connections between patient ratings on eight websites and objective
measures involving 1,299 internists.
“This is what we should keep in mind: A very high score in patient
satisfaction is not wholly connected with clinical quality,” Gao says.
“If you want to use the online ratings to infer how good a doctor is
clinically, take them with a grain of salt.”
The Centers for Medicare & Medicaid Services
are working on an online resource that would allow consumers to compare
data on health-care outcomes of different physicians, called the
Physician Compare Initiative, but it remains controversial because
doctors doubt it will be possible to correct for things such as the
general health of a physicians' patients and whether patients adhere to
doctors' recommendations.
The study, “Vocal Minority and Silent Majority: How Do Online Ratings
Reflect Population Perceptions of Quality," is forthcoming in MIS Quarterly: http://go.umd.edu/ZTL.