Researchers compared the ratings of 1,425 doctors in three metropolitan areas — Denver, Kansas City and Memphis — on the popular site RateMDs.com against thorough surveys of patient satisfaction conducted by Checkbook.org, a nonprofit consumer research organization. The surveys were designed by the Agency for Healthcare Research and Quality (AHRQ) within the U.S. Department of Health and Human Services.
The study confirmed that there was a correlation between the online ratings and the more thorough examinations of patient satisfaction. This suggests that the ratings were representative of a broad spectrum of the patient population. More surprisingly, physicians who did poorly in the government evaluations tended to receive fewer online ratings than those who did well - the opposite of what one would expect if patients with bad experiences dominated the ratings.
"The concern that ratings aggregation sites will become digital soapboxes for disgruntled patients appears to be unfounded," wrote Gordon Gao and Ritu Agarwal of the Smith School, Brad N. Greenwood of Temple University (and a Smith PhD), and Jeffrey McCullough of the University of Minnesota in the study. Agarwal and Gao co-direct the Smith School's Center for Health Information and Decision Systems (CHIDS).
In other areas of the economy, unhappy customers tend to be the most vocal. Why might that not be true in health care? The authors offer several possible explanations. First, it's conceivable that the patients of the worst doctors might have less access to the Internet or be less familiar with online reviews. Second, patients might be worried that if they leave reviews, health-care providers might retaliate against them in some way, even if the reviews are anonymous. Finally, customers might just evaluate health care in a different way than they evaluate products on Amazon.
The effectiveness of online ratings is a subject of intense interest that is only increasing: 37 percent of patients have consulted a ratings website when they sought healthcare. According to the new study, online star ratings tended to be most helpful for distinguishing doctors in the middle 50 percent of performance (as measured by the government surveys). A "hyperbole effect" was evident for doctors in the highest-performing and lowest-performing quartiles: Their rankings tended to group together, meaning that small differences in star ratings had no significance.
One caveat is that the study was limited to an evaluation of patient satisfaction, as opposed to objective measures of patient outcomes or protocols doctors followed. A study published in the February 2015 issue of JAMA Internal Medicine by Gao and four co-authors found little statistically significant connections between patient ratings on eight websites and objective measures involving 1,299 internists.
“This is what we should keep in mind: A very high score in patient satisfaction is not wholly connected with clinical quality,” Gao says. “If you want to use the online ratings to infer how good a doctor is clinically, take them with a grain of salt.”
The Centers for Medicare & Medicaid Services are working on an online resource that would allow consumers to compare data on health-care outcomes of different physicians, called the Physician Compare Initiative, but it remains controversial because doctors doubt it will be possible to correct for things such as the general health of a physicians' patients and whether patients adhere to doctors' recommendations.
The study, “Vocal Minority and Silent Majority: How Do Online Ratings Reflect Population Perceptions of Quality," is forthcoming in MIS Quarterly: http://go.umd.edu/ZTL.