Official Vaccine Numbers May Be Overestimated, New Study Shows

Editor’s note: Find the latest COVID-19 news and guidance in Medscape’s Coronavirus Resource Center.

Bigger surveys are not necessarily more accurate, a study published online today in the journal Nature explains. In the case of projecting first-dose COVID-19 vaccine uptake, some of the biggest surveys were way off.

Authors, led by doctoral student Valerie C. Bradley, with the Department of Statistics, University of Oxford, Oxford, United Kingdom, write that increasing the data size narrows confidence intervals but can magnify survey bias, an effect known as the ‘Big Data Paradox.’

Valerie Bradley

The team focused on three surveys of attitudes and behaviors toward first-dose uptake: Delphi–Facebook (which gets about 250,000 responses per week and had more than 4.5 million responses from January to May 2021); Census Household Pulse (which gets about 75,000 every 2 weeks and had 600,000 responses from January to May 2021); and the Axios–Ipsos Coronavirus Tracker, the smallest at 1000 responses per week.

The surveys have been used to help fill in gaps in information gathering because the actual data collected by the Centers for Disease Control and Prevention (CDC) is often delayed. Real-time data can help shape government policy surrounding vaccines and messaging.

But the largest surveys are overestimating by orders of magnitude beyond the statistical uncertainty provided by the surveys, the authors write.

“By May 2021, Delphi–Facebook overestimated uptake by 17 percentage points and Census Household Pulse by 14, compared to a benchmark from the [CDC],” according to the paper.

Axios–Ipsos online panel with about 1000 responses weekly, and more than 10,000 responses from January to May 2021, also overestimated uptake but only by 4.2 percentage points.

In some cases, the two large surveys got very different results asking an almost identically worded question. For example, Delphi–Facebook’s state-level estimates for willingness to receive a vaccine from the end of March 2021 are 8.5 percentage points lower on average than those from the Census Household with differences as large as 16 percentage points.

However, the Axios–Ipsos aligns closely with the CDC benchmark.

“Our central message,” the authors write, “is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.”

“In addition to the damage done to efforts to combat COVID, these surveys play into a popular misconception — that data size can overcome low data quantity,” Bradley told Medscape Medical News.

“The estimates of COVID vaccination from these two large surveys are not only biased, but overconfident in their incorrect numbers. While it’s easy to be skeptical of big data that is collected via convenience sampling, we don’t always apply the same skepticism to large surveys that do take some quality-control steps,” she said.

What’s Causing the Discrepancies?

Among the reasons for the discrepancies, the authors say, is that the three surveys use different designs and recruiting methods, which may introduce different biases in the estimates.

Weighting is also a factor: “The overrepresentation of white adults and people with college degrees explains part of the error of Delphi–Facebook,” the authors write. “The racial groups that Delphi–Facebook underrepresents tend to be more willing

and less vaccinated in the samples. In other words, reweighting the Delphi–Facebook survey to upweight racial minorities will bring willingness estimates closer to Household Pulse and the vaccination rate closer to [the] CDC.”

Co-author Seth Flaxman, PhD, with the University of Oxford, told Medscape Medical News the Facebook survey did not adjust for education or race, and the Census survey could not adjust for political partisanship.

Dr Seth Flaxman

“Ultimately, however, the big online surveys left people out, and no amount of adjustment can solve that. If you don’t have internet access, you’re not going to be included,” Flaxman said. “People in rural areas are less likely to be included. So when it comes to measuring vaccine uptake, access, and hesitancy, it’s really hard. The same barriers that exist to getting vaccinated make people less likely to be included in these surveys, whether that’s in terms of a convenient location and time off from work, or  in terms of having a healthcare provider you trust to help you make up your mind.”

He said although there is reason to question other survey estimates on everything from social distancing to why people are hesitant, “the only estimates we know are wrong are on uptake and the directly related questions — hesitancy and willingness.”

“Our study period ended in May, but first-dose vaccine uptake has crept steadily upwards since then approaching 85% of the adult population in the US,” Flaxman noted. “On the one hand, this is encouraging — even one dose is protective against severe disease, especially in those who were previously infected. But on the other hand, it means that 3 out of 20 adults are not protected.”

He said the surveys help communities know how to focus efforts and that accurate information is crucial. If the surveys show, for instance, that people are spending their energy disinfecting surfaces, “We urgently need to get the message out that COVID-19 is airborne,” he said.

Flaxman said his team is trying to raise the alarm that data quality can be a huge issue, and even random samples — the gold standard for surveys — can still have considerable bias because of who is willing or able to respond.

Nature. Published online December 8, 2021. Full text

The study authors have disclosed no relevant financial relationships.

Marcia Frellick is a freelance journalist based in Chicago. She has previously written for the Chicago Tribune, Science News, and Nurse.com, and was an editor at the Chicago Sun-Times, the Cincinnati Enquirer, and the St. Cloud (Minnesota) Times. Follow her on Twitter at @mfrellick

Source: Read Full Article