Test-Negative Design for Influenza and COVID-19 Vaccine Effectiveness

The U.S. Department of Health and Human Services (HHS) has decided not to publish a study led by the Centers for Disease Control and Prevention (CDC) that estimated the effectiveness of COVID-19 vaccines during the most recent winter season. The decision comes after the study had already passed internal reviews by the CDC’s scientific and editorial teams.

The unpublished research focused on vaccine effectiveness (VE), which the CDC defines as how well vaccination works under real-world conditions to protect people against health outcomes such as symptomatic illness, hospitalization, and death. This differs from vaccine efficacy, which measures performance under the controlled conditions of a clinical trial.

The Test-Negative Design

The study in question utilized a methodological approach known as the test-negative design. What we have is a specific type of case-control study frequently used by researchers globally to evaluate the effectiveness of vaccines against respiratory viruses, including influenza and COVID-19.

In a test-negative design, researchers compare vaccination status among people who seek care for respiratory symptoms. Those who test positive for the virus are classified as cases, while those who test negative are classified as controls. This method helps account for “healthcare-seeking behavior,” as both groups demonstrated a similar willingness to seek medical attention for similar symptoms.

According to a policy brief from KFF, the CDC has a history of using this specific design. The agency recently published a study on influenza vaccine effectiveness using the same approach and has employed it in previous years for COVID-19 vaccine studies.

Justification for Withholding Publication

The decision to block the study’s publication was supported by Jay Bhattacharya, who serves as the Director of the National Institutes of Health and acting Director of the CDC.

The decision not to publish the study was due to concerns about its methodology, and that CDC was “upholding its responsibility to ensure that every piece of information it shares is rigorously validated, accurate and worthy of public trust” through additional review of the study prior to publication.

Jay Bhattacharya, NIH Director and acting CDC Director

The statement was issued via an op-ed, citing the need for further review to ensure the accuracy of the information shared with the public.

Observational vs. Experimental Research

The use of observational methods like the test-negative design is standard for vaccines already in wide use. Because the COVID-19 vaccines are FDA-approved and widely distributed, conducting a new randomized controlled trial—where participants are randomly assigned to receive a vaccine or a placebo—is generally considered unethical.

public health agencies rely on observational approaches to generate real-world estimates of protection. These estimates are typically expressed as a percentage; for example, a vaccine has a VE of 50% if the targeted health outcome is 50% less common in the vaccinated group than in the unvaccinated group.

While the test-negative design is widely accepted in epidemiology, it possesses specific strengths and limitations that researchers must balance when estimating how well a vaccine prevents severe outcomes, such as emergency room visits or hospitalizations, across diverse populations.