Every year around this time, 120 million Americans roll up their sleeves to get their annual flu shots. Since 2010, the U.S. Centers for Disease Control and Prevention has recommended yearly jabs for every healthy American over the age of six months. The goal is to curb the spread of infection and minimize the risk for potentially dangerous complications such as pneumonia, particularly among the elderly and the very young. But science on the vaccine’s efficacy is scant among those two vulnerable groups. And although healthy adults do get some protection, it may not be as robust as they expect.

One oft-cited claim, based on several large meta-analyses published more than a decade ago, is that seasonal flu shots cut the risk of winter death among older people by half. But the research behind that claim has been largely debunked. A 2005 study published in the Archives of Internal Medicine noted that influenza only causes about 5 percent of all excess winter deaths among the elderly—which works out to one death from flu per 1,000 older people each season—so it’s impossible for the shot to prevent half of all their winter deaths. The following year, a study reported that as vaccine coverage increased among the elderly in Italy in the late 1980s, there was no corresponding drop in excess deaths. In another 2006 paper, Lisa Jackson, an infectious disease epidemiologist at the Group Health Research Institute in Seattle, and her colleagues showed that although vaccinated seniors were 44 percent less likely to die during flu season than unvaccinated seniors were, the vaccinated ones were also 61 percent less likely to die before flu season even started. “Naturally, you would not expect the vaccine to work before the thing it protects against is going around,” says Lone Simonsen, a research professor in global health at George Washington University and a co-author of the 2005 study in the Archives of Internal Medicine.

Researchers now attribute these odd findings to a “healthy user” effect. People who don’t get vaccinated often “are the most frail or [those] whose health has gone down dramatically in the last few months,” explains CDC epidemiologist David Shay. People who choose to get flu shots, in other words, are already healthier and therefore the least likely to die.

So how much does the vaccine truly help older people? In January 2012, Michael Osterholm, an epidemiologist at the University of Minnesota’s Center for Infectious Disease Research and Policy, and his colleagues published a meta-analysis in The Lancet Infectious Diseases that analyzed the results of all randomized controlled clinical trials conducted between 1967 and 2011 on the effects of flu shots. It found that there have been no clinical trials evaluating the effects of the traditional flu vaccine in the elderly. The only vaccine shown to protect against infection or death in older adults, it said, is the live-attenuated vaccine—an inhalable vaccine that contains a live, modified version of the virus—which is not approved in the U.S. for adults over age 50.

The traditional vaccine may not work so well in older people because of an idea known as immune senescence, which posits that as people age, their immune systems weaken, resulting in poor vaccine response, especially to inactivated strains. Although the U.S. Food and Drug Administration licensed a high-dose vaccine for seniors in 2009 that could theoretically overcome this problem, no studies have yet been published on how effective it is. “The higher dose produces a higher level of antibodies, but we don’t really know what that correlates to,” says Jackson. A 2010 systematic review published by the Cochrane Collaboration, an independent, nonprofit organization that promotes evidence-based medicine, concluded that “until such time as the role of vaccines for preventing influenza in the elderly is clarified, more comprehensive and effective strategies for the control of acute respiratory infections should be implemented.”

The dearth of controlled research on seniors stems in part from the fact that the U.S government considers such clinical trials unethical. Based on an idea known as clinical equipoise, scientists can’t test, in a randomized controlled trial, a treatment that the larger medical community already considers to be effective, because doing so would involve denying treatment to half of the participants, potentially putting them at risk. “We’re in a difficult spot,” Shay says—since the CDC already recommends flu shots to seniors, the agency can’t suddenly turn around and ask them to participate in a clinical trial that might deny them the standard of care.

What about kids? In 2010, the U.S. Advisory Committee on Immunization Practices began recommending flu vaccination for all healthy children older than six months, an expansion that they claimed was “supported by evidence that annual influenza vaccination is a safe and effective preventive health action with potential benefit in all age groups.” Yet a July 2012 Cochrane Collaboration systematic review concluded that for kids under the age of two, the currently licensed vaccines “are not significantly more efficacious than placebo.” The review highlighted a single small study conducted on children under two—the only controlled study that has evaluated the efficacy of the shot currently licensed for young kids—which found, overall, that vaccines provided no statistically significant protection over the course of two flu seasons. “One season, the vaccine did something to prevent some symptoms, but in the second, nothing,” says co-author Tom Jefferson, an epidemiologist with the Cochrane group. In kids older than 2, however, flu vaccines do seem to work; according to the Cochrane analysis, the shot reduces the absolute risk that a child will catch the flu by about 3.6 percent, whereas the live (inhaled) vaccine reduces the absolute risk by about 17 percent.

In healthy adults under the age of 65, flu vaccines work, too. A 2010 Cochrane review, also co-authored by Jefferson, estimated that during “good” vaccine years—when the vaccines match the circulating viral strain well, which Jefferson says happens about half the time—the vaccine reduces the relative risk that an adult under 65 will catch the flu by about 75 percent. In absolute terms, however, this means adults have about a four percent chance of catching the flu if they don’t get the vaccine and about a one percent chance if they do. Shay notes that while this estimate is reasonable, some flu seasons are worse than others, so the risk may be higher than 4 percent in some years (and some people) and lower than 4 percent in others. (And of course, the vaccine won’t protect against the nearly 200 viruses that cause flu-like symptoms but aren’t actually the flu.) Although scientists generally believe that the flu vaccine slows the spread of the virus through communities, there are no data showing that this is true, because “those studies are very difficult to do,” Shay explains.

So should people still dutifully line up for their flu shots? Older kids and healthy adults do get some protection from them; just perhaps not as much as they want or expect. But for seniors and toddlers, there may never be a clear answer to this question, particularly because the U.S. government is unlikely to conduct additional clinical trials. On Monday, Osterholm and a group of five other scientists at the University of Minnesota’s Center for Infectious Disease Research and Policy published a report highlighting the need for better alternatives. Although the current options may—for most people—be better than nothing, “we can no longer accept the status quo,” they wrote. “The perception that current vaccines are already highly effective in preventing influenza is a major barrier to pursuing game-changing alternatives.”