As the pharmaceutical giant Pfizer was reminded in May, arriving first has its rewards, but they come with the risks of venturing into uncharted territory. This past spring the Federal Aviation Administration banned pilots and air traffic controllers from taking the company’s popular smoking-cessation aid, varenicline, which is sold in the U.S. as Chantix. Amid 6.5 million prescriptions written worldwide since 2006, the drug had spawned highly publicized reports of acute psychiatric episodes that included seizures, psychosis and suicidal depression. In May the nonprofit Institute for Safe Medication Practices documented 988 such “adverse events,” prompting the aviation ban.

The Food and Drug Administration has now added strong warning language to varenicline’s medication guide, and Pfizer is reviewing evidence that might help explain the rare but severe incidents. Although the bad publicity may dampen sales of the drug, observers say that some adverse events are not unexpected when a new drug hits the market, especially one that is the first of its kind. Varenicline is not just a novel smoking-cessation tool; it is the first of an entire class of medications specifically designed to target a powerful family of receptors on the surface of brain cells. Known as neuronal nicotinic acetylcholine receptors, they can mediate pain, mood, memory, attention and other cognitive functions.

Abbott Laboratories, Targacept and AstraZeneca have nicotinic receptor drugs in clinical trials for memory impairment, adult attention-deficit hyperactivity disorder and pain. The National Institute on Drug Abuse is testing varenicline itself as a treatment for cocaine and alcohol dependence. Preclinical studies are looking at other new nicotinic receptor compounds for Parkinson’s disease, Alzheimer’s disease, depression, ulcerative colitis and inflammation as well, attesting to the broad influence of this receptor family.

The effects of nicotinic receptors are so pervasive, in fact, that some of the mechanisms involved are not completely understood. “It’s a story that’s still evolving, and it’s very complicated, so going in with a drug like varenicline, I’m not surprised that there are side effects,” says Lorna Role, who studies the receptors’ biology at Columbia University and Stony Brook University. This type of acetylcholine receptor, which also responds to nicotine, acts as “a volume control” for other neurotransmitters, according to Role. “A little nicotine turns up transmitter release,” she explains. “It’s been shown to increase the release of dopa­mine, glutamate, GABA—every major neurotransmitter.”

Activating a subtype of nicotinic receptor known as alpha4beta2 causes dopamine to be released in a part of the brain involved in reinforcing reward, for example, and that receptor is the primary target of varenicline. The drug works as a “partial agonist,” meaning it binds to the receptor, producing moderate stimulation intended to stave off nicotine withdrawal. In so doing, it blocks nicotine from getting to the receptor as well, which prevents a smoker from receiving a dopamine surge from a cigarette.

In cell studies, varenicline also acts as a potent full agonist for another receptor subtype called alpha7 that is associated with some of the positive cognitive effects of nicotine, such as enhanced focus. Variations in the alpha7 receptor gene are implicated in the difficulties schizophrenics tend to have with shutting out sounds or other stimuli. “I was hopeful that varenicline could be used for schizophrenia,” Role says, “then the first report came out of it causing a psychotic episode, and it was hands off.”

Given the complexity of the neuro­psychological systems affected by nicotinic receptors, most of the episodes involving varenicline may never be explained. Pfizer representatives point out that smokers as a group have higher than average rates of anxiety and depressive disorders, suggesting that mild or undiagnosed preexisting mental illness might have played a part in some of the reactions to the drug. Moreover, symptoms such as agitation and suicidal thinking are well-documented side effects of tobacco withdrawal, notes Anjan Chatterjee, a director of medical affairs for Pfizer: “So it’s hard to decide, is it the smoker’s past history, or is it varenicline?”

Antidepressants in the class known as selective serotonin reuptake inhibitors (SSRIs), which debuted more than 20 years ago, have also been associated with adverse events such as suicidal thinking. The first generation of such drugs, which included Prozac, was notorious, too, for lesser side effects, including stomach upset and sexual dysfunction. Subsequent generations of SSRIs addressed some of those issues by building in blockers of certain serotonin receptor subtypes to eliminate unwanted drug actions. 

“As with serotonin, people discovered there are subtypes of [receptor] subtypes. I think as time goes on there will be more sophisticated [nicotinic] drugs coming out,” says Edward D. Levin, a behavioral pharmacologist at Duke University, who has consulted for Targacept and for the National Institutes of Health.

“It’s just like the SSRIs,” Role agrees. “I think refining the compounds in terms of the balance of their activities is really key, but that’s not to say that’s trivial. It’ll take time.” Targeting nicotinic receptors “has enormous therapeutic potential,” she says, adding that the biggest joke on the tobacco industry may be that they missed seeing it.

Note: This story was originally printed with the title, "First in Class".