The 1990s and 2000s were glorious decades to be a psychiatrist. It seemed as though each year several new, potentially life-changing medications were brought to market. Options became available to tackle long-standing, treatment-resistant disorders. It did not hurt that big pharma had deep pockets to promote the new offerings. Stigmas were falling, more people were seeking help than ever before, and just about any psychiatrist could brag about patients who, with the right medication and a little time, returned looking more confident and reporting fewer symptoms.

Today the shine has dimmed. The mental health of the nation may have even declined in the past 20 years. This trend is what Thomas Insel, former director of the National Institute of Mental Health, calls one of the “inconvenient truths” of mental illness. Suicide rates per 100,000 people have increased to a 30-year high. Substance abuse, particularly of opiates, has become epidemic. Disability awards for mental disorders have dramatically increased since 1980, and the U.S. Department of Veterans Affairs is struggling to keep up with the surge in post-traumatic stress disorder (PTSD).

The most discouraging assessment came in 2013 from an in-depth analysis by the U.S. Burden of Disease Collaborators. Hundreds of investigators gathered data on 291 diseases and injuries between 1990 and 2010. Combining premature death and disability to calculate the burden of each disease, they found that the toll of mental disorders had grown in the past two decades, even as other serious conditions became more manageable.

Over the past two decades mental illness has become the second most common cause of disability in the U.S., as measured by Social Security disability claims. Only musculoskeletal disorders have risen faster, although both have tapered off with the improving economy. Source: Annual Statistical Report on the Social Security Disability Insurance Program, 2013. Social Security Administration, December 2014

More people are getting treatment and taking medications today than ever before, so what is going on? I would argue that a lack of precision and objectivity in diagnosing and treating mental illness has stalled our progress. We must embrace new strategies in research and prevention to move forward.

The Inconvenient Truth

The American Psychiatric Association, the American Psychological Association and big pharma explain the deterioration of mental health nationally by proposing that not enough people are getting treatment. But this suggestion seems a bit self-serving.

Another explanation points to the vague nature of psychiatric diagnoses. Social Security awards for disability have exploded in two areas: musculoskeletal and mental disorders, both of which are often diagnosed on the basis of a patient's subjective reports rather than hard measures such as scans or blood tests. Furthermore, both seem to expand and contract with the economy. Thus, changes in the prevalence of mental disorders may not necessarily reflect changes in the biology of mental illness.

It is also possible that we are hampered by not having new treatments for patients seeking help. As it turns out, drugs developed in the past 20 years perform like older medications. Abilify is no more effective for treating schizophrenia than the very first antipsychotic, Thorazine. New antidepressants lift mood no better than the tricyclic antidepressants discovered in the 1950s. Lithium, first used in 1949, remains the gold standard for bipolar disorder. Adderall provides no further advantages for attention-deficit/hyperactivity disorder than the Benzedrine first administered for it in 1937.

There are exceptions—we appear to be better at treating the depressive phase of bipolar disorder, and Clozaril is a more effective treatment for schizophrenia than its predecessors—but much of what seemed so revolutionary 20 years ago was more illusion than substance. The new medications tap into the same brain mechanisms as the old ones, albeit with fewer side effects. Finding novel treatments for mental illness has become so discouraging that several pharmaceutical companies have shut down or reduced neuroscience research.

The problem is that the brain is exceedingly complex. Behavior, emotions and cognition are manifestations of networks of cells that are turned on or off at the right time. The capacity to affect specific cells in the brain without altering other cells remains a massive challenge.

Finding New Solutions

Australia is experiencing the same problem as the U.S.: more people have access to treatment than ever before, but national mental health has not improved. In a 2014 discussion of the dilemma, psychiatric researcher Anthony F. Jorm of the University of Melbourne argued that prevention could be the best response, citing numerous studies of preventive interventions. For example a 2011 Cochrane review—a top-quality, multistudy analysis—revealed that teaching cognitive-behavioral therapy skills in the classroom can help reduce the incidence of depression among students.

Taking prevention one step further, Insel has advocated for research initiatives to discover biomarkers for mental illness. Laboratory tests or genetic markers that can identify at-risk individuals might allow us to intervene early, before symptoms begin.

In addition, it is paramount that we discover new mechanisms to treat mental illness. This progress will not come quickly—the brain does not give up its secrets easily. But there is one unique, promising treatment that is struggling to get approval: psychedelic-assisted psychotherapy. Preliminary evidence suggests that drugs such as LSD and psilocybin could be used episodically, together with psychotherapy, to enhance the healing process.

The best evidence so far is with MDMA, also known as ecstasy, for people with PTSD. In a 2016 review, Michael C. Mithoefer, a Medical University of South Carolina psychiatrist, and his colleagues identified encouraging results from two phase II clinical trials in which people who received therapy supplemented with MDMA at just two or three sessions had fewer symptoms than those given therapy and a placebo.

Unfortunately, the Drug Enforcement Administration classifies psychedelics as drugs with “no currently accepted medical use and a high potential for abuse,” which makes them hard for scientists to access. Big pharma has little interest in studying these molecules because most cannot be patented, and the government is not currently funding such research.

None of this is to say that mental health workers and their patients should stop what they are doing. We all have success stories to tell. As psychiatrist Peter D. Kramer points out in his new book Ordinarily Well (Farrar, Straus and Giroux), the advent of today's antidepressants has largely eliminated the immobilizing melancholy that was once all too common. Based on my own clinical experience, mental health treatment does improve symptoms and quality of life by about 20 to 40 percent for most patients. That's a whole lot better than nothing but not nearly good enough.