Despite a widely hyped meta-review, the evidence doesn’t really stack up.
Depression is one of the biggest health issues in society today.
It’s hard to know for sure how many people experience depression, but current best estimates put it at about 1 in 10 people experiencing symptoms consistent with major depressive disorder every year, and a scarily high 1 in 5 people experiencing depression at least once in their lifetime.
From personal experience, I can say with some certainty that depression sucks.
On top of that, treatments for depression are notoriously problematic. The most common drugs prescribed to treat the condition, selective serotonin reuptake inhibitors, have uncomfortable side-effects which cause many people to stop taking them over time. For many people with depression, effective treatment can be a serious struggle.
But according to recent headlines, there’s hope! It seems that scientists have shown that exercise – simple physical workouts – is a more effective treatment for depression than medicines. According to some news stories, exercise is 50% more effective than drugs when it comes to mental health.
Unfortunately, the evidence doesn’t show this at all. Exercise is definitely good for you, but most of the evidence looking at exercise for depression is simply not good enough for us to make any useful conclusions.
The study
The study that has created such a huge hubbub is an umbrella review of systematic reviews into depression and exercise, recently published in the British Journal of Sports Medicine.
This sort of study is essentially a review of reviews – each systematic review looks at all the evidence and studies on a specific topic, say exercise for depressive symptoms in people undergoing chemotherapy, and then the umbrella review aggregates all of those systematic reviews into one enormous meta-study.
This particular umbrella review looked at a total of 97 review papers covering more than 1,000 randomised trials, which means a genuinely huge pool of data.
After aggregating together all of these studies, the paper in question found that “Physical activity is highly beneficial for improving symptoms of depression, anxiety and distress across a wide range of adult populations”, which is quite a ringing endorsement from a scientific paper.
In particular, the authors noted in their discussion that the effect size seen for exercise in the treatment of depression in their full model was comparable to or better than those seen for medications, which lead to the breathless media headlines.
And, at first, this sounds very rosy. Systematic reviews of randomised trials are the gold standard in medical research, after all, and in this study the researchers found that lumping in all of these reviews together found a big benefit.
At the outset, however, it’s clear that the headlines are completely incorrect. You see, this umbrella review didn’t assess exercise compared to medicines — in fact, they explicitly excluded studies that compared exercise to medications directly.
This means that it’s simply not possible to make any claims about how the results of this review relate to drugs, and all of the people who did are simply wrong.
So the review didn’t look at exercise vs meds, it just looked at exercise against care as usual. But on top of that, there’s a big issue with reviews, and umbrella reviews in particular.
You see, a review article is only as good as the research it includes — if you aggregate the results of mostly terrible research into one big statistical model, what you’re left with at the end is meaningless noise.
The quality of the included research is far more important than the quantity, especially when it comes to systematic reviews that themselves aggregate together potentially questionable research.
And when you look at the quality of the reviews that the authors included in their main model, most of the research was very unreliable.
Indeed, the authors of the umbrella review rated every study that they included, and of the 97 studies that went into their main estimate of the effect of exercise on depression, 77 were rated as “critically low quality” using a study-rating tool called AMSTAR.
In fact, of the 97 systematic reviews that the umbrella review identified, just 10 were rated as “high” quality. Of these, 3 looked only at anxiety disorders, leaving just 7 looking specifically at exercise for depression. And if we look only at these high-quality reviews, the message about exercise seems, at best, to be much more uncertain than the headlines suggest.
High-quality research
The first thing to note here is that the AMSTAR system is more of a baseline for quality control than a perfect estimate of whether a study is good or not. It lists very basic elements that you’d expect in systematic reviews, like whether the authors actually defined the target population that they were interested in analysing.
What this means is that when I say “high-quality” research, what I really mean is that these reviews meet the minimum standards that you’d expect from systematic reviews that we use as evidence.
That being said, if we look at all these high-quality reviews, the evidence seems quite slim. Here are all of the reviews:
- Exercise interventions on health-related quality of life for cancer survivors
- Effects of exercise on depression and anxiety in persons living with HIV: A meta-analysis
- Exercise for women receiving adjuvant therapy for breast cancer
- Yoga in addition to standard care for patients with haematological malignancies
- Dance/movement therapy for improving psychological and physical outcomes in cancer patients
- Aerobic physical exercise for adult patients with haematological malignancies
- Exercise for depression
Of these papers, five looked at exercise for depressive symptoms in people either diagnosed with or recovering from cancer, one looked at exercise for depression in people with HIV, and just one looked at the broader question of exercise for depression.
Moreover, of these seven reviews, five were published by the Cochrane collaboration, known as the gold standard in evidence-based medicine.
If we consider the reviews looking at specific patient populations, the results are not all that positive. For example, exercise for women receiving breast cancer therapy appeared to make “little or no difference in depression” based on the 2016 Cochrane review.
On top of that, if we look at the large 2013 Cochrane review of exercise for depression more broadly, we see an even more interesting pattern. This review found a fairly impressive benefit in their main model, but when they restricted the analysis only to high-quality clinical trials that benefit disappeared.
In other words, the highest-quality evidence from the best review seems to indicate that exercise had minimal benefit on depression, although this is a bit out of date now and was also quite uncertain at the time.
All of this makes more sense when you look at some of the evidence that is included in many of these systematic reviews. For example, this relatively low-quality Portuguese study from 2011 that looked at just 29 people, or this 1997 study that looked at 32 older people doing resistance training.
Most of these studies have extremely poor controls, and lack basic elements of statistical reliability such as pre-registration. If we restrict our analysis only to studies that meet the requirements for high-quality clinical research, much of the benefit seen for exercise in the papers seems to disappear.
Bottom line
What does this all mean? Well, the take-home is not that exercise is useless for depression, or mental health generally. We couldn’t possibly make that claim based on the evidence.
Rather, what we see is that when you look at mostly bad studies, there’s a massive benefit for exercise on depression. When you restrict your view to only those papers that appear to have been conducted in a rigorous fashion, the benefit for exercise doesn’t disappear, but it does reduce substantially to the point where it’s hard to differentiate from no benefit at all.
This doesn’t mean that exercise is ineffective — mostly it means that even the few good studies on the topic aren’t really that good. In all of the systematic reviews on the topic, the largest single study I could find was this 2018 paper looking at 12 weeks of exercise for people with mild/moderate depression.
The paper found that both cognitive behavioural therapy and exercise caused about a 3-point improvement on a 60-point depression rating scale over 12 weeks when compared to a control group, which is meaningful but not enormous.
And in this context, it’s hard to make any specific claims about exercise for depression. There are so few pieces of research examining how exercise compares to drugs for the condition that we really can’t say anything meaningful at all on the topic.
So the headlines were wrong, but unfortunately there doesn’t appear to be a satisfying yes/no answer to the question we really want to know about.
Exercise programs seem likely to have some benefit for depressed people, but how big that benefit is, and how it compares to medications, is really anyone’s guess.
Hopefully we see some better research in the future. For now, it’s hard to make any serious recommendations other than that we need more useful data.
Gideon Meyerowitz-Katz is an epidemiologist and PhD candidate who tweets as The Health Nerd @GidMK.
This piece was originally published at Medium.
