The Problem with Observational Studies (Epidemiology)

  • Does saturated fat clog our arteries?
  • Should we embrace low-fat or high-fat diets?
  • Is low-carb the answer to our nutrition quandaries, or is eating more whole grains?
  • What’s better, a plant-based or a paleo diet?
  • Are soy-based burgers healthier than beef burgers?
  • Is fruit juice healthy, or is it a sugar bomb?
  • Are eggs good for us or are they too high in cholesterol?

What is an observational study?

There are two primary ways of undertaking studies to find out what happens when we eat one thing versus another: observational studies and randomized trials.

The problems with observational studies

In short, the primary issue with drawing conclusions from observational studies is that they’re often wrong. In fact, they may be wrong more often than they’re right.

By Jim Borgman for the Cincinnati Inquirer

Healthy user bias

The teeth brushing study mentioned earlier, which implies that frequent teeth brushing prevents heart disease, suffers from what is called healthy user bias.

By Brian Crain for The Washington Post

Unhealthy user bias

Just as there is healthy user bias, there is also unhealthy user bias. For decades, public health advice has told us to limit cholesterol and opt for low cholesterol egg whites instead of high cholesterol egg yolks. The rebels that ignore such health advice and opt for high cholesterol foods tend to also be the people that ignore other health advice to exercise more, eat fruits and vegetables, stop smoking, and drink less alcohol.

By xkcd

Confounding variables

In the case of teeth brushing and ashtray ownership, the associated lifestyle habits–diet, exercise frequency, number of cigarettes smoked per day, etc–are called confounding variables, because they confound (mix up or confuse) cause and effect.

Food Frequency Questionnaires

If observational studies are all about observing what someone eats, and correlating that to disease, you may be wondering how researchers objectively and quantifiably determine what exactly goes into someone’s mouth over the course of many years or decades. Turns out, they don’t.

A typical food frequency questionnaire (source). If you’re an avid kiwi eater like I am or a lover of papayas and persimmons, you’re out of luck. Those fruits, and dozens of others, are not included in this questionnaire.

Reliance on biomarkers

Determining whether a certain variable leads to death or disease requires following study participants for a long time, possibly many decades. Doing so is extremely time consuming and expensive., and most researchers would like to see the results of their observational studies be published during their career rather than after their retirement.

As a result of observational studies, a 1995 pamphlet from the American Heart Association recommends choosing snacks such as low-fat cookies, candy, and sugar (source)
By Harley Schwadron for Reader’s Digest

Randomized controlled trials

As we’ve seen, the results of observational studies are almost entirely worthless from a policy standpoint since they have just as much a chance of being wrong as they have of being right.

By Dan Wasserman for The Boston Globe

Conclusion

The tides may finally be starting to turn against observational studies. After decades of promoting egg whites and low cholesterol diets based on results of observational studies, the American Heart Association, in a January 2020 publication on cholesterol, did the equivalent of throwing its hands in the air (bolding mine):

By Tom Toles for The Washington Post

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store