When is “Science” Truly Science?
The phrase “new research shows…” might not always give the most dependable health information. In this information age, we are constantly bombarded with and have easy access to copious amounts of info and data about health-related research findings including ones claiming to have supporting scientific evidence. But how do you know the scientific evidence is based on sound science? As consumers, its important to determine whether the findings of new scientific studies are reliable and which are not?
The Power of a Headline
Remember when we were told that drinking red wine = an hour in the gym? Catchy headlines surfaced after researchers at the University of Alberta published a study demonstrating that an antioxidant found in red wine was heart healthy (and gave rats an extra boost during exercise). However, these “cover stories” largely misrepresented the data found from the research!
These days, our desire for a healthy lifestyle has made us gullible for any research that touts miracle health benefits. And it seems that companies, scientists, marketers, and non-profit organizations use scientific findings to sway public opinion so they can sell products or convince people to their point of view.
Furthermore, the media has fed this interest by featuring medical and scientific experts as well as products and diets on social media, TV and radio, and print publications— making it easier for consumers to get their hands on information. Sound bites are easy to report but it is important (and often very difficult) to go beyond the ‘quick report.’
And most importantly, is this “scientific information” always reported correctly?
In simple terms, here is how you distinguish science from pseudoscience. Answer these questions and you will know if you are reading something that is valid.
- If science is being reported, who is reporting it? Are they selling something? Do they have an objective?
- If you are reading a scientific journal, is it peer reviewed? Has the scientist cited opposing views? Is the scientist unbiased?
As non-scientist readers and listeners, how do we evaluate scientific research being presented to us in order to determine its quality and accuracy?
The problem, it seems, is that media outlets and publications are not always transparent about:
- How they decide what to report, and
- The methods they use to determine that the scientific discoveries are fact-based and supportable.
For example, a recent study analyzed what was “more likely” to be reported in newspapers. Using a database of studies classified into lifestyle (e.g., smoking) and non-lifestyle (e.g., genetic risk), the main finding was that journalists preferentially cover positive initial findings about lifestyle-related subjects and rarely inform the public when findings are later overturned.
There is a call for the media to be more accurate in their reporting of science.
Some non-profit groups are trying to improve the quality of science being reported. Organizations like the Science Media Centre, founded in the UK in 2002, are trying to help scientists engage more effectively with the media. The Centre will connect scientists with journalists so that there can be a conversation— particularly when it comes to controversial science-related issues.
Additionally, following some high profile retractions, several major scientific journals and the Center for Open Science developed guidelines on sharing data and methods that over 100 journals and 30 organizations have signed onto. These organizations along with individual science journalists are calling for news outlets to do a better job of reporting science to the public.
Let’s be better informed consumers…
When you hear about or read a second-hand review of a particular scientific study or a so-called science-based claim, you should read the original study or related studies yourself.
Relying on reports written by someone other than the study author(s) increases the possibility of getting a flawed, biased interpretation of the study’s findings. Reading the primary source (as opposed of relying on second-hand information) will get you closer to what the researchers say they discovered.
If you’re not a scientist and have never cracked the cover of a scientific journal, this may seem daunting to you and that’s understandable! But if you wish to read original publications, we’d like to equip you with some tools to help you better understand what you are reading (see the infographic below). If you decide reading scientific studies is not for you, this article provides some critical issues to look for when findings are being interpreted by others.
Additionally, a reliable method of fact checking is to check the internet to see if other scientists who work in the same field have critiqued the report. Scientists have opinions, and sometimes their opinions cloud their reasoning – just as with non-scientists. However, if multiple scientists are pointing out the same flaws in a study, then there’s a good chance the criticism has merit. For example, in our previously published post, Dear New York Times, D2D reported on the response of various scientists to a New York Times article on genetically modified crops. The scientists’ critiques were detailed and pointed regarding the choice of data and the analytical methods used.
The Power of the Scientist: Good Studies require “Good Scientific Practice”
According to the National Academies of Science, the leading U.S. science body, good scientific practices include:
- Precision when defining terms, processes, context, results, and limitations;
- Openness to criticism and refutation; and
- Addressing bias and avoid overstatement.
Let’s explore each of these good scientific practices a bit further:
Scientific Precision = Addressing Uncertainty.
“Researchers have a professional obligation to perform research and present the results of that research as objectively and as accurately as possible.”
– National Academies of Science and Engineering; Institute of Medicine
All scientific data and processes have limitations and therefore include a measure of uncertainty to account for the unknown. For example, if you run 200 meters twice daily for two weeks you will post different times. This is why numerical data in a scientific or technical paper should never be only one value, but should include a range of plausible values.
When designing or conducting a scientific study, one of the key tasks for an investigator is to identify and control for errors or variations as much as possible and to estimate the magnitude of the remaining errors. Going back to our running example, to eliminate as much variability in your data as possible, you would run on the same indoor track at the same time each day, one hour after you eat a bowl of oatmeal and a banana. Your results might not be the same if you sometimes ate eggs instead of the banana and oatmeal and ran later at different times each day.
Another factor that affects the researcher’s ability to detect an effect such as differences between treatment and controls is the size of a study. This is referred to as the statistical “power” of a study and determines the confidence with which conclusions can be drawn and interfering or lurking variables (referred to as confounding variables) can be understood. When it comes to sample size, bigger is usually better. You can think of this in terms of the average: the average of a large number of samples is more informative than the average of a smaller sample set.
Scientists must be open to criticism and refutation…
Science is all about discovery and exploration – a pursuit of knowledge at the expense of opinions. When researchers discuss their work, they should compare their findings to what is already known and address how it fits as one piece into the larger puzzle. If their results conflict with others’ work, they should discuss what they believe is the reason for this. If their results were unanticipated or introduced unanswered questions, these should be discussed along with suggestions for further research that may provide the missing information.
You may have seen the term “peer-reviewed” used to describe scientific and technical studies. What does this mean and why is it important? When a paper is “peer-reviewed” it means it was submitted to other experts in the particular field of research to judge the quality of the work.
“Researchers must remain open to new ideas and continually test their own and other’s ideas against new information and observations.”
– National Academies of Science and Engineering; Institute of Medicine
“Methods of communication that do not incorporate peer review or a comparable vetting process could reduce the reliability of scientific information.”
– National Academies of Science and Engineering; Institute of Medicine
The practice of peer-review offers a valuable way of evaluating and improving the quality of scientific studies. Peer-reviewed journals are publications that follow a process of subjecting an author’s scholarly research to the inspection of other experts in the same field before publishing a study. Journals that do not go through the peer-review process are missing an important quality control mechanism. And by publishing a paper in a non-peer-reviewed journal, scientists run a greater risk of having to correct or retract flawed work after it was published versus making corrections prior to publication during the peer-review process.
Scientists must address/deal with bias…
Just as no measurement is free from error, human interpretation is not free from bias. However, when conducting research, scientists must design experiments to provide unbiased, useful data that, when analyzed, either do or do not support the hypothesis.
Of course, this is easier said than done since bias constructs are innate and difficult to recognize in ourselves. But the scientific process takes this into account and scientists must give significant effort in addressing it.
Although study funding sources may have an influence on the findings, it is important to not automatically jump to this conclusion.
Public-private partnerships between non-profit / for-profit entities and academia have become more common in the past several decades as government funding is reduced. These arrangements are often contractually administered to protect the researchers from undue funder influence that could bias research findings.
In designing a study, scientists incorporate methods (e.g., randomized assignment to groups, investigator “blinding” so they do not know which subjects are being treated, etc.) to eliminate or control bias as much as possible. Whatever bias is not eliminated or controlled by study design, must be considered and discussed when researchers interpret their results
Other sources of bias such as conflicts of interest are more overt. Some peer-reviewed journals mandate researchers declare potential conflicts of interest. Even if a conflict of interest statement does not appear in the article, a reader can do their own research to determine if the author and/or a funding source benefits in any way from reporting the results as they were reported.
One of the most notorious (and more recent) cases of a retracted scientific paper was in part due to a charge of conflict of interest. In 1998, The Lancet, a well-regarded British medical journal, published a study by Dr. Andrew Wakefield where he concluded that the measles, mumps, and rubella vaccine should be separated and given over an extended period of time because it was linked to autism in children. A British journalist discovered that Dr. Wakefield had a patent on a measles vaccine, use of which may have increased if the vaccines for each disease were indeed separated and that his research was funded by lawyers representing families looking to sue vaccine makers for damages. The Lancet retracted the paper in 2010 after a British medical panel ruled he had been dishonest, violated basic research ethics rules, and demonstrated a “callous disregard” for children’s suffering. Following publication of Dr. Wakefield’s research, vaccination in Britain and America plunged and measles cases surged. To this day, anti-vaccine groups continue to spread the view that vaccines are associated with autism even when study after study has failed to provide evidence.
How do you evaluate a study?
Now that we’ve reviewed the basic architecture of a scientific article and the National Academies’ good scientific practice, let’s consider how to critically evaluate the actual research findings and conclusions.
In addition to the National Academies’ publication, three prominent scientists published concepts for interpreting scientific claims in the acclaimed peer-reviewed journal, Nature. The authors created the list with politicians in mind to provide them with some basic understanding so they could ask their advisors informed questions. However, the authors also stated that if everyone in society understood these concepts it “would be a marked step forward,” and we here at D2D couldn’t agree more!
Are you ready to try your hand at spotting some erroneous or misleading data?
Two professors at the University of Washington developed a course called “Calling Bullshit in the Age of Big Data”, and as part of the coursework provide case studies illustrating statistical distortions, misleading data, and other violations of scientific principles and practices. These case studies provide great examples of how data is used intentionally and unintentionally in a way that misleads the reader if you are not aware or knowledgeable about what to look for.
Take a moment and test your “BS” acumen by reading some case studies here.
The Bottom Line:
Do not unquestioningly believe everything you read whether claiming scientific evidence or touting new discoveries. It’s not the individual study that matters so much — many quality studies are needed to replicate findings and create robust evidence supporting a theory. Being a well-informed reader requires some work to equip yourself with the skills to evaluate and critique science-based claims. Learn to critically analyze research findings and judge the merits of what you hear and read.
Graf C, Wager E, Bowman A, Fiack S, Scott-Lichter D, Robinson A. 2007. Best practice guidelines on publication ethics: a publisher’s perspective. International Journal of Clinical Practice Supplement, 61(Suppl 152):1-26. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1804120/
Hoogenboom BJ, Manske RC. 2012. How to write a scientific article, International Journal of Sports Physical Therapy, 7(5):512-517. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3474301/
Science Communication. Sage Publishing, https://us.sagepub.com/en-us/nam/journal/science-communication#description.
Public Understanding of Science, Sage Publishing, https://us.sagepub.com/en-us/nam/journal/public-understanding-science#description.
Bibliography of science communication articles (2006-09): https://colinschultz.wordpress.com/2010/03/23/so-you-want-to-be-a-science-journalist-well-heres-a-bibliograpy/