A large amount of information is available to us via media, books, websites, and other sources. Making sense of all this information can be difficult at times.
As we know, there are several reasons why people may want to evaluate different studies or conclusions. This includes things like finding out which treatments work best, determining if a treatment is safe, or just wanting to see how well another thing has worked.
It is very important to understand what makes up strong research studies before assigning value to a given conclusion. There are some components that seem to always make up a good study, but not every study fits that description. It is also important to recognize when a study is too biased to produce meaningful results.
This article will discuss some tips for evaluating scientific research. You will learn about some factors that influence the reliability of a study as well as strategies to identify unreliable studies.
Look at the methodology used
A key part of evaluating any research is looking at how well the researchers conducted their experiment. Did they test the hypothesis under study in a controlled setting with random assignment?
Research that lacks methodological quality is less likely to be accurate than studies that have special attention paid to ensuring internal validity. External validation via replication or direct comparisons are also important considerations when assessing the reliability of a finding as well.
It’s very common to see media stories touting new findings from academic labs, but these reports often fail to include enough information about the experimental design to determine if the results can be trusted.
Look at the sample size
A small study population is one of the biggest warning signs for scientific research. If the researchers are not very transparent about their numbers, then it can be difficult to determine whether or not they cherry-picked their participants or if there were simply just not that many people who took their product seriously enough to participate in the survey.
By having a large participant pool, the researchers are able to draw conclusions from a larger group which makes things more believable. However, this also means that these findings may not be as representative of the overall population like a smaller scale study would be.
Look at the time period examined
A key element in evaluating any research study is determining how long the researchers conducted their studies. If they only studied a short amount of time, then the results may not be representative of what’s happening now or in the past.
Research studies that seem very impressive are usually limited in both duration and number of participants. This makes it difficult to determine if the findings are generalizable (apply to other people) or specific to those few individuals who took part in the study.
By comparison, studies with longer term outcomes can help us understand whether the results remain the same over time or change as times changes.
Similarly, studies that include more participants give greater confidence in the accuracy of the results. Although there is no ‘right�’ length of time for a study, we can say that one year is generally considered the cutoff point for meaningful data. Anything shorter than a year should be treated with some caution.
Look at the results
A large part of evaluating any research study is looking at the results. This includes assessing whether the conclusions are supported by the data in the studies, as well as determining how much of the findings were due to chance or random events.
When reading through an article, you should be focused on what the authors of the article intended for people to learn from their work.
You shouldn’t be trying to determine if the researchers made false claims, but rather if they presented adequate information so that others can compare their methods with their own to see whether there exist similar ways to apply the knowledge gained.
By using clear writing and examples, your should be able to get some sense of how confident the authors feel about the quality of the research and the applicability of the findings.
Consider the limitations of the research
Recent criticisms of psychology as a field that produces overstated claims about the effectiveness of its interventions is not new. Skeptics have pointed out the lack of methodological quality in much psychological research for decades, going back at least to Thomas Liggins’s 1981 article “The Fraudulent Psychology Industry.”1
In his article, Liggins wrote:
“Psychology has become more interested in what feels good to you than what works. We are continually being bombarded with messages telling us that there are easy ways to achieve happiness or success through self-help strategies that increase our feeling levels of confidence, hope, motivation, etc.”2
He continued by stating:
“Many so called’self help’ programs actually do little to no longer to reduce mental health problems. In fact, they can sometimes make things worse because individuals develop delusions of grandeur due to false beliefs about their efficacy.”3
More recent critiques echo these themes. For example, psychologist Matt Fitzgerald published an op-ed in The Los Angeles Times in May arguing that many studies claiming benefits from various therapies are unlikely to work and may even be harmful4.
Fitzgerald cited several examples of questionable findings including studies suggesting that acupuncture helps treat pain,5 cognitive behavioral therapy (CBT)6 reduces suicidal thoughts and behaviors,7 and mindfulness8 practices improve emotional regulation9.
Look at the authors’ conclusions
As mentioned before, researchers are not always able to determine whether or not their findings prove the hypothesis of the study. This is because most studies have a limited scope that only look into one factor. For example, a research paper may test whether eating raw vegetables can prevent cancer from developing by studying how many vegetables you eat and what kind you eat.
However, it may be too early in the disease process for such an outcome to occur so there is no clear proof that raw veggies help prevent cancer.
Is it possible the research was influenced by the researcher?
A study can be completely undermined if there are reasons why the results seem strong for the hypothesis being tested. For example, researchers who studied whether or not eating chocolate helps reduce stress may have been motivated to find an answer that benefits their own chocolate habit.
Researchers want to know whether or not they should give up chocolate because the studies they read seemed to indicate this is good for your health. So, they add chocolate to the mix and test their theory.
It is important to evaluate how much influence the researchers in the study had on the outcome of the experiment. Did they work for companies that sell supplements or pharmaceuticals? Were they paid per body weight or per person? These factors can heavily influence the result.
Is it possible the research was influenced by the source?
A study’s results can be skewed due to sources of funding. For example, studies that show a product is effective may simply be funded by the company that produces the product.
Studies conducted in an area with strong pre-existing evidence are less likely to find effectiveness for a given treatment because people already agree that it works.
Research has shown that physicians tend to over-estimate the benefits of new treatments and patients often don’t tell their doctors about them because they want faster relief or worry that their doctor will not prescribe it for fear of being seen as unhelpful.
By looking at all the available scientific evidence, however, we can more confidently determine if a particular approach really does work.
