Recent events have made it clear that there is no easy way to assess the quality of scientific research. A study with potentially life-changing information has been withheld due to concerns about its reliability.1
In another instance, an already published study was criticized for methodological flaws that called into question its conclusions.2 The researchers behind this study were forced to admit that they may not have followed all relevant ethical guidelines.
It seems that even when studies are free from serious errors, people still disagree about their importance or validity. This can create confusion as to whether a particular treatment works, or if established practices are effective.
As experts in both psychology and medicine, we feel very confident in our own areas, but we also understand how difficult it can be to evaluate work outside of yours. That’s why we decided to make this article!
To help you sort through the overwhelming amount of information, we will focus only on two types of studies—clinical trials and systematic reviews. We will also exclusively discuss what makes a good clinical trial or review, not just whether a study is well conducted or not.
This article will definitely add some clarity around the value of psychological treatments like CBT for depression. It may also prove helpful for those who want to determine the effectiveness of other therapies and interventions.
Look at the methodology
A key part of evaluating any scientific research is looking at the methodological quality of the study. How was this piece of research conducted? What tools were used to gather and analyze the data?
Most importantly, how reliable are these results? Even if an experiment seems really effective, you have to consider whether there could be other reasons for the findings. Was there a better, more credible test that showed different results? Or did the researchers conduct another test with similar results?
When assessing studies, make sure to not only look at what effects they found but also evaluate how well the investigators carried out their experiments.
Is the sample big enough to be representative?
A small study population or sample size is a major warning sign of poor research quality. Because disease often has symptoms, there are many more people experiencing those symptoms than without them. If the studies being compared have too few participants, then it may not be possible to determine if one intervention works better than another!
Studies with very little participant number tend to draw conclusions from smaller datasets that do not represent the larger population. When you read a paper with only five participants, you cannot know whether these five individuals were simply extremely healthy already or if this intervention actually does work.
Furthermore, because of the low numbers, it can’t be determined how much effect each treatment had on the outcome. It is impossible to tell if any changes observed in the study were due to chance or the test product.
When interpreting results, make sure to consider both benefits and risks of interventions, even for large studies. Just because something seems safe and common practice doesn’t mean it will work for your specific health condition.
Has the research been peer reviewed?
One important thing to consider when evaluating any new health claim or theory of disease is whether the study has been verified by other experts. A recent popular example comes from studies linking chocolate with lower blood pressure.
A systematic review conducted in 2018 looked at all published, randomized controlled trials (RCTs) on this topic. It found that eating five servings of chocolate per week will not reduce your blood pressure.
You may be familiar with some of these terms: RCT means a clinical trial done under rigorous scientific conditions. Systematic reviews use standardized methodology to evaluate the evidence across many studies.
By having independent reviewers assess the same body of literature, they can get an accurate picture of how well studied a given hypothesis is.
Has the research been published?
A study that has not been published anywhere is considered to be fraudulent. It is also known as sham science. This includes studies that have not been peer reviewed, unpublished master’s degrees, or PhD dissertations that have never been made public.
There are many reasons why someone might choose to keep their findings private, but in most cases it is because they do not want others to critique their work or find flaws with it. By hiding what you know, you prevent other people from sharing in this knowledge either by direct competition or constructive feedback.
By staying anonymous, these researchers take advantage of the trust that is placed in academic institutions like universities. If students at your school may need help understanding an idea or concept, there will be no one to teach them. Or if colleagues ask you about something, you will go without answers because you decided not to share your insights.
This can create an uncomfortable situation for those who look up to you, and for yourself when you should be educating yourself and developing new skills. You could even hurt your own career by keeping secrets that could easily be found through doing some basic research.
Is the research recent?
Recent means that the study had to be published in a peer-reviewed journal or reported at a conference where other experts review the material. This looks like an academic degree of quality, but actually it’s not!
Most reputable scientific institutions require people who want to present their work to first prove they have done adequate background research and made sure what they are proposing is correct. They must put effort into proving their hypothesis wrong before getting credit for being right!
This process helps ensure that the results are sound and believable, and use appropriate statistical methods to produce reliable conclusions.
It is very important to note that just because someone else has published something similar to your topic does NOT mean that their findings are accurate! A lot of times, researchers will say the same thing as someone else’s research, only modified slightly or even completely opposite what the original report said.
By using sources with good credibility, you can avoid adding false information to your diet regime.
Is the research applicable to your situation?
The second major factor in evaluating any health claim is whether the findings of the study can be applied to you or not. Most studies have limited applicability, as they are done under very specific conditions that may not always apply to the general population.
Studies often test just one hypothesis about a particular topic. For example, most clinical trials look at if eating three carrots per day is more effective than drinking two glasses of milk per day for helping prevent skin cancer.
However, what happens when you eat three carrots a day is that your hands become bright orange. Testing this theory by drinking two glasses of milk every night would be quite difficult!
Another limitation of some studies is that they only test one variable. In other words, they test whether having five fruits and vegetables a week is better than having two. But they never test whether having twice as many fruits and veggies is even better.
What is the source?
First, you need to determine what source your information comes from. There are three main sources: research studies, statements made by experts, and published articles or essays.
Studies are an important component of any successful lifestyle change. When done properly, studies can prove or disprove theories about effective health and wellness strategies. They can also show how different methods work for changing behaviors or treatments.
By studying past studies, we learn more about potential benefits of certain diets or exercises. This helps us identify if there are ever-effective approaches or whether something works in some people but not others.
Sadly, many people’s beliefs are influenced too much by false assumptions and misinformation. So it is very important to evaluate the source and credibility of each piece of evidence before forming opinions.
Experts and experienced individuals make valuable contributions by sharing their knowledge and insights with the rest of us. If someone calls themselves an “expert” then they should probably be given at least some respect until proven otherwise!
But just because someone says things that seem logical does NOT mean that they are correct. Take his word for it only when confirmed through other credible sources.
Who funded the research?
Even if you don’t agree with everything an expert says, at least look into who is paying for their studies! Unfortunately, not everyone publishes how they are paid or even if they are being paid at all.
Studies have shown that people say things that earn them money, so making sure those studies are worth it depends on whether the researcher was sponsored by the manufacturers of products under investigation.
It is important to remember that strong marketing campaigns can sway results, and sometimes researchers are too influenced by funding to draw conclusions that align more favorably with sponsors.
By looking into who is sponsoring a study, we can determine the credibility of the information much better than relying only on what the experts tell us themselves.
