How To Identify and Assess Well-Designed Trial Research

When searching for evidence-based research, it’s important to distinguish good research from poor research.  The following tips may help you in this endeavor—

1. Identify trials with large groups of participants. Trials performed with high numbers of subjects have greater impact than trials with fewer subjects. For instance, a trial with 500 participants is much more likely to have a statistically valid conclusion than a trial with 20 participants.
2. Trials should include randomization. When sorting participants into groups, group assignments should be random. So for instance, if a trial recruits 250 participants, they should be randomly assigned to the experimental and control conditions (e.g., sealed envelope assignment or computer generated random assignment may be used).
3. Check for adequate blinding. Trials should involve blinding, meaning that researchers, experimenters, subjects, and assessors should not know which group subjects are in during the experiment.
4. Trials should describe methods used. If you’re looking for whether subjects were randomized to groups and whether the subjects were blinded and the information is not included, this indicates that the trial’s methodology is unknown and therefore may be questionable. A good question to ask is, “Can a study’s quality be evaluated by the information the authors provide?”
5. Look for systematic reviews and meta-analyses. Even if one trial on a certain subject has been well conducted with a large number of subjects, strong conclusions may not be accurate based on a single trial. Systematic reviews and meta-analyses identify all available trials on the same subject and can therefore provide more substantive evidence of effect. Ten trials with the same conclusion provide better evidence than one.
6. Assess placebo methods, shams, and controls. Did the researchers have a control group? What kind of sham acupuncture was used, if applicable, and was it actually inert? Many types of acupuncture shams can be active shams, and may have subtle or overt physiological influences on the body. Did the researchers account for this possibility?
7. Check for bias. Many types of bias can exist in a trial. For instance, selection bias may occur if, as one example, researchers recruit only Chinese patients who have been using acupuncture as a means of medical treatment for years. Citation bias may occur if researchers are unwilling to publish unfavorable results. Confounding bias can occur when a third independent factor is not accounted for when examining two outcomes of interest—for example, an association found in a study between coffee consumption and heart attack, might later turn out to be confounded by cigarette smoking, if researchers did not initially account for this possibility. The Cochrane Collaboration has published information on assessing risk of bias, and developed a commonly used tool: http://handbook.cochrane.org/chapter_8/8_assessing_risk_of_bias_in_included_studies.htm.
8. Check the study’s validity. A study should have internal validity, meaning that it should rule out alternative explanations for the effects observed. It should also have external validity, meaning that a study’s findings in a certain population should be generalizable to the population at large. Additionally, a study should have construct validity, meaning that a study should have been designed in a way that means it has the capacity to measure what it sets out to measure. For example, any outcome measures used (i.e., surveys, scales) or sham measures used (i.e., type of Sham needle, etc.) should have been independently validated in prior studies.
9. Identify the impact factor of the journal the study was published in. An impact factor is a measurement that shows how often articles within a journal have been cited by other articles. As such, it suggests the relative importance of the journal within a certain field. A higher impact factor means that studies published within that journal are more likely to be important within their field. For example, JAMA’s impact factor in 2014 was 35.289. In contrast, the impact factor ofthe International Journal of Mosquito Research in the same year was 0.453. While a study published in a low impact factor journal isn’t necessarily flawed, chances are it won’t have received the same level of rigorous review or have the kind of far-reaching, field-shifting implications of a study published in a high impact journal. 
10. Determine whether the research has been peer reviewed. Peer reviewed research has been evaluated by external experts with experience in the subject matter. It is considered higher quality.
11. Check for the use of appropriate statistical methods. Accurate analysis of data depends on the use of appropriate statistical tests. Did the research group consult a statistician and were the tests used to analyze data appropriate for the type of study and outcomes they wanted to explore?
12. Assess the appropriateness of conclusions. In discussion and analysis of data, researchers should note whether findings are statistically significant, and should be careful not to overstate outcomes. Even in a well-designed trial, further research and affirmation of outcomes in equivalent studies are needed before trial outcomes can be accepted as factual. In translating research summaries into news articles, some journalists have a tendency to overstate conclusions in an effort to create news headlines. If you’re getting research summaries from a news outlet, make sure to read the original article and decide for yourself. Again, one article is never enough to draw a firm conclusion about any topic. 

These guidelines should help you in sorting through research studies, summaries and articles to help you find the most accurate information available.

Click here for information on finding research in PubMed and other common databases.