January 6, 2017

Study methodology

STUDY VALIDITY

Internal validity
Internal validity reflects the quality of the study design, implementation and data analysis in order to minimize the level of bias and determine a true 'cause and effect'

External validity
External validity describes the circumstances under which the results of the research can be generalized


METHODOLOGY

Sample size
This is the number of subjects included in the study from a given population. Larger sample sizes tend to be more representative of the population

Randomization
Participants are randomly assigned to one of two or more interventions. Randomization minimizes the risk of confounding variables, reduces the risk of bias, and allows for examination of direct relationships. A major limitation of using an RCT is that it is impossible to obtain a true random sample of the population (2). This can make it difficult to generalize the results.

Concealed allocation
The person responsible for determining whether a subject was eligible for inclusion in the trial should be unaware of the group the subject is allocated (6). If allocation is not concealed, the decision about whether or not to include a person in a trial could be influenced by knowledge of whether the subject was to receive treatment or not. This could produce systematic biases in an otherwise random allocation (6).

Blinding
This is a method to prevent study participants, as well as those collecting and analyzing data from knowing who is in the intervention group and who is in the control group (1). When subjects are blinded, it is less likely that the results of the treatment are due to a placebo effect (6). Blinding assessors prevents their personal bias from affecting the results (6).

Baseline comparability
Baseline comparability involves a comparison of the baseline values of the groups (intervention and control). There should be statistically significant difference between groups. An appropriate randomization should ensure that groups are similar at baseline. This may provide an indication of potential bias arising by chance with random allocation (6). A significant difference between groups may indicate an issue with randomization procedures (6). 

Intervention
The intervention should be described in enough detail for reproducibility. An inadequate description decreases internal validity as it is unclear of the exact mechanism that led to the change in outcomes.

Adequate follow-up
The number of subjects who completed the trial to provide follow-up data for statistical analysis must be sufficient. The PEDro group states that data collected from a minimum of 85% of subjects increases internal validity (6). It is important that measurements of outcomes are made on all subjects who are randomized to groups. Subjects who are not followed up may differ systematically from those who are, and this potentially introduces bias. The magnitude of the potential bias increases with the proportion of subjects not followed up (6).

Intention-to-treat analysis
This is a strategy that ensures that all subjects allocated to either the treatment or control groups are analyzed together as representing that treatment arm, whether or not they received the prescribed treatment or completed the study (1). When patients are excluded from the analysis, the main rationale for randomization is defeated, leading to potential bias (6).

Between-group comparisons
This comparison is a statistical comparison of one group with another. It is performed to determine if the difference between groups is greater than can plausibly be attributed to chance (6).

Point estimates (effect size) and variability
A point estimate or effect size is a value that represents the most likely estimate of the true population (4). Some examples include the mean difference, regression coefficient, Cohen's d, correlation coefficient.

It is important to consider the variability of the effect size (point estimate). A few examples of variability include: the standard deviation, standard error of the mean, and a range of value. The standard deviation is an estimate of the degree of scatter (variability) of individual sample data points about the mean of a sample (3).

Study limitations
A description of the limitations of the study design and methodology allows for transparency as it should describe potential biases. This includes an explanation of possible errors in the internal and external validity.


REFERENCES

1. Akobeng A. Understanding randomized controlled trials. Arch Dis Child 2005;90:840-844.

2. Carter R, Lubinsky J, Domholdt E. Rehabiliation Research. 4th ed. 2010. Elsvier; St. Louis, Missouri.

3. Gaddis G, Gaddis M. Introduction to biostatistics: part 3, sensitivity, specificity, predictive value, and hypothesis testing. Ann Emerg Med 1990;19:145-151.

4. Nakagawa S, Cuthill I. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev 2007;82(4):591-605.

5. Pannucci C, Wilkins E. Identifying and avoiding bias in research. Plast Reconstr Surg 2010;126(2):619-625.

6. Physiotherapy Evidence Database. PEDro Scale (1999). http://www.pedro.org.au/english/downloads/pedro-scale/. Accessed on January 27, 2015.