Practical Significance of Meta-analysis Findings Lipsey
Applied topics:
Interpreting the Practical Significance of Meta-Analysis Findings
Mark Lipsey Co-Chair, The Campbell Collaboration Co-Editor-in-Chief, Campbell Systematic Reviews Director, Peabody Research Institute, Vanderbilt University, USA
The Campbell Collaboration
The problem
? The effect size statistics that constitute the direct findings of a metaanalysis often provide little insight into the nature, magnitude, or practical significance of the effects they represent.
? Practitioners, policymakers, and even researchers have difficulty knowing whether the effects are meaningful in an applied context.
? Example: The mean standardized mean difference effect size (Cohen's d or Hedges g) for the effects of educational interventions with middle school students on standardized reading tests is about . 15 and statistically significant.
? Seems small: Is .15 large enough to have practical significance for improving the reading skills of middle school students?
? Most important to recognize: There is no necessary relationship between the numerical magnitude of an effect size and the practical significance of the effect it represents!
The Campbell Collaboration
1
A widely used but inappropriate and misleading characterization of effect sizes
?Statistical effect sizes assessed by Cohen's small (.20), medium (.50) and large (.80) categories
? Impressionistic norms across a wide range of outcomes in social and behavioral research
? Almost never are these the appropriate norms for the particular outcomes of a particular intervention
?Comparing an obtained mean effect size with norms can be informative, but those norms must be appropriate to the context, intervention, nature of the outcomes, etc. [more on this later]
The Campbell Collaboration
Two approaches to review here
1. Descriptive representations of intervention effect sizes: ? Translations of effect sizes into forms that are more readily interpreted. ? Supports better intuitions about the practical significance of the effect size.
2. Direct assessment of practical significance: ? Assessing statistical effect sizes in relationship to criteria that have recognized practical value in the context of application. ? Requires that appropriate criteria be used; different criteria may yield different conclusions.
The Campbell Collaboration
2
Useful Descriptive Representations of Intervention Effect Sizes
The Campbell Collaboration
Back translation to an original metric
? Useful when the original metric is readily interpretable; not so useful when it is in arbitrary units.
? Example: Mean Phi coefficient for effects of intervention on the reoffense rates of juvenile offenders < .20 allegedly trivial.
? Computation of Phi Coefficient as an effect size:
Don't
Reoffend Reoffend
(failure) (success)
Tx
a = p b = 1-p a+b=1
Ct
c = q d = 1-q c+d=1
a+c= p+q
b+d= (1-p)+ (1-q)
Phi = (ad-bc)/SQRT((a+b)(c+d)(a+c)(b+d))
The Campbell Collaboration
3
Back translation to original metric: Phi coefficient example
? Mean reoffense rate for the control groups in the studies was .50. ? Some algebra (or trial & error in a spreadsheet) yields the reoffense
rate of the average treatment group required to produce Phi = .20 ? [Note: Similar procedure would work for odds ratio ES as well]
Don't
Reoffend Reoffend
(failure) (success)
Tx
.30
.70 1.00
Ct
.50
.50 1.00
.80
1.20
Phi = .20
Phi = .20 thus means an average .20 reduction in the reoffense rate from a .50 average baseline value; That is, a 40% decrease in the reoffense rate.
Hardly trivial!
The Campbell Collaboration
Back translation to original metric: Standardized test example
? Suppose the mean standardized mean difference effect size for intervention effects on vocabulary tests is .30
? The most frequently used measure of vocabulary in the contributing studies was the Peabody Picture Vocabulary Test (PPVT)
? The PPVT has a normed standard score of 100 with a standard deviation of 15. Differences in standard scores are readily understood by researchers and practitioners familiar with standardized tests
? The control groups in the studies using the PPVT had a mean standard score of 87.
? How much improvement in the PPVT standard score is represented by an effect size of .30?
The Campbell Collaboration
4
Back translation to original metric: PPVT
The Campbell Collaboration
Intervention effect sizes represented as percentiles on the normal distribution
Percentile values on the control distribution of the intervention effect in standard deviation units
The Campbell Collaboration
5
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.