ࡱ> jlghiq Kbjbjt+t+ ЉAAYW]$PD@$ ( [D$$$$$$$$$%'$% ^ $   % V  $$7$$$ $?|ceN$Its the Effect Size, Stupid What effect size is and why it is important Paper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002 Robert Coe School of Education, University of Durham, Leazes Road, Durham DH1 1TA Tel 0191 374 4504; Fax 0191 374 1900; Email r.j.coe@dur.ac.uk Abstract Effect size is a simple way of quantifying the difference between two groups that has many advantages over the use of tests of statistical significance alone. Effect size emphasises the size of the difference rather than confounding this with sample size. However, primary reports rarely mention effect sizes and few textbooks, research methods courses or computer packages address the concept. This paper provides an explication of what an effect size is, how it is calculated and how it can be interpreted. The relationship between effect size and statistical significance is discussed and the use of confidence intervals for the latter outlined. Some advantages and dangers of using effect sizes in meta-analysis are discussed and other problems with the use of effect sizes are raised. A number of alternative measures of effect size are described. Finally, advice on the use of effect sizes is summarised. During 1992 Bill Clinton and George Bush Snr. were fighting for the presidency of the United States. Clinton was barely holding on to his place in the opinion polls. Bush was pushing ahead drawing his on his stature as an experienced world leader. James Carville, one of Clinton's top advisers decided that their push for presidency needed focusing. Drawing on the research he had conducted he came up with a simple focus for their campaign. Every opportunity he had, Carville wrote four words - 'It's the economy, stupid' - on a whiteboard for Bill Clinton to see every time he went out to speak. Effect size is simply a way of quantifying the size of the difference between two groups. It is easy to calculate, readily understood and can be applied to any measured outcome in Education or Social Science. It is particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison. It allows us to move beyond the simplistic, Does it work or not? to the far more sophisticated, How well does it work in a range of contexts? Moreover, by placing the emphasis on the most important aspect of an intervention the size of the effect rather than its statistical significance (which conflates effect size and sample size), it promotes a more scientific approach to the accumulation of knowledge. For these reasons, effect size is an important tool in reporting and interpreting effectiveness. The routine use of effect sizes, however, has generally been limited to meta-analysis for combining and comparing estimates from different studies and is all too rare in original reports of educational research (Keselman et al., 1998). This is despite the fact that measures of effect size have been available for at least 60 years (Huberty, 2002), and the American Psychological Association has been officially encouraging authors to report effect sizes since 1994 but with limited success (Wilkinson et al., 1999). Formulae for the calculation of effect sizes do not appear in most statistics text books (other than those devoted to meta-analysis), are not featured in many statistics computer packages and are seldom taught in standard research methods courses. For these reasons, even the researcher who is convinced by the wisdom of using measures of effect size, and is not afraid to confront the orthodoxy of conventional practice, may find that it is quite hard to know exactly how to do so. The following guide is written for non-statisticians, though inevitably some equations and technical language have been used. It describes what effect size is, what it means, how it can be used and some potential problems associated with using it. 1. Why do we need effect size? Consider an experiment conducted by Dowson (2000) to investigate time of day effects on learning: do children learn better in the morning or afternoon? A group of 38 children were included in the experiment. Half were randomly allocated to listen to a story and answer questions about it (on tape) at 9am, the other half to hear exactly the same story and answer the same questions at 3pm. Their comprehension was measured by the number of questions answered correctly out of 20. The average score was 15.2 for the morning group, 17.9 for the afternoon group: a difference of 2.7. But how big a difference is this? If the outcome were measured on a familiar scale, such as GCSE grades, interpreting the difference would not be a problem. If the average difference were, say, half a grade, most people would have a fair idea of the educational significance of the effect of reading a story at different times of day. However, in many experiments there is no familiar scale available on which to record the outcomes. The experimenter often has to invent a scale or to use (or adapt) an already existing one but generally not one whose interpretation will be familiar to most people.   (a) (b) Figure  SEQ Figure \* ARABIC 1 One way to get over this problem is to use the amount of variation in scores to contextualise the difference. If there were no overlap at all and every single person in the afternoon group had done better on the test than everyone in the morning group, then this would seem like a very substantial difference. On the other hand, if the spread of scores were large and the overlap much bigger than the difference between the groups, then the effect might seem less significant. Because we have an idea of the amount of variation found within a group, we can use this as a yardstick against which to compare the difference. This idea is quantified in the calculation of the effect size. The concept is illustrated in Figure 1, which shows two possible ways the difference might vary in relation to the overlap. If the difference were as in graph (a) it would be very significant; in graph (b), on the other hand, the difference might hardly be noticeable. 2. How is it calculated? The effect size is just the standardised mean difference between the two groups. In other words:  Effect Size =  Equation  SEQ Equation \* ARABIC 1 If it is not obvious which of two groups is the experimental (i.e. the one which was given the new treatment being tested) and which the control (the one given the standard treatment or no treatment for comparison), the difference can still be calculated. In this case, the effect size simply measures the difference between them, so it is important in quoting the effect size to say which way round the calculation was done. The standard deviation is a measure of the spread of a set of values. Here it refers to the standard deviation of the population from which the different treatment groups were taken. In practice, however, this is almost never known, so it must be estimated either from the standard deviation of the control group, or from a pooled value from both groups (see question 7, below, for more discussion of this). In Dowsons time-of-day effects experiment, the standard deviation (SD) = 3.3, so the effect size was (17.9 15.2)/3.3 = 0.8. 3. How can effect sizes be interpreted? One feature of an effect size is that it can be directly converted into statements about the overlap between the two samples in terms of a comparison of percentiles. An effect size is exactly equivalent to a Z-score of a standard Normal distribution. For example, an effect size of 0.8 means that the score of the average person in the experimental group is 0.8 standard deviations above the average person in the control group, and hence exceeds the scores of 79% of the control group. With the two groups of 19 in the time-of-day effects experiment, the average person in the afternoon group (i.e. the one who would have been ranked 10th in the group) would have scored about the same as the 4th highest person in the morning group. Visualising these two individuals can give quite a graphic interpretation of the difference between the two effects.  REF _Ref19356035 \h Table I shows conversions of effect sizes (column 1) to percentiles (column 2) and the equivalent change in rank order for a group of 25 (column 3). For example, for an effect-size of 0.6, the value of 73% indicates that the average person in the experimental group would score higher than 73% of a control group that was initially equivalent. If the group consisted of 25 people, this is the same as saying that the average person (i.e. ranked 13th in the group) would now be on a par with the person ranked 7th in the control group. Notice that an effect-size of 1.6 would raise the average person to be level with the top ranked individual in the control group, so effect sizes larger than this are illustrated in terms of the top person in a larger group. For example, an effect size of 3.0 would bring the average person in a group of 740 level with the previously top person in the group. Table  SEQ Table \* ROMAN I: Interpretations of effect sizes Effect SizePercentage of control group who would be below average person in experimental groupRank of person in a control group of 25 who would be equivalent to the average person in experimental groupProbability that you could guess which group a person was in from knowledge of their score.Equivalent correlation, r (=Difference in percentage successful in each of the two groups, BESD)Probability that person from experimental group will be higher than person from control, if both chosen at random (=CLES)0.050%13th0.500.000.500.154%12th0.520.050.530.258%11th0.540.100.560.362%10th0.560.150.580.466%9th0.580.200.610.569%8th0.600.240.640.673%7th0.620.290.660.776%6th0.640.330.690.879%6th0.660.370.710.982%5th0.670.410.741.084%4th0.690.450.761.288%3rd0.730.510.801.492%2nd0.760.570.841.695%1st0.790.620.871.896%1st0.820.670.902.098%1st (or 1st out of 44)0.840.710.922.599%1st (or 1st out of 160)0.890.780.963.099.9%1st (or 1st out of 740)0.930.830.98 Another way to conceptualise the overlap is in terms of the probability that one could guess which group a person came from, based only on their test score or whatever value was being compared. If the effect size were 0 (i.e. the two groups were the same) then the probability of a correct guess would be exactly a half or 0.50. With a difference between the two groups equivalent to an effect size of 0.3, there is still plenty of overlap, and the probability of correctly identifying the groups rises only slightly to 0.56. With an effect size of 1, the probability is now 0.69, just over a two-thirds chance. These probabilities are shown in the fourth column of  REF _Ref19356035 \h Table I. It is clear that the overlap between experimental and control groups is substantial (and therefore the probability is still close to 0.5), even when the effect-size is quite large. A slightly different way to interpret effect sizes makes use of an equivalence between the standardised mean difference (d) and the correlation coefficient, r. If group membership is coded with a dummy variable (e.g. denoting the control group by 0 and the experimental group by 1) and the correlation between this variable and the outcome measure calculated, a value of r can be derived. By making some additional assumptions, one can readily convert d into r in general, using the equation r2=d2/(4+d2) (see Cohen, 1969, pp20-22 for other formulae and conversion table). Rosenthal and Rubin (1982) take advantage of an interesting property of r to suggest a further interpretation, which they call the binomial effect size display (BESD). If the outcome measure is reduced to a simple dichotomy (for example, whether a score is above or below a particular value such as the median, which could be thought of as success or failure), r can be interpreted as the difference in the proportions in each category. For example, an effect size of 0.2 indicates a difference of 0.10 in these proportions, as would be the case if 45% of the control group and 55% of the treatment group had reached some threshold of success. Note, however, that if the overall proportion successful is not close to 50%, this interpretation can be somewhat misleading (Strahan, 1991; McGraw, 1991). The values for the BESD are shown in column 5. Finally, McGraw and Wong (1992) have suggested a Common Language Effect Size (CLES) statistic, which they argue is readily understood by non-statisticians (shown in column 6 of Table I). This is the probability that a score sampled at random from one distribution will be greater than a score sampled from another. They give the example of the heights of young adult males and females, which differ by an effect size of about 2, and translate this difference to a CLES of 0.92. In other words in 92 out of 100 blind dates among young adults, the male will be taller than the female (p361). It should be noted that the values in Table I depend on the assumption of a Normal distribution. The interpretation of effect sizes in terms of percentiles is very sensitive to violations of this assumption (see question 7, below). Another way to interpret effect sizes is to compare them to the effect sizes of differences that are familiar. For example, Cohen (1969, p23) describes an effect size of 0.2 as small and gives to illustrate it the example that the difference between the heights of 15 year old and 16 year old girls in the US corresponds to an effect of this size. An effect size of 0.5 is described as medium and is large enough to be visible to the naked eye. A 0.5 effect size corresponds to the difference between the heights of 14 year old and 18 year old girls. Cohen describes an effect size of 0.8 as grossly perceptible and therefore large and equates it to the difference between the heights of 13 year old and 18 year old girls. As a further example he states that the difference in IQ between holders of the Ph.D. degree and typical college freshmen is comparable to an effect size of 0.8. Cohen does acknowledge the danger of using terms like small, medium and large out of context. Glass et al. (1981, p104) are particularly critical of this approach, arguing that the effectiveness of a particular intervention can only be interpreted in relation to other interventions that seek to produce the same effect. They also point out that the practical importance of an effect depends entirely on its relative costs and benefits. In education, if it could be shown that making a small and inexpensive change would raise academic achievement by an effect size of even as little as 0.1, then this could be a very significant improvement, particularly if the improvement applied uniformly to all students, and even more so if the effect were cumulative over time. Table  SEQ Table \* ROMAN II: Examples of average effect sizes from research InterventionOutcomeEffect SizeSourceReducing class size from 23 to 15Students test performance in reading0.30Finn and Achilles, (1990)Students test performance in maths0.32Small (<30) vs large class sizeAttitudes of students0.47Smith and Glass (1980)Attitudes of teachers1.03Setting students vs mixed ability groupingStudent achievement (overall)0.00Mosteller, Light and Sachs (1996)Student achievement (for high-achievers)0.08Student achievement (for low-achievers)-0.06Open (child-centred) vs traditional classroom organisationStudent achievement-0.06Giaconia and Hedges (1982)Student attitudes to school0.17Mainstreaming vs special education (for primary age, disabled students)Achievement0.44Wang and Baker (1986)Practice test takingTest scores0.32Kulik, Bangert and Kulik (1984)Inquiry-based vs traditional science curriculumAchievement0.30Shymansky, Hedges and Woodworth (1990)Therapy for test-anxiety (for anxious students)Test performance0.42Hembree (1988)Feedback to teachers about student performance (students with IEPs)Student achievement0.70Fuchs and Fuchs (1986)Peer tutoringAchievement of tutees0.40Cohen, Kulik and Kulik, (1982)Achievement of tutors0.33Individualised instructionAchievement0.10Bangert, Kulik and Kulik (1983)Computer assisted instruction (CAI)Achievement (all studies)0.24Fletcher-Flinn and Gravatt (1995)Achievement (in well controlled studies)0.02Additive-free dietChildren's hyperactivity0.02Kavale and Forness (1983)Relaxation trainingMedical symptoms0.52Hyman et al. (1989)Targeted interventions for at-risk studentsAchievement0.63Slavin and Madden (1989)School-based substance abuse educationSubstance use0.12Bangert-Drowns (1988)Treatment programmes for juvenile delinquentsDelinquency0.17Lipsey (1992) Glass et al. (1981, p102) give the example that an effect size of 1 corresponds to the difference of about a year of schooling on the performance in achievement tests of pupils in elementary (i.e. primary) schools. However, an analysis of a standard spelling test used in Britain (Vincent and Crumpler, 1997) suggests that the increase in a spelling age from 11 to 12 corresponds to an effect size of about 0.3, but seems to vary according to the particular test used. In England, the distribution of GCSE grades in compulsory subjects (i.e. Maths and English) have standard deviations of between 1.5 1.8 grades, so an improvement of one GCSE grade represents an effect size of 0.5 0.7. In the context of secondary schools therefore, introducing a change in practice whose effect size was known to be 0.6 would result in an improvement of about a GCSE grade for each pupil in each subject. For a school in which 50% of pupils were previously gaining five or more A* C grades, this percentage (other things being equal, and assuming that the effect applied equally across the whole curriculum) would rise to 73%. Even Cohens small effect of 0.2 would produce an increase from 50% to 58% a difference that most schools would probably categorise as quite substantial. Olejnik and Algina (2000) give a similar example based on the Iowa Test of Basic Skills Finally, the interpretation of effect sizes can be greatly helped by a few examples from existing research. Table II lists a selection of these, many of which are taken from Lipsey and Wilson (1993). The examples cited are given for illustration of the use of effect size measures; they are not intended to be the definitive judgement on the relative efficacy of different interventions. In interpreting them, therefore, one should bear in mind that most of the meta-analyses from which they are derived can be (and often have been) criticised for a variety of weaknesses, that the range of circumstances in which the effects have been found may be limited, and that the effect size quoted is an average which is often based on quite widely differing values. It seems to be a feature of educational interventions that very few of them have effects that would be described in Cohens classification as anything other than small. This appears particularly so for effects on student achievement. No doubt this is partly a result of the wide variation found in the population as a whole, against which the measure of effect size is calculated. One might also speculate that achievement is harder to influence than other outcomes, perhaps because most schools are already using optimal strategies, or because different strategies are likely to be effective in different situations a complexity that is not well captured by a single average effect size. 4. What is the relationship between effect size and significance? Effect size quantifies the size of the difference between two groups, and may therefore be said to be a true measure of the significance of the difference. If, for example, the results of Dowsons time of day effects experiment were found to apply generally, we might ask the question: How much difference would it make to childrens learning if they were taught a particular topic in the afternoon instead of the morning? The best answer we could give to this would be in terms of the effect size. However, in statistics the word significance is often used to mean statistical significance, which is the likelihood that the difference between the two groups could just be an accident of sampling. If you take two samples from the same population there will always be a difference between them. The statistical significance is usually calculated as a p-value, the probability that a difference of at least the same size would have arisen by chance, even if there really were no difference between the two populations. For differences between the means of two groups, this p-value would normally be calculated from a t-test. By convention, if p < 0.05 (i.e. below 5%), the difference is taken to be large enough to be significant; if not, then it is not significant. There are a number of problems with using significance tests in this way (see, for example Cohen, 1994; Harlow et al., 1997; Thompson, 1999). The main one is that the p-value depends essentially on two things: the size of the effect and the size of the sample. One would get a significant result either if the effect were very big (despite having only a small sample) or if the sample were very big (even if the actual effect size were tiny). It is important to know the statistical significance of a result, since without it there is a danger of drawing firm conclusions from studies where the sample is too small to justify such confidence. However, statistical significance does not tell you the most important thing: the size of the effect. One way to overcome this confusion is to report the effect size, together with an estimate of its likely margin for error or confidence interval. 5. What is the margin for error in estimating effect sizes? Clearly, if an effect size is calculated from a very large sample it is likely to be more accurate than one calculated from a small sample. This margin for error can be quantified using the idea of a confidence interval, which provides the same information as is usually contained in a significance test: using a 95% confidence interval is equivalent to taking a 5% significance level. To calculate a 95% confidence interval, you assume that the value you got (e.g. the effect size estimate of 0.8) is the true value, but calculate the amount of variation in this estimate you would get if you repeatedly took new samples of the same size (i.e. different samples of 38 children). For every 100 of these hypothetical new samples, by definition, 95 would give estimates of the effect size within the 95% confidence interval. If this confidence interval includes zero, then that is the same as saying that the result is not statistically significant. If, on the other hand, zero is outside the range, then it is statistically significant at the 5% level. Using a confidence interval is a better way of conveying this information since it keeps the emphasis on the effect size which is the important information rather than the p-value. A formula for calculating the confidence interval for an effect size is given by Hedges and Olkin (1985, p86). If the effect size estimate from the sample is d, then it is Normally distributed, with standard deviation: Equation  SEQ Equation \* ARABIC 2 (Where NE and NC are the numbers in the experimental and control groups, respectively.) Hence a 95% confidence interval for d would be from d 1.96 ( ([d] to d + 1.96 ( ([d] Equation  SEQ Equation \* ARABIC 3 To use the figures from the time-of-day experiment again, NE = NC = 19 and d=0.8, so ([d] = ((0.105 + 0.008) = 0.34. Hence the 95% confidence interval is [0.14,1.46]. This would normally be interpreted (despite the fact that such an interpretation is not strictly justified see Oakes, 1986 for an enlightening discussion of this) as meaning that the true effect of time-of-day is very likely to be between 0.14 and 1.46. In other words, it is almost certainly positive (i.e. afternoon is better than morning) and the difference may well be quite large. 6. How can knowledge about effect sizes be combined? One of the main advantages of using effect size is that when a particular experiment has been replicated, the different effect size estimates from each study can easily be combined to give an overall best estimate of the size of the effect. This process of synthesising experimental results into a single effect size estimate is known as meta-analysis. It was developed in its current form by an educational statistician, Gene Glass (See Glass et al., 1981) though the roots of meta-analysis can be traced a good deal further back(see Lepper et al., 1999), and is now widely used, not only in education, but in medicine and throughout the social sciences. A brief and accessible introduction to the idea of meta-analysis can be found in Fitz-Gibbon (1984). Meta-analysis, however, can do much more than simply produce an overall average effect size, important though this often is. If, for a particular intervention, some studies produced large effects, and some small effects, it would be of limited value simply to combine them together and say that the average effect was medium. Much more useful would be to examine the original studies for any differences between those with large and small effects and to try to understand what factors might account for the difference. The best meta-analysis, therefore, involves seeking relationships between effect sizes and characteristics of the intervention, the context and study design in which they were found (Rubin, 1992; see also Lepper et al. (1999) for a discussion of the problems that can be created by failing to do this, and some other limitations of the applicability of meta-analysis). The importance of replication in gaining evidence about what works cannot be overstressed. In Dowsons time-of-day experiment the effect was found to be large enough to be statistically and educationally significant. Because we know that the pupils were allocated randomly to each group, we can be confident that chance initial differences between the two groups are very unlikely to account for the difference in the outcomes. Furthermore, the use of a pre-test of both groups before the intervention makes this even less likely. However, we cannot rule out the possibility that the difference arose from some characteristic peculiar to the children in this particular experiment. For example, if none of them had had any breakfast that day, this might account for the poor performance of the morning group. However, the result would then presumably not generalise to the wider population of school students, most of whom would have had some breakfast. Alternatively, the effect might depend on the age of the students. Dowsons students were aged 7 or 8; it is quite possible that the effect could be diminished or reversed with older (or younger) students. This illustrates the danger of implementing policy on the basis of a single experiment. Confidence in the generality of a result can only follow widespread replication. An important consequence of the capacity of meta-analysis to combine results is that even small studies can make a significant contribution to knowledge. The kind of experiment that can be done by a single teacher in a school might involve a total of fewer than 30 students. Unless the effect is huge, a study of this size is most unlikely to get a statistically significant result. According to conventional statistical wisdom, therefore, the experiment is not worth doing. However, if the results of several such experiments are combined using meta-analysis, the overall result is likely to be highly statistically significant. Moreover, it will have the important strengths of being derived from a range of contexts (thus increasing confidence in its generality) and from real-life working practice (thereby making it more likely that the policy is feasible and can be implemented authentically). One final caveat should be made here about the danger of combining incommensurable results. Given two (or more) numbers, one can always calculate an average. However, if they are effect sizes from experiments that differ significantly in terms of the outcome measures used, then the result may be totally meaningless. It can be very tempting, once effect sizes have been calculated, to treat them as all the same and lose sight of their origins. Certainly, there are plenty of examples of meta-analyses in which the juxtaposition of effect sizes is somewhat questionable. In comparing (or combining) effect sizes, one should therefore consider carefully whether they relate to the same outcomes. This advice applies not only to meta-analysis, but to any other comparison of effect sizes. Moreover, because of the sensitivity of effect size estimates to reliability and range restriction (see below), one should also consider whether those outcome measures are derived from the same (or sufficiently similar) instruments and the same (or sufficiently similar) populations. It is also important to compare only like with like in terms of the treatments used to create the differences being measured. In the education literature, the same name is often given to interventions that are actually very different, for example, if they are operationalised differently, or if they are simply not well enough defined for it to be clear whether they are the same or not. It could also be that different studies have used the same well-defined and operationalised treatments, but the actual implementation differed, or that the same treatment may have had different levels of intensity in different studies. In any of these cases, it makes no sense to average out their effects. 7. What other factors can influence effect size? Although effect size is a simple and readily interpreted measure of effectiveness, it can also be sensitive to a number of spurious influences, so some care needs to be taken in its use. Some of these issues are outlined here. Which standard deviation? The first problem is the issue of which standard deviation to use. Ideally, the control group will provide the best estimate of standard deviation, since it consists of a representative group of the population who have not been affected by the experimental intervention. However, unless the control group is very large, the estimate of the true population standard deviation derived from only the control group is likely to be appreciably less accurate than an estimate derived from both the control and experimental groups. Moreover, in studies where there is not a true control group (for example the time-of-day effects experiment) then it may be an arbitrary decision which groups standard deviation to use, and it will often make an appreciable difference to the estimate of effect size. For these reasons, it is often better to use a pooled estimate of standard deviation. The pooled estimate is essentially an average of the standard deviations of the experimental and control groups ( REF _Ref4486165 Equation 4). Note that this is not the same as the standard deviation of all the values in both groups pooled together. If, for example each group had a low standard deviation but the two means were substantially different, the true pooled estimate (as calculated by  REF _Ref4486165 Equation 4) would be much lower than the value obtained by pooling all the values together and calculating the standard deviation. The implications of choices about which standard deviation to use are discussed by Olejnik and Algina (2000).  Equation  SEQ Equation \* ARABIC 4 (Where NE and NC are the numbers in the experimental and control groups, respectively, and SDE and SDC are their standard deviations.) The use of a pooled estimate of standard deviation depends on the assumption that the two calculated standard deviations are estimates of the same population value. In other words, that the experimental and control group standard deviations differ only as a result of sampling variation. Where this assumption cannot be made (either because there is some reason to believe that the two standard deviations are likely to be systematically different, or if the actual measured values are very different), then a pooled estimate should not be used. In the example of Dowsons time of day experiment, the standard deviations for the morning and afternoon groups were 4.12 and 2.10 respectively. With NE = NC = 19, Equation 2 therefore gives SDpooled as 3.3, which was the value used in  REF _Ref468180341 \h Equation 1 to give an effect size of 0.8. However, the difference between the two standard deviations seems quite large in this case. Given that the afternoon group mean was 17.9 out of 20, it seems likely that its standard deviation may have been reduced by a ceiling effect i.e. the spread of scores was limited by the maximum available mark of 20. In this case therefore, it might be more appropriate to use the morning groups standard deviation as the best estimate. Doing this will reduce the effect size to 0.7, and it then becomes a somewhat arbitrary decision which value of the effect size to use. A general rule of thumb in statistics when two valid methods give different answers is: If in doubt, cite both. Corrections for bias Although using the pooled standard deviation to calculate the effect size generally gives a better estimate than the control group SD, it is still unfortunately slightly biased and in general gives a value slightly larger than the true population value (Hedges and Olkin, 1985). Hedges and Olkin (1985, p80) give a formula which provides an approximate correction to this bias. In Dowsons experiment with 38 values, the correction factor will be 0.98, so it makes very little difference, reducing the effect size estimate from 0.82 to 0.80. Given the likely accuracy of the figures on which this is based, it is probably only worth quoting one decimal place, so the figure of 0.8 stands. In fact, the correction only becomes significant for small samples, in which the accuracy is anyway much less. It is therefore hardly worth worrying about it in primary reports of empirical results. However, in meta-analysis, where results from primary studies are combined, the correction is important, since without it this bias would be accumulated. Restricted range Suppose the time-of-day effects experiment were to be repeated, once with the top set in a highly selective school and again with a mixed-ability group in a comprehensive. If students were allocated to morning and afternoon groups at random, the respective differences between them might be the same in each case; both means in the selective school might be higher, but the difference between the two groups could be the same as the difference in the comprehensive. However, it is unlikely that the standard deviations would be the same. The spread of scores found within the highly selected group would be much less than that in a true cross-section of the population, as for example in the mixed-ability comprehensive class. This, of course, would have a substantial impact on the calculation of the effect size. With the highly restricted range found in the selective school, the effect size would be much larger than that found in the comprehensive. Ideally, in calculating effect-size one should use the standard deviation of the full population, in order to make comparisons fair. However, there will be many cases in which unrestricted values are not available, either in practice or in principle. For example, in considering the effect of an intervention with university students, or with pupils with reading difficulties, one must remember that these are restricted populations. In reporting the effect-size, one should draw attention to this fact; if the amount of restriction can be quantified it may be possible to make allowance for it. Any comparison with effect sizes calculated from a full-range population must be made with great caution, if at all. Non-Normal distributions The interpretations of effect-sizes given in Table I depend on the assumption that both control and experimental groups have a Normal distribution, i.e. the familiar bell-shaped curve, shown, for example, in Figure 1. Needless to say, if this assumption is not true then the interpretation may be altered, and in particular, it may be difficult to make a fair comparison between an effect-size based on Normal distributions and one based on non-Normal distributions.  Figure  SEQ Figure \* ARABIC 2: Comparison of Normal and non-Normal distributions An illustration of this is given in Figure 2, which shows the frequency curves for two distributions, one of them Normal, the other a contaminated normal distribution (Wilcox, 1998), which is similar in shape, but with somewhat fatter extremes. In fact, the latter does look just a little more spread-out than the Normal distribution, but its standard deviation is actually over three times as big. The consequence of this in terms of effect-size differences is shown in Figure 3. Both graphs show distributions that differ by an effect-size equal to 1, but the appearance of the effect-size difference from the graphs is rather dissimilar. In graph (b), the separation between experimental and control groups seems much larger, yet the effect-size is actually the same as for the Normal distributions plotted in graph (a). In terms of the amount of overlap, in graph (b) 97% of the 'experimental' group are above the control group mean, compared with the value of 84% for the Normal distribution of graph (a) (as given in Table I). This is quite a substantial difference and illustrates the danger of using the values in Table I when the distribution is not known to be Normal.   (a) (b) Figure  SEQ Figure \* ARABIC 3: Normal and non-Normal distributions with effect-size = 1 Measurement reliability A third factor that can spuriously affect an effect-size is the reliability of the measurement on which it is based. According to classical measurement theory, any measure of a particular outcome may be considered to consist of the true underlying value, together with a component of error. The problem is that the amount of variation in measured scores for a particular sample (i.e. its standard deviation) will depend on both the variation in underlying scores and the amount of error in their measurement. To give an example, imagine the time-of-day experiment were conducted twice with two (hypothetically) identical samples of students. In the first version the test used to assess their comprehension consisted of just 10 items and their scores were converted into a percentage. In the second version a test with 50 items was used, and again converted to a percentage. The two tests were of equal difficulty and the actual effect of the difference in time-of-day was the same in each case, so the respective mean percentages of the morning and afternoon groups were the same for both versions. However, it is almost always the case that a longer test will be more reliable, and hence the standard deviation of the percentages on the 50 item test will be lower than the standard deviation for the 10 item test. Thus, although the true effect was the same, the calculated effect sizes will be different. In interpreting an effect-size, it is therefore important to know the reliability of the measurement from which it was calculated. This is one reason why the reliability of any outcome measure used should be reported. It is theoretically possible to make a correction for unreliability (sometimes called attenuation), which gives an estimate of what the effect size would have been, had the reliability of the test been perfect. However, in practice the effect of this is rather alarming, since the worse the test was, the more you increase the estimate of the effect size. Moreover, estimates of reliability are dependent on the particular population in which the test was used, and are themselves anyway subject to sampling error. For further discussion of the impact of reliability on effect sizes, see Baugh (2002). 8. Are there alternative measures of effect-size? A number of statistics are sometimes proposed as alternative measures of effect size, other than the standardised mean difference. Some of these will be considered here. Proportion of variance accounted for If the correlation between two variables is r, the square of this value (often denoted with a capital letter: R2) represents the proportion of the variance in each that is accounted for by the other. In other words, this is the proportion by which the variance of the outcome measure is reduced when it is replaced by the variance of the residuals from a regression equation. This idea can be extended to multiple regression (where it represents the proportion of the variance accounted for by all the independent variables together) and has close analogies in ANOVA (where it is usually called eta-squared, (2). The calculation of r (and hence R2 ) for the kind of experimental situation we have been considering has already been referred to above. Because R2 has this ready convertibility, it (or alternative measures of variance accounted for) is sometimes advocated as a universal measure of effect size (e.g. Thompson, 1999). One disadvantage of such an approach is that effect size measures based on variance accounted for suffer from a number of technical limitations, such as sensitivity to violation of assumptions (heterogeneity of variance, balanced designs) and their standard errors can be large (Olejnik and Algina, 2000). They are also generally more statistically complex and hence perhaps less easily understood. Further, they are non-directional; two studies with precisely opposite results would report exactly the same variance accounted for. However, there is a more fundamental objection to the use of what is essentially a measure of association to indicate the strength of an effect. Expressing different measures in terms of the same statistic can hide important differences between them; in fact, these different effect sizes are fundamentally different, and should not be confused. The crucial difference between an effect size calculated from an experiment and one calculated from a correlation is in the causal nature of the claim that is being made for it. Moreover, the word effect has an inherent implication of causality: talking about the effect of A on B does suggest a causal relationship rather than just an association. Unfortunately, however, the word effect is often used when no explicit causal claim is being made, but its implication is sometimes allowed to float in and out of the meaning, taking advantage of the ambiguity to suggest a subliminal causal link where none is really justified. This kind of confusion is so widespread in education that it is recommended here that the word effect (and therefore effect size) should not be used unless a deliberate and explicit causal claim is being made. When no such claim is being made, we may talk about the variance accounted for (R2) or the strength of association (r), or simply and perhaps most informatively just cite the regression coefficient (Tukey, 1969). If a causal claim is being made it should be explicit and justification provided. Fitz-Gibbon (2002) has recommended an alternative approach to this problem. She has suggested a system of nomenclature for different kinds of effect sizes that clearly distinguishes between effect sizes derived from, for example, randomised-controlled, quasi-experimental and correlational studies. Other measures of effect size It has been shown that the interpretation of the standardised mean difference measure of effect size is very sensitive to violations of the assumption of normality. For this reason, a number of more robust (non-parametric) alternatives have been suggested. An example of these is given by Cliff (1993). There are also effect size measures for multivariate outcomes. A detailed explanation can be found in Olejnik and Algina (2000). Finally, a method for calculating effect sizes within multilevel models has been proposed by Tymms et al. (1997). Good summaries of many of the different kinds of effect size measures that can be used and the relationships among them can be found in Snyder and Lawson (1993), Rosenthal (1994) and Kirk (1996). Finally, a common effect size measure widely used in medicine is the odds ratio. This is appropriate where an outcome is dichotomous: success or failure, a patient survives or does not. Explanations of the odds ratio can be found in a number of medical statistics texts, including Altman (1991), and in Fleiss (1994). Conclusions Advice on the use of effect-sizes can be summarised as follows: Effect size is a standardised, scale-free measure of the relative size of the effect of an intervention. It is particularly useful for quantifying effects measured on unfamiliar or arbitrary scales and for comparing the relative sizes of effects from different studies. Interpretation of effect-size generally depends on the assumptions that control and experimental group values are Normally distributed and have the same standard deviations. Effect sizes can be interpreted in terms of the percentiles or ranks at which two distributions overlap, in terms of the likelihood of identifying the source of a value, or with reference to known effects or outcomes. Use of an effect size with a confidence interval conveys the same information as a test of statistical significance, but with the emphasis on the significance of the effect, rather than the sample size. Effect sizes (with confidence intervals) should be calculated and reported in primary studies as well as in meta-analyses. Interpretation of standardised effect sizes can be problematic when a sample has restricted range or does not come from a Normal distribution, or if the measurement from which it was derived has unknown reliability. The use of an unstandardised mean difference (i.e. the raw difference between the two groups, together with a confidence interval) may be preferable when: - the outcome is measured on a familiar scale - the sample has a restricted range - the parent population is significantly non-Normal - control and experimental groups have appreciably different standard deviations - the outcome measure has very low or unknown reliability Care must be taken in comparing or aggregating effect sizes based on different outcomes, different operationalisations of the same outcome, different treatments, or levels of the same treatment, or measures derived from different populations. The word effect conveys an implication of causality, and the expression effect size should therefore not be used unless this implication is intended and can be justified. References Altman, D.G. (1991) Practical Statistics for Medical Research. London: Chapman and Hall. Bangert, R.L., Kulik, J.A. and Kulik, C.C. (1983) Individualised systems of instruction in secondary schools. Review of Educational Research, 53, 143-158. Bangert-Drowns, R.L. (1988) The effects of school-based substance abuse education: a meta-analysis. Journal of Drug Education, 18, 3, 243-65. Baugh, F. (2002) Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement, 62, 2, 254-263. Cliff, N. (1993) Dominance Statistics ordinal analyses to answer ordinal questions Psychological Bulletin, 114, 3. 494-509. Cohen, J. (1969) Statistical Power Analysis for the Behavioral Sciences. NY: Academic Press. Cohen, J. (1994) The Earth is Round (p<.05). American Psychologist, 49, 997-1003. Cohen, P.A., Kulik, J.A. and Kulik, C.C. (1982) Educational outcomes of tutoring: a meta-analysis of findings. American Educational Research Journal, 19, 237-248. Dowson V. (2000) Time of day effects in school-children's immediate and delayed recall of meaningful material. TERSE Report http://www.cem.dur.ac.uk/ebeuk/research/terse/library.htm Finn, J.D. and Achilles, C.M. (1990) Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27, 557-577. Fitz-Gibbon C.T. (1984) Meta-analysis: an explication. British Educational Research Journal, 10, 2, 135-144. Fitz-Gibbon C.T. (2002) A Typology of Indicators for an Evaluation-Feedback Approach in A.J.Visscher and R. Coe (Eds.) School Improvement Through Performance Feedback. Lisse: Swets and Zeitlinger. Fleiss, J.L. (1994) Measures of Effect Size for Categorical Data in H. Cooper and L.V. Hedges (Eds.), The Handbook of Research Synthesis. New York: Russell Sage Foundation. Fletcher-Flinn, C.M. and Gravatt, B. (1995) The efficacy of Computer Assisted Instruction (CAI): a meta-analysis. Journal of Educational Computing Research, 12(3), 219-242. Fuchs, L.S. and Fuchs, D. (1986) Effects of systematic formative evaluation: a meta-analysis. Exceptional Children, 53, 199-208. Giaconia, R.M. and Hedges, L.V. (1982) Identifying features of effective open education. Review of Educational Research, 52, 579-602. Glass, G.V., McGaw, B. and Smith, M.L. (1981) Meta-Analysis in Social Research. London: Sage. Harlow, L.L., Mulaik, S.S. and Steiger, J.H. (Eds) (1997) What if there were no significance tests? Mahwah NJ: Erlbaum. Hedges, L. and Olkin, I. (1985) Statistical Methods for Meta-Analysis. New York: Academic Press. Hembree, R. (1988) Correlates, causes effects and treatment of test anxiety. Review of Educational Research, 58(1), 47-77. Huberty, C.J.. (2002) A history of effect size indices. Educational and Psychological Measurement, 62, 2, 227-240. Hyman, R.B, Feldman, H.R., Harris, R.B., Levin, R.F. and Malloy, G.B. (1989) The effects of relaxation training on medical symptoms: a meat-analysis. Nursing Research, 38, 216-220. Kavale, K.A. and Forness, S.R. (1983) Hyperactivity and diet treatment: a meat-analysis of the Feingold hypothesis. Journal of Learning Disabilities, 16, 324-330. Keselman, H.J., Huberty, C.J., Lix, L.M., Olejnik, S. Cribbie, R.A., Donahue, B., Kowalchuk, R.K., Lowman, L.L., Petoskey, M.D., Keselman, J.C. and Levin, J.R. (1998) Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 3, 350-386. Kirk, R.E. (1996) Practical Significance: A concept whose time has come. Educational and Psychological Measurement, 56, 5, 746-759. Kulik, J.A., Kulik, C.C. and Bangert, R.L. (1984) Effects of practice on aptitude and achievement test scores. American Education Research Journal, 21, 435-447. Lepper, M.R., Henderlong, J., and Gingras, I. (1999) Understanding the effects of extrinsic rewards on intrinsic motivation - Uses and abuses of meta-analysis: Comment on Deci, Koestner, and Ryan. Psychological Bulletin, 125, 6, 669-676. Lipsey, M.W. (1992) Juvenile delinquency treatment: a meta-analytic inquiry into the variability of effects. In T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, T.A. Louis and F. Mosteller (Eds) Meta-analysis for explanation. New York: Russell Sage Foundation. Lipsey, M.W. and Wilson, D.B. (1993) The Efficacy of Psychological, Educational, and Behavioral Treatment: Confirmation from meta-analysis. American Psychologist, 48, 12, 1181-1209. McGraw, K.O. (1991) Problems with the BESD: a comment on Rosenthals How Are We Doing in Soft Psychology. American Psychologist, 46, 1084-6. McGraw, K.O. and Wong, S.P. (1992) A Common Language Effect Size Statistic. Psychological Bulletin, 111, 361-365. Mosteller, F., Light, R.J. and Sachs, J.A. (1996) 'Sustained inquiry in education: lessons from skill grouping and class size.' Harvard Educational Review, 66, 797-842. Oakes, M. (1986) Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York: Wiley. Olejnik, S. and Algina, J. (2000) Measures of Effect Size for Comparative Studies: Applications, Interpretations and Limitations. Contemporary Educational Psychology, 25, 241-286. Rosenthal, R. (1994) Parametric Measures of Effect Size in H. Cooper and L.V. Hedges (Eds.), The Handbook of Research Synthesis. New York: Russell Sage Foundation. Rosenthal, R, and Rubin, D.B. (1982) A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166-169. Rubin, D.B. (1992) Meta-analysis: literature synthesis or effect-size surface estimation. Journal of Educational Statistics, 17, 4, 363-374. Shymansky, J.A., Hedges, L.V. and Woodworth, G. (1990) A reassessment of the effects of inquiry-based science curricula of the 60s on student performance. Journal of Research in Science Teaching, 27, 127-144. Slavin, R.E. and Madden, N.A. (1989) What works for students at risk? A research synthesis. Educational Leadership, 46(4), 4-13. Smith, M.L. and Glass, G.V. (1980) Meta-analysis of research on class size and its relationship to attitudes and instruction. American Educational Research Journal, 17, 419-433. Snyder, P. and Lawson, S. (1993) Evaluating Results Using Corrected and Uncorrected Effect Size Estimates. Journal of Experimental Education, 61, 4, 334-349. Strahan, R.F. (1991) Remarks on the Binomial Effect Size Display. American Psychologist, 46, 1083-4. Thompson, B. (1999) Common methodology mistakes in educational research, revisited, along with a primer on both effect sizes and the bootstrap. Invited address presented at the annual meeting of the American Educational Research Association, Montreal. [Accessed from < HYPERLINK "http://acs.tamu.edu/~bbt6147/aeraad99.htm" http://acs.tamu.edu/~bbt6147/aeraad99.htm>, January 2000] Tymms, P., Merrell, C. and Henderson, B. (1997) The First Year as School: A Quantitative Investigation of the Attainment and Progress of Pupils. Educational Research and Evaluation, 3, 2, 101-118. Vincent, D. and Crumpler, M. (1997) British Spelling Test Series Manual 3X/Y. Windsor: NFER-Nelson. Wang, M.C. and Baker, E.T. (1986) Mainstreaming programs: Design features and effects. Journal of Special Education, 19, 503-523. Wilcox, R.R. (1998) How many discoveries have been lost by ignoring modern statistical methods?. American Psychologist, 53, 3, 300-314. Wilkinson, L. and Task Force on Statistical Inference, APA Board of Scientific Affairs (1999) Statistical Methods in Psychology Journals: Guidelines and Explanations. American Psychologist, 54, 8, 594-604.  PAGE 3  PAGE 1  This calculation is derived from a probit transformation (Glass et al., 1981, p136), based on the assumption of an underlying normally distributed variable measuring academic attainment, some threshold of which is equivalent to a student achieving 5+ A* Cs. Percentages for the change from a starting value of 50% for other effect size values can be read directly from  REF _Ref19356035 Table I. Alternatively, if ((z) is the standard normal cumulative distribution function, p1 is the proportion achieving a given threshold and p2 the proportion to be expected after a change with effect size, d, then, p2 = ({(-1(p1) + d } [Mean of experimental group] [Mean of control group] Standard Deviation Iw _ hijk|}<G  ####u$v$$$$$$$$$% %O&Q&&&((-(.(/(0())d*f***********++++-+G+B*hjJ6UH*jCJUmHmH jU jU6CJOJQJmH 6mH 6CJ CJ0NIJ,nw   _ ` a b gh$v$d$%d$&d$'d$$$$IJ,nw   _ ` a b ghluXYr{wplh   e g h    *                  H I J K      '  'hluXYrj !!u$((R(^(($$$ ]j !!u$((R(^(()|))Y*Z*^*b*g*l*q*v*w*{****ǿzrme]UM K  P  T  X  Y  ^  c  h  m  q  u  v    S      q  }       e  P   ()|))Y*Z*^*b*g*l*q*v*t[$$ֈ $ v*w*{******t$[$$ֈ **************************ûyqiaYQLD                                 $  )  .  3  7  ;  <  A  F ********t$[$$ֈ *******************tpp$[$$ֈ ****+++ +++++!+"+&+*+.+3+8+=+>+B+F+J+O+T+û~vniaYQIA                                                  **+++ +++++!+"+&+*+.+3+8+=+>+ppp[$$ֈ $>+B+F+J+O+T+Y+Z+^+b+f+k+p+u+v+z+~+++pp[$$ֈ $G+I+c+e+++++++++++ , ,,,:,<,B,D,l,n,t,v,7/8/M/N/O/U/V/W/00001111111122 2 22233<<|?}???????<@V@@@@@@@BAdAAAAAB9B\B]BBBBCVC}CCC1DHDCJ56mHj6U jUH*[T+Y+Z+^+b+f+k+p+u+v+z+~+++++++++++++++þyqiaYTLD   !  "  '  ,  1  5  9  =  >  C  H  M  Q  U  Y  Z  _  d  i  m  q  u  v  { +++++++++++++++++++ppp[$$ֈ $++++++++++++++++++,,, ,!,&,+,0,û~vqiaYQIA                                                  ++++++++++++,,, ,!,&,+,p[$$ֈ $+,0,1,5,9,Q,V,[,`,a,e,k,,,,,,,,[$$ֈ $0,1,5,9,Q,V,[,`,a,e,k,,,,,,,,0578j<u?v?????Ž|xtpliaYQ       Z e   %   ;  <  =  B  G  L  d  j  n  o  t  y  ~         ,0578j<u?v???????@7@<@V@$$$$G$$l4\ yp# R$$$$$$$$$???@7@<@V@W@X@|@@@@@@@@@@@@@@A=ABAƾ|tld_WOG                       ,  L  M  N  S  w  x  y             V@W@X@|@@@@@@@@@aLxG$$l4\ yp#R$$$$$$$$G$$l4\ yp# R @@@@@@A=ABAdAeAfAaG$$l4\ yp# RG$$l4\ yp#R$$$$$$$$ BAdAeAfAAAAAAAAAABBB9B:B;BWB\B]B^BBBBƾ|tld_WOG   )  q  r  s  x                      8  9  :  ;  @  i  j  k   fAAAAAAAAAABBB9BG$$l4\ yp#R$$$$$$$$ 9B:B;BWB\B]B^BBBBBaG$$l4\ yp#R$$$$$$$$G$$l4\ yp# R BBBBBBCCECQCVC}C~CCCCCCD,D1DHDIDWDmDrDƾ|tld_WOG b  x                    !  Q  R  y  ~                   BBBBBCCECQCVC}C~CCCCCCD,D1DX$$$$$$$$G$$l4\ yp# R1DHDIDWDmDrDDDDDDD$axG$$l4\ yp# R$$$$G$$l4\ yp# R$$$$ HDrDDDDDD@EbEEEEE FFFF\FuFFFGGG"GvKwKWW]X`X#Z&ZJZ`Z````aa a!a"a#a-a.a4a5aaaaaaaaaaaaaaaaaaaaaaaa b b b bIbJbNbObYbZb js56 j5556H*mH jU jUmH j0JU66CJCJQrDDDDDDDDDDDDD!E;E@EbEcEdEEEEEEEEƾ|tld_WOG   (  ;  <  =  B  k  l  m                      !  &  <  =  >  ] DDDDDDD!E;E@EbE4aG$$l4\ yp# R$$$$$$$$G$$l4\ yp#R bEcEdEEEEEEEEEa0G$$l4\ yp#R$$$$$$$$G$$l4\ yp# R EEEEF FFFKFWF\FuFvFFFFFFFGGGGGHoLkO"Rƾ|tld_YUSOKd `                    $  2  Y  Z  s  x                   EEEF FFFKFWF\FuFvFFFFFFFGG\D8$$$$$$$$G$$l4\ yp# RGGGGHoLkO"R#RiRbTqWZZ8[!``$a%a$G$$l4\ yp# R$$$$"R#RiRbTqWZZ8[!``$a%a}a~aaaa bbAdBdwdqgj+psuwzzz{{~¾Ҷ|uqoiea            9 0 p  4        * + ` a       U      .  &%a}a~aaaa bbAdBdwdqgj+psuwzzz{{~$$$$Zbebfbgbhblbmb7fOw2:b{ bw2GYZ`abcefgmnopstXYZH*H* jFmH j0JU0JmH0J j0JU6:0Jj8U jUN*=>?@ABCDEFGHIJK$$  =K5mH * 00P. A!"#S$%7 0. A!"#$%F{O˓1_)JFIF``C   (1#%(:3=<9387@H\N@DWE78PmQW_bghg>Mqypdx\egcC//cB8BccccccccccccccccccccccccccccccccccccccccccccccccccW" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ? ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (OmPK۾ݴBV^!կoDB% ;IkG?^<֖຅svh?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?h?Q@ c}9G c}9]h?5liJ*Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@s?R &Dq؃MtQ٧Xlϛ+I=6cJT[N%]Q bDVW~ŧ[ZoD7mgX((e\,æ궋h.VRڭJ8a#ukZ+ !gf 6zvV :l{3=c @>_5g3YA\?tf]y}ٺn09Xiy}~Ɖp ]aMw 1H@$< đPA}B &|Gg{tȽפwŶ^&ӷe y | JI.-,s2U[9QW^^>\Kq6d mS+{*3ׂ@a.^I x-b[?b$j0vzprEtw-ޤנ F<ܜ8?1''hZ((((((((u-JI sJ)d?*֠e-my,VEVy5]GNչgrP]F$dg (((((:vnwjqw,",w1pB>TQEQEQEgW72]A2?m/OS׵8ip.L42`H'鴏U(XZjQDXȮgy!h^H&ꄌ?N5^Q?2 1wG#U( zt֐>r4na=J"w1^, ɶ=Ijj(.8 "&Dlg`rO8ݎwE31,^V1 ef08$'EQEQEQEQTJG^Jȿf{w.2FNhQ@Q@Q@Q@Q@Q@ciFiIJ&cY1pp2:b1e3ynfU@U,#mQEQEQEQEQEQEQEQEaֺ֗VsjW_2r#91=>I ?R|D/UxѶHohZh4m S<6`o~ޥu=]X Y|s|tg I#|DKJ MwdtϥMwYzq[i% 3p7p~lg f~ۨ~>Vv3qgO\R" Ic$#8'_dQpdWuu@DZU/i+ֳY?w$K BF 䁎ynxE3"s]bx*ݿnϔsA>լA%_oݻ?:Q@Q@Q@Q@Q@wT }2˳gRHR>bcqB<E[ZK#Hq#pHuP;U} ";C1'WvH$Ԟzy]Z0 ,Aϩ;r3|u=xv X"`mV߼}ۢ]rzUaKȡʹ?ȺU-Idv¨#C?vQEQETs - Q}C)5O GUP*(cEgah t ?X:hQYZ??-V?@V@O GUP*(cEQEҢdLGBY`PTU((п_پF^f=6coÿ__nۏ]SӴty-#}y 8 wOg33RQEQEQEQEV@O GUP*(cEgah t ?X:hQYZ??-V?@V@O GUP*(cEgah t kimg(b@\MEPEPEPEPEP}ag}-RG<^B>'<SۊТiv"ԒHzUx#H9sRQ@Q@Q@Q@Q@Q@Q@!ғZmnsCpMy߁5G\c,ǮEnQEQU3- e=Gz[_[qc[S7z;ӽXs Vc\.w0kqa@q;7M=A@[,W%q$I U1FO=yj((((((()HFHꈀ3&Ula] ѐ 9g tK$=>fx] gzyᶅ8b_0U&;mN8!upc*AթnSTz|2qyN9 t 4CYMWsnca Nqܮ23jPc;AIgeKXWgqϖӽs>,ú*^[$̩>zphđT#i՟wYY%8VF߁ qҬYz4OݾFѲЃa}owg'H Pk>Q`;=P.6M dsW?-GYÖ?jw툛vFG9v86K=Q;StVp:Ɨ,Zj^\4̩ <(So P9q\`89r|?A_ynˇހ}O޴,hOuːAr@5 t* 7o18QSVLnlCM 3$OoFGz_-]*|! ڀ4(((((((((((((+'Y4XL@?8g޵{-c*y5-p3׽W}52+f]QEQEQEe̺ݼ˺)n:W?6GT}H1l#kuh 珼rA':W@z 9kuU^=Hb:F>)?-qIJ=vUu;դdg8'Lds7#R@os)L)8I!u SDt_#Kyc/#RO34}ҷy!VcےEMkimg(b@\MYayVnZPdb*4A!oak?C m,w6r۟>Ywck*(M R~ͻ61 }N29>M7?&E?~ O-0W>oj*݅϶ZAq;|\g('U}>,V=v䁏b`8^Ҽ?gg1!.?X#zgquu 0#=Nq1mPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPDd?9-#<  C A2[5*?Ev7Dp`!/5*?Ev@ T(x?Wxy|Ut I " (6e}(?C OEy@1,Ev "8l# *,#Lu}?|NˍrJ;dq*;?krMJrʍEʺS 1?]?Y 1RxA]qwJ;TNJA}{s]_7x^K'o?UtL,{erZ< Οyk*ڟsu_-FG]KTѽL-JN߬\q|1:lˠ }[4!)geMn6'4p̧1&6 qw]*]kJy?:fmOz)Ll<}x.Lu]M<_UxnLk홲1O; >=I,3(>`E GE8Vƿ[4ҙu2.րq1_[`^zz;k:MbW6xm2lա 4փnX _Fm 趀"uC,n_=,Q}&.CUXN)=Q23B= wp?$I@5J,gPj4\wr3:6ׯ7boPʝDc&ixΥlGQbKJ*:S_oo}^'i}cN>"ޅe^`'+lG`;3o0o8l7؉mDÔm؅);Ra^ĶJ, ?B>Þ *3Wd`8* bʈ!¼ 8x(#)=0S¼gűuƗlD6{xF[˼LRa^:I,t tė~|U^#v<cĿjL*Ln*߱qP񯐄&'% R {RR(ʔBQ8 J&B)_BW2LcuTd%iwLp3B./`-ű{rq,9LXsa),٬g<0y8< s Δ!q/^~)}`1~¼~ `.DÓO;ZR$ڐ ۉc;m+ӥ>94Ϡ SJ{*Vtq,98S@^EX^"fuQcuT\$%_논. ?/a7k.`T]b% 6mQQc(bs'z'%)kdS~ g=-s(#F W'5az/ 'JS]LF ]oB>87J*\8A] !@.L1%\}2ӄ>̹i$VWbAND̺̻|\yGp;gp .ys1ν@*\8A#P}?{' .3SksVW{?4l54AR;(q?@Q6 I(sȵ' ;Iw0sp$,IkOǞcg ;0S3%\{ n j0sյpŲ,Qoe .JEup[O}ܝn)V{Hk~kc}".v+Z&/v1/-j>0yJ,_\neuoLu+l7QބYnU,:'NP"y sp"0e{#Sks+cW3ݛ;ƹil\u&UŭB؁H]Δ rmn5O~-\X WUbu(D[`l\U 6¼#1,wS𺪋EV`rJcQzKsJ.9y{L>OU"G8}XrKu;nVqglV1cj 0y88|.",ub&0oIsı7U#ΞD*86{U}4T?G5×=8xI5ahQ/ _ǒ=T8{FJ,G=}}81]U V {cއF5vbr[Ւy¼NODiK6(\=dHyı5 N\E*}86nrl կ:-*>nQWWműdՁSIEőXUumո>A'XĊa, Kf^Ս$v"&'c;UiRlG@;G$8Abi8^qzU),ý^uz2E PG$Vb2E+K)tauTݤddלЪg#רpUo$wUgQQXHKRaTWzD臮z/SPfx?Exhٻe,12j|C"َ8Aԋttۣha#' FP߀)F(*K/A͖,yfcU%"VQzKts=Gց/Y21X_Dgkv+8ZVWe5|5@}U}{~K5[7>eqlFUR53z V׃8J5!z'FîM|V_dN.qu+qc1Ww*nvIEoqFE` >~z}8)8'qT˫0pI{[ID9JQe?ȵq_x?a#s hl\m Fb k()k6(94Nw{;ι. >'i!2$V$}__39I_HiKX0S }o׿` oyLG}A8D_.2v.4,HkgSm%ytW;2]_bJ qV&G;fkv#Y)GR5Ŕ};A\?h5Ocj.pl!fl"WVZ]٭˘M[ۿN*\qĶJ,WD?˛CʼCu"|&&_؛Is8EȔtHdJ*r-N15LT57vp6¼س59&7 ǒÝB0s 6.F;N*;.c (44@3sk41)+ &+ql屹tz)GAS[gXȓ8|r 0n|C*0"ҿ~ &HCİVPY "B5Xܢ"TjHZC`-FnEd9̜; B_Zx7Ͼpgww{x-$X=6xH:"pVk[om1{y#=ěMt<@=̈́RFwüIzub6Jp>oU:2dNc<$oBjh}*ֿͼ~z8Mp>lm-i;B> ^4!=8|**=l8OӴƵ)~/$>ɟXEghJSp>em'|cx( >KW%G8`WN{xIz7HKhisp>-Z[$?h?7M˻%_#D\jNї]{C~DZSyp׸1QiDe7 MSKH1Viuw#^'y:$ \_;Jx|ͦPQw3<|/uhջ8V5Iceu?PQՏ: /J .vRi|>Տ22u7Pf5/ˌu_QPQ ҝsV[[Tbll \P"lؼc`s挍F6ۏjG_VՄ[vll.,lblypAaf K%bS$l>wa|dflűepAaƖ[[T`lα9YpA9aFR K{ۋ,\ [16ql.(dlޱy9pAyasƎFba#c= )6Ր}K|lwɭWu-=OC_<{WGWޏW㖸*y#y'dg>?==5v_Dd?9-#<  C A2L3{Etx{׀0/p`!L3{Etx{׀0I T(x?WqxytU뾵/H"ёEPAQaQ# hPQ@Ea E 0nlB0 JTD PYeM K=޻EUwsnկ$88Nm'H`EQG {db3}$#5C4YI1}~x-Xig*xNZ;I <+c;c+b5F{#^+p]Sm^;s252X95Dc;3zGR5_V6{a; ^fy?xTsV$?wެʟW'5s'1:v1:zq+ffr >PÎ]nMӉ1w\Mգ)h=5B1Sjs8\'bqysF}Pêr,.^U v,(Qks7RR4Uo42\Ѝ\ZL0ngsy N>tV `s IngwV.&a\N DK'>As.eyͧU|s.o<¹(euE!SME(Y_ј 5pyláf,'⸼_E7 "[B 2.v,/qjYUF_<";#e3^slS1ǜ ݌=]>NaMZB%RT7gNrjQsA|ƪR|`7U j>SGǎBf.e t_].Oe|WDCz`yu9%R.NOĿ\΍g#ErV;AĿ\^H*RKrԍ gpߺ0WTJKww' ;Ur^xX׏%P%Qy^k.&&DJ[$Ss׳+RIi|s/<o]ݥ-y~C}kR}ٽɽ .;217S"èybG&FI[HW;21Q¨IypU;2q9[fXf`ؒ"r.uu fPncɾ " n=]bGjTݔr:tؑtm3ZRnPT}e\+pmHYr9rtRt OB݈jF@]cI;ɹHOM6c*Nծ ŎT-9)=}ϩWƎmJ]F,^/z->䫃cj}ϩŎm!][W?h8Gq&K-{T]W[ *:PI@-6pl+K \* I78[أdݎjU0Z&MIףH+߀&/3IU|@su.XF, =] g9z9zj ejToBF+/ -}F; 4̆>\r PPKEp?Øأj7!e\k38hGy'uz>=ˡT7טwo#h8 Tٟe=*O6D%`T%flǾV0՘أeDnR#![ժ[s{J#>d^ D=wd﫞Hb]aKqlo5hV}-6E8hzh`7W= K cqlzѷy:=zY'2Kcz^Pسj2Ǡ,GkD3B Pc,y&aCU>b8=UKB\0d~@}FOYZ2["Qk|5ֱ;,wv!K7hGF,m3oe/ڋX1ߌcZSD'B6WB+?ɚ,Si~ >0KUR Xs-U[s^S]ĵJU2截g|Θ2'RĚd)/LP^m|{e+C4ܜy3IOFs:wHA=oR]zy;_-k|1oPREy|9H30>K⋍y|&ʑ i9#ϓr/02 F c4>Dsk|?f sN9osMro/ y^mr;!^K+]BV.X"cɐSx\OygrMG VlBr Z&GLnr󓢔Lnlrˑ9<*~ďMT-ƲGBb^?|08hr׋Æ6!kom4wqouJی+qœS(as"g -! M -@ ._1Lb>̹24QT3!ꋑh2FC#򐂵L27Ƣ%.K/q*C=27Dsh#K qTPRh&4\{츳#zuMLff.z8:sＯ(7ϦnE >K 3n鬱1|0U|Gėq[̇OyH}+yx?!ψIԘg|'4JkŸ'1Oc"$sn|[ hn+爻"q7'ms |uG'H`r]Ч\[|h6Ҋl%UL|"(7"&oM E)bW$+%f.>S4sƷ3TBkոϮaFc^~;fB in +wF,fQ1i&ĘUf'aU%}O_jʧLA ~"@chVWA?6{]~,SƮ=Xcy.5e{QĮh֒dkvbm:Ƙ`mz)h΢5dEQ%CVcv٣HA_Y^.kl׋F/X? E( 8I/n) sJa6$70YRGCwQ䂿ȝ0-B0,R OR< 0Q. .&wRrɏ0.LKѴX\h&PMz>5AHh!pBs~uPz,4_Ƃe"x4hF=_z:\g@K1πoRW@z F_ccq٣o&Elp N/#U/κИC[-_GO^%Ʋe;K ]pnrwA-E V,_0XASC99+L`#LnF*Ҝ_{>t=yX{=7T)OR> س]M֗u+ ,6#}(7Y7BK:Zܣ{*[e3l&g()XEgKȼH簥1\e<SCyHAX)W "N CHhع4F=m.]̶\~4田zvNvH\}"~xGv\oGfcΎiv*ԧXVG4ЧMQ!bݑB(bW>OX\+EGU]q<{{$HB! -R0TA#mp/TЎXdG!SۂŝBiPhY0Mbh*!Bi1i5P([~sw=8̙s?{]^v_n}_VQZn7̺ շAC(GYɹ@*d]*y}m2S!Ks'AC/)qOge_Q\W+ⶺEאb7nNP ٛ-oS]N^,WҺ,dwȻYPcd/\=-eVMt;d"zK G*\Vr, Ȏe}Pfє?`Cy|/k(aHa#o\C}3/=a2 أr!{Xԫyl SQLul\&'N@R@V)8Q=!ױiMz=.o7AVM)%Fm˧lLz3{D`e+?c+<ۀ4(4Q_z"&/k:_bo1.߅2(lNiwe7OIWƛQ2 (K[7 9^Ky?~P e:G3|Z>-+zF@*h F*sV.f-9߆׋;\ 픚ʻZ度T.]s|W;]KNGNnR>缄KC(wBv:G}>(pqWݕ].ˡ?o|q>?jls '/Co:ovJmI9o|3;b?[y'Sj_Bjy缛#^wyZWg%peSCvh59a~qt)[n̏4?'I~U#t/qW>%c|-pdޓr]%R}~mo|m,{_wjSN2Zx1g{x7_4rfe%c93xЉ$#Dl<\q }]HlNE=]7j6la`SbVl!TTˁMH2=l+̶2p*Zw81[ΨZ6 ZdVl%RW4 \ BGƉm;6ώ$Cߏjk`BĈlZTD%`NE5Ć 6l8!d&,`~V$ \k58#HlNE=Qb8ad0 `C@6D,7Xl"' l"́ 6l=p*bz[ l8Ȇ4J`+SQlA ռ rb۳7IGךM6N.Fl`"JkVJbC6AbYIF?jtJ=#U駬}Sh}o_GOȩ-ǯ:?y(yg6C|^~{DyK  _Ref19356035{DyK  _Ref19356035(Dd+a  c AnC:\~My Documents\web site\research\effectsize\eqn2.jpgR' nXjQ{l'7FF{' nXjQ{lJFIF``C   (1#%(:3=<9387@H\N@DWE78PmQW_bghg>Mqypdx\egcC//cB8Bcccccccccccccccccccccccccccccccccccccccccccccccccc" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ? ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( m$G9|W6οp %u1QӷW>Y/eGQ-*@珿(KE[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[E[@<E[A?x,<;ۜ=2}kB((((((((((((O%}q m+y H<Q+H[=$;:" u;cSU_xrVter̤nTdz((((((i1N;b5,򷢁LY{tZʒ bs8f#\=@&༲Ztr#807^R'DK@]׬EŹ"K ?4gC#ԵIuy,dbD{k̓pqz3z| ףhsn쒘$qG=V1Ir|@yL`q@VA[OO +5}׍)%N}[M-!~I2j??@N١{3G O ⶠVkyc&Sۂ+"}Y=q(yYV6~+ c߮? nC",O @ #A ~{LH\ύ.=#UYT1L%`Hǯt#M˪ldGosr}:{H/w:Cɾ!2qu댿jT KL !.j.]FQ/.K2X`0p8OzHtR'8\u8;kWD"Lٯmw}YW9$s?mJ. [ytRG\ Wҵ;]^.pA{W+-z6{Uu I=>zA[O Y}j6B OYs;U<\ⴴomd#?xpHH#*{u cibIݼ;F8rޭ:,Ia|g9ؠ TQX?jqo"ng##€.iڔ:ڼ~p\d;0j}AFBW1s#zaa5qykY;0UuFK77Ӭ1${9?Z'V:M;Kie[{B%(0)~6}7/]$jLJ*aI=1ܐE| 8ᛚx_v=+>i$GpzsA"^?/\ @jh5#N$P18#=4WS,^M粁xfKH崖9d8!mى*όhLø49;{v iw2ڴbM dU= 1x$ C_b)-e7]K6qGҹ}Ěn`]}w7UQ :br-͹N>MGb;g][D ygcrI98v((((((5]y%pCC$Тf\\ȱ2I^3}Cn6#:ր7{%4X9]dJ XmeYae]{J(((((((ig,q7=OI@Q@O ]*Ԇ$@{aTַ<l㐩-rX~+rڐ#|סibӭ7"Hc3€,QEQEQEQEQE /^ ٻP|g·@Cȶwn ڻ*.s~鞇&l4P>hЩ<4p<>q'G<Cqo:|"8<? hRh:$kg X}qt?Eul`@uWd?-z>mF!+dT308]<k(-makk5#cCozڲ4Wrݾ$ZnzL_ң`en O+ryVp.{}kj` !qє#ր3Iz`(.dm>d yO#5&6hJM\`))uX8QvJ $w8(<7 I ֌Q!z340Լ_]"֒ ''Lx~]YwzjP=B%QF 8zz XeSH4Fe ϧ.,v} nz?AMHeynf$r(s׌ ? GUUK ;-c~7yQݎՊ(/sѯ-S-żɒW{XMayeyo+dVT0>|uvTP7xHi:)v:;$ !Bwp1\6ڊ]j[HT`x7N: S@k/ؠ祿2cls$d9NjKx {͊ȷ.79A5Q@/ooug ĶRr@n9w//I>Dt6W$>Qw粢90 5{S{qrrpRdWEkZ]w$gǩϯ9tcԧ2Sc⬤gszmI|oLn`: zH( ( ( ( ( ( ( |u2Z`id Z&RcmX7Z^+0JFyf=I +]EcX,0(OBOzO80kT>'Gb<7,Oxn$xl99dwmq%p\E+eY1C@T_Cgpd* q /fݎy$nCsj^$CE}x "Vg#*8> wV8hI؃!#`{(?NmK 7x†{g<e>51}J8%R^b\a(ۏ豽igM6?/n_xvnEx!sa'i{whҨWXMݶ6Y5ƲHS;K*V'q1cP^]OO ]K @N F?/WonWvOMckV+W[_:U@^2N8#??SL;jvvݸwsB|;@'Zfe[>LL-$ EP=OT-o&&p} pl!VGP18 w{Wyj|WAR` L#qlfn98qҀx1Ozc{W5[;::ʀ=c;gp;;QWITt8- ODdtYX` 2QEQEQEQEQEQEQEQEQEQEQEQEW}S--Upeb?(`caH?9z^&ԡKJ̹33\ Ҭl-e87~JO8¯\+yO8|1Ђ8a[Mn[yQ>nnJJA6 tW+kޥCwm&nI#kw8S:S]U4sŘ9&W7McY\$V-j>'~\gp:@r~,:ޣqyp]5 BsGzx#H9sT'o"Q4=e7\20@8d`zLmFӭ5MRk{8D ;.=q ٪I\GG.>x/T:{]Lv c >QX:VmؕΌ`Xe }CK_o[grŃ&;`g$Ө4zǿge}q\?GQ|wx$+d3cCW;q J!6 $@Rt/sn:`Hi{ys]p<]8<`ұt)4k5J&`#iY_8* rS~G(((((((((((((((((((((((m巙wE*u2Wx3H;AڽAFY^-.\gTtTQEQEQEQEQEQEQEQEQEQEQEQEQEM<]]n0y> (4 ]iqNM14 @ddY-*y"T#tp8Q 4fN?.y"F0F 0yՊ(hVx%Q߀*J(6R[]A%@ʊRyv`E4 a۳D5#?6uHqҢ9 K:IhX9R}z:+Yx̍ Dby15QEW+w-:KBY9giw?;wy6( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (?}DyK _Ref468180341"5Dd 9#ON>  # A"24XKSXZd#f2{l4`p`!d4XKSXZd#f2{n`{@V24x}xU)@K( A@ ((ҥ*`A(UA)RP@P@H'w͜LȻw'ƣ{԰ۢ?@<9 E@bh=,
    EqW9dF:(djUknjxYCVW꟩]Xn[uMj%!Z!EkojC-Z[{)~QwGY"-AX:wդGOo+ 'bH_5l㮰8-5v\k{1#[h  HH4 bBZ 0u(eV$J;1s1c?p'sLS܊I#b]B_ "ńI5ai$,' [ΐĩ=-]!Ƌ4UlDۉp]({D,<^RV3 Xyo V+{;m^KᏕxw+}n+ j}׵Xo _'i_c.V K9KǡvZ ;[rnuPn#cGsǾ[vb:4tkCjҢvdI=꡶ؑ mچb6Y~-k}؃A7ΎL`8.ER}bG&EQН#XG{ 'l }gvdB;Ԧ}נA;2͠\QDMcs:ךvd8Dmvdh{{c= ikҘG\ Q:+8ڊaG`m V}M˰#/tűE;h_D,b;hvjw+;hBy\>mwADZT<6)W<(>GԠs|ZҎ:%ѯjuP*ح0cxQ_W3YvB:Ԯ.;jhw'_ ٦_rݎhY4õ*iQ(QDerNKD?i tHqn~P tHC)L[TDT@@2-.r1?&棩ba`0$hGP~8] b"Lm^p׽hgѽb:gD 8tK.>3ҏtUyѽBD?s6A=Y_ezψ1Μz;n-yuy,'9ƜC;GR=k 'IRIqDA7I]D\7I:/F. ``WA :/. ;:gIg4yIK,#4hb8%H;2]̐NA,s>:HV"|,u yk~O>'a_B;vw.:-m%)fhGZ.7j;:ߔVi)+]ǜ?v-$qrpI<$R6#(DTM*5g2w$O")Ov&sJCy_O3ShkDhA ^+4 |_:ّهcKd\ *L[u]3.p8 !߀npbЎ0#^lcZ95\3/pUVʆc *-3\1ʟ>d/n@#= P#`*DuhG D\~Ͻsf/Y9k(ߚ啣fqs>jb~L>h;9raP>7S#s3Yi6*(-Efei}@cD<|`vRwYne.>8b%|Qe6*ӘvӾ (zq=:Мf9Wyd/3_SsC-E;E;0gLsR|_i+_4@ptޠT0?S{v"s.jb~ \ s0<&ەZ<$R9h1(h 2f<5W4Qљe|ٸWKƟIrʸ\6"S]T EjmU36P',豿@KA-Fsj X@QTaQgO vQi_714Pӑ=HE:TԑyQ4Fm#TmdQ-ԖLkj(zf{sh F'I,{h11TPs5dmtJW+ƋjZ911?jPPs<2g/^cW=Hǜ=wAکy}zTG=^`s6CGEW=2]2t^W?ר]C7Oߌ\}:EߥN?U1T3 (z8uMQg\2K\p/!V0m~iBSjC;bچ_{sN? j~Eg_~yBPI@# dj?R;jY旬c.1ntdcZ4U״_`4~6V;hEo$F+&$CEW^f :6iZBmf.+ Mꢶ6E^k#zT5`4CEW1^E:7պk6Z'Z\jeVS멕Vj1E/C* (+9hxmgm(Cױ}uvE]U'iس:is:(}qBγ[^uO}G;2]P?Q?櫫4꟯a1b226tޤ~~3]̯ jڪFwZc߽G#CU#gYճ|Z=S/hUs1槩r[Әri)|vπs1Q)ORJo2IJiGIEmeRCdZzfںJ#!2p''&z5<7GJ3_ڶz>9PEϴ<ސ1t PtwUܖsoOyi2m+yCEW GʛEo$̹+#G^(דzM0!d u| ~&?I]wK=']c$~tQo#g|U?/Ǥ? 1/[z^@_#ÝwH՟]׷KDbg$.QR@ˑJat8#[J6I ƻy 34PXMĜK2&HERicT7IYRev Tk` U3F7JjKO2 ΃ wΐs=#]jlTZ&Fs$5e~IR~(ڹ82ܹR''e~Q9/Gɐ$t9pbtm<_4ACv~;/%@WΧq<8)7ӌʼn)8 e#ÝocR\b50U"ֹPFdg;qXNzq!syS\#~hWƛLiܿ1猽x3 ƿ q33\h@5qV.~mk̹xeA&zv w(EFLb46:2E P_{d-?oޭkFpj\ڈ|Fx @~ =)nGF5Q1SƓ9Ua`JCBh1d28s>m"Zr>(,w9vmw[w`/>yì1D]сXED|UAOOowoq5{S ۇg.#[D&x9- ? 8GH8J ^ CrzYr] "F_X_%P\҉tQqv/jB-r0ߠ({KcW}wg>c3tu!.똧D{*&d2g eDq$<$Iaql?Y&^'OyEH1&B6)my.(Y67Z$ g?i圆 wn*~ō7"?bJV_sd-H3BN36{~◤x _~sqFt/' pb.cʻx#3.,w.^"ś 2ǘW?IM>70|\oEXGGv^ yPs~"t{Dg{DJq|cy/2X11^SbAz93|%@ <:,դTg()yK|(9GLu9s]OzHORC_Q/ z&J hԔ$l-iHiC-o(5bmdۀK;!u]2N̥GNz,E]O;Kw?ʿsg^/H/nQ_:FN'HDi +sP;,:F_d #_43(?2/fcBi]#K?E#-R~gy72si:/{zTZN?gG)?N\οJczYLH;]i+9ߕ>l>ϔs9pg:o_Q(CLY|c"i }JH( H>O+5⪢2|v}{|Po,x<"K޷d}[VSmYs>/?_Ϡb99ӻHx?y!bVe}ʅD7r\{X~{P {®D&3>\{d./?] G,C.KNޑyoBreVvxgr=9{n0ru撠ӽJo[\yʛλW3m}5j[x)- ކJ{o}_hڸS]Ko2&JGK#vTz{*Ô^L;LϴsWQw222;WI{5f"T7;Cy;_y.899*V%GݢL~~neگL{QYw(󽗔/yrY }aΩs{]Y8Ey!:{T?Wy ]*nj{zWp ޺7tu"sNWb~=~UOz;1m__cW]dxc۽c_3߽+ޱyzQ/jSU}>aT_o5}Z鋮2*=oU'5w^jɨ{Qϫ0W+WU>ui+9yQڗUji}9*D*k}k |M_o[ qZ3Zo79:0w曯} Z0+ uU ^Nm8:Fj|Q6]զv6w_{@ʯks}|~|1UE 22_A}/җL>>d dsU}=:U7};̹y0;|1<sγhJk5dέ`(Vq7H?տcS>>C肏b*2 '?W}|2rzڻc!]=~>.)O)~%djsg?7_]]g~^#EZqF9Q5,R~>kl(!p%da.8b5yfW04? 2ΈbԷ0 }MqFcF ;̾gA5g\c9c776F[1kkf}^2p!FOXcϯ3A~>qב<5tk/5&c2u?jgw_ow|/=S a_9W|ʙ|M?|圊 ^-6oc{hfg0f#?u-l!I8/?`y 3^!l?dKw3!Ѝt7cXX80,k3K_>#=%gF|g"aY%0W43Ef&X'llXi6 ,2>7k8#6 Ke4,50OpmffW O+gmק<}OÜSE^spZ8ǜC^~hLhNƚ'syc^}7  <2@͗"p?U^6|gu%z k2Xt)NeȏJL$E~3N3qp8 2-\*@r_8@|ƻ G8g{lE [5vu!OqM`7R,ERHV)Ab%R>tvb.=fQ8b૤Qp8i kA}_Yf34 d3fd;8 $H]og,px18 N,2 \#.!K|`Ւhz}tM3dz+\ =h7"8GqM0| ']C;y#.]Q r! gy/ r>L1b'fbcwFr߿|!~N%%BHɐ->(#v(K$=CWHuDu3{ -dz%hE;*Q1 jV]b+Ѯ-+tS{Nkmgle+4d/cO-C߶ 'oQ#Z^;ekTymlU3w]lYmr ?Ŗ7HY&jƺWyekG*d| 8@KQXgqUGT%ν~N89<3*vYg4j_s Ź0dG :~:Fqqf;jtߣ{]&r_wI3ߎZ:+39;+=xǽM_~ ا:]tOQ XGT<_Ǫ'j{WH6a} oni:E}7޽"׻|d-#o9tۑwQ{tZiZ9,]K@W4`mԖIpoS\>mWűۂi:ڞhp*$׎Σui:3ZvkQ6*aGm`vv[{m۔gl}vԆ\±/uwPJQXG[<G Qt m&${,6vD>Cq졠hGm`DԾ!WʔgQ{:{Ә..Q%Z55oG`m|~RQ v6Dm;hO~6&o~Cϛ8::Vv_TuuVdwP{t9vA~ʱrS Xx-T;hS uq۠M괇:+@DNt;ҁuQ;{)PMkua8F#XGQ{tّ qUbulG:69/NtoSDǽM_~)اtHѶFmwDDZѹ?#Xyd"_{j$֠kY={ j?Lt!ϳvd@ۈ}N'@wю `E*IFR gG6.j:vdh렶f{jqohا;t/ۑ}S$y:.#XyA_~;j'դY j'Wߏy^:aG&d//)VdhdR*ّ ֨m Nvdh;]{aU{ZWux{<ڑ aj7$ע_QX@/t_ݿDk.<]m;nc؂:EZu xOLKPj_;hGѿOFߏF~[dT2>,[%S:UQ*~lbZ*:$TŴ XL\LJb*bnYL`߻*wrrM) qö~4g4YcΝ?'MϢEˀ2Ma j{mAyzKǕ1wO{!Ey@ڢ)(D~r2 |N1ܿi߻gj]CuPBM ׵ovb2<]l4MV>wSo^ KCz @Q6{F{܀w_+TPŒ<O; 7++z א)g=SQ~*M'[My(Qc=_ڣ_UCOg~Uįբ2 ˜<g<_AIܿd/-adiSCW)+ovL< !C>)C>b*l!t/ڱ. 0 CX}&!C!_ t@_U(Q._:BꀾO:DD;h;#;@ ^ zB^8hxOЏ~Co߀ll7>s B?ZЭGo jUC3 OA)>!v@|? S;{ N K.BKsWcbH0&DcrH2@L \6C B;y?C?? 7!( 2|9Е}YW4D\~iЮ Я&>i!ZA5m tmAhl.΂|7u}7 gOw=1 FCo&g@~:覃~:gLY+!+! t@}by.͠ o3>[!> q ?? 7--M ȟyП~!~x{)Sg7 rBsy<1'CLDM C$KKB4M _uS!_ t@_ U!j@<tς%k^@4<]!t]@u]!xb8kHЍx x5_6S!?tS@?MS!!A>@~9薃>x,F7@~=փ~=[7@lq0G -wAw?3?@ ~q= ԐF}912!B<|1ȧ.Šd䦅ǯ !_ t@_ U!@Tx) tA 5!9;A#:#;A<b(0_{@a5\~ o1 b2b%/2-RC,"ON Zʛd% ߪc裺cпEtneX}X> nIqc~vT}zQ[#BpԖ G`<GñugC뷾 5g] T(kؐAYwC#ZoL۱}Zϓf;iXQqV.WyMW*2D(ڛ1=ЂBe42Dd 9#>  # A"21g[XE\H~1ԕp`!v1g[XE\H2z`{@VD1x}Utt:N!$ɈH A02)2f@I " " H(ITA%#9Sz\Ⱦϩ۾$5)O$'*ǐv#|E[ (oʗ?U=cueNɾ$_=3%Ţ.ա8)Isj6iGQf)0nq1dsb*h!MC67?CaŐźg1>rSY$4̿!9y<#PrFG-sIn!!3lta~!Dۮ7sm<;!u56>clQ96>QcαV䇮rmQoc^ٸdQvWl<'3ݎ'$ٸVR+;<ËI6ސTKAL_% l+B}ar'cw(e#FII6[ifFh!'/'ٸ _6&c>WQ=h$W0[OFez]N ɎNMHAg ~mJU]kZH6~Jhj3qM\N35bTexxE9%PTiRVʯlmyp[,3B>+Vm;(sh-?K/WR]7(wI1}".Njj'ڐTP qSd?ʭS7}gԿ}wճ>C;{B;1 r߾M orG4WB?å z'>kOzꋸt_OՏrz=Iȟyb [yl\Ѹ72L0,|bvekFG1kԛceid \?ϯd)w(sWQշq*K~yCcF6fm`m9M\m4kKySxq0PE/h?7 ֻf+:kx.8d b 6҃q@_p138\i\ 0v8:bTet3&筍ׄT(/QEHٰ|qp%2*\, ׃fHy=_kl/ҙ䋺?6NF)G{d{_:h)k>uZi{96αUZ\x|nwAJw56>cly96xssx5x$aonfm.GLZy滓IJ9ajKa)j3jF۠وf704w릂',Ⱥ pVb+8aj2 ]7: u_Ln&)-~vQsgd]8'0',`]0~A XW?vFYD%l1pY11q/;aj~Dmн}}Pto8aj@8DynnppYy2Y]f',`]fPBLJ0^pu~}UgT',`]m}FT+t߻/ k',ȺpNk5;ajv4jGnv:jA7ݎڟKغ}ەud]#8vqt;,`]- JNXJmڶ?n]}ݺd]7qb9ajv' ]lz=vp=|nćvXZ!C%?vCk',ȺpNL[ t-u-P;@7 XW;SЭrծBFDnV'ljpYy9]m',`]mw>bG#꡶^tվڗuzݺNkȻud]^8gsy.~%G~Нt՞D?%tq s%p}Y!IGmzR&)-A+&/Yf7P&8MKߺ=xo29sdpDd]y8g3j7''u?/D8H&u" X.">MB8~)Y_|6Pyw_:N\Ia| 'ۗ0 ND4EO2/z <\?v//Gwnd+ lc@tӅ0t$a_׋NlotHBTiUeD/`{Hb;)EwRIt.eN'aԯ@m Τ?t{) cF#fz7u׉&ǔ/822 #Yu eiL1ԙ耧dg_t&W=9˵#BgL 1ԁ< |_"Q3]=c).g6w__$֤ s[K9vьoI-HT"cyPO@>~_"07!B7¶-`C"D=a yϷM`Cȏv{FdGž'MsCP=r"wx SAMOzjB+A \8jNaCB|$UVLBsP ̥c HC'[0ECTDj&TrP̥"c, %Ii220f0+vOzU(F^B+ *BfaYBA2ۉ L13= ?S(dK!P>=V,&[\dP'6/쩘O8 ]B>.aCr+`ˠ Rm ]T ?*A ȦPcH%ȴ~Q$Q'ɢ,Z 6Ð_ֆKG6vV~S܃1C=*>F>.`C]yx{j1C<2$T Fo^.`Cnjw81Ev΂1C_q!Jފ8(4~ Oˇ0wX4('e>Վq 0ATS{QR9h $x^2ICF˜FmMF3?dR0Έ{l_2nR;ԋ:w1m my[oj$mF`(OzV|FKՌ16bZ0|8O*a, ˥@E`Cyj6><.t^ ۤ]J67u=~cC¶fM.~gǏ3~P;"KW :߄q ]կ |U?qC:')`1Q3:9wGr%}܀:?O~IU\\DgOt39@.B _@~0m\X_"+4|_9"P TcʒIie:ӶR>.w s>@!7|XM>Gq<ί!Ü3E+ОUZj R+Gi)j @9jsUVG)*5s+x {s\G5Ri-u.5sLE@iȰSamh=U뢤RgUJ.)(2609RRΩo(m>?ao:<32>P)v]W]@]abuHD:GYBQgQR@W V|~|.T @G]ƨ˱hu2T] SW)cv vPR(+{0MJ7uSN_RwE=_JQzF)SI ("\s%MhmEW)ޠs(HEY1Eŧ2O}#:UXG Wݥ-J2-Jjwb\"m\)Q a'EJ8J3vRj_GߋcR-{SQ @ C9uREQ{ TjPXZiWڨ T@iŎ%bpVptt%j 2nw'^jDF ;ʔ~:VޑR/cԋ]T0զg6P|wq;|bG~]|5xIMo7MG7%wiLXUvHv~!Fɠ>i5a+(:R6_ʧ}*iRn7J*ڂ;R1mTV"FQ$v.=yUa/VF#U:JյNR%/u$UsjJ :Z yB++դڲRHP.R'-B6(u%~x" P9 sc}PSc}@8ip9!\..jDžڏmp];AW/dmX[&HrAWQrә=ޑq{}#炮/e>eM"X1/o T}"վ- j ]vB)PBϠBq (nȸ=wCY8u.ԥP+Vԫ UBP]Ύ5@+#V?'T՟2;#Ԥ.BcP-4 Nz:jBkL@WdxEMGϊſB=)(22?}3|;רFV@[1=8п#Z~41v1QO?gT;Js^[q3i!uD]^a}0TFCT:Ñqm6DįpSԥ +_[VF ;̯ڶ+vBƭޑߠ;NKK{vT?bz~^U/?Ce"Uq++?#SR?H]"xR AՆT(b:2nSznnW8Suݣ.xGȸ~C2'WG#z.EGB={qu(6r>.b귎+@&@Λ0 sny\a{FWsy\Ew=W4pՍ\cWXHgpX}?Pd(%:75 c;::P, E; sEFW2pmF7ܖD:\'O-eݙ{N]Kn;kÊh6Q`ԣ:ܻTP|ņV]&U)0hB]j˜>[O}{#EBx ]dTOiV#Zq.-8r;y̍l:uN1C;=~6ք}~6t# \τ1;Y{: yqVΏ`5#ߥ+]U#NS C"n$œEr!}Qrz0g}yyU󟾧ѹW%eW1C%=~g}ňwˋCyrWrǮ9嫇 w_;rlLNS>veqK㾖 N?Х5u9Bw < lp =atN_lI$o&9qONKؓ3ȟtǨ0f$< =}-'6MfѶ mH8&zosƾ>rw<'JcL 9}8?|#BfY'W -pOL!+R.g{_M~ͥrsA6D˜:AT}\I?s~_UT37|BW -N35߹{ W,r\;Gیc; ?2O(7lԹ 6٧& 8Aй9֜0?+P*&^dxg!Ü ѹod~ŷ4&ɾhcz`sVw07R}|'}a^3Ke^̿x K}S 2nL4׼6rooŠFaIص}4SQCƭ ͠0 F]B(RZ')sGfeaYZ`ߍM7f= V3fp*C]*3K aVk6]efl!o?37[=#Î!jt^e֚ߛuvv9_g~)4W?:&ɾI=b92 t.2.|P8KnQ{ټ.c&Mx˼A>ɾQ~hoYÜf@A1)"G+%%PR$b4PA J`} ~(JpH <NOSgui%NJ遖.b@X0Jd 3<"V|"F,h' ԥX bŒbj`L`Վ+Q".jk_M_ kQ+i`Qx\ ;6vmйU]7aC`u>,v~ިGA ܢZA`gYa. dX@4~ .PQe9nVC&e}p-[̮IS6C=mA&ʎNpB]v*[YG;xF)x]9<ˮS~ ~T[Ep \H]BE+`T;T b]u1s3hyyuΫ^s*T_WPe}^CP] w ^FC̮Wy5||W E=XSw1z0?o]˟!gb\8 + ©oal}P@{X.d8]&\L.ysq0c^MQ\sUy^$R4\:7[]f+;;=½t+[ wνavgJVݵqD9o#oQlh@k"ɱ[m=\Né}t!4+~X gB=g V eNx L<~_ .+ez}sx::oп~Q'|H%|R?>Mgwg5~IتD~˜Qvaf=[u[^?гây #.8Qj<8p$)4#5|"|wMpHYQK6J *aX0_qP3ƳJF%D)\1ThږFƋZ;+_0"⽳I¶442FZ 8oqaL?CH7H/H v^x#}sCj4FG,cL$ԯ]Mxxُӏ"o#%7eO1C #;r%7fwb4dG|2؀G{"Nڙ0fwh_{'.϶,2@Sό+`]W#nGwU85(`[p;`՛`Q^|\FOFB˜~d4.GǍ!P: kCF=H]Ø~mÊ}A+9` zPps2+1 +N7DDo{W8$Ьw hqdoQ0ߴ⺨BvDu3925]a+6%^?FM?jgyØ?qk_5'RM+RԻzŅah.r;kØ_"d$?AR \ i E+t\INl.}$'Kڠ$BR;UH*'T|$$u C u)ctNJݒ˓H䧁-kIf(3ٻJ&% ^%KCiWliX{h;QXw騝]a`䊐eWT1[g=V\\lt*:Q.ٻꄊ̭ȶjB]js+ME;ϜY4&.bWZrĉ,`U+&i0Ny/Yξ\ = ΂Y2V.o0W-w_T&NdAunsa@Yڷr5P?A'2r',`URGNX ѹ\u]at֝NXͱ1 O3I',`sɝFm-еp6ǚܨl5Q;ޭ1[7\g8aAusVcA krc ]]:+NXWϟ<3I^',`]mt\Qto0W<ݻ/ke',ȺΕqN{=z;aj{=ԾNX٨߁nvjCt a`nדNXͱ1.Ag6ǚyp?5AW kr3j;. kruDCxn(n@p} ;8gvt_9acMnC+ytGu> XWe(?.1׻/Rd]"8A^\ tmumP; @7 XW;KPt:ajE6Dn3'lzpYy/ι]\C',`s|x TLR krlA LDyn]p :[8g2v1t󜰀uP.Vߢw@w XW{$?H~o*}!$Q'59 #NX[]ڕ5o_WS~?߿vfeHOKO*"~ͩ˫N˸L+b3er06S6)l3r0m<0Nm{bw~߿Zީh{t/Dd 9#>  # A"2.Tl1dsnE _Q[F.p`!.Tl1dsnE _Q[FNs`{@V.x͝tKXY2,KlDrFTDDE "Iኂb #P@ɰdDԫUӳUO}{{s^)׿gt=5DH_.r%%%%)% y 2EƤ*7" %%=?eh~:tK⒒BAW&2b)w##5Gmo֦(^a9ꅾyZt{o*W?CfAC[ԟJyߺ-:(-'K/=+\؂$K}m"\{rTqTt{{ŸW|rW|oǽR{՗~w{`'DKiT-홪=Hg*&}B_XN}=B:F,hF/E|}4R z4̀>pA=L׉ A'. Aϼ\˵iz^R>_R =_+BH.%)O{$IfQ+G#~'}D<cn(D ȃ~|+M s+N1_7nb2LlI'5F|J^[[<'3%iJͬTF4l gzݤVR9LsҗzX%f|*M2JwJsRmC?|n_W 7z7>LNɛx*D|@Gj{YyDQvT W]g}2~tOolTAQ˼_(䮊(ߦ|!=W&K*wJ6UR(}!Q(g٢>":j_g}}V,PkmTG^nSP gՕrA.VsOԩB:A8ΨA)2xQBkmE Ц C98m0C ~E bp\s3Z Qۊ#b}}ZWC84CSRY-W }AB*E^K>mcs_W6 )x?:_/7+i 5bvTf[lV7Gjcts3_/7n~|B?mGV=N~uF=RhEF2ɸf%$xz>9o,'qs=7wV1b]gf'9do^F[]9E-5t&C4+2k?e#H,n鐘K|,ҝ$⹸ux:5|3^ZvuG6j}-'z}M`)no5m}M>MI~_g _dZg}M"_O^|m}_OiJK_!nQasKE+Qs~j- ۂAX³bI!uRA))UMhf J6'`%eߖ:\u) WR98bfZsp ̮ mH =jFq8bO06fH#jZspfjLc~!mjW{5֘탪G$bKƳ P060;r3Rjp -+0"ҀnHf6<QU;\gT\ FqF>P;՚dm֩ }P.B}  FfV1r3RAv]! ܦ'6`0}⌊z<՚h@>"bS)AKU#Zu5 ֘퍪^pY WS Xb,䦧Zpt.rȭO&=j_FplwFŻz,՚h@>ƶiH#Fz4jmLy".qʓҫS|n#9S11zAcGS1Xb,䦧Zcpt.rȭO=j_FplwFŻz,h@>ƶi@#2>"r]O"p(OeQyrz~*i@ƩVn@nިgXg9jT+рUT+7Nr.T+7ȮAT{3*ދc2*ԣV(nSU@.[A6ƎкQC[Z!'Zbn}Npŝ/ a4 m`c 7- X>]et]5u=jobjWFŻz4h@>۔`%)~kn)}Ԣ-S! )Ol/Tݔ}NJ} ~Fr?M]:ԤR (OL2.J N)g@*veT GSF |mS`%JuSRDL*UӫS{ڴWPͧEj=p9ꅪ[;\uR3K5xd?MM:VSի4uYP5`W.BjUF6#CTUP }oZ~^AVkhMF JzKͣTD-,[\er*TL8V6*BJp2rKU.퉪k;\u·VC˦}<+@TXcZN<:* &_tgCU$!~u$tk$9B!Kfܖ%@Hcs)J6KEW;3;4~ik-#*Gu6D-W!ɿi>ddJY=,RKO'\+Hm"\]b?_.BT!%Hϓ|~Op sk;\u3WG/0)藪pT?B aߑ1?3w/U,TφK2/[>K~y?Y"$ y7bw0V藪=!p3~ޡo$_W0?~Gu6D k>CDc)5e/Uo,TN~XF:ucpr*fH_BT!R'D~4GTr17;'CTM =Qu }Fxb"aiFR5 G@<y5oC3n'{}~Cf"!SBY$0yƍd>q #w~jIVZ"$KBfdWEdQW앏GU"W;2F%rH/\g藪(9U5!:Lhn`_`ߌeL605T:y:2-|ļǼfbCT:y:| Ŝ`v5J*w+:-̼p'm{O2̜T+Pk6Yf2<^G/0*NN^`3 nJbFT:9'eE{Gi3 160yPEGcyH 1v/cykaكF1~ *}/q8yE78bqcd۬53T:y:Yg|3?67' s#/7~b^N~ Nk\@9EwO1.sy U5P)pð(in%<1ƻӰT:y:($H75#`TfQN5j!940j;FiF}ƫ *tLyPԃWF!iD3JQF )͉f\Kt nfY *}F2{gزn lF1kg3؞G@~{zw3{yB.Ʊ8VfMN1rދmI5J'D?[۰Ҿd~@|lePɋH,"돑Hg3X}SO ɣ'<avWS_&) IM/aYF>Tz9*=$C:|2>!.}x-Ҷ-^UA[],Q+fj3g'ێcedվ ivFH;yY;U|I^Ҿ"/j&s,;W;βAϽ`Ưpgi?Q|]FjqƟ2L*Gpl4u>ь7DŹo[z̲,T] fvQջ @(78@Zziz?.UWJ*tUHW|ۡ: gA%QMCeM˲1;W&Msse*=Lcٍt6Js;:|\t/:_yD3+ˌ7^y ʛe,Yve)˙ *tt^?w :Q};1Q>ZGD+);ieddM϶1z8BODGXvm$fFDgZ]=ͲJ#t~-OU )GN̊3NgF/ QŚY,;1*Zj`ԴxVOԳnƬ,{s4β}AWw:em@(6Zg̲D[*VhՖeD+l"ZU-/ZÊF ,%ZJò%r=hP*:9wJ.Br:*ͬScՌFا$P#p:$6[ɝvF~_ސ;3T-DZ2PkŚ/^[q?Vq5N`= wX0N!=Vw^Mf`n, TzSfg,:[t tbqUGkU'Yyx.O(OdK.KY"^cvz:g(K?0]e_i_Zh^dbg`ƹWaf`blcLN|mH?Y זּ{ˑN3UT:2:\Q:_ 9J,KͪȖt΋v-ɗ A$:!jO3uTX:GYOاbvl#*cUbSJ'E׊U\9D{Y'V-eybM{NlPbĞFcbQ9=$bP1p F$_(n;1bg냪/tr_c]cěc݁~;#w{mZj *|:xTkk%3V_/V g>w3^U~jJ'EsőH~0VQcqX.#1GűL9Ḅ STL7yaFOvya:]΋3gb% ̎]RkvFyAoa7/(v@X;(76NT::!ߍm>>>cd_#y >†ا0Q>>T؇jw;\m o-^-w-c w /CEs¯sosF}ol}Q>E\E鼩Ÿ& $ONq? 3<\B gc/> Ff'ώ UPU =*jG Bp2(sU7%̨F0\h cc:B[T-CBP}#X_0[w\u |2*m avT2*%T(aOC2{2fGFGGC?VQxGf <ϢiϢ? Ϝ9i¿aq6dgT2/f-,GiY|2*>pUe_5»at&̃>|j]<0TRa;ַ-JWݷjW+h6E6fϣ$];*]  {݅6dBd'狵oWkuv]/w)umnW>i_)ֱkv- UeZbO{!^ tx}VXvKQlgw[1r':dwC>ǂo{]>@݇=l B_7۰()^i8﵇}9t8eq/ً%eF^">lclaL ND| D{8.?abq?Li!>t8qgۿDl?ͳgy t8Jo yM7 ئ_i?;.ge!r:x>Wvk݀H:^"_~'mAgvoi}(0@i?su'w>6ʼK!!~`}6?p" xG 1{tž%}kOdYw6?brGSNxK)ߵKZ霽J22K_H$Z#?*Tb#F%xqIuFudY^ND6:l;̏|~T,愎H f4$,Ge6.;RM3f;W坺rTc3֓+l-P3XԑH2?6X k՜лYn"u:c]:eNW]nR7cMqB3Nn<.pjKkmM2\"㶝~,>2wgY#vܝms@/|!8d~\ÏY ͽ`[=r_+@WE}-(K=?e~}Ï X"ɘ<ؑ92JEe*GpQNx :ک¯fa.ѧ0z3)N5bÿϘft3K$? vw&D_tF0\[}: eΪ[B]},uw>KCUFF gKts(s8G?`э]\~(GU6;YVU8>lp;eRv9NW`*{6͕MN{Ӝe+߰Ϙ= R~rW:C ÿO<s:Au?9_1F\/8Ӕ?ٝ8YvqqEw.#p)] ^rw:K2ݍ@ (~(![%goR=TpesJEj|ZSk69O{::sϹ"n!U[שM]~Ŗw-\}|:}z [׺ ?yzs : p+sD}H>rTIf]sWw܅,JeVvԡnǸjs;:e=wi;Wvϫݨ6UqUYJNrhOڳnum{e\N# n5YnM Uf_T3vo^v{k}n;ʲ#Ř}Vj?hZAtOp;Z[҃kE[3>ph K?cqhƹ3~ྨ}׀4Q^60ʧ6smMnvj||*|ۈce{j6g8v-~t#vYn']~P6:4Fn~֭^&pW]xVu?ـ:cEw7R='As=~cn<N_abı2{TQ8}̃WX|/Ϡߍoүwji"K3OxZNdlH>7BW]P=~m :4xWi$1ĵf,WX ~H5C{DSp[DM4c6@W&UB!^Wt@3뫐aVInLa^w: ~=Sy:<,DrAb$:1l ~UE<NcG f\k%sb^1 _-Ǟñ2ijoޒxx^1{&k}!"tx;z*~98//DNk>{Gq&pt_+8:ێNpIl3&QO$SZ)!~4gb}3' %~[% ~]Y&oYeY!%*uxE)զ;plKƽ9O'D%F}~T$W;Ot u,hVZ^=+>YxL2IfEۓo>Ƹ>Y돫ZH54Cf >F~j.:dc7f&fYO<ZgEYȨȣY/#}~ TSצ?"/%"yFJ2?=Jrp@_3ՒoȻH^)ٔ|+Wڑ^`mɸ7gm'ɡ-0w@Mp/Yu ~ |šOf!e%|w/#>_ONrEd:[Dre?jw2k@GU;^z|4Gɳy>?1Y* inLf<HpchI6 1(T+ɒ:6)qIq?% 3PMIW y*D$NjIVfdՒdx7CdB7 y}2Nw'Z4(_i8Վdx=8% CIN5/}:T$+ԃ4'NfHvmpJ ullT4;ƍNΡ9MvEKgg`rg51{Cve9 %|=\s!;O1;Uh0Z*A50=2*]>] ܊S\MGLvxI{J g ud񫿠_ڍj{vx89}]̮Ee_ UpC|XŅTͩCPi9\Q*TZNIr8FN5Z/׷ }*VgTs«lDZr2|;hVHbdNu:)"6 Q=^ElRNxHe9tyN pZfsڍj{Nx89mT+`C9|XŜT+i/\B\Cڸ\-pkzqqUr 2 N0DUܤTknTf(Wj {Qm/^o;mΨx3PZ# ȇpXEi +3סS`-Ƙo±ˇ+RFr/fV1rS ;<_ rfKM"p".TʇW܆cgT9PZ䃸 ~+7ȪVǐ+J"plf;Bp 1P{Z^b&ZAv"fQ O!e-И.Pܺ \BBxem8yS1+m.` V[cpZj\Q57a*+ ǮW5р UTk ni&gT1bxuyH~hK+6*Q"WPm^x}q>Tkq Xȩ~knUsR AQ WǮ W|P{Z =qXpMH& |u1zn]jM ^ٝWފc2*FjTFrOf8V1rRjp*B*ph>V@> FollowedHyperlink >*B* phBOB table$d`a$B*CJhphvGKKLMNOPQRSTUVK   KLMNOPQRSTUVY   K n ,n G+HDZb Kh(v****>++++,,V@@fA9BB1DDbEEG%a5K**T++0,?BABrDE"RK| u $-$/$7+N+V+|;;;] ]"]] ^ ^{{{||}}~~āρe|~l=K t t tt t t t tt t tX !!tv,-OR$O˓1_)Mn2$XKSXZd#f2{l42$g[XE\H~12$Tl1dsnE _Q[F.@$,0 (  T  C O NB  S DO  C AnC:\~My Documents\web site\research\effectsize\eqn4.jpgG"B S  ?\Kq t 7Y7tqW4 _Ref19356206 _Ref468180341 _Ref19356035 _Ref19356122 _Ref471893470 _Ref4486165 _Ref19356229 _Ref19356269u$v;}}^eL0$;~~L5925    7 ; mu ((j,v,`.e.]1d1v<{<<<==B=K=W=\======>>&>l>n>>>>?? ?#?%?V?_?l?u???@@1@6@;@@@y@~@@@@@@@@@@@IANASAZAAAAA BB\BbBBBBBC CA%c;>˻=wż޼B޽RSþʾپv>x%,?yfC|FO"+&V2dc_ab2XYrsL Robert CoeCC:\My Documents\Academic research\effect size\effect size BERA2.doc Robert CoeUC:\WINDOWS\Application Data\Microsoft\Word\AutoRecovery save of effect size BERA2.asd Robert CoeCC:\My Documents\Academic research\effect size\effect size BERA2.doc Robert CoeCC:\My Documents\Academic research\effect size\effect size BERA2.doc Robert CoeIC:\My Documents\Academic research\Conferences\bera02\effect size BERA.doc Robert CoeTC:\WINDOWS\Application Data\Microsoft\Word\AutoRecovery save of effect size BERA.asd Robert CoeIC:\My Documents\Academic research\Conferences\bera02\effect size BERA.doc Robert CoeIC:\My Documents\Academic research\Conferences\bera02\effect size BERA.doc Robert CoeIC:\My Documents\Academic research\Conferences\bera02\effect size BERA.docTeresa Farthing,\\LIBRARY\USERLIB\BEI\EDUCOL\ACCESS\2182.doc1k  hh^h`OJQJo(1k@HH@HHK@G:Times New Roman5Symbol3& :ArialCFComic Sans MS"qhi&i&SiFKd[|\#20d8Guide to calculating, interpreting and using effect size Robert CoeTeresa FarthingOh+'0 4@ \ h t 9Guide to calculating, interpreting and using effect sizeguid Robert Coelobe normal.dotlTeresa Farthing2reMicrosoft Word 8.0g@@> X@2c@2cKd՜.+,D՜.+,l( hp   CEM Centre|[1 9Guide to calculating, interpreting and using effect size Title(RZ _PID_GUID _PID_HLINKSAN{24DFA037-CEF3-11D6-9C06-0002B363AB4C}A@D**http://acs.tamu.edu/~bbt6147/aeraad99.htme7C:\~My Documents\web site\research\effectsize\eqn2.jpgc7C:\~My Documents\web site\research\effectsize\eqn4.jpg  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ACDEFGHIJKLMNOPQRSTUVXYZ[\]^`abcdefkRoot Entry F\ ccmData U1TableB(WordDocumentЉSummaryInformation(WDocumentSummaryInformation8_CompObjjObjectPoolcc  FMicrosoft Word Document MSWordDocWord.Document.89q