Lab 16

Effect Size, Method of Thresholds, and Review

Effect Size When we reject a null hypothesis, we would like to give an estimate of the the size of the effect we believe is present. This estimate is called an Effect Size. Effect sizes are standardized measures of the effect of a variable. There are many kinds of effect sizes but for now we will learn just one: Cohen's d. Cohen's d measures the size of the difference between 2 means. For now, we will only concern ourselves with the case when the populations we are comparing have the same standard deviation. In this case, Cohen's d is this: This formula should look familiar. It is just like the z-score formula! It measures how many standard deviations the means of the 2 distributions are apart. When the 2 groups have different standard deviations, Cohen's d has a slightly different formula as seen here. We won't concern ourselves with this case in this class. So if you have a sample mean of 20, a population mean of 10, a standard deviation of 5, and a sample size of 100, you would conduct a z-test and conclude that the sample does not come from this population (If you worked everything out, the z-test would equal 20 and you would reject the null hypothesis, assuming alpha was .05). Now we want to give an estimate of the size of the mean population differences. Note that for Cohen's d, sample size is unimportant. Also, Cohen's d can be negative, depending on which mean is considered the first mean and which is the second mean. In this case, Cohen's d would be (20 - 10) / 5 = 10 / 5 = 2. This means that the population from which the sample was drawn is estimated to be 2 standard deviations above the original population. Suppose that you conclude that a sample mean of 4 did not come from a population with a mean of 8 and a standard deviation of 16. Cohen's d would be (4 - 8) / 16 = -4 / 16 = -.25 Blackboard 1 and 2) Answer the 2 questions in Blackboard about effect sizes. Method of Thresholds Suppose that you know that 22% of members of Group 1 passed a test and 88% of Group 2 passed the same test under the same conditions. Even though Group 2 has 4 times the passing rate of Group 1, it would be a mistake to conclude that Group 2 is 4 times better than Group 1 at whatever ability the test measures. Although you either pass the test or you don't, the underlying ability measured by the test is not something you either have or you don't. Rather, ability is a continuous trait that can be low, high, or anywhere in between. Ability is not related in a linear way to the passing rate. We assume that if you have enough of the ability measured by the test, you are likely to pass it. The point at which you are likely to pass is called a threshold. There are complex ways of looking at what a threshold is but we are going to keep it very simple: If your ability is above the threshold, you pass. If not, you don't. If we can assume that the ability measured by the test has a normal distribution, we can use the properties of the normal curve to make an estimate of the "effect size" of the difference in the 2 groups' abilities. The method is simple: Cohen's d = NORMSINV(.88) - NORMSINV(.22) = 1.95 Here is what is happening graphically: The shaded blue is the 22% of Group 1 that passed and the shaded pink is the 88% of Group 2 that passed. Converting these passing rates to z-scores with the NORMSINV function in Excel allows us to compute the difference in means in standard deviation units. We may not know what the scores of the 2 groups were in the original scale of the test but that is not needed here. We know that the Cohen's d of 1.95 means that Group 2 is about 1.95 standard deviations ahead of Group 1. The method of thresholds can be used also for behaviors that we don't normally think of as "tests." For example, if 95% of Hispanic voters voted for a candidate and 50% of the rest of the population voted for that candidate, we can estimate how much more Hispanic votes found the candidate appealing (assuming that all the diverse motives for voting can be aggregated into 1 variable). In this case: Estimated Cohen's d =NORMSINV(.95) - NORMSINV(.50) = 1.64 Blackboard 3) Use the method of thresholds to answer question 3. Review Questions Blackboard 4) Calculate the standard error. From Lab 11. Blackboard 5) Probability of a mean falling below a value. From Lab 11. This spreadsheet might be helpful. Blackboard 6) Null hypothesis question. From Lab 12. Blackboard 7) z-test question. From Lab 13. Blackboard 8) Type I and II errors. From Lab 14. Blackboard 9) Power. From Lab 14. Blackboard 10) Confidence Interval. From Lab 15.