farmgirl
05-10-2004, 03:30 PM
With all the discussion on the board regarding polls and their accuracy, I thought it might be beneficial for you to read an article that we were required to read for one of my journalism classes. Dr. Blake sums things up pretty well.
By Ken Blake, Ph.D.
Assistant Professor of Journalism
Middle Tennessee State University
Author's note: With apologies to God and co-author Moses, I wrote this in 1995 while earning my doctorate at the University of North Carolina, Chapel Hill. At the time, I was the field director for the Carolina Poll, the statewide poll conducted once a semester by the School of Journalism and the model for MTSU's Middle Tennessee Poll. Anyway, the school's librarian was building a web page for the school and asked me to contribute some guidelines for telling good polls from bad ones. Much to my surprise, I've since run across it in several different places on the Web, all with my name attached. The moral of this story: Do good work, because you never know what will dog you for the rest of your life. Or perhaps: Don't appropriate God's literary devices.
Fortunately for me, the original form of this thing was pretty solid. Being older and wiser now, though, I've added a few caveats in the version below. I think you'll quickly see how you could adapt it to a lecture, although I'll leave to you the choice of whether to risk the "10 commandments" sacrilege. I've included some suggested follow-up activities below.
1. Thou shalt know how the people interviewed for the poll were chosen. The best polls choose people at random. There are many different schemes for doing so. For example, a pollster can program a computer to generate telephone numbers at random. Another strategy involves putting people's names on a list in some non-systematic fashion and then choosing every sixth, tenth or "whateverth" one starting from some random point on the list. Also, good polls make at least two or three attempts to contact each randomly-chosen person. Why? Because some people are hard to reach by phone. People who work at night, for example, or people who travel a lot. Furthermore, these people's opinions on some issues can differ substantially from opinions held by easy-to-reach people. If your poll is based only on interviews with easy-to-reach people, then the opinions of harder-to-reach people won't be represented. The basic rule is that everyone whom the poll purports to represent should have an exactly equal chance of being interviewed. The results of polls based on non-random samples -- the "dial-in" polls so popular with television newscasts, for example -- are generalizable to no one except the people who were interviewed.
2. Thou shalt know the poll's response rate. Here's a basic way to calculate a poll's response rate using any calculator.
Enter the number of people interviewed for the poll.
Press the division key.
Enter the number of people the poll attempted to interview (for example, the number of questionnaires mailed out, or the total number of working, residential telephone numbers dialed in a poll claiming to represent residents of some population).
Press the "equal" key.
The resulting number should be about .66 or higher. In other words, 66 percent or more of the people asked to take part in the poll should agree to do so. The lower the response rate, the greater the risk that those who declined to be interviewed belong to some significant subgroup whose attitudes and opinions the poll will fail to represent.
3. Thou shalt know the poll's error margin. Here's a basic way to calculate a poll's error margin using any calculator.
Enter the number .25.
Press the division key.
Enter the number of people interviewed for the poll.
Press the "equal" key.
Press the "square root" key.
Press the multiplication key.
Enter the number 1.96.
Press the "equal" key.
Press the multiplication key.
Enter the number 100.
Press the "equal" key.
The resulting number represents the poll's statistical accuracy at the 95 percent level of confidence. For example, a result of 4.2 means that if 60 percent of the people interviewed for the survey said they preferred chocolate ice cream over vanilla, then one can be 95 percent confident that the actual percentage of people favoring chocolate over vanilla in the population the poll purports to represent lies somewhere between 55.8 (that's 60 minus 4.2) and 64.2 (which is 60 plus 4.2). What's a good error margin? It depends. An error margin of 4.2 will let you predict the winner of an election if the poll shows one of two candidates has 80 percent of the vote. Subtracting the error margin from 80 percent reveals that, at the very worst, the candidate has 75.8 percent of the vote and is clearly still ahead. If, however, the poll shows one of two candidates with 53 percent of the vote, then you can't predict a winner (without using some really hairy statistics that you'll need a computer and some fancy software to calculate) because the candidate could have anywhere from 57.2 percent of the vote to only 48.8 percent of the vote. It's also important to keep in mind here that a poll's error margin represents the statistical accuracy only. Poor or biased question wording, low response rates, data entry errors and a host of other problems can detract from a poll's accuracy.
4. Thou shalt know the poll's confidence level. The 95 percent confidence level is the most common. Confidence levels exceeding 95 percent are even better. Confidence levels less than 95 percent should be increasingly mistrusted as they get further from 95 percent.
5. Thou shalt recalculate the error margin when drawing conclusions about a subgroup in a poll's sample. If 600 randomly chosen North Carolinians are interviewed for a poll, the poll's error margin at the 95 percent level of confidence is plus-or-minus four percent. But conclusions about the, say, 312 people in the sample who are female have an error margin of plus-or-minus 5.5 percent at the 95 percent confidence level. A pollster who wants an error margin of plus-or-minus four percent at the 95 percent level of confidence when drawing conclusions about North Carolina females needs to talk to 600 randomly selected North Carolina females, not 312.
6. Thou shalt make a big deal out of a crosstabulation only if the difference is statistically significant. A crosstabulation compares how people in different subgroups answered a question. For example, a poll might find that 60 percent of the women interviewed plan to vote for Candidate A compared to only 55 percent of the men interviewed. So, Candidate A enjoys stronger support among women in the general population than among men in the general population, right? Not necessarily. As a rule of thumb, it takes a difference of 10 percentage points or more between subgroups in a random sample to accurately suggest that a similar difference exists in the general population from which the sample was drawn. Ideally, you should get a statistician to calculate something for you called a Chi Square. Or, you can pick up just about any basic statistics book and figure out how to calculate a Chi Square on your own. It's too involved to detail here, but you can do it with any calculator and the table of probabilities in the back of the statistics book.
7. Thou shalt generalize a poll's results only to the population sampled. A poll based on a sample of UNC-Chapel Hill students age 18 to 21, for example, does not necessarily represent the views of all people age 18 to 21. Nor does a poll based on a sample of North Carolina state residents necessarily represent the views of all Americans.
8. Thou shalt know who paid for the poll, and who conducted it. Exercise a little healthy skepticism if whoever paid for the poll has a vested interest in its outcome or if whoever conducted it doesn't have a solid reputation for telling the truth.
9. Thou shalt know the wording of the question. Leading questions are easy to write. For example, "You certainly don't plan to vote for putting that crook, John Smith, back into office, do you?" might tend to yield results unfavorable to John Smith, whatever his actual character might be. Balanced questions present all possible answers as equally attractive. For example, "Do you plan to vote for John Smith, or for Jane Doe?" Also, try to find out the wording of previous questions. The answer to one question can be influenced by questions asked earlier in the poll. For example, a respondent asked about support for the death penalty after being asked several questions about the threat of crime might answer differently than a respondent asked about support for the death penalty during a poll that made no mention of crime. The computer programs used in conducting many polls make it possible to randomize the order in which questions are presented. Such randomization can help minimize the impact of these so-called "question order effects" in polls.
10. Thou shalt consider the ability of the people interviewed to give an informed, well-considered answer. There's not point in doing a poll on whether a prominent political figure wears boxers or briefs. Only he knows, and he's probably not telling. What everyone thinks he's wearing isn't going to alter the reality of what he is wearing. If there are any appropriate ways to investigate the question, polling is certainly not among them. Similarly, ordinary people should not be asked whether they approve or disapprove of, say, U.S. policy on Argentinian beef imports. Most people have no idea what the policy is, or if there even is one. Even if the pollster patiently explains the policy, the resulting answer is likely to be a snap judgment ****e to reverse itself the next day. The best polls probe not only a respondent's opinion but also the level of deliberation and conviction associated with the opinion.
By Ken Blake, Ph.D.
Assistant Professor of Journalism
Middle Tennessee State University
Author's note: With apologies to God and co-author Moses, I wrote this in 1995 while earning my doctorate at the University of North Carolina, Chapel Hill. At the time, I was the field director for the Carolina Poll, the statewide poll conducted once a semester by the School of Journalism and the model for MTSU's Middle Tennessee Poll. Anyway, the school's librarian was building a web page for the school and asked me to contribute some guidelines for telling good polls from bad ones. Much to my surprise, I've since run across it in several different places on the Web, all with my name attached. The moral of this story: Do good work, because you never know what will dog you for the rest of your life. Or perhaps: Don't appropriate God's literary devices.
Fortunately for me, the original form of this thing was pretty solid. Being older and wiser now, though, I've added a few caveats in the version below. I think you'll quickly see how you could adapt it to a lecture, although I'll leave to you the choice of whether to risk the "10 commandments" sacrilege. I've included some suggested follow-up activities below.
1. Thou shalt know how the people interviewed for the poll were chosen. The best polls choose people at random. There are many different schemes for doing so. For example, a pollster can program a computer to generate telephone numbers at random. Another strategy involves putting people's names on a list in some non-systematic fashion and then choosing every sixth, tenth or "whateverth" one starting from some random point on the list. Also, good polls make at least two or three attempts to contact each randomly-chosen person. Why? Because some people are hard to reach by phone. People who work at night, for example, or people who travel a lot. Furthermore, these people's opinions on some issues can differ substantially from opinions held by easy-to-reach people. If your poll is based only on interviews with easy-to-reach people, then the opinions of harder-to-reach people won't be represented. The basic rule is that everyone whom the poll purports to represent should have an exactly equal chance of being interviewed. The results of polls based on non-random samples -- the "dial-in" polls so popular with television newscasts, for example -- are generalizable to no one except the people who were interviewed.
2. Thou shalt know the poll's response rate. Here's a basic way to calculate a poll's response rate using any calculator.
Enter the number of people interviewed for the poll.
Press the division key.
Enter the number of people the poll attempted to interview (for example, the number of questionnaires mailed out, or the total number of working, residential telephone numbers dialed in a poll claiming to represent residents of some population).
Press the "equal" key.
The resulting number should be about .66 or higher. In other words, 66 percent or more of the people asked to take part in the poll should agree to do so. The lower the response rate, the greater the risk that those who declined to be interviewed belong to some significant subgroup whose attitudes and opinions the poll will fail to represent.
3. Thou shalt know the poll's error margin. Here's a basic way to calculate a poll's error margin using any calculator.
Enter the number .25.
Press the division key.
Enter the number of people interviewed for the poll.
Press the "equal" key.
Press the "square root" key.
Press the multiplication key.
Enter the number 1.96.
Press the "equal" key.
Press the multiplication key.
Enter the number 100.
Press the "equal" key.
The resulting number represents the poll's statistical accuracy at the 95 percent level of confidence. For example, a result of 4.2 means that if 60 percent of the people interviewed for the survey said they preferred chocolate ice cream over vanilla, then one can be 95 percent confident that the actual percentage of people favoring chocolate over vanilla in the population the poll purports to represent lies somewhere between 55.8 (that's 60 minus 4.2) and 64.2 (which is 60 plus 4.2). What's a good error margin? It depends. An error margin of 4.2 will let you predict the winner of an election if the poll shows one of two candidates has 80 percent of the vote. Subtracting the error margin from 80 percent reveals that, at the very worst, the candidate has 75.8 percent of the vote and is clearly still ahead. If, however, the poll shows one of two candidates with 53 percent of the vote, then you can't predict a winner (without using some really hairy statistics that you'll need a computer and some fancy software to calculate) because the candidate could have anywhere from 57.2 percent of the vote to only 48.8 percent of the vote. It's also important to keep in mind here that a poll's error margin represents the statistical accuracy only. Poor or biased question wording, low response rates, data entry errors and a host of other problems can detract from a poll's accuracy.
4. Thou shalt know the poll's confidence level. The 95 percent confidence level is the most common. Confidence levels exceeding 95 percent are even better. Confidence levels less than 95 percent should be increasingly mistrusted as they get further from 95 percent.
5. Thou shalt recalculate the error margin when drawing conclusions about a subgroup in a poll's sample. If 600 randomly chosen North Carolinians are interviewed for a poll, the poll's error margin at the 95 percent level of confidence is plus-or-minus four percent. But conclusions about the, say, 312 people in the sample who are female have an error margin of plus-or-minus 5.5 percent at the 95 percent confidence level. A pollster who wants an error margin of plus-or-minus four percent at the 95 percent level of confidence when drawing conclusions about North Carolina females needs to talk to 600 randomly selected North Carolina females, not 312.
6. Thou shalt make a big deal out of a crosstabulation only if the difference is statistically significant. A crosstabulation compares how people in different subgroups answered a question. For example, a poll might find that 60 percent of the women interviewed plan to vote for Candidate A compared to only 55 percent of the men interviewed. So, Candidate A enjoys stronger support among women in the general population than among men in the general population, right? Not necessarily. As a rule of thumb, it takes a difference of 10 percentage points or more between subgroups in a random sample to accurately suggest that a similar difference exists in the general population from which the sample was drawn. Ideally, you should get a statistician to calculate something for you called a Chi Square. Or, you can pick up just about any basic statistics book and figure out how to calculate a Chi Square on your own. It's too involved to detail here, but you can do it with any calculator and the table of probabilities in the back of the statistics book.
7. Thou shalt generalize a poll's results only to the population sampled. A poll based on a sample of UNC-Chapel Hill students age 18 to 21, for example, does not necessarily represent the views of all people age 18 to 21. Nor does a poll based on a sample of North Carolina state residents necessarily represent the views of all Americans.
8. Thou shalt know who paid for the poll, and who conducted it. Exercise a little healthy skepticism if whoever paid for the poll has a vested interest in its outcome or if whoever conducted it doesn't have a solid reputation for telling the truth.
9. Thou shalt know the wording of the question. Leading questions are easy to write. For example, "You certainly don't plan to vote for putting that crook, John Smith, back into office, do you?" might tend to yield results unfavorable to John Smith, whatever his actual character might be. Balanced questions present all possible answers as equally attractive. For example, "Do you plan to vote for John Smith, or for Jane Doe?" Also, try to find out the wording of previous questions. The answer to one question can be influenced by questions asked earlier in the poll. For example, a respondent asked about support for the death penalty after being asked several questions about the threat of crime might answer differently than a respondent asked about support for the death penalty during a poll that made no mention of crime. The computer programs used in conducting many polls make it possible to randomize the order in which questions are presented. Such randomization can help minimize the impact of these so-called "question order effects" in polls.
10. Thou shalt consider the ability of the people interviewed to give an informed, well-considered answer. There's not point in doing a poll on whether a prominent political figure wears boxers or briefs. Only he knows, and he's probably not telling. What everyone thinks he's wearing isn't going to alter the reality of what he is wearing. If there are any appropriate ways to investigate the question, polling is certainly not among them. Similarly, ordinary people should not be asked whether they approve or disapprove of, say, U.S. policy on Argentinian beef imports. Most people have no idea what the policy is, or if there even is one. Even if the pollster patiently explains the policy, the resulting answer is likely to be a snap judgment ****e to reverse itself the next day. The best polls probe not only a respondent's opinion but also the level of deliberation and conviction associated with the opinion.