EDMUND BURKEThose of us who are political junkies read with great interest lots of polls during election campaigns, but we should ignore surveys that ask voters whether something will make them more likely or less likely to vote for a particular candidate. For example, Gallup asked voters whether Romney’s 47-percent statements will make them more likely or less likely to vote for him. Gallup reported that 36 percent of registered voters said that Romney’s remarks made them less likely to vote for him, 20 percent said Romney’s remarks made them more likely to vote for him, and 43 percent said it made no difference. If you want more worthless data, such as how independent voters answered this question, you can find them here.
Why do I think these kinds of surveys are worthless? People cannot reliably tell us what motivates them to make decisions, including decisions about which candidate to vote for or what product to buy. People are significantly influenced by emotional rather than rational responses. They themselves may not consciously understand what motivated them to make a particular decision.
Here’s a small example from my own life. Often I find myself comparing products. Maybe I want to try a new breakfast cereal. I find a number of candidates that look good to me on the shelf. I like to think I’m a pretty rational decision-maker, and I read the nutritional information on the sides of the boxes. I look at the calories, fiber content, whether the cereal is made from whole grain. I also consider which cereal appeals to me the most, which I think I’d enjoy. Finally, I narrow it down to four cereals. They are not identical – they have different pros and cons – but in my mind they are equally balanced. I agonize for a moment or two and then randomly throw one in my shopping cart. On the same shopping trip, I want to get a sun screen. I examine the ones available. I look at the SPF factor, whether it’s water resistant, promises “won’t run into eyes,” and the price. There is nothing else that I (consciously) care about. I narrow it down to two – say Coppertone and Banana Boat – that are both SPF 50, make the same claims, and are identically priced. It’s a toss-up. Again, I randomly throw one in my shopping basket. When I unload my purchases at home, I discover that I bought the same sun screen last summer. More surprisingly, I have an unopened box of the exact same breakfast cereal in my cupboard. Oh yeah, I forgot, I bought it some weeks ago after believing then, as now, that I was genuinely torn among a wide assortment of choices. Even when I don’t know why, I make the same choices over and over again. I’ll bet this happens to you too.
Steve Jobs was asked whether he used consumer surveys to help design the iPad. “No,” he said. “It’s not the consumer’s job to know what they want.” Matthew Willcox, Executive Director of the Institute of Decision Making, writes that Jobs probably meant that consumers aren't good at telling you want they want. You can access an interesting short piece by Willcox about what drives decision-making here.
That does not mean that all surveys and focus groups are worthless; it means they have to be designed and interpreted with great care. If my law school were to ask incoming students who were also accepted by a competing law school why they chose us, I’d take their answers with a grain of salt. It would be more revealing to ask what they thought about the two schools: Which school has a better reputation? Which school has a better faculty? Which school is harder to get into? Which school did their parents prefer? Which school did their friends say was better? What guide books and web sites did they consult? Which school has smaller classes, more clinics, higher job placement rates? The issue isn't which school is actually harder to get into or has smaller classes. We know the objective answers to those questions. It’s what applicants thought the facts were. Then we can try to correlate what they thought with what they did.
Comparing polls that were taken shortly before and after Romney’s 47-percent remarks can tell us a great deal. Voters and consumers can tell us whether they would choose A or B. But a survey that asks whether one is more or less likely to vote for Romney because of his remarks is treacherous because, at best, people have a tenuous grasp on why they choose as they do.