Via James Kwak, I found Nate Silver’s posts on questioning the legitimacy of Strategic Vision’s polls. There are a lot of fascinating statistical issues here but I wanted to focus on the answers to a poll commissioned by the Oklahoma Council of Public Affairs.
The purpose of this poll was to measure how well Oklahoma High School students would do on a US citizenship test. One thousands students were polled and not a single one got more than 6 questions right. Here are the questions:
As Silver points out this is suspect, but what as an educator and former social sciences tutor I find even more suspect is the answers that students supposedly gave.
Here are the first six:
Looking at these answers it looks as if this poll must have been multiple choice.
For the first two questions, ‘What is the supreme law of the land’ and ‘What do we call the first ten amendments to the Constitution’, students answered the Monroe doctrine, 2% and 4% of the time. Unless the Monroe doctrine was covered recently this is a pretty large number of students to offer it as the wrong answer to a question like this.
Think about this from a student’s perspective. The first question uses the phrase “Supreme Law.” This means that answer has to be something important. The natural thing to do is to answer the most important thing you can think of. From free response I would not be surprised to get answers like “the bible”, “democracy” or “freedom”. The student might also simply free associate with things that are like Supreme or like Law. Hence, I would not be surprised to get answers like “Federal government”, “Supreme Court”, “Congress” or even “Roe vs. Wade.”
Since none of these answers were given, but Monroe doctrine was. I find this hard to believe unless the poll was multiple choice and the six answers were the only answers a student could give.
However, look at questions 7 and 10
The only two answers given for seven were “Democrat and Republican” and “Communist and Republican.” Were these seriously the only two answers provided in a multiple choice question. Not only is there is obvious silliness of using “Communist” and not even “Socialist”, but of course dropping to only two options destroys the power of the question. However, perhaps Strategic Vision is suggesting that there were other answers but they were chosen by no one. Strange that they wouldn’t include them and strange that no one randomly picked another answer given that 46% didn’t know.
Question 10 is even worse, however. The problem here is that “The Governor” is a correct answer to the question “Who is in Charge of the Executive Branch.” When designing a multiple choice test listing a technically true answer that is not the one you are looking for is a cardinal sin. I can’t believe that even the sloppiest of polls would do that.
In short, the set of answers seem highly inconsistent with being either a multiple choice or free answer test. This suggests that the answers were forged and done so by someone who does not have experience testing high school / college age students.