|
Korean Job Discussion Forums
"The Internet's Meeting Place for ESL/EFL Teachers from Around the World!"
|
| View previous topic :: View next topic |
| Author |
Message |
Fox
Joined: 04 Mar 2009
|
 Posted: Sat May 14, 2011 9:48 pm Post subject: Study: Pundits Are No More Accurate Than Coin Tosses |
 |
|
Article here.
| Quote: |
Are Talking Heads Blowing Hot Air?
An Analysis of the Accuracy of Forecasts in the Political Media
By Holly Donaldson '11, Russell Doubleday '11, Scott Hefferman '11, Evan Klondar '11 and Kate Tummarello '11
Executive Summary
Introduction
In this paper, we report on the first-ever test of the accuracy of figures who made political predictions. We sampled the predictions of 26 individuals who wrote columns in major newspapers and/or appeared on the three major Sunday television news shows (Face the Nation, Meet the Press, and This Week) over a 16 month period from September 2007 to December 2008. Collectively, we called these pundits and politicians �prognosticators.� We evaluated each of the 472 predictions we recorded, testing it for its accuracy. Based on an analysis of these predictions, we answer three questions:
Which prognosticators are most accurate? We found wide disparities in the predictive accuracy of these individuals, and we divided them into �the good, the bad, and the ugly.�
Which characteristics are associated with predictive accuracy? We examined the effects of age, education, ideology, and other factors on accuracy.
What is the purpose of media pundits? We discuss whether the ordinary citizen should look to pundits for deeper analysis of events, or whether pundits are simply a more enjoyable way to learn about the events of the day. We also consider alternative viewpoints, including the notion that pundits are useful as representatives of opposing points of view in the country, and the idea that they are simply entertainers.
Methodology
In order to test the accuracy of prognosticators we needed a data set of their predictions. We chose to evaluate pundits during a 16-month period between September 2007 and December 31st 2008. This period was selected because we believed many predictions would be made as a result of the 2008 elections. Additionally, picking a period that ended a few years ago gave us ample time to test each prediction�s accuracy.
To obtain our final prognosticator sample we first generated a sample of 22 print media columnists and 36 TV prognosticators based on who was most widely syndicated and who appeared on the network Sunday morning talk shows Meet the Press, This Week, and Face the Nation more than 5 times within our evaluation period. From our full sample of 58 we then randomly selected 25 prognosticators to create our final sample. Later, we added George Will as our final (26th)prognosticator, due to his enormous presence in the media scene.
We randomly selected a sample of columns and transcripts for each prognosticator, which we scoured for predictions. To identify predictions, we compiled a �dictionary� of predictive language. We rated each predictive word on a scale from 1 (will not happen) to 5 (will absolutely happen,) with 2 through 4 ranking words in between.
Other variables recorded for each prediction include the demographic information about the individual who made the prediction, the topic of the prediction, whether the prediction was made on TV or in a newspaper, and a few related variables. Finally, we evaluated each prediction to see whether it happened or not.
Results
We discovered that a few factors impacted a prediction's accuracy. The first is whether or not the prediction is a conditional; conditional predictions were more likely to not come true. The second was partisanship; liberals were more likely than conservatives to predict correctly. The final significant factor in a prediction's outcome was having a law degree; lawyers predicted incorrectly more often. (R-square of .157) Partisanship had an impact on predictions even when removing political predictions about the Presidential, Vice Presidential, House, and Senate elections.
A number of factors impacted whether a prediction was extreme. In other words, we measured whether certain things led to pundits saying that an event absolutely will or will not happen. Again, conditional predictions were significant, and made something more extreme. Predictions about the GOP primaries were less extreme, as were predictions about the Vice Presidency. Predictions about the partisan makeup of the House were more likely to be extreme. As prognosticators aged, they became less extreme. (R-square of .145)
Implications
We have discovered a number of implications from our regressions and analysis of the data. First, we have discovered that six of the analyzed prognosticators are better than a coin flip (with statistical significance.) Four are worse, and the other 16 are not statistically significant. A larger sample can provide better evidence addressing the question of if prognosticators on the whole are better than a coin flip. We understand that being better than a coin flip is not a high bar to set, but it is a serious indictment of prognosticators if they are, on average, no better than a flipped coin.
According to our regression analysis, liberals are better predictors than conservatives�even when taking out the Presidential and Congressional election questions. Whether this holds true from election season to election season needs further evaluation; liberals may have implicit benefits from Obama winning the 2008 election. Tentatively, however, we can assert that liberals are better predictors than conservatives. Additionally, individuals with law degrees were less accurate than those who did not possess law degrees.
A final important implications is that we did not discover that certain types of predictions tended to be more or less accurate than others. For example, we did not see that economic predictions were more accurate than healthcare predictions. This suggests that prognosticators on the whole have no unique expertise in any area�even on political predictions, like the Presidential or party primary elections. |
The full study is also available at the link. I don't think these findings should really surprise anyone here. I suppose the idea that conservatives are more prone to make erroneous predictions that liberals might be controversial (though given the conservatives and liberals included in the study, I believe it; a large percentage of the conservatives included are patently full of shit regarding almost everything they say, and it's no surprise that fact checking confirms this). Paul Krugman was also ranked the single most accurate individual in the study.
I think the study would have been better if it had been complemented by an equivalent study done on non-mainstream media, both to see how their predictions stacked up, and to see if the trends remained the same.
The nine predictors substantially more accurate than a coin toss:
-Paul Krugman
-Maureen Dowd
-Ed Rendell
-Chuck Shumer
-Kathleen Parker
-Nancy Pelosi
-David Brooks
-Eugene Robinson
-Hank Paulson
The five predictors less accurate than a coin toss:
-Sam Donaldson
-Carl Levin
-Joe Lieberman
-Lindsey Graham
-Cal Thomas
George Will and Hillary Clinton managed to just barely stay out of the "less accurate than a coin toss" category, with Clinton managing precisely 50%, and George Will only slightly above that. |
|
| Back to top |
|
 |
Kuros
Joined: 27 Apr 2004
|
| Posted: Sat May 14, 2011 11:43 pm Post subject: |
|
|
| Quote: |
| We chose to evaluate pundits during a 16-month period between September 2007 and December 31st 2008. |
Worthless.
| Quote: |
| liberals were more likely than conservatives to predict correctly. |
During an election year where Democrats swept. Hrmmm. |
|
| Back to top |
|
|
Fox
Joined: 04 Mar 2009
|
| Posted: Sun May 15, 2011 12:41 am Post subject: |
|
|
| Kuros wrote: |
| Quote: |
| We chose to evaluate pundits during a 16-month period between September 2007 and December 31st 2008. |
Worthless. |
That's over a year's worth of predictions, considered after enough time has passed for those predictions to be determined true or false. More data never hurts, but calling such a span worthless seems hasty.
| Kuros wrote: |
| Quote: |
| liberals were more likely than conservatives to predict correctly. |
During an election year where Democrats swept. Hrmmm. |
It's worth noting they didn't limit themselves to political predictions. In fact, it seems like they were trying to be as generous as possible; the prediction that kept George Will above the "worse than a coin toss" lie was a prediction about baseball of all things. It's also why they additionally pointed out that liberals were more likely to be correct even when political matters were subtracted.
That said, Democrats doing well may well have been a factor that helped the liberal side of the equation to at least some degree, but going that route enhances their damnation of the punditocracy rather than mitigates it, since it renders even the people who were correct correct merely because of luckily being on the "right side," rather than having engaged in thoughtful analysis. You're essentially implicitly arguing that these talking heads are even more full of shit than the study implies. That may be true. |
|
| Back to top |
|
|
Kuros
Joined: 27 Apr 2004
|
| Posted: Sun May 15, 2011 6:33 am Post subject: |
|
|
| Fox wrote: |
| Kuros wrote: |
| Quote: |
| We chose to evaluate pundits during a 16-month period between September 2007 and December 31st 2008. |
Worthless. |
That's over a year's worth of predictions, considered after enough time has passed for those predictions to be determined true or false. More data never hurts, but calling such a span worthless seems hasty. |
This is not the first time I've seen this study. I've seen it discussed elsewhere.
I would want a decade or so of data to go on. And if I had to pick a year, it definitely wouldn't be an election year, particularly such a dramatic partisan victory as took place at the end of that period. |
|
| Back to top |
|
|
ontheway
Joined: 24 Aug 2005
Location: Somewhere under the rainbow...
|
| Posted: Sun May 15, 2011 6:56 am Post subject: |
|
|
We have to look at the intent of the pundits - each one individually.
It is likely that in many cases the intent of the pundit is to move public opinion rather than to make an accurate prediction.
Further, a pundit could predict a win in order to encourage supporters and a bandwagon effect or a pundit could pedict a loss in order to increase turnout by stimulating the fear of losing.
I have never actually considered the "predictions" of these talking heads to be anything more than a mix of interesting discussion and blather. |
|
| Back to top |
|
|
Kuros
Joined: 27 Apr 2004
|
| Posted: Sun May 15, 2011 11:08 am Post subject: |
|
|
| ontheway wrote: |
It is likely that in many cases the intent of the pundit is to move public opinion rather than to make an accurate prediction.
Further, a pundit could predict a win in order to encourage supporters and a bandwagon effect or a pundit could pedict a loss in order to increase turnout by stimulating the fear of losing.
I have never actually considered the "predictions" of these talking heads to be anything more than a mix of interesting discussion and blather. |
I think that's what Fox was getting at, and I agree with both of you. |
|
| Back to top |
|
|
Nowhere Man
Joined: 08 Feb 2004
|
| Posted: Sun May 15, 2011 1:01 pm Post subject: ... |
|
|
| Quote: |
| I would want a decade or so of data to go on. |
Yer, and conviction is a factor. 472 predictions seems an awful lot, especially if we're going to toss in baseball quotes.
let's say you're testing weather forecasters. Why would including their baseball predictions be a valid aspect of research?
In addition to conviction, I'd have to add specialization, the areas that each pundit is versed in and knows intimately.
I'm no fan of George Will, but judging Chomsky by his Super Bowl predictions is a bit silly.
Too much quantitative, not enough qualitative. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
