A Critical Review of The Unexplained Exit Poll Discrepancy

Rick Brady (rick@alohalee.com)

March 20, 2005. 

 

A day after Edison Media Research/Mitofsky International made available their much anticipated report (NEP Report)^[1] on the 2004 Presidential election exit polls in January, the University of Pennsylvania (UPenn) issued a press release^[2] announcing that Dr. Steven F. Freeman, an "expert" on the presidential election exit poll errors, has access to a satellite link and is available for interviews. 

Although Dr. Freeman is highly credentialed,^[3] his publication and presentation credits to date are devoid of any published research on exit polls.  His website does however include a link to one working paper, The Unexplained Exit Poll Discrepancy,^[4] several op-ed pieces, and indicates that two additional working papers and a book on the subject are forthcoming. 

Does Dr. Freeman’s exit poll research qualify him as an “expert” in the subject field?  I suggest not, but that in no way precludes him from conducting research on exit polls and his findings should be judged solely on the logic of presentation and validity of scientific methods.

The UPenn press release quotes Dr. Freeman regarding the NEP Report:

Although the authors of the report state that, “the differences between the exit poll estimates and the actual vote [are] most likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters,” they provide little data or theory to support this thesis,” said Freeman.  “Rather, the report only confirms the exit poll official count discrepancy that I documented in my Nov. 12 paper, corroborates the data I collected, and rules out most types of polling error.”^[5]

I intend to demonstrate that Dr. Freeman’s only publicly available research on the 2004 NEP Presidential exit polls is seriously flawed.  However, despite the flaws, Dr. Freeman is correct in concluding that statistical “explanations for the discrepancy thus far provided are inadequate.”^[6]

 

The Unexplained Exit Poll Discrepancy

In this review of The Unexplained Exit Poll Discrepancy, I start with each of Dr. Freeman’s conclusions,^[7] work through the methods he used to reach each of those conclusions, and present a counter-analysis to demonstrate that, in all but his final assertion, Dr. Freeman’s conclusions are wrong and improperly drawn.

Conclusion I:  In General, Exit Poll Data Are Sound

Dr. Freeman provides data and largely qualitative descriptions of exit polls from German, Utah, Mexico, and Ex-Soviet Block to make his point that, in general, exit poll data are sound.  Given the lack of data to support his conclusions for exit polls in Mexico and Ex-Soviet Block nations, I have limited my scope here to the German and Utah exit polls where Dr. Freeman failed to consider that disparate methodology could account for the greater accuracy of those exit polls when compared to media funded US Presidential exit polls.  It seems that Dr. Freeman thinks that all exit polls are created equal, which is far from true. 

German Exits

Dr. Freeman analyzes data from several German exit polls of national elections where where the predicted result matched closely the tallied result.  Mystery Pollster (MP) Mark Blumenthal uncovered an opinion^[8] prepared by the ACE project, which is funded by the United Nations and the United States Agency for International Development.  As excerpted by MP, the opinion states:

[Exit poll] reliability can be questionable. One might think that there is no reason why voters in stable democracies should conceal or lie about how they have voted, especially because nobody is under any obligation to answer in an exit poll. But in practice they often do. The majority of exit polls carried out in European countries over the past years have been failures.^[9] (Emphasis added)

In a telephone conversation with Dr. Freeman, one month prior to the publication of the final version of The Unexplained Exit Poll Discrepancy, MP urged the professor to check the German exit poll methodology before suggesting that the accuracy of the foreign polls have any relevance to the discussion of the 2004 NEP polls.^[10]  Following the conversation, MP contacted Dr. Dieter Roth of FG Wahlen, the organization that generated the German exit poll data used by Dr. Freeman.  Dr. Roth provided some information about methods saying, “I know that Warren Mitofsky's job is much harder than ours, because of the electoral system and the more complicated structure in the states.”^[11]

Put simply, the German exit polls are designed better than the NEP exit polls,^[12] and the potential for both sampling and non-sampling error is reduced by their methods, which explains why exit polls in Germany have a greater chance at achieving accuracy than US Presidential exit polls.  Dr. Roth’s data regarding methods and his statements were available to Dr. Freeman before he published his paper; but, this information was not incorporated into the working paper.

BYU Exits

Dr. Freeman explains how the Brigham Young University (BYU) exit poll of Utah voters in the 2004 Presidential election came within .03 percent of predicting tallied Bush proportion and 0.1 percent of the tallied Kerry proportion, but as with the treatment of the German exit poll data, Dr. Freeman does not provide information about the BYU poll methods.^[13]  MP queried the BYU exit poll website and found information on the poll’s methods.^[14]  As was the case with the German exits, when compared to the NEP methods, the BYU methods are far superior - hence one reason that the BYU exits have a greater chance of achieving accuracy than the NEP exits. 

But there is something more peculiar about Dr. Freeman’s selection of the BYU poll to as evidence that exit poll data, including the 2004 NEP Presidential exit poll data, are generally sound.  Why did he not consider the NEP exit poll results for Utah; the state where the BYU exit poll nailed the election result?  If he had, he would have realized that the NEP exit poll of Utah was off by 2.7 percent in Kerry’s favor.^[15]  Rather than demonstrating that the NEP exit poll data are “generally sound,” Dr. Freeman’s use of the BYU exit poll provides evidence to suggest that the inferiority of the NEP exit poll methods could explain the observed discrepancies. 

No Apples-to-Apples Comparison?

Dr. Freeman’s review of exit poll data excluded an important piece of literature on media-funded US Presidential exit polls.  The bibliography hit all the major exit poll literature sans one chapter by Warren Mitofsky and Murray Edelman written in 1995.^[16]  In that chapter on the 1992 VRS exit polls, the authors wrote:

The difference between the final margin and the VRS estimate (in 1992) was 1.6 percentage points. VRS consistently overstated Clinton’s lead all evening...Overstating the Democratic candidate was a problem that existed in the last two presidential elections.^[17]

Certainly this year's NEP Presidential exit polls showed greater unidirectional bias than other years, but that is not the case that Dr. Freeman built.  He chose to highlight data from exit polls that employed highly disparate methods when compared to the methods typically used for media-funded US Presidential exit polls and did so while ignoring pertinent literature on these exit polls that demonstrated chronic Democratic bias.

Conclusion II:  Analysis of Exit Poll Data Reveal Statistically Significant Discrepancies in OH, PA, and FL

In short, Dr. Freeman concluded that John Kerry’s predicted exit poll proportion significantly exceeded the Senator’s tallied proportion.  This section begins with a review of the problems with Dr. Freeman’s data and methods, followed by the implications of these problems for his conclusions. 

Freeman's Data and Methods

Dr. Freeman's null hypothesis states that, assuming independent state polls with no systematic bias, Kerry's predicted proportion should not significantly exceed his tallied proportion.

To test his null, the professor compared data extrapolated from exit poll data posted on CNN's website shortly after midnight on election eve to election tally data.^[18]  The CNN data were presented in tabular format and reported the predicted proportions for Bush, Kerry, and Other candidates by gender.  From the Male/Female split, which was posted on the CNN website as a whole number, Dr. Freeman extrapolated to achieve a value significant to a 10^th digit.  Although Dr. Freeman reported both Bush's and Kerry's "predicted" (exit poll) proportion of the vote, his statistical analysis is based only on Kerry's proportion; therefore, I have only reproduced these data for Kerry’s proportion in Exhibit 1.^[19]

Dr. Freeman’s paper recognizes that exit polls are not simple random samples, but cluster samples, and therefore have higher standard errors than typical phone surveys of similar sample size.^[20]  The difference between these standard errors is known as the design effect.  Dan Merkle and Murray Edelman calculated the design effect of the 1996 Presidential election exit polls to be 1.7, the square root of which is 1.3, leading the authors to state that the 1996 exit polls showed "a 30(percent) increase in the sampling error computed under the assumption of simple random sampling."^[21]  This 30 percent adjustment is referred to as the design effect square root (DESR).  To account for the design effect associated with the 2004 exit polls, Dr. Freeman applied Merkle and Edelman’s DESR to the standard error of Florida, Ohio, and Pennsylvania. 

 

Having estimated the standard error for each state, the professor performed a single-tailed test for comparing the results derived from a single sample to a mean of samples (or established standard) and determined that Kerry's proportion significantly exceeded the election result in all three battleground states at the 95 percent confidence level.  If a finding is "significant" (p-value <.05), then one can reject the null hypothesis.  If the result is "not significant" (>.05), then statisticians do not reject the null hypothesis; in fact a non-significant finding is just that – non-significant.  Freeman rejected the null hypothesis by stating that the observed discrepancies are "impossible" to have been due to chance or random error (i.e., the discrepancies in each state were significant).^[23]

Dr. Freeman's "statistical" analysis fails on three main points, which affects Freeman’s conclusions.  The analysis: 1) violates the "rule of significant digits"; 2) improperly estimates the design effect; and 3) employs a single-tail test when his assumptions require a two-tail test.   

Rule of Significant Digits

According to Dr. Freeman, the exit poll predicted that Kerry would win 49.7 percent of the vote in Florida, 52.1 percent in Ohio, and 54.1 percent in Pennsylvania.  As explained in Exhibit 1 above, Dr. Freeman arrived at these proportions by building partials from an extrapolation of the CNN data by gender, which was posted on the website in whole proportions.  In doing so, the professor violated the rule of significant digits, which states:

In a calculation involving multiplication, division, trigonometric functions, etc., the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied, divided etc.^[24]

Acknowledging the limits of his data, Dr. Freeman makes the following statement in a footnote to his extrapolations:

Displaying these numbers out to one decimal point is not meant to imply that the numbers are precise to that level of significance, but rather to provide as much data as accurately as I can.  Among the limitations of the CNN exit poll data are the lack of significant digits.  I did not want to unnecessarily degrade the data further by rounding numbers derived from calculations.^[25]

Essentially, the data used by Dr. Freeman are imprecise and some call a dataset like this “fuzzy,” while others describe it as “noisy.”  While acknowledging the limitations of the data in his footnote, Dr. Freeman does not seem to understand the implications of these limitations for statistical analysis.  In short, analysis of fuzzy data, yields fuzzy results.

 

Exhibit 2 shows the range of possible values for Kerry's predicted proportion given the error bounds of the data when a significant digit is considered (10^th).^[26]  If Dr. Freeman would have analyzed the error bounds associated with his data, he would have realized that statistical analyses (calculated Z-scores and p-values in particular) are highly sensitive the number of significant digits. 

 

 

Now that I’ve established the fuzziness of the data, there is also a degree of fuzziness associated with the estimation of the design effect that should also be considered. 

What Design Effect? Why?

As mentioned, the 1.3 DESR was applied to the standard error of a poll of the same sample size assuming a simple random sample to account for the exit poll’s design effect. 

Warren Mitofsky explained to me in an e-mail that the DESR calculated for each state in the 2004 exit polls ranged from 1.5 to 1.8 depending on the average number of samples per precinct.  When asked whether it was appropriate to use the 1.3 DESR factor calculated by Merkle and Edelman for the 1996 election to estimate the 2004 exit poll design effects, Mr. Mitofsky replied that “[t]he Merkle/Edelman paper is not what we computed this year...both Merkle and Edelman participated in this latest calculation.”^[27]

Dan Merkle of ABC News wrote the following regarding the use of this factor for analysis of the 2004 Presidential Election exit polls:

What was in the Merkle and Edelman chapter is only a general estimate based on work at VNS in the early 1990s.

 

The design effect will vary state by state based on the number of interviews per precinct and how clustered the variable is. More work was done on this in 2004 by Edison/Mitofsky. Edelman and I did participate in this. I would suggest using the design effects calculated by Edison/Mitofsky for their 2004 polls.^[28]

Complicating the computation of the DESR is that the fact that there are likely two different factors used for the intercept interviews and the telephone interviews.  Dan Merkle wrote, that Mitofsky’s DESRs, “only appl[y] to the intercept interviews,” and that “there may be a separate (smaller) design effect for the telephone survey component.^[29]

I checked with Jennifer Agiesta of Edison Media Research whether there was a smaller DESR associated with the telephone survey component than that which was conveyed by Mitofsky. Ms. Agiesta replied:

According to Warren, we did a new study since the one that Dan Merkle and Murray Edelman did some years ago and the design effects Warren reported to you were the latest ones computed. The whole advisory council, including Dan Merkle and Murray Edelman, participated in it and agreed that the information on design effects that Warren sent you is correct.^[30]

 

Although I'm not certain that Ms. Agiesta understood my question and I have a follow-up question pending with her, it should be clear that application of a 1.3 DESR is not appropriate; the DESR varies by state and is at least 1.6 in all three states, but could be as high as 1.8 in Florida.^[31]  This information was shared with Dr. Freeman before his paper was published. 

How Many Tails?

Dr. Freeman's null-hypothesis stated that, assuming independent state polls with no systematic bias, Kerry's predicted proportion should not significantly exceed his tallied proportion.  Exhibit 3 is a reproduction of Dr. Freeman's Figure 1.2,^[32] which depicts a normal distribution for Ohio given his calculated standard deviation for all three states.^[33]

 

Exhibit 3

Normal Distribution of Kerry’s Tallied Distribution

with Kerry’s Predicted Proportion

The normal distribution depicts the range of possible proportions that could occur if the exit poll were conducted 100 times.  Kerry's "tallied percentage of the vote" is the established standard, or, in this case, the mean of samples. The 95 percent confidence interval shows the range of proportions that would result for 95 of 100 exit polls and is commonly referred to as the "margin of error."  With this figure, Freeman attempts to show that the Ohio exit poll is outside the margin of error and therefore we can be 95 percent confident that the discrepancy is "significant" and cannot be explained by random error alone. 

Notice though that there are two "tails" outside the 95 percent confidence interval: left and right.  The right tail consists of the 2.5 exit poll results of 100 that could be expected to significantly exceed the tallied percentage, whereas the left tail represents the 2.5 exit poll results of 100 that could be expected to be significantly lower than the tallied percentage.  Unless the professor sets aside his assumption of no bias in the exit poll and an accurate election tally, he must include the probability of a significant finding at BOTH ends of the normal distribution.  By insisting on a single-tail test, he is hinting that either the exit poll is biased or the tally is wrong.  This insinuation is inappropriate prior to, or in the process of, testing a null hypothesis that assumes no bias and an accurate tally.  Dr. Freeman's failure to properly apply a two-tail test means that his p-values are ½ what they should be. 

Implications for Dr. Freeman's Analysis and Conclusions

Using Dr. Freeman's Data, I calculated Z-scores and p-values under multiple scenarios to consider range of possible p-values given the fuzziness of the dataset and to highlight the effect of using the 1.3 DESR calculated for the 1996 elections and a single-tail test in violation of the null hypothesis.  The results of these tests are presented in Exhibit 4.

Exhibit 4 Results of Z-tests for Three Rounding Scenarios, Two DESR Assumptions, and Both 1-tail and 2-tail p-values
State	1.3 DESR (Merkle & Edelman, 2000)									1.6 DESR (Mitofsky, 2004)
	Freeman’s 10th			LB			UP			Freeman’s 10th			LB			UP
	Z	p-values		Z	p-values		Z	p-values		Z	p-values		Z	p-values		Z	p-values
		1-tail	2-tail		1-tail	2-tail		1-tail	2-tail		1-tail	2-tail		1-tail	2-tail		1-tail	2-tail
FL	2.15	.02	.03	1.99	.02	.05	2.72	.00	.01	1.75	.04	.08	1.61	.05	.11	2.21	.01	.03
OH	2.44	.01	.01	2.03	.02	.04	2.65	.00	.01	1.98	.02	.05	1.65	.05	.10	2.15	.02	.03
PA	2.21	.01	.03	1.80	.04	.07	2.41	.01	.02	1.80	.04	.07	1.47	.07	.14	1.96	.03	.05
Notes:  Freeman’s 10th=Freeman’s  extrapolation from the whole proportions; LB=Lower Bound of the true proportion significant to a 10th; UB=Upper Bound of the true proportion significant to a 10th. p-values in red indicate a significant finding. p-values in green indicate non-significant finding. p-values that are .05 were determined to be significant or not significant by taking the digit out to a 100th.

To review, a p-value of <0.05 represents a significant finding.  If the p-value is >0.05 then statistically, nothing can be said about the discrepancy - it is not significant.  As shown in the exhibit, when the 1.3 DESR is assumed, all findings are significant with the exception of the 2-tail p-value for Pennsylvania when the lower bound of the exit poll data is considered.  However, when the 1.6 DESR is applied, there are several scenarios where the p-values (1- or 2-tail) are not significant.  When the inappropriate single-tail findings are removed, the majority of findings when rounding is considered are not significant. 

Dr. Freeman’s analysis leads him to conclude the following about the data:

Assuming independent state polls with no systematic bias, the odds against any two of these statistical anomalies occurring together are more than 5,000:1...The odds against all three occurring together are 662,000-to-one. As much as we can say in social science that something is impossible, it is impossible that the discrepancies between predicted and actual vote counts in the three critical battleground states of the 2004 election could have been due to chance or random error.^[34] (emphasis added)

When the data set is analyzed correctly, there possibility remains that all three states are not significant.  In fact, what we learn from my analysis of the data is that it is impossible to determine the significance of discrepancies in Florida, Ohio, and Pennsylvania. 

Conclusion III:  Explanations for the Discrepancy Thus Far Provided are Inadequate. 

Although it is technically possible that the observed discrepancy could be explained by random sampling error alone, that conclusion is not reasonable.  Sampling error could account for some of the error, but it is highly improbable that it accounted for all in the three states analyzed by Dr. Freeman.  In fact, this point was so obvious that the Jim Rutenberg of the New York Times wrote after reading a report prepared by Edison/Mitofsky the day after the election, “the surveys had the biggest partisan skew since at least 1988, the earliest election the report tracked.”^[35]  So why did Dr. Freeman set out to statistically prove a point conceded by the survey designers the day after the election?  I suggest that he did so in an attempt to lend himself credibility as an “expert”^[36] on the matter in time for the pending release of the NEP Report. 

Dr. Freeman published his paper in December, 2004.  At the time, there was little by way of explanation of what went wrong with the exit polls.  The much anticipated NEP Report on the subject was not available and no one really had an idea when it would be released.^[37]  Nearly two pages of The Unexplained Exit Poll Discrepancy are dedicated to largely dismissive discussion of anticipated non-fraud related explanations for the discrepancies that had been suggested by talk radio show hosts and bloggers.  Rather than nitpick the hasty generalizations and strawman quality of this part of his paper, I will simply say that I believe his Conclusion III stands:  The exit poll-election result discrepancy is still not fully explained, despite the NEP Report that was released on January 19, 2004.

NEP Report^[38]

The 77-page report prepared by the designers of the 2004 Presidential exit polls for the NEP includes a ton of information.  “Say what you will about its conclusions, this report is loaded with never before disclosed data” wrote Mystery Pollster.^[39]  The following bullets include a sample of the summary findings from the report:^[40]

·        Sample section of polling locations were not the problem.

·        No systematic problem in how the exit poll data were collected or processed was discovered.

·        Discrepancies do not support allegations of fraud due to rigging voting equipment.

·        The observed higher than average Within Precinct Error (WPE) in many precincts was likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters.

A group of academics that included Dr. Freeman circulated a paper^[41] under the organization US Count Votes,^[42] which is critical of the NEP Report.  Unlike Dr. Freeman’s The Unexplained Exit Poll Discrepancy, this paper dealt quickly with the significance of the discrepancies as the NEP Report readily admitted that the systematic Democratic bias in the 2004 NEP Presidential exit polls could not be explained by sampling error.  The US Count Votes authors conclude that only one of two hypotheses are worthy of exploration: 1) the exit polls were subject to a consistent bias of unknown origin; or 2) the official vote count was corrupted.^[43]  The question then becomes; did the NEP Report prove the first hypothesis?

Differential Non-Response

Of primary concern to the US Count Votes authors is NEP Report conclusion that differential non-response is the likely explanation for the discrepancy.  “No data in the report supports the hypothesis that Kerry voters were more likely than Bush voters to cooperate with pollsters, and the data suggests the opposite may have been true” wrote the authors.^[44]  The paper included a chart of data collected from the NEP Report that was “not analyzed or mentioned in the text.” ^[45]  The chart is reproduced in Exhibit 5. 

Exhibit 5

Response to Exit Polls Slightly Higher

in Republican Precincts

If these data were significantly correlated, the US Count Votes authors conclude the finding would suggest that “in precincts with higher numbers of Bush voters, response rates were slightly higher than in precincts with higher number of Kerry voters.”^[46]  However, the NEP Report stated that “there was no significant difference between the completion rates and the precinct partisanship.”^[47]  In situations like these, I would prefer to see “completion rate” regressed against “precinct partisanship,” but I take the NEP Report authors’ word that the relationship is not significant.  US Count Votes should have pushed for more information supporting their non-significant finding, but instead they ignore the statement of non-significance and graphically present the data to suggest an alternative hypothesis, which seems to me to be inappropriate unless they are inferring professional incompetence or outright deception from the NEP Report authors. 

Regarding the differential non-response explanation from the NEP Report, researchers Traugott, Highton, and Miller of the National Research Commission on Elections and Voting^[48] write the following:

What must necessarily become more subjective, having eliminated the most directly testable explanations for the bias, is an attempt to explain what other sources there might be among many unobserved and unmeasured possible explanations.  The NEP Report concludes that the most likely source of the errors is differential response patterns by Kerry and Bush voters leaving the polls – that is, Kerry voters were more likely to be interviewed while Bush voters were less likely.  The authors include a simulation of the likely magnitude of the differential response rates, and then they speculate about contributing factors.  These conclusions must be inferred (of necessity, since no information is available on the refusers) from the finding that the average WPE was greatest where younger and more highly educated people were interviewers, irrespective of gender.  The analysis also suggests that interviewers hired later and who describe themselves as “somewhat” or “not very well” trained also were associated with data that produced higher average WPE’s.  From these analyses, the NEP leaders conclude they must pay more careful attention to interviewer recruitment, including trying to hire older interviewers, and to their training.^[49]

Given the frequency of NEP Report conclusions that included qualifiers such as “likely,” “may,” and “could,” I understand how US Count Votes are concerned with the analysis.  In effect, the null hypothesis that differential non-response was not a factor, was never (from what I can tell) statistically rejected by the NEP Report.  However, the contention that “[no] data in the report supports the hypothesis that Kerry voters were more likely than Bush voters to cooperate with pollsters” is not in the least bit accurate.^[50]  The NEP Report presented volumes of information that “suggests” support for their hypothesis.  Nonetheless, the US Vote Counts authors turned their attention to another set of data in the NEP Report that they suggest implies election fraud is a more plausible explanation of the exit poll discrepancy.  

WPE by Vote Equipment

The NEP Report included mean and median WPE by type of equipment used at the polling place.  Categories of voting equipment included paper ballot, mechanical voting machine, touch screen, punch cards, and optical scan.  The NEP Report rejects the fraud hypothesis because they found, “no systematic differences for precincts using touch screen and optical scan voting equipment” and “the differences are similar to the differences for punch card voting equipment, and less than the difference for mechanical voting equipment.”^[51] 

US Count Votes reproduced a table from the NEP Report showing that the median WPE for precincts with paper ballots was -0.9, but anywhere from -5.5 to -10.3 for all other voting methods, including optical Scan, punch cards, touch screen, and mechanical voting.  The implied conclusion being that automated voting technology could have been tampered with, accounting for the substantially different WPE when compared to WPE in precincts with paper ballots.  Regarding the disparate WPE by vote equipment, US Count Votes states:

[The NEP Report authors] implicitly dismiss the possibility that errors for all four automated voting systems could derive from errors in the election results and their breakdown for voting equipment ignores whether results are tallied in the precinct or at a central location.^[52]  

But why would the average WPE be higher for mechanical vote equipment (presumably the old fashioned lever pull-lever equipment) and punch cards than for electronic voting equipment?  According to authors Dr. Freeman and Dr. Josh Mitteldorf:

[T]his fact merely suggests that all three of these systems may have been corrupted. Indeed, there is little question about problems associated with both punch card systems (recall the Florida debacle in 2000) and mechanical voting machines, which are generally unreliable, vulnerable to tinkering and leave no paper trail.^[53]

Missing from this discussion is the NEP findings of these data when disaggregated into urban vs. rural precincts.  The NEP Report explains that “[t]he low value of the WPE in paper ballot precincts may be due to the location of those precincts in rural areas, which had a lower WPE than other places.^[54]  In other words, in rural areas, where paper ballots are most prevalent, the average WPE was lower than the aggregate average WPE; in the urban areas, only five sampled precincts included in the analysis used paper ballots.

What we have here is an apparent case of Simpson’s Paradox,^[55] which seems ripe for more rigorous statistical analysis.  US Count Votes makes the correct call:

The Edison/Mitofsky Report does not report having done an ANOVA (analysis of variance^[56]) of voting machine type that might confirm their claim that there is no difference between precincts using different types of voting machines.^[57]

ANOVA is an obvious test for these data and why the NEP Report does not include the results of this test is baffling.  If a significant difference by voting equipment is found in the aggregate, the data can be disaggregated by rural v. urban and retested. If the differences are not significant, then the NEP Report authors should convey this fact and explain in more detail what it is about the data that could cause the discrepant significance findings.  If the discrepancies are significant, then some other explanation of the discrepancy should be pursued, as suggested by US Count Votes.

US Count Votes makes the following statement in the conclusion to their paper:

The Edison/Mitofsky report fails to substantiate their hypothesis that the difference between their exit polls and official election results should be explained by problems with the exit polls. They assert without supporting evidence that (p. 4), “Kerry voters were more likely to participate in the exit polls than Bush voters.” In fact, data included within the report suggest that the opposite might be true.

Their analysis of the potential correlation of exit poll errors with voting machine type is incomplete and inadequate, and their report ignores the alternative hypothesis that the official election results could have been corrupted.

The issue raised in the first paragraph quoted above could be dealt with by releasing the regression analysis of completion rates by precinct partisanship.  The issue raised in the second paragraph could be resolved with an ANOVA of WPE by vote equipment and vote equipment location.  The NEP Report authors acknowledged that they “need to do more investigation into the causes of the statistical skew in the exit poll data for the general election.”^[58]  For these reasons, I concur with Dr. Freeman’s Conclusion III of The Unexplained Exit Poll Discrepancy: Explanations for the discrepancy thus far provided are inadequate.  I look forward to further analysis from Edison Media Research and Mitofsky International on this topic. 

Concluding Remarks

This paper has demonstrated that Dr. Freeman is not an “expert” on exit polls or the 2004 Presidential exit poll discrepancies as suggested by the UPenn press release.  In fact, his paper, The Unexplained Exit Poll Discrepancy, is highly flawed.  His argument that “in general, exit poll data are sound”^[59] fails having suppressed evidence^[60] and the conclusion that “it is impossible that the discrepancies between predicted and actual vote counts in”^[61] Ohio, Florida, and Pennsylvania was not substantiated statistically.  Nevertheless, Dr. Freeman is right in concluding that explanations of the discrepancy to date are inadequate and Edison/Mitofsky should address the concerns of US Count Votes in subsequent analysis of their data. 

Dr. Freeman wrote a book based on his research that is due out in a couple of months^[62] and has a couple of working papers in progress^[63] that I am told will be published in an academic or professional journal.  If The Unexplained Exit Poll Discrepancy is any indicator of the quality of research included in these forthcoming works, I suggest that his publishers take a closer look at the manuscripts.

 

NOTES:

 

^[1] See http://www.exit-poll.net/election-night/EvaluationJan192005.pdf.

² See http://www.appliedresearch.us/sf/UP-MitofskyComment.htm.

³ See http://www.appliedresearch.us/sf/.

⁴ See http://center.grad.upenn.edu/center/get.cgi?item=exitpollp. 

⁵ See http://www.appliedresearch.us/sf/UP-MitofskyComment.htm.

⁶ Page 17 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp.

⁷ Ibid.

⁸ See http://www.mysterypollster.com/main/2004/12/what_about_thos.html.

⁹ See http://www.aceproject.org/main/english/lf/lfd08e.htm.

¹⁰ See http://www.mysterypollster.com/main/2004/12/what_about_thos.html.

¹¹ Ibid.

¹² See http://www.exit-poll.net/election-night/MethodsStatementStateGeneric.pdf.

¹³ See pg. 8 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp.

¹⁴ See http://exitpoll.byu.edu/about.asp.

¹⁵ Dr. Freeman’s dataset indicates that the exit poll predicted Kerry would win 29.1 percent of the vote in Utah, whereas the election tally showed that Kerry received 26.4 percent. 

¹⁶ See pg. 18 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp.

¹⁷ Mitofsky, Warren J. and Murray Edelman.  1995.  “A Review of the 1992 VRS Exit Polls.” In Presidential Polls and the News Media. Eds. Lavrakas/Traugott/Miller.  Boulder, CO: Westview Press. (pp. 81-100)

¹⁸ I could not locate the source of Freeman's election tally data.

¹⁹ Pg. 4 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp describes the extrapolation method used for Ohio.

²⁰ For a more thorough discussion of standard errors associated with cluster samples, see: 1) Frankel, Martin. 1983. Sampling Theory. Handbook of Survey Research. Eds. P. Rossi, J. Wright., and A. Anderson. Orlando, FL: Academic Press. (pp. 47-62); 2) Kalton, Graham. 1983. Introduction to Survey Sampling. Beverly Hills, CA: Sage. (pp. 28-47); 3) Kish, L. 1965. Survey Sampling. New York: Wiley; 4) Mendenhall, William, Lymann Ott, and Richard Scheaffer. 1971. Elementary Survey Sampling. Belmont, CA: Duxbury Press. (pp. 121-141, 171-183); 5) Sudman, Seymour. 1976. Applied Sampling. New York: Academic Press. (pp. 69-84, 131-170); and 6) Williams, Bill. A Sampler on Sampling. New York: Wiley. (pp. 144-161, 239-241).

²¹ See pg. 72 of Merkle, Daniel M. and Murray Edelman (2000). "A Review of the 1996 Voter New Service Exit Polls from a Total Survey Error Perspective." In Election Polls, the News Media and Democracy, ed. P.J. Lavrakas, M.W. Traugott, pp. 68-92. New York: Chatam House.  .

²² Freeman’s analysis relies on screenshots of the CNN Exit Poll webpage saved shortly after midnight on election eve.  The data likely represent the final exit poll data posted on the website before they were weighted to conform to the election result.  Weighting the exit poll data to conform to the election result is not unusual.

²³ See pg. 13 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp.

²⁴ See http://www.physics.uoguelph.ca/tutorials/sig_fig/SIG_dig.htm. 

²⁵ See footnote #12 on pp. 4-5 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp. 

²⁶ It would be preferable to include at least one more significant digit for more precise analysis, but to remain consistent with Freeman’s extrapolations I stick with the 10^th.

²⁷ Mitofksy, Warren J. 2004. Electronic communication to Rick Brady, December 7.

²⁸ Merkle, Dan. 2004. Electronic communication to Rick Brady, December 15.

²⁹ Merkle, Dan. 2004a. Electronic communication to Rick Brady, December 17

³⁰ Agiesta, Jennifer. 2004. Electronic communication to Rick Brady, December 23.

³¹ According to Mitofsky, states where the average number of interviews per precinct (N)= 50; the DESR was calculated as 1.8.  Where N=40, the DESR was calculated as 1.6.  Finally, with N=30, the DESR was 1.5.  The NEP State Methods statement (see: http://www.exit-poll.net/election-night/MethodsStatementStateGeneric.pdf) gives the number of precincts and both intercept and telephone interview totals for the final exit polls (the actual breakdown of intercept v. telephone interviews included in Freeman’s data is not known, but the NEP data provides a good estimate).  In Florida, the average number of interviews per precinct (ANP) was 50.2 when all interviews are considered and 43.3 when only intercept interviews are considered, meaning the DESR could be either 1.6 or 1.8.  No telephone interviews were conducted in either Ohio or Pennsylvania and therefore the ANP was 41.7 and 42.5 respectively, resulting in a DESR of 1.6 for those states. 

³² See pg. 12 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp. 

³³ The 95 percent confident interval for Ohio was constructed with the 1.3 DESR.

³⁴ See pg. 13 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp.

³⁵ See http://www.nytimes.com/2004/11/05/politics/campaign/05poll.html?ex=1257397200&en=8a888e4368266498&ei=5090&partner=rssuserland

³⁶ See http://www.appliedresearch.us/sf/UP-MitofskyComment.htm.

³⁷ On November 21, 2004, Washington Post Polling Director, Richard Morin, penned a piece that started, “It will be a few more weeks before we know exactly what went wrong with the 2004 exit polls.”  It was actually two months before the NEP Report was published. 

³⁸ See http://www.exit-poll.net/election-night/EvaluationJan192005.pdf'.

³⁹ See http://www.mysterypollster.com/main/2005/01/impressions_on_.html.

⁴⁰ See pgs. 3-4 of http://www.exit-poll.net/election-night/EvaluationJan192005.pdf

⁴¹ See http://www.uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf#search='NEP%20Exit%20Poll%20Report%20US%20Count%20Votes'.

⁴² See http://www.uscountvotes.org/.

⁴³ See pg. 3 of http://www.uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf#search='NEP%20Exit%20Poll%20Report%20US%20Count%20Votes 

⁴⁴ Ibid, pgs. 3-4.

⁴⁵ Ibid, pg. 4.

⁴⁶ Ibid.

⁴⁷ See pg. 37 of http://www.exit-poll.net/election-night/EvaluationJan192005.pdf.

⁴⁸ See http://elections.ssrc.org/.

⁴⁹ See pgs. 11-12 of http://elections.ssrc.org/research/ExitPollReport031005.pdf. 

⁵⁰ See pgs. 3-4 of http://www.uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf#search='NEP%20Exit%20Poll%20Report%20US%20Count%20Votes.

⁵¹ See pg. 4 of http://www.exit-poll.net/election-night/EvaluationJan192005.pdf.

⁵² See Pg. 4 of http://www.uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf#search='NEP%20Exit%20Poll%20Report%20US%20Count%20Votes.

⁵³ http://www.inthesetimes.com/site/main/article/1970/

⁵⁴ See pg. 40 of http://www.exit-poll.net/election-night/EvaluationJan192005.pdf.

⁵⁵ See http://plato.stanford.edu/entries/paradox-simpson/. 

⁵⁶ See http://www.psychstat.smsu.edu/introbook/sbk27.htm.

⁵⁷ See pgs. 4-5 of http://www.uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf#search='NEP%20Exit%20Poll%20Report%20US%20Count%20Votes.

⁵⁸ See pg. 12 of http://www.exit-poll.net/election-night/EvaluationJan192005.pdf.

⁵⁹ See pg. 17 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp. 

⁶⁰ See http://wiki.cotch.net/index.php/Suppressed_Evidence.

⁶¹ See pg. 13 of http://center.grad.upenn.edu/center/get.cgi?item=exitpollp. 

⁶² See http://www.appliedresearch.us/sf/hypotheses.htm.

⁶³ See http://www.appliedresearch.us/sf/epdiscrep.htm.

 

---

 

Got to love this... Feel free to post.  Rick

FIRST E-MAIL:

Brady's work not strong and we will be posting a critique of some of his
exit poll work in about one week.

What are your credentials and background in statistics?  Your own papers
do not seem to be worth responding to from what I've seen of them.  Do you
have credentials and a position at a university?

Kathy
_________________

 

I responded at length and said I was happy to field criticism from US
Count Votes.  To which she responded:
 

_________________

Rick,  I am sorry but I can't send it USCV's group of statisticians due to your
lack of credentials. If you had a PhD in something that required
statistics or had to do with elections, I could.  The same thing goes  for
me. I have a master's degree only, so no one would have to respond  to
anything I said in the academic world either - unless I sign with a  group
of PhDs.

Kathy

_________________
 

I was a bit puzzled at this new rule in academia, so I sent along the most
recent version of the paper and she replied:

_________________
 

Rick,  Get a statistician with a PhD to sign it with you and then I'll read it
and send it on. I don't have time. I'm buried in work.

Kathy

_________________

Last I checked Dr. Freeman's paper and the US Count Votes response to the
NEP Report were NOT in the academic world (not published in peer reviewed
journal or book).