Regular Stubborn Mule guest James Glover (@zebra) turns his statistical expertise on some apparently contradictory polls gauging opinions on climate change.

Two polls came out today on the question of whether people believed climate change is real and if so whether it is caused by human activity. The first was a Newspoll published in The Australian and the second was by Essential Media and was commented upon by Essential’s Peter Lewis on ABC’s The Drum. Intriguingly, the Newspoll suggests 73% of Australians believe in climate change with a significant human contribution (so called Anthropogenic Global Warming or AGW). Now The Australian has copped a bit of flak lately for its alleged anti-climate change agenda, but leaving that aside this poll suggests that AGW should be practically a closed book politically as an overwhelming majority believe in it. Essential Media describes themselves as a research tool for progressive political campaigns. Essential’s poll indicates that only 45% of people believe in climate change caused by human activity. In the accepted narrative of such things the results would have been around the other way and the tweetsphere would be apoplectic accusing The Australian and News Corp of once again distorting Newspoll results for their own right-wing political agenda. So what is happening here?

First a note on sampling error. Essential polled 1896 people while Newspoll contacted 1,123 people. For polls where the expected split is approximately 50% a good rule of thumb for margin of error (MoE) is 1/√sample size. In the two polls here this gives MoEs of 2.2% and 3.0% respectively. MoE represents two standard deviations from the sample average so differences of 25% are extremely unlikely (like 10^-10 probability unlikely) to be explained by a unfortunate random choice of sample from the general population.

The most likely explanation is that one or both of these polls suffer from an underlying sampling bias. This would be easy enough to generate artificially—just poll people in Newtown if you want to get more people who believe in AGW or in Bob Katter’s seat for the opposite result (is this a little glib? Maybe, but you know I am right). But legitimate pollsters like Essential and Newspoll rely on the rigour of their sampling technique. Especially as every time you publish a controversial result, a large section of the population who disagree with it will accuse you of bias. There are a number of techniques to reduce bias—one is to ask coquestions whose population statistics are well accepted. For example if in your poll you found that 46% of the respondents were female and 54% were male you can readjust the result to reflect the actual population average of 51:49. I assume both polling organisations follow standard methodologies to minimise bias. Often though their actual methodologies are proprietary so question marks remain. A famous political polling agency was well know to always come up with polling results that reflected the political opinions of its founder after “adjustment for bias”.

Some indication that there isn’t an overwhelming bias are some additional questions about voting intention. Here are the results:

I would have to say that the differences in the numbers are on the borderline of being consistent with the MoEs I estimated. In any event the Newspoll which has a higher number believing in AGW has less Coalition voters (though about the same Labor+Green votes). It seems unlikely that the votes for Independents and other alone could account for the 28% difference in the polls on the question of AGW.

So that leaves us with the polls themselves. I have assumed so far that they asked the same questions, but there are major differences. Here are the actual questions and results:

Essential

Newspoll

Now what appears at first to be a headline difference between the polls is more subtle. It is quite hard (I tried) to map the answers between them exactly to compare the results. For example Essential doesn’t ask if the respondents directly if they don’t believe in climate change at all (18% in Newspoll) so presumably the climate skeptics get lumped under “Don’t know” (19%) which will also include those who believe in climate change but don’t know if it is caused by human activity or don’t know if climate change is real. That Newspoll total of “don’t knows” and skeptics is 23%, a bit higher than Essentials “Don’t know” of 19% but within the MoEs as reflected by the voting intention results.

However we can try to compare the two main results which boil down to “Believe climate change is real and human activity is significantly affecting it” of 73% (Newspoll’s headline result combining “solely” and “partly” caused by human activity) vs Essential’s “Climate change is happening and is caused by human activity” of 45%. The difference appears huge. The only thing I can think to explain this is that when not offered the choice of “solely” vs “partly” caused by human activity the Essential respondents threw their lot in with “caused by a normal fluctuation in the Earth’s climate”. In other words the results are consistent if most people who believe that “climate change is real” but don’t believe it is “solely caused by human activity” believe it is “partly caused by human activity” but mostly due to “natural fluctuations in the Earth’s climate”.

What is clear here is that the wording of polls is important and that both polls failed to tease out the subtle distinctions in people’s views on climate change (though Newspoll did a better job of this than Essential). There are also question marks about the sampling bias as shown in the voting intention results. But the headlines of both polls will superficially look like totally different results. And that is a problem when the results are used to support political rather than scientific views on anthropogenic global warming as fact or fiction.

Possibly Related Posts (automatically generated):

• Keep the date, and Vote (24 August 2017)
• Left, Right and Climate Change (7 January 2010)
• No move expected by the Reserve Bank (31 May 2010)
• Poll Dancing (15 July 2013)