Evaluating the accuracy of Catholic polls

Several recent polls on Catholic issues have prompted some alarming and sensationalistic headlines. 

“Catholics paving the way on same-sex marriage.” “More than half of all Catholics believe Church leaders are out of touch and that women should be ordained priests.” 

However, as with everything, you have to read the fine print. 

“A problem that the media has when it goes out and writes stories based on these (polls) is that they don’t always give the distinction in the margin of error,” said Mark M. Gray, director of the CARA Catholic Poll and a research associate for the Center for Applied Research in the Apostolate at Georgetown University. 

Media outlets, Gray told Our Sunday Visitor, “might report a percentage of people who approve of the pope without saying that number could be 8 percentage points or lower, and that it is just a blurry snapshot of public opinion.”

Evaluating numbers

Gray said a scientifically sound poll on Catholic issues should have a margin of error of 3 percentage points, include Spanish-speaking respondents, well-phrased questions and communicate with respondents by cellular and land-line telephones. 

The polls should also be based on a random sampling of between 800 to 1,000 respondents of the target population; thus any poll purporting to reflect Catholic opinion on various issues requires a large enough sample. 

“The 800 number would be considered minimally necessary for a peer review publication,” Gray said. “Social scientists would need that in order to say they have proof of what people think. A sample of say, 500, is insufficient.” 

However, some recent surveys on Catholic public opinion have targeted smaller samples, such as one poll conducted in late February and early March by Quinnipiac University in Connecticut. 

The Quinnipiac poll sampled 1,944 registered voters nationwide. Out of those, only 497 respondents self-identified as Catholics. 

Among those Catholics polled, Quinnipiac reported that 54 percent of Catholic voters favored legalized same-sex marriage, leading Peter A. Brown, the assistant director of the Quinnipiac University Polling Institute, to issue a press release declaring that Catholics are “leading the U.S. toward accepting same-sex marriage.” 

Brown told OSV that the Quinnipiac Polling Institute is well respected among social scientists, and added that its surveys randomly select people, take into account landline and cell phone users, are within a 4 percentage point margin of error, and are conducted by live telephone calls with people rather than automated “robo-calls.” 

A 2010 study by New York University political science professor Patrick J. Egan reported that polls generally are accurate when reporting the percentages of people who support same-sex marriage, though adding that they tend to underestimate the opposition to same-sex marriage ballot initiatives by almost 7 percentage points.

Inaccurate reports?

Pia de Solenni, an ethicist and cultural analyst who writes on Catholic issues, raised questions about Quinnipiac polling just 497 Catholics to represent the opinions of more than 70 million Catholics in the United States. 

Indeed, the Quinnipiac poll’s numbers show some distinctions. Among those Catholics who vote and attend Mass weekly, only 36 percent said they supported same-sex marriage, with 55 percent opposed. 

Interestingly enough, the same poll also found that while 55 percent of Catholics think Pope Francis needs to move the Church in new directions, almost the exact same percentage — 52 percent — think the Church is already headed in the right direction. 

Solenni wrote on her blog that many people will identify themselves as Catholic even when they are not practicing, and they likely give different answers on the question of same-sex marriage. 

“When asking Catholics what they think about same-sex marriage, a responsible poll would first report why someone identifies as a Catholic, particularly if the poll is suggesting that this religious group wants its leadership to change Church teaching,” Solenni wrote. 

However, in sociology and in the Church, the basic definition of belonging to a population is self-identification, Gray said. 

“If you’re baptized and you consider yourself Catholic, you’re a Catholic from a sociological and Church perspective,” Gray said. “It doesn’t mean that weekly or Massgoing Catholics are a more important group in polling.” 

Gray, who doesn’t criticize polls for not distinguishing between Mass-attending and nonpracticing Catholics, said CARA’s polls find little differences between Catholics who attend Mass weekly and monthly. The big differences in opinion are between that cohort and those who attend church on Easter or Christmas, or not at all. 

“That’s the real divide when we do a survey,” Gray said.

In the data

A Marist Poll conducted in early March for the Knights of Columbus made the distinction between practicing and lax Catholics. That poll — though it only surveyed 515 Catholics across the nation — found that 82 percent of those who identified themselves as practicing had positive impressions of Pope Emeritus Benedict XVI’s pontificate. 

The Marist Poll, which says it is the nation’s first college-based survey center to involve undergraduate researchers, also found that more than 70 percent of Catholics overall said the former pontiff had a positive impact on the moral direction of the world. 

A Washington Post-ABC News poll in March reported similar findings, saying that 68 percent of Catholics thought Pope Benedict XVI was either an “excellent” or “good” pope. 

“Large-scale polling by nationally recognized operations such as Gallup and Washington Post are usually excellently conducted and have few statistical concerns,” Stephen F. Schneck, director of the Institute for Policy Research & Catholic Studies at The Catholic University of America, said. “Polls by Pew and CARA are also superb. These polls are designed with all the right safeguards for sampling and are generally of a size to make them very reliable.” 

A Washington Post-ABC News poll in March reported that 58 percent of Catholics favor women priests, 60 percent believe the Church is out of touch and that 54 percent believe the Church’s “traditional policies” — without naming what those are — should be changed to reflect “the attitudes and lifestyles of Catholics today.” 

However, that poll only surveyed 201 Catholics, and it has a margin of error of almost 12 percentage points, which would mean that less than half of the nation’s Catholics, according to the poll, may actually favor wholesale changes. 

The Washington Post-ABC news poll also found differences between nonpracticing and weekly Mass-attending Catholics, 56 percent of whom said they favored the Church’s teaching on the male priesthood. Still, don’t expect social scientists and media outlets to split hairs when polling Catholics for their views. 

“Intentional, regular Mass-attending Catholics might object, but I’m sure we don’t want pollsters deciding who counts and doesn’t count as Catholics,” Schneck said. “So, we’ll continue to see polling firms allowing people to self-identify as a Catholic.” 

Brian Fraga writes from Massachusetts.