Dumb Statistics question

[SDGuy]

I'm sure everyone here has heard about the Pew Research Center poll on Muslims in America:

http://pewresearch.org/pubs/483/muslim-americans

The results of this have been repeated ad nauseum in the media, but I'm just wondering about the methodology.

According to Pew Research, they used a "a national sample of 1,050 Muslims", and "Pew Research Center estimates the total population of Muslims in the United States at 2.35 million"

So, they can extrapolate the entire population using a 1 out of 2238 sampling?

Somehow this doesn't seem right to me. I work at a large Naval R&D facility. There are approximately 4000 personnel here. Using this methodology, and taking a sample of people here (myself and my officemate), I can determine that half of the employees here were born in Michigan and moved to Los Angeles during their teen years, whereas the other half, or about 2000 people, grew up in Serbia and moved to the United States eight years ago.

Any statisticians out there care to comment?

[kj]

It's very important that it be a _random_ sample, but it probably isn't. kj.

[JoeH]

Not a statistician, but know some of the basics. Basically, if you can get a relatively representative random sample of a group, you can make a very good estimate, plus or minus a couple percent, with an even smaller sample than 1 in 2000. And the mathematics behind statistical sampling show that you do not need to double the size of the sample if you double the size of the group. I forget the actual amount a sample has to be increased to get the same reliability when doubling the group, but it was something like only a few percent larger. This is where methodology becomes very important. If the sample is not random, but say "self-selected", then the results can be skewed from the actual. Even more important can be the wording of the questions and the interview techniques. People do have a tendency to give the "expected" or what they perceive as the "expected" answer in situations where they are answering questions placed by an interviewer. This is as opposed to anonymous surveys with no interviewer.

Now your example has taken the idea that the methodology is to take a random 1 in 2000 sample. But the is not the actual way the sample size is determined. There are formulas that give a sample size for a given size group and the statistical accuracy the sampler wants. Again, without looking up the actual numbers and the equations, so if you wanted the accuracy to be plus/minus 5%, it might take 100 persons in your group of 4,000, but for a group of 8.000 it might only take a sample size of 110 to get the same accuracy. Essentially statistics is using the rules of chance to determine what the probability is that the random sample picks out the equivalent of all heads for a small group of coins picked up off the floor at random after dumping a box of coins out.

P.S. I have not actually heard about the Pew report, and just took a quick look at the news report. Here is an important quote from that, "The Pew Research Center conducted more than 55,000 interviews to obtain a national sample of 1,050 Muslims living in the United States." What this means is that there is this large mass of interview reports, 55,000 plus. They then used some means, probably outlined in the full report, to select reports at random from that bunch of reports. If done correctly, that often corrects for interviewing methodology errors and any subjective ranking of the interview reports. I suspect many persons are looking over that full report to see if they can find any methodology errors. Unfortunately, that kind of thing often only gets reported in journals, not in the mass media news except in the back pages.

[OWC Jamie]

Like any poll, you have to analyze where the data comes from.


Too many polls are done in such a manner to gaurantee the desired results.
It's not hard to do at all.

Even "random" polls can have had targetted pollees.

[spearmint]

400 is a good sample if it is truly random. With this issue it would be hard to find.

[laarree]

Part of the last bullet point of the report page is alarming to me:

"...Many doubt that Arabs were responsible for the 9/11 attacks. Just 40% of Muslim Americans say groups of Arabs carried out those attacks."

Only 40%??! Are the others so caught in the grip of their delusions that they
can't believe that Arabs carried out the 9/11 attacks? Even the Saudi government
doesn't dispute this.

[Dennis S]

I worked at a bank and we figured that 270 was a very accurate sample size for our customer base of 18.000.