We need to have a talk.

This thing of glibly taking press releases or articles about survey results as anything meaningful? This has got to stop.

I’ve been following technology for years but I don’t think it really hit me until I started doing nothing but following technology about a year ago just how bad this problem is. I’ve already opined on the problems with a survey Pew Research conducted on iPad market share last summer. But I have compiled some other examples that will, I have no doubt, leave you slack-jawed with amazement. Assuming you weren’t already slack-jawed for some other reason. Like, say, dental surgery or gum-chewing fatigue.

Anyway, back in November, a publication no less august than the LA Times (no less august because it actually was the LA Times) ran the headline:

“Report: Kindle Fire HD to outsell iPad mini 2:1 in holiday season”

The source for the “report” cited in the headline? I swear to you on a stack of mint condition bagged and boarded copies of Frank Miller’s run on Daredevil, issues 227-233, that I am not making this up: an outfit called CouponCodes4U. Their evidence for this totes reliable claim? For a three-week period their site had twice as many searches for “Kindle Fire HD” as “iPad mini”.

That’s it.

LA Times.

[golf clap]

This kind of garbage is not limited to struggling dead tree publications, though. Writing for The Street back in July of 2012, Chris Ciaccia said:

“iPad Mini Sees Tepid Demand : Poll”

According to an online poll Ciaccia conducted (scientific value: zero), “only” 43 percent of respondents said they would “definitely” buy an iPad mini. How that would be considered “tepid” I know not, but that is not even what his clown college poll asked.

The options for answers for this poll were, in reverse order:

• Maybe—Others are selling 7-inch tablets, but I would need to see how Apple does it.
• No—I agree with Steve Jobs. A 7-inch tablet is “dead on arrival.”
• Yes — I’ll buy anything Apple related.

(Emphasis mine.) 43 percent of respondents said they would buy “anything Apple related”. Ciaccia turned that around, saying:

…only 43% of the 815 people who responded to the poll said they would “buy anything Apple related.”

You can see the sleight of hand there, as if the question were “Would you consider buying anything Apple related?” as opposed to, you know, the opposite.

OK, that one’s just Ciaccia making things up. No one else, to my knowledge, cited it as meaningful and I’m not looking to see if anyone did because I need to hold on to what few shreds of hope I have for humanity.

But there’s willfull ignorance like the Times’, deliberate jerkery like Ciaccia’s and grasping for results that satisfy pre-conceived biases. In June of last year, PC World’s resident open-source fan, Katherine Noyes, wrote “Android Tablets Beating Out iPad in Business and IT: Report”.

You shall be surprised to discover that that is not what the report said. No, what the report from IDG said is that 44 percent of first-time tablet buyers said they were planning on buying an Android tablet compared to 27 percent for the iPad. 71 percent already owned a tablet, most of which were iPads. Buying intentions don’t really mean a lot, apparently, because back here in the future, Citrix seems to think there’s “More evidence that iOS is beating back Android in the enterprise”. Now, yes, that’s just one source and we should never take anyone’s single word for it, that’s part of my point, but Good Technologies seems to agree. They are, however, only measuring what they can measure — the device usage at companies that use their services — but, regardless, IDG’s results did not support Noyes’ desired conclusion.

We’re not done yet! In February, Forbes contributor Louis Bedigian cited an online poll taken by Yahoo Finance, saying “63% Of Investors Don’t Want An Apple iWatch”.

Which might be interesting if a) the poll he cites were not an online poll and b) anyone had one damn idea what an iWatch might be. Also, I’m not a professional poll-designing guy (not an actual job title), but the list of possible responses to “How interested would you be in buying an iWatch?” seems a little off to me.

• Very interested (18,325) — 18 percent
• Somewhat interested (19,529) — 19 percent
• I already have a watch (65,179) — 63 percent

The third option might as well have been “cheddar cheese”. It doesn’t answer the question. I can have a watch and still be very interested in buying an iWatch. If I knew what the hell an iWatch was.

Finally, the most recent atrocity was committed by the BBC:

“Apple Brand Less ‘Inspiring’, Survey Says”

As John Gruber pointed out, the survey in question should be, well, in question, as it was conducted by a subsidiary of Samsung’s ad agency.

What are the lessons to take away from this madness? Before you report any numbers:

1. Consider the source. In many of these cases the source is not pushing science, they’re pushing themselves. CouponCodes4U was interested in one thing: getting its name in publications like the LA Times. And mission accomplished.
2. Check the rigor with which the survey was conducted. Online polls that anyone can answer are utterly meaningless, of course, but polls will often be restricted to “visitors to our site”, which is itself a form of selection bias. Look at the questions. Often they are either poorly designed or deliberately designed to skew the results in a salacious way.
3. Check the actual results versus the reported result. Surveys can purport to measure things they can’t actually measure. Writers can attempt to mine answers they want to hear from the results in ways that are not supported.
4. Check the results against other results. Pew’s numbers, for example, didn’t tie to several sales estimates. Has iPad market share declined? Sure. But it probably hadn’t by as much as Pew said at the time.

If you must report these survey results because you toil endlessly for a cruel and unforgiving corporate beast that will only be sated by increased page views, at least caveat it appropriately. You owe it to your readers if not your eternal soul.

ADDED: Just to emphasize the point that I’m not saying you should only question surveys that swing anti-Apple, here’s a prime example from 2011 I just recalled. Web developer Tarandeep Gill faked a report showing that Internet Explorer users have lower IQs. Many members of the Apple web community, including me, glibly passed it along because ha-ha soooooo true, right? So, mea culpa.