CR: 数据可不可靠要看它的读者是不是反映整个社会人口的构成。


   如果它的读者的构成 (Demorgraphic) 和整个社会人口的构成不一样,结果就不能反映整个社会的观点。比如,CR的读者年纪大的人多于社会的构成 (见下文)。由此CR对问题的看法就更偏于年纪大的人的观点。文章的另一点我也同意,样本量 (Sample size) 起点太低: 100 个回复以上就够了。几亿人的国家,100 个人的观点CR就敢下结论,太草率了。

http://blog.bluespringsfordparts.com/233/consumer-reports-rating-methodology-flawed/


Consumer Reports Car Rating Methodolgy is Flawed

Generally speaking, Consumer Reports provides useful and informative buying advice and product reviews for a wide range of products. From flat screen TVs to vacumn cleaners to car seats,Consumer Reports often uses a careful and refined testing procedure that generates some great advice.

As a long-time reader and current subscriber, I am 100% satisfied with Consumer Reports…except for their automotive reliability rankings.

Here’s why: Car rankings are based exclusively on surveys offered by Consumer Reports (CR) readers. This, in my view, is a fatally flawed approach.

1. CR subscribers aren’t representative of the general public. Quantcast.com, which estimates demographic and user data for millions of websites, has provided the following demographic “snap shot” of ConsumerReports.org (see the original report here):

ConsumerReports.org demographic data

Demographic data about the ConsumerReports.org website audience, as determined by Quantcast.com

As you can see, the typical ConsumerReports.org visitor is more likely to be wealthy ($100k+ annual household income) and college educated. While there’s nothing wrong with being wealthy or educated, I suspect these consumers are a bit biased against American car brands.

For anyone who thinks that Quantcast’s data might be off, check out this 2009 study of CR’s auto buying guide, which was sponsored by CR. According to the data on page 34, the average CR reader (either online or via magazine subscription) is wealthier and more educated than average.

2. CR data is noisy. By “noisy,” I mean varying quite a bit from year to year. In this year’s study, Volvo and Chrysler fell 10 and 8 spots in the rankings, while GMC, Cadillac, and Audi skyrocketed 10, 14, and 16 (!) slots.

Consumer Reports auto reliability data is noisy

How can one brand’s reliability ranking surge from the bottom 5 to the top 10 in just one year? Because Consumer Reports data is very “noisy,” and hence not terribly accurate.

Are we honestly supposed to believe that Audi was ranked as one of the least reliable brands last year, and yet somehow ranked top 10 in reliability this year? This is obviously a result of a limited amount of data, which brings me too…

3. CR uses as few as 100 surveys to rate vehicles! That’s right folks – 100 measly surveys is all it takes for Consumer Reports to assess a specific vehicle’s reliability rating.

100 data points is hardly enough to form a scientific evaluation – it’s embarrassing that CR would admit to this methodology, but they’ve done precisely that:

…The scores are presented as a percentage better or worse than the average of all cars. The minimum sample size is 100 vehicles, but Consumer Reports often gets many more.

While CR might “often” get 100’s or surveys, this hardly seems like a good system. It also explains Audi’s wild change in rankings, doesn’t it?

The bottom line: Don’t trust Consumer Reports quality and reliability data, at least as far as automobiles are concerned.

At best, use CR automotive rankings as a supplement to other data sources. See their official 2012 rankings here.


 

所有跟帖: 

不全是这样,它推荐的产品有的根本就不耐用! -chinomango- 给 chinomango 发送悄悄话 chinomango 的博客首页 (0 bytes) () 11/11/2014 postreply 19:05:45

人家测试的都是新产品而不是旧东西并不是耐久性 -MoonRiverMe- 给 MoonRiverMe 发送悄悄话 (0 bytes) () 11/12/2014 postreply 08:42:20

没质量就是垃圾,啥理由都不行 -chinomango- 给 chinomango 发送悄悄话 chinomango 的博客首页 (0 bytes) () 11/13/2014 postreply 08:20:38

CR倒是有破坏性测试。你想看苹果经不经存你会去看苹果甜度测试? -MoonRiverMe- 给 MoonRiverMe 发送悄悄话 (0 bytes) () 11/13/2014 postreply 10:37:55

和福布斯一样,没被收买才怪呢 -日理万机- 给 日理万机 发送悄悄话 (0 bytes) () 11/11/2014 postreply 19:13:48

Who brought CR? It's always indepent -MoonRiverMe- 给 MoonRiverMe 发送悄悄话 (0 bytes) () 11/12/2014 postreply 07:06:20

This article is flawed! -Yangtze430030- 给 Yangtze430030 发送悄悄话 Yangtze430030 的博客首页 (410 bytes) () 11/11/2014 postreply 21:35:31

No. CR's reliability data come from its readers. That's why the -yanif- 给 yanif 发送悄悄话 (477 bytes) () 11/12/2014 postreply 04:25:16

But those educated people who answered the survey are owners of -Yangtze430030- 给 Yangtze430030 发送悄悄话 Yangtze430030 的博客首页 (1276 bytes) () 11/12/2014 postreply 07:00:50

You are so naive o_o -隐睾- 给 隐睾 发送悄悄话 (61 bytes) () 11/12/2014 postreply 08:22:19

Can't agree with you more! -Yangtze430030- 给 Yangtze430030 发送悄悄话 Yangtze430030 的博客首页 (0 bytes) () 11/12/2014 postreply 09:04:21

本坛有”恨车一族“ o_o -隐睾- 给 隐睾 发送悄悄话 (31 bytes) () 11/12/2014 postreply 09:56:36

看看盖勒普(Gallup)如何统计和发表其统计结果就知道CR的方法太小儿科了。 -southmountainer- 给 southmountainer 发送悄悄话 (4679 bytes) () 11/12/2014 postreply 18:17:00

是传教式销售,CR花很多很多年建立的客户群,客户,应该很多老客户是80年代90年代买过日本车,使用坚信CR说的 -soccer88- 给 soccer88 发送悄悄话 soccer88 的博客首页 (0 bytes) () 11/12/2014 postreply 07:04:44

I might not be an educated person but -MoonRiverMe- 给 MoonRiverMe 发送悄悄话 (168 bytes) () 11/12/2014 postreply 10:34:27

This article did not say anything sJ -Yangtze430030- 给 Yangtze430030 发送悄悄话 Yangtze430030 的博客首页 (0 bytes) () 11/12/2014 postreply 07:06:06

The average reader of CR is over 60 years old. Yak. That also -其乐无穷- 给 其乐无穷 发送悄悄话 (54 bytes) () 11/12/2014 postreply 08:59:15

文科生写的文章! -Yangtze430030- 给 Yangtze430030 发送悄悄话 Yangtze430030 的博客首页 (75 bytes) () 11/12/2014 postreply 09:08:57

任何数据,在用数据下结论前,一定要搞明白数据是如何采集的,有什么假设,否则会得出错误的结论。 -southmountainer- 给 southmountainer 发送悄悄话 (39 bytes) () 11/12/2014 postreply 17:58:23

We are only talking about cars here -MoonRiverMe- 给 MoonRiverMe 发送悄悄话 (453 bytes) () 11/12/2014 postreply 09:44:47

请您先登陆,再发跟帖!