Consumer Reports says it stopped recommending Microsoft’s Surface Laptop and Surface Book based on reliability data of past Surface products, which the company admits is « not 100 percent perfect. »
When Consumer Reports stopped recommending the Microsoft Surface Laptop and latest Microsoft Surface Book, it did so based not on those specific products’ reliability. Rather, the company used reliability surveys of earlier Surface devices (including Surface 3, Surface Pro 3, and the early Surface Book) , according to a report the publication released Thursday.
Microsoft apparently privately confirmed this as well, via an internal memo obtained by Thurrott.com this week, which shows that while return rates of products like the Surface Pro 3 and Surface Book were initially atrocious, they improved substantially over time. Later Surfaces, like the Surface Hub, also never really suffered from the same issues.
Though it named four specific products, a Consumer Reports spokesman said it was penalizing the Surface brand as a whole. “We can’ t recommend any of [Microsoft’s] laptops or tablets until we get data that shows better reliability, ” a Consumer Reports spokesman said in an email.
Microsoft, for its part, said that the company respectfully disagreed with Consumer Reports’ findings, and that its real-world return and support rates are on par with competing devices and “significantly lower than 25 percent, ” Microsoft Devices corporate vice president Panos Panay wrote .
Consumer Reports pulled its recommendation of the Surface Laptop, even though the notebook wasn’ t covered in the survey.
Consumer Reports, which bases its reliability reports on surveys that it sends to its subscribers, compiled data on 90,741 tablets and laptops that its members bought from 2014 until the beginning of 2017, it said. That data included devices from a variety of manufacturers, including Microsoft Surface devices.
The reliability predictions represent extrapolations made from data reported on older devices, such as the Surface 3, Surface Pro 3, and other devices sold during that time, the CR report said. Actual problems that consumers experienced included freezing or unexpected shutdowns, and touchscreens that weren’ t responsive enough, the publication said. Using that past data, Consumer Reports then generated a reliability rating for the brand as a whole.
The Microsoft Surface Pro 3 tablet suffered some early hardware problems.
The spokesman acknowledged the problem with using past data on a new product that wasn’ t even shipping when the survey took place. “It’s not 100 percent perfect, ” the spokesman said of the publication’s methodology.
Still, the spokesman added, it represents the best data the company has on the brand as a whole. “Our analysis revealed that Microsoft as a brand has a comparatively worse reliability rate than other brands, so we can’ t recommend Microsoft tablets and computers – no matter the model family, ” the Consumer Reports spokesman said. “This wouldn’ t necessarily change if Microsoft came out with a brand new tablet or computer next week.”
Consumer Reports’ findings surprised many at Microsoft and elsewhere, which had seen steadily increasing reliability metrics from generation to generation. Surface Pro 3 owners had struggled through battery issues and overheating, while the Surface Pro 4 and Surface Book suffered screen flickering and their own power problems. But by 2017, the Surface Book had become fairly reliable.
Given that the Surface Laptop began shipping in June, after the survey concluded, many wondered how Consumer Reports could reach its conclusions. “I own and have used every version of Surface products for extensive periods of times, not just reviews, ” Patrick Moorhead, principal analyst at Moor Insights, said in an email. Moorhead’s experiences, while anecdotal, are far more positive. “Since 2012 I have only had an issue with two systems: software issues with Windows RT systems in 2012 and in 2016 with the first batch of Skylake-based systems. After six months, the Skylake driver issues disappeared.”
Moorhead’s largely trouble-free experience extends to Microsoft’s latest models. “I have both the Surface Pro (5) and Surface laptop and have not experienced an issue with stock (non-beta) software.”
Panos Panay, Microsoft’s Corporate Vice President of Surface Computing, demonstrates the new Surface Book at the company’s press event in New York City on October 26,2016.
Consumer Reports is no stranger to controversy. The publication made headlines earlier this year when it said it uncovered issues with the Apple MacBook Pro’s battery life. Its testing used a developer setting that included a undiscovered bug, however, which affected its results. Those tests prompted Consumer Reports to withhold a buying recommendation, though CR later reinstated the recommendation after the bug was resolved.
Microsoft has its own side of the story. The company maintains an internal testing facility and generates its own estimates of how its Surface devices compare to its competition—which is basically Apple. Its internal defect rate was “not even close” to 25 percent, according to Microsoft employees. Surface Computing vice president Panos Panay wrote that the “incidents per unit” were less than 1 percent, or presumably less than 1 out of 100 devices.
Consumer Reports noted that it ranked the new Surface Pro very highly in recent Consumer Reports tests, including display quality, battery life, speed, and ergonomics. (PCWorld’s Surface Pro review nicked the tablet for lack of innovation, while our Surface Book review was more positive) . The CR spokesman also acknowledged in an email that models within a brand may vary, and that changes in design and manufacture might affect future reliability. By then, though, the damage had been done.
