Microsoft: To ensure Windows 10 update quality, these are the tools we use

[The AchieVer]

Microsoft: To ensure Windows 10 update quality, these are the tools we use

Microsoft offers Windows 10 users a look at how they use data to improve the quality of Windows 10 updates.

 

 

 

 

The Windows 10 version 1809 release didn't quite go as smoothly as Microsoft hoped, prompting lots of questions about how it tests the quality of its builds before release. To answer these questions, Microsoft has now revealed one of the key tools it's developed to measure the health of Windows updates.  

 

These days Microsoft is heavily reliant on diagnostics data to ensure Windows 10 is humming along nicely across the world, and key to this ability is Release Quality View (RQV), an internal dashboard Microsoft uses to assess the ongoing health of millions of systems before and after updates roll out.  

 

According to Jane Liles and Rob Mauceri from the Windows Data Science team, the company approaches every new release with the question: "Is this Windows update ready for customers?".

 

"This is a question we ask for every build and every update of Windows, and it's intended to confirm that automated and manual testing has occurred before we evaluate quality via diagnostic data and feedback-based metrics," they write in a blogpost.

 

So while Microsoft does depend heavily on diagnostic data, the pair of data scientists emphasize the internal checking by engineers that takes place even before Windows Insider testers receive a build.     

First, a build that goes out must pass initial quality testing. Then Microsoft engineers who "aggressively self-host Windows" give it a thorough once-over for potential problems. Diagnostics data is involved at all stages. 

 

"We look for stability and improved quality in the data generated from internal testing, and only then do we consider releasing the build to Windows Insiders, after which we review the data again, looking specifically for failures," explain Liles and Mauceri. 

 

They're responsible for ensuring that Microsoft's metrics are "reliable, repeatable, precise, true and unbiased" before the big "Is Windows ready" question is broached. 

 

The data science team is always aiming to ensure current metrics are higher than the quality levels of the previous release.

 

To understand what 'quality' looks like through data, the team counts the unique 'active' monthly devices – a number that Microsoft doesn't publish – and then they look at how the upgrade process went, as well as the "general health of the user experience". 

 

Then they look at certain user scenarios that indicate success for Windows, which include "success rates for connecting to Wi-Fi, or opening a PDF file from Microsoft Edge, or logging in using Windows Hello".

Central to all this data-driven analysis is the RQV dashboard, which includes over 1,000 other measures. This dashboard is used to assess the customer experience while a build is still in the hands of engineers and Insiders, and after an update becomes generally available. 

 

The dashboard is also critical to Windows managers' 'readiness sessions', where engineering teams review RQV measures and decide what bugs to fix. 

 

"Starting with the lowest scoring problem areas, we run down the list of areas whose measures are proportionally farthest from their targets. The engineering owners for those areas are then called on to explain what is causing the problem, who is on point to resolve it, and when they expect the quality of that area, as represented by the measures, to be back within target."

 

RQV developers also built the capability to see whether a bug fix actually resolves the problem, which can be seen by a particular measure returning to a healthy range.  

 

"Fixes that engineers check into future builds are tracked through the system, so reviewers can see when a fix will be delivered via a new build and can monitor impact as the build moves through its normal validation path: through automated quality gates, to self-hosted devices in our internal engineering 'rings', and to our Windows Insiders."

 

The company acknowledges there are gaps and specifically notes customer feedback as an area it is investing in "to help us identify gaps or inconsistencies in our diagnostic data-based measures". 

It's also looking at how to "provide insight into the experiences you have on your actual devices." And of course the company is creating new machine-learning models for "earlier detection through text analytics". 

This view of the trend for networking measures in internal and Windows Insider builds between January 25 and February 19, 2019, shows an improved score over the previous build.

 

 

 

Source

[Karlston]

I stopped reading after "Microsoft: To ensure Windows 10 update quality".

 

I couldn't stop laughing...

[shorty6100]

"To ensure Windows 10 update quality, these are the tools we use"-more telemetry, more spying.

[Mutton]

To ensure Windows 10 update quality, these are the tools we use...

...our USERS!

[Karlston]

Microsoft's data-focused approach to Windows Updates requires an update

Microsoft uses a data-focused approach to determine if updates are ready for wider distribution and it appears that the company is very content with the approach.

 

Issues of recent updates suggest that Microsoft may need to rethink its strategy when it comes to Windows Updates.

 

Microsoft started a new series on the official Windows Experience blog some time ago in which company representatives lift the veil in regards to the company's strive for quality.

 

Microsoft told the public that update quality is better than ever, and that the same is true for the quality of released device drivers.

 

In "Data, insights and listening to improve the customer experience", information is provided on how Microsoft uses data to determine the quality of updates.

 

The fundamental question that Microsoft asks for each release is "Is this Windows Update ready for customers?". Updates go through different stages during development:

1. Automatic and manual testing of updates.
2. Evaluation of quality based on diagnostic data and feedback from Microsoft engineers.
3. Distribution to Windows Insiders and further evaluation of updates.
4. Distribution to the general Windows population.

The metrics that Microsoft gathers and monitors need to be equal to or better than the metrics of the previous update.

By the time we are ready to ship to our customer base, our metrics must be, at a minimum, at or above the quality levels for the previous release, the idea being that every update should make the Windows 10 experience better.

Is the approach sufficient?

 

Data plays an important role when it comes to the release of updates. Microsoft, and any other company for that matter, may use it to make sure that certain features behave as intended. It is certainly possible to catch bugs by just looking at data but if you look at recent updates, you will notice that things were not as smooth as Microsoft's data focused appraoch suggested.

 

Microsoft had to pause the Windows 10 version 1809 feature update because it caused a lot of issues. To name just a few:

• A Data loss issue
• Another data loss issue
• No audio issue and bluescreens
• Font display issues
• Mapped network drives issues

All of these issues were not detected during tests conducted by Microsoft, by diagnostic data, and by feedback that Microsoft engineers and participants of Microsoft's Insider program provided.

 

A game performance related bug was not detected in the most recent update for Windows 10 version 1809. Microsoft had to update the support article to add the performance affecting bug to the list of known issues of the update.

The problem

 

Microsoft engineers and participants of the Insider program may not provide a sufficient sample size to provide data for all major use cases. Gaming might be such a case. It seems unlikely that Microsoft engineers spend a lot of time playing games on their devices. Even if they would, they could never test new versions of Windows on tens of thousands of games that are available for PC. The bulk of Insider participants may not be interested in games as well. Gaming is just one area where Microsoft's approach falls short.

 

It is certainly unrealistic to expect Microsoft to catch all issues in all updates before release. The sheer number of hardware and software configurations makes that an impossible task.

 

But major issues, like game performance in popular games, should not hit the general population.

 

That's one reason why it is a good idea to install updates a while after release and not as early as possible: you never know what is going to happen.

 

Source: Microsoft's data-focused approach to Windows Updates requires an update (gHacks - Martin Brinkmann)