For the former Microsoft software developer Jerry Berg everything was better in the past: For 15 years he worked on the launch of Windows operating systems like Windows XP, Vista and 8. In a Youtube video, he explains why from his point of view, the current Windows 10 for new content updates contains many bugs. The reason: The company runs test scenarios automatically in virtual machines. “They do not represent diversity, they all act the same, they all have the same hardware configuration and all have the same memory,” he says.
That’s why it was up to the employees themselves to find fault with self-hosting. To do this, developers have equipped their machines with new Windows builds and tested this version in realistic operation. Bugs were noticed so immediately. Meanwhile, this method is probably less widespread. Instead, Microsoft relies more on Windows Insiders and their diagnostic data. If the community reports an error, the corresponding scenario will run on the test systems of the Windows developers. If the bug is confirmed by it, be it at the next instance to fix it.
Telemetry does not tell the whole story
However, many error messages from the Windows Insider are not really working parts of the OS and no bugs. Therefore, these would partly not forwarded. In addition, only a very small proportion of insiders are ready to report certain errors in a manner that is usable for Microsoft. “Fundamentally, Microsoft has replaced flesh-and-blood people who create automatic test sequences by ourselves – the consumers -“ says Berg.
In addition, user telemetry data would not help identify external dependencies on crashing or misbehaving processes. In many situations, it is not the reported program itself that is responsible for bugs, but some other parallel software. Although it is possible to create complete snapshots of a configuration and situation, such full crash dumps would very rarely be uploaded due to their sheer size. Microsoft would most often have to deal with small dumps – snapshots of the process crash within memory – and manually assemble the associated test environment.