What affect will Windows 10 forced updates have on Research Software?
If you change it, It will break
“It only works on the Windows version of MATLAB 2010a. The code doesn’t work on Linux or other versions of MATLAB.” explained the researcher. He needed to run his program hundreds of times and his solution was to lock himself into a computer room over the weekend, log into the two dozen managed desktop machines there and manually start his code running on each one. Along with colleagues from the University of Manchester, I was trying to understand why he couldn’t use the Linux-based 3000+ core Condor pool we’d built since it was perfectly suited to his workflow.
Now, when he said ‘doesn’t work‘ what he meant was ‘gives different results’ and the only ones he liked were the ones that came from MATLAB 2010a on Windows. Naturally, I offered to take a look at his code with a view to figuring out what was going on but he simply wasn’t interested. Once he determined that we couldn’t (or more accurately, wouldn’t) stop his current workflow he thanked us for our interest and left.
Different strokes results for different folks
This wasn’t the first time I had discovered research code that gave different results when run on different operating systems or runtimes and it probably won’t be the last. A relatively high profile case that caught my eye recently was a publication that demonstrated that the results of a program called FreeSurfer varied according to operating system, workstation type and software version.
I suspect that the phenomenon might be more prevalent than we think because I suspect (but confess to having no evidence) that a large number of computational research results come from research code that’s only ever been run on one operating system, with one set of dependencies on one particular piece of hardware.
Experience has shown me that some researchers deal with this lack of robustness by keeping their working environment as constant as possible. Don’t. Touch. Anything!
The shifting sands of Windows 10
A huge percentage of researchers conduct their research using Windows and the Windows environment is about to change in a rather fundamental way: Updates will be automatic and mandatory.
Since the operating system will be constantly shifting under their feet, researchers are no longer going to be able to keep that aspect of their environment stable.
I wonder if or how this will change things in the world of research software. Perhaps it will go by unnoticed, perhaps more test-suites will be written or perhaps something else?
Any thoughts?
Sounds the MATLAB code used particularly brittle methods if it’s that sensitive to OS and hardware for a given MATLAB+JRE release. It might be useful for the Research Software Engineering community to collate and publish information on which commonly-used MATLAB/Python/R/Julia functions/methods yield outputs that are significantly dependent on OS/architecture. Alternatively the academic community could move to VMs as deliverables but I presently doubt that we have the time and resources to educate researchers in how to package research computing environments using Vagrant/Packer/Docker.
I think the bigger problem with research software is that most academics have never heard of version control. There’s more risk from a student changing research code (or data) than from Microsoft changing its code.
I think this will increase the popularity of Windows 7.
Another reason to use unit tests.
I had this exact issue with Matlab last year albeit on OSX, excited as I was to try out Yosemite before anyone else I installed it and found that Matlab does a lookup of the OS version when it loads and because 10.10 was not on the list, Matlab would not run. Taught me not to beta test on platforms I regularly use, I am in two minds whether to upgrade to Windows 10 yet in case HyperWorks 13 stops running too.
@John: You are right, version control would help. But, in defence of the students, one will find out that an error appeared/the results came out like that around the same time a student was working on the code. Eventually, one will see different results shortly after KB3035583 came out, but there will be many more fixes to the OS than students working on the code.