{"id":3586,"date":"2011-06-10T12:19:17","date_gmt":"2011-06-10T11:19:17","guid":{"rendered":"http:\/\/www.walkingrandomly.com\/?p=3586"},"modified":"2011-08-12T17:30:52","modified_gmt":"2011-08-12T16:30:52","slug":"in-defense-of-inefficient-scientific-code","status":"publish","type":"post","link":"https:\/\/walkingrandomly.com\/?p=3586","title":{"rendered":"In defense of inefficient scientific code"},"content":{"rendered":"

One part of my job that I really enjoy is the optimisation of researcher’s code. \u00a0Typically, the code comes to me in a language such as MATLAB<\/a> or Mathematica<\/a> and may take anywhere from a couple of hours to several weeks to run. \u00a0I’ve had some nice successes recently in areas as diverse as finance, computer science, applied math and chemical engineering among others. \u00a0The size of the speed-up can vary from 10% right up to 5000% (yes, 50 times faster!) and that’s before I break out the big guns such as Manchester’s Condor pool or turn the code over to our HPC specialists for some SERIOUS (yet more time consuming in terms of developer time) optimisations.<\/p>\n

Reporting these speed-ups to colleagues (along with the techniques I used) gets various responses such as ‘Well, they shouldn’t do time-consuming computing using high level\u00a0languages. \u00a0They should rewrite the whole thing in Fortran’ <\/strong>or words to that effect. \u00a0I disagree!<\/p>\n

In my opinion, high level programming\u00a0languages\u00a0such as Mathematica, MATLAB and Python have democratised scientific programming. \u00a0Now, almost anyone who can think logically can turn their scientific ideas into working code. \u00a0I’ve seen people who have had no formal programming training at all whip up models, get results and move on with their research. \u00a0 \u00a0Let’s be clear here – It’s results that matter not how you coded them.<\/p>\n

It comes down to this. \u00a0CPU time is cheap. \u00a0Very cheap. \u00a0Human time, particularly specialised human time, is expensive.<\/strong><\/p>\n

Here’s an example: \u00a0Earlier this year I was working with a biologist who had put together some MATLAB code to analyse her data. \u00a0She had written the code in less than a day and it gave the correct results but it ran too slowly for her tastes. \u00a0Her sole programming experience came from reading the MATLAB manual and yet she could cook up useful code in next to no time. \u00a0Sure, it was slow and (to my eyes) badly written but give the gal a break…she’s a professional\u00a0biologist\u00a0and not a professional programmer. \u00a0Her programming is a lot better than my biology!<\/p>\n

In less than two hours I gave her a crash course in MATLAB code optimisation; how to use the profiler,\u00a0vectorisation\u00a0and so on. \u00a0We identified the hotspot in the code and, between us, recoded it so that it was an order of magnitude faster. \u00a0This was more than fast enough for her needs, she could now analyse data significantly faster than she could collect it. \u00a0 I realised that I could make it even faster by using parallelised mex functions<\/a> but it would probably take a few more hours work. \u00a0She declined my offer…the code was fast enough<\/strong>.<\/p>\n

In my opinion, this is an optimal use of resources. \u00a0I spend my days obsessing about mathematical software and she spends her days obsessing about experimental biology. \u00a0She doesn’t need a formal course in how to write uber-efficient code because her code runs as fast as she needs it to (with a little help from her friends). \u00a0The solution we eventually reached might not be the most CPU-efficient one but it is a good trade off between CPU-efficient and developer-efficient.<\/p>\n

It was easy…trivial even..for someone like me to take her inefficient code and turn it into something that was efficient enough. \u00a0However, the whole endeavour relied on her producing working code in the first place<\/strong>. \u00a0Say high-level languages such as MATLAB didn’t exist….then her only options would be to hire a professional programmer (cash expensive) or spend a load of time learning how to code in a low level language such as Fortran or C (time expensive).<\/p>\n

Also, because she is a beginner programmer, her C or Fortran code would almost certainly be crappy and one thing I am sure of is ‘Crappy MATLAB\/Python\/Mathematica\/R code is a heck of a lot easier to debug and optimise than crappy C code<\/strong>.’ \u00a0Segfault anyone?<\/p>\n","protected":false},"excerpt":{"rendered":"

One part of my job that I really enjoy is the optimisation of researcher’s code. \u00a0Typically, the code comes to me in a language such as MATLAB or Mathematica and may take anywhere from a couple of hours to several weeks to run. \u00a0I’ve had some nice successes recently in areas as diverse as finance, […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[7],"tags":[],"class_list":["post-3586","post","type-post","status-publish","format-standard","hentry","category-programming"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p3swhs-VQ","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/3586","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3586"}],"version-history":[{"count":11,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/3586\/revisions"}],"predecessor-version":[{"id":3798,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/3586\/revisions\/3798"}],"wp:attachment":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3586"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3586"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}