## A Parallel programming tip for Mathematica

June 29th, 2009

Someone emailed me recently complaining that the ParallelTable function in Mathematica didn’t work.  In fact it made his code slower!  Let’s take a look at an instance where this can happen and what we can do about it. I’ll make the problem as simple as possible to allow us to better see what is going on.

Let’s say that all we want to do is generate a list of the prime numbers as quickly as possible using Mathematica.  The code to generate the first 20 million primes is very straightforward.

primelist = Table[Prime[k], {k, 1, 20000000}];


To time how long this list takes to calculate we can use the AbsoluteTime function as follows.

t = AbsoluteTime[];
primelist = Table[Prime[k], {k, 1, 20000000}];
time2 = AbsoluteTime[] - t


This took about 63 seconds on my machine. Taking a quick look at the Mathematica documentation we can see that the parallel version of Table is, obviously enough, ParallelTable. What could be simpler? So, to spread this calculation over both processors in a dual-core laptop we simply need to do the following

t = AbsoluteTime[];
primelist = ParallelTable[Prime[k], {k, 1, 20000000}];
time2 = AbsoluteTime[] - t


The result? 98 seconds!  So, it seems that the parallel version is more than 50% SLOWER! My correspondent was right – ParallelTable doesn’t seem to work at all.

Before we send a message to Wolfram Tech Support though, let’s dig a little deeper. It turns out that a single evaluation of
Prime[k] for any given k is very quick – even for high k. Prime[100000000] evaluates in a tiny fraction of a second for example (try it! – it really is astonishingly quick.)

What I suspect is happening here is that when you move to the parallel version, the kernels spend most of the time communicating with each other rather than actually calculating anything.  I think that ParallelTable does something roughly like the following.

Kernel1:Give me a k.
Master:have k=1.
Kernel2:Give me a k.
Master:have k=2
Kernel1:I’ve done k=1 and the answer is 2, give me another k
Master: have k=3
kernel2:I’ve done k=2 and the answer is 3, give me another k etc

So, the kernels spend more time talking about the work that needs to be done rather than actually doing it. Also, for various reasons, kernel1 and kernel2 might get out of sync and so the Master kernel may end up with a list out of order.  If so then it will need to reorder the lists at the end of the calculations and so that’s even more time taken.

So, my approach was to try and reduce the amount of communication between kernels to something like

Master: Kernel 1 – you go away and get me first 10 million primes. Don’t bother me until it’s done.
Master: kernel 2 – you go away and get me the second 10 million. Don’t bother me until it’s done.

The following Mathematica code does this.

t = AbsoluteTime[];
job1 = ParallelSubmit[Table[Prime[k], {k, 1, 10000000}]];
job2 = ParallelSubmit[Table[Prime[k], {k, 10000001, 20000000}]];
{a1, a2} = WaitAll[{job1, job2}];
time2 = AbsoluteTime[] - t


This works in 40 seconds compared to the original time of 62 seconds. Not a 2x speedup but not too shabby for so little extra work on our part.

The moral of the story? Make sure that your code spends more time actually doing work than it does just talking about it.

Disclaimer: I am still learning about Mathematica’s parallel tools myself so don’t take this stuff as gospel. Also, there are almost certainly more efficient ways of getting a list of primes than using the Prime[k] function. I am only using Prime[k] here because it is an example of a function that evaluates very quickly.

June 25th, 2009

Update (10th Feb 2010): There is an update to this post here.

I received a lot of feedback from my recent post Is Mathcad Dying? and, thanks to a couple of readers, I have learned of some new (new to me at least) information sources concerning this product and they all point to a major new mathematical product coming from PTC called Mathcad Prime.

The first ‘new’ Mathcad information source is the blog, Engineering with Mathcad, and its associated Twitter feed.  I am not sure if this blog is official or not because all of the posts are currently anonymous but the author(s) seems to know what is going on in the Mathcad world and it is well worth a look.

The second is another blog, simply called Mathcad, which is written by the Mathcad product management team.  It’s only been going since the beginning of the month but there are already some interesting posts there including The right tool for the job and Mathcad and Knovel Math.

So it looks like PTC DO have something up their sleeve in the form of a new product called Mathcad Prime.  Until very recently I hadn’t heard of it and I am yet to see some screenshots, videos or demos myself but it was apparently demonstrated at a PTC user conference recently. Google seems to know very little about it  at the moment though.

## EuroSciPy 2009

June 22nd, 2009

I’ll be attending the 2009 EuroSciPy scientific Python conference in Leipzig next month – July 25th and 26th to be precise – and would love to hear from anyone else who’ll be attending.

## Proof that math exams have become easier over the last 169 years!

June 22nd, 2009

Solution to problem of the week #6

Way back in April, I posed the following problem.  ‘Consider a square pyramidal pile of identical cannonballs of radius r such that the bottom layer contains 16 cannonballs (such as the pile in the diagram above). Find the volume (in terms of r) of the pyramid that envelops and contains the whole pile’

Since then I have received several answers (check out the comments section of the original post for a few of them) and all but one of them were wrong.  In my opinion, this is nothing to be too ashamed about since I couldn’t solve the problem either and I am not about to berate my readers for failing to do something that I couldn’t do myself!

So if I couldn’t do it then how did I know that all of these answers were incorrect?  Well clearly I had cheated and had access to a worked solution.  ‘My’ problem was in fact problem number 15 from one the 1840 undergraduate final exams at Cambridge University and a worked solution is given in the (now fully digitized, thanks to google) text Solutions for the Cambridge Problems 1840,1841.

The solution is

You’ll find the official worked solution to this problem on page 20 of ‘Solutions for the Cambridge Problems 180,1841’ (sadly, the diagram is missing) but James Graham-Eagle of the University of Massachusetts Lowell sent me not one but two different solutions in this pdf file and they are much easier to follow in my humble opinion.  Thanks for that James.

I’m glad that this problem wasn’t in my final exam!

## Octave 3.2 released

June 19th, 2009

A major new release of the free alternative to MATLAB, GNU Octave was released a couple of weeks ago (on June 6th to be precise).  The Octave team have added many new features and so this release looks well worth checking out.  The full set of user visible changes can be found on the Octave website but a few highlights include.

• Compatibility with Matlab graphics has been improved – many new graphics functions added including ezplot,
isosurface and plotmatrix.
• New functions for reading and writing images. The imwrite and imread functions have been included in Octave.
• New experimental OpenGL/FLTK based plotting system. An experimental plotting system based on OpenGL and the FLTK toolkit is now part of Octave.
• Object Oriented Programming.
• Octave now includes a single precision data type. Single precision variables can be created with the single command, or from functions like ones, eye, etc.
• 64-bit integer arithmetic has been added. Arithmetic with 64-bit integers (int64 and uint64 types) is fully supported.
• Improved array indexing. The underlying code used for indexing of arrays has been completely rewritten and indexing is now significantly faster
• Various other performance improvements.

June 16th, 2009

I’ll start off this post by mentioning that I don’t like PTC’s Mathcad very much and think that is a very weak product compared to its competitors.  Professionally I have had a lot of grief with it and personally I cannot see why anyone who can also choose from Mathematica, MATLAB and Maple (and I am lucky enough to be in this position) would ever bother with it. Most of the things I choose to write about Mathcad concern its bugs.

So, read the following in the knowledge that the writer is heavily biased.

Just recently I have found myself wondering if the product is doomed.  Let’s look at the evidence.

• PTC’s Mathcad hasn’t seen a major new release in over 2 years.  Version 14 was released on 12th February 2007 and since then its competitors have gone from strength to strength.  In the same time MATLAB has seen 5 major new releases going from 2006b to 2009a and Mathematica has been improved beyond recognition in the transformation from version 5.2 to 7.01.  Then we have Maple which was at version 11 back in 2007 and is now at version 13.

Before anyone states the obvious, yes I know version numbers on their own mean very little but the increase in functionality in Mathcad’s competitors over the last 2 years or so has been substantial whereas Mathcad itself has gone nowhere.

• MathCad’s symbolic engine, Mupad, has since been bought by rival math software vendor and maker of MATLAB, The Mathworks.

When I first started working with Mathcad, it came with a cut-down version of the Maple kernel which took care of all of its symbolic calculations.  There was nothing particularly unusual about this as several other maths packages did exactly the same (MATLAB’s symbolic toolbox immediately springs to mind) but in the transition from v13 to v14, Mathcad swapped the Maple Kernel for Mupad.

Mupad was a nice product and, although there were problems with the transition, Mathcad could have done a lot worse in its choice of symbolic engine.  Of course, since version 14 of Mathcad was released the owners of this symbolic engine, Sciface Software, were completely bought out by MATLAB makers, The Mathworks, and now The Mathworks use Mupad as the basis for their symbolic toolbox.

So where does that leave Mathcad?  Will The Mathworks strike a licensing deal with PTC for the Mupad technology or will PTC have to find a replacement symbolic engine for version 15 of Mathcad?

• The maintenance releases don’t add very much

Every software manufacturer has maintenance releases which tend to be little more than a set of bug-fixes and extra tidbits of functionality to keep users happy between major releases.  Although we shouldn’t expect too much of them – we at least expect something worth justifying the download.  Here is the changelog from the latest maintenance release of Mathcad – Mathcad M030.

New features
* Windchill 9.1 M010 Workgroup Manager (WGM) support
* New installer – You can now customize your Mathcad installation by selecting the language, components, and directories.
* User interface translation into Russian
Problems fixed in Mathcad 14 M030
* 1436139: Addresses specific problems involving rapid consumption of memory upon repeated recalculation of a worksheet.
* 1502717: Fixes improper result returned when evaluating 2^31.
* 1507803 Addresses a specific memory leak issue relating to an uninitialized handle.
* 1507032 Addresses a compatibility problem resulting from an improvement in how numeric tolerance in integrals was inferred.
* 1547641 Fixes PDF file generation problem on Window XP x64 machines.
* 1437427 Fixes in-line evaluation to update display to match available value.
* 1587915 Addresses a WRITEPRN restriction by increasing the maximum allowable string length from 128 to 1024.
* 1403321 Addresses specific memory release issue involving large worksheet computations.

Although I am sure this is good news for Russian users I have to say that I’m underwhelmed!

So what do you think – Is Mathcad doomed or am I making a big fuss over nothing? Comments from Mathcad users are particularly welcomed.

## Math Software houses who Tweet

June 15th, 2009

When I first came across Twitter, the micro-blogging site that allows people to follow your interests (and your life if you like) by answering the question ‘What are you doing?” I thought it was completely and utterly pointless.  Most of my friends agreed.

I move in very geeky circles though and while at BarCamp Sheffield last year, someone convinced me to sign up and give it a try.  To start with, all it did was confirm my initial suspicions – it seemed to be pointless.  I knew when my geek friends were in the office, I knew what they were having for lunch, when they were having problems with Java and when they were being bit on the toe by their pet ferrets, Ruby and Perl.  Pointless and yet strangely compelling.

In a week I was hooked.

Since then, not only have I found it to be compelling but I have also found it to be immensely useful.  Just last week, for example, I was stranded in Manchester and needed a cheap hotel room.  Although my phone has Internet, it’s very slow, so I didn’t want to have to spend ages googling for the best deal so I tweeted to my small band of followers that I was stuck and I needed help.  They came through for me with first class form and I was soon in a nice, comfortable room that included breakfast for under 50 quid.

I have used it to help solve technical problems, arrange impromptu gatherings, follow the world’s reactions to Wolfram Alpha’s launch in real time,get more readers for this blog, get advice on learning German and, of course, to tell the world what I am having for lunch.

Some people find it significantly more useful though – Dell used it to boost sales of their computers by \$3million apparently!

More recently, I have noticed that the makers of various mathematical software packages have started getting in on the tweeting act and the results can be interesting.  Following the tweets of organisations such as Mathworks and OriginLab can be a great way to keep abreast of blog posts, interesting technical tidbits and new releases among other things.  Some of the better Twitter feeds (Labview’s for example) encourage a public two-way dialog between the application vendor and its users – a trend that I hope will continue.

Here are a few I have found so far, some better than others.

• labview – An official feed from National Instruments
• Maplesoft – The official feed of the makers of maple
• MATLAB_Webinars – Seems to be an ‘official’ Mathworks feed.
• matlabDoug – The feed of Mathwork’s employee, Doug Hull.
• OriginLab – Official feed from OriginLabs – The makers of plotting software, Origin.  Feed only just started.
• sagemath – Not sure if this is official.  Could do with being updated more often.
• Wolfram_Alpha – Official feed of Wolfram Alpha.  Includes lots of interesting Wolfram Alpha inputs.

Let me know if you find any more and I’ll add them to the list.

## New math problem of the month series

June 14th, 2009

A new math problem of the month series has started up over at problemas | teoremas – a dual language English, Portuguese blog written by Walking Randomly reader, Américo Tavares.  Américo successfully solved some of the problems I set here in the past and now he has started his own series.

Check out the first problem here but try not to look at the comments section if you don’t want any clues.

## Should Fortran be taught to Undergraduates?

June 10th, 2009

Back in 1997 I was a 2nd year undergraduate of Physics and I was taught how to program in Fortran, a language that has survived over 40 years due to several facts including

• It is very good at what it does.  Well written Fortran code, pushed through the right compiler is screamingly fast.
• There are millions of lines of legacy code still being used in the wild.  If you end up doing research in subjects such as Chemistry, Physics or Engineering then you will almost certainly bump into Fortran code (I did!).
• A beginner’s course in Fortran has been part of the staple diet in degrees in Physics, Chemistry and various engineering disciplines (among others) for decades.
• It constantly re-invents itself to include new features.  I was taught Fortran 77 (despite it being 1997) but you can now also have your pick of Fortran 90, 95, 2003, and soon 2008.

Almost everyone I knew hated that 1997 Fortran course and the reasons for the hatred essentially boiled down to one of two points depending on your past experience.

• Fortran was too hard!  So much work for such small gains (First time – programmers)
• The course was far too easy.  It was just a matter of learning Fortran syntax and blitzing through the exercises. (People with prior experience)

The course was followed by a numerical methods course which culminated in a set of projects that had to be solved in Fortran.  People hated the follow on course for one of two reasons

• They didn’t have a clue what was going on in the first course and now they were completely lost.
• The problems given were very dull and could be solved too easily.  In Excel!  Fortran was then used to pass the course.

Do you see a pattern here?

Fast forward to 2009 and I see that Fortran is still being taught to many undergraduates all over the world as their first ever introduction to programming.  Bear in mind that these students are used to being able to get interactive 3D plots from the likes of Mathematica, Maple or MATLAB and can solve complex differential equations simply by typing them into Wolfram Alpha on the web.  They can solve problems infinitely more complicated then the ones I was faced with in even my most advanced Fortran courses with just a couple of lines of code.

### Learn Fortran – spend a semester achieving not very much

These students study subjects such as physics and chemistry because they are interested in the subject matter and computers are just a way of crunching through numbers (and the algebra for that matter) as far as they are concerned.  Despite having access to untold amounts of computational and visualisation power coupled with easy programming languages thanks to languages such as MATLAB and Python, these enthusiastic, young potential programmers get forced to bend their minds around the foibles of Fortran.

For many it’s their first ever introduction to programming and they get forced to work with one of the most painful programming languages in existence (in my opinion at least).   Looking at a typical one-semester course it seems that by the time they have finished they will be able to produce command line only programs that do things like multiply matrices together (slowly), solve the quadratic equation and find the mean and standard deviation of a list of numbers.

Not particularity impressive for an introduction to the power of computation in their subject is it?

Fortran syntax is rather unforgiving compared to something like Python and it takes many lines of code to achieve even relatively simple results.  Don’t believe me?  OK, write a program in pure Fortran that gives a plot of a Sin(n*x) for integer n and x ranging from -2*pi to 2*pi.  Now connect that plot up to a slider control which will control the value of n.  Done?  Ok – now get it working on another operating system (eg if you originally used Windows, get it to work on Mac OS X).

Now try the same exercise in Python or Mathematica.

This is still a very basic program but it would give a much greater sense of achievement compared to finding the mean of list of numbers and could easily be extended for more able students (Fourier Series perhaps).  I believe that many introductory Fortran programming courses end up teaching students that programming means ‘Calculating things the hard way’ when they should be leaving an introductory course with the opposite impression.

### Learn Fortran – and never use it again.

Did you learn Fortran at University?  Are you still using it?  If you answer yes to both of those questions then there is a high probability that you are still involved in research or that advanced numerical analysis is the mainstay of your job.  I know a lot of people in the (non-academic) IT industry – many of them were undergraduates in Physics or Chemistry and so they learned Fortran.  They don’t use it anymore.  In fact they never used it since completing their 1st semester, 2nd year exam and that includes the computational projects they chose to do as part of their degrees.

The fact of the matter is that most undergraduates in subjects such as Physics end up in careers that have nothing to do with their degree subject and so most of them will never use Fortran ever again.  The ‘programming concepts’ they learned might be useful if they end up learning Java, Python or something along those lines but that’s about it.  Would it not be more sensible to teach a language that can support the computational concepts required in the underlying subject that also has an outside chance of being used outside of academia?

### Teach Fortran – and spend a fortune on compilers

There are free Fortran compilers available but in my experience these are not the ones that get used the most for teaching or research and this is because the commercial Fortran compilers tend to be (or at least,they are perceived to be) much better.  The problem is that when you are involved with looking after the software portfolio of a large University (and I am) then no one will agree on what ‘the best’ compiler is.  (Very) roughly paraphrased, here are some comments I have received from Fortran programmers and teachers over the last four years or so.

• When you teach Fortran, you MUST use the NAG Fortran compiler since it is the most standards compliant.  The site license allows students to have it on their own machines which is useful.
• When you teach Fortran, you MUST use the Silverfrost compiler because it supports Windows GUI programming via Clearwin.
• We MUST avoid the Silverfrost compiler for teaching because it is not available on platforms such as Mac and Linux.
• We MUST avoid the NAG compiler for teaching because although it has a great user interface for Windows, it is command line only for Linux and Mac.  This confuses students.
• If we are going to teach Fortran then we simply MUST use the Intel Fortran Compiler.  It’s the best and the fastest.
• We need the Intel Fortran Compiler for research because it is the only one compatible with Abaqus.
• A site license for Absoft Fortran is essential.  It’s the only one that will compile <insert application here> which is essential for my group’s work.
• We should only ever use free Fortran compilers – relying on commercial offerings is wrong.

I’m only getting started!  Choose a compiler, any compiler – I’ll get back to you within the week with an advocate who thinks we should get a site license for it and another who hates it with a passion.  Let’s say you had a blank cheque book and you gave everyone exactly what they wanted – your institution would be spending tens of thousands of pounds on Fortran compilers and you’d still be annoying the free-software advocate.

Needless to say, most people don’t have a blank cheque book.  At Manchester University (my workplace) we support 2 site-wide Fortran compilers

• The NAG Compiler – recommended for teaching.
• The Intel Fortran Compiler – recommended for research

We have a truly unlimited site license for NAG (every student can have a copy on their own machine if they wish) which makes it perfect from a licensing point of view and many people like to use it to teach.  The Widows version has a nice GUI and help system for example – perfect for beginners.  There are Mac and Linux versions too and although these are command-line only, they are better than nothing.

The Intel Fortran Compiler licenses we have are in the form of network licenses and we have a very limited number of these – enough to support the research of every (Windows and Linux) Fortran programmer on campus but nowhere near enough for teaching.  To get enough for teaching would cost a lot.

There are also pockets of usage of various other compilers but nothing on a site-wide basis.

Of course, not everyone is happy with this set-up, and I was only recently on the receiving end of a very nasty email from someone because I had to inform him that we didn’t have the money to buy his compiler of choice.  It’s not my fault that he couldn’t have it (I don’t have a budget and never have had) but he felt that I was due an ear lashing I guess and blamed all of his entire department’s woes on me personally.  Anway, I digress….

The upshot is that Fortran compilers are expensive and no one can agree on which one(s) you should get.  If you get them all to please (almost) everybody then you will be very cash poor and it won’t be long before the C++ guys start knocking on your door to discuss the issue of a commercial C++ compiler or three….

### Learn Fortran – because it is still very useful

Fortran has been around in one form or another for significantly longer than I have been alive for a very good reason. It’s good at what it does. I know a few High Performance Computing specialists and have been reliably informed that if every second counts in your code execution – if it absolutely, positively, definitely has to run as fast as humanly possible then it is hard to beat Fortran. I take their word for it because they know what they are talking about.

Organisations such as the Numerical Algorithm’s Group (NAG) seem to agree with this stance since their core product is a Fortran Library and NAG have a reputation that is hard to ignore.  When it comes to numerics – they know their stuff and they work in Fortran so it MUST be a good choice for certain applications.

If you find yourself using research-level applications such as Abaqus or Gaussian then you’ll probably end up needing to use Fortran, just as you will if you end up having to modify one of the thousands of legacy applications out there.  Fortran is a fact of life for many graduate students.  It was a fact of life for me too once and I hated it.

I know that it is used a lot at Manchester for research because ,as mentioned earlier, we have network licenses for Intel Fortran and when I am particularly bored I grep the usage logs to see how many unique user names have used it to compile something.  There are many.

I am not for a second suggesting that Fortran is irrelevant because it so obviously isn’t.  It is heavily used in certain, specialist areas.  What I am suggesting is that it is not the ideal language to use as a first exposure to programming.  All of the good reasons for using Fortran seem to come up at a time in your career when you are a reasonably advanced programmer not when you first come across the concepts loops and arrays.

Put bluntly I feel that the correct place for learning Fortran is in grad school and only if it is needed.

### If not Introductory Fortran then Introductory what?

If you have got this far then you can probably guess what I am going to suggest – Python.  Python is infinitely more suitable for beginner programmers in my opinion and with even just a smattering of the language it is possible to achieve a great deal.  Standard python modules such as matplotlib and numpy help take care of plotting and heavy-duty numerics with ease and there are no licensing problems to speak of .  It’s free!

You can use it interactively which helps with learning the basics and even a beginner can produce impressive looking results with relatively little effort.  Not only is it more fun than Fortran but it is probably a lot more useful for an undergraduate because they might actually use it.  Its use in the computational sciences is growing very quickly and it also has a lot of applications in areas such as the web, games and general task automation.

If and when a student is ready to move onto graduate level problems then he/she may well find that Python still fulfills their needs but if they really do need raw speed then they can either hook into existing Fortran libraries using Python modules such as ctypes and f2py or they can roll up their sleeves and learn Fortran syntax.

Using Fortran to teach raw beginners is a bit like teaching complex numbers to kindergarten children before they can count to ten.  Sure, some of them will need complex numbers one day but if it was complex numbers or nothing then a lot of people would really struggle to count.

So, I am (finally) done.  Do I have it all wrong?  Is Fortran so essential that Universities would be short changing students of numerate degrees if they didn’t teach it or are people just teaching what they were taught themselves because that’s the way it has always been done?  I am particularly interested in hearing from either students or teachers of Fortran but, as always, comments from everyone are welcome – even if you disagree with me.

If you are happy to talk in public then please use the comments section but if you would prefer a private discussion then feel free to email me.

(I am skating closer to my job than I usually do on this blog so here is the hopefully unnecessary disclaimer.  These are my opinions alone and do not necessarily reflect the policy of my employer, The Univeristy of Manchester.  If you find yourself coming to me for Fortran compiler support there then I will do the very best I can for you – no matter what my personal programming prejudices may be.  However, if you are up for a friendly chat over coffee concerning this then feel free to drop me a line. )

## Sage Math 4.0 released

June 9th, 2009

While catching up on all of the news I missed while on Vacation, I discovered that the world’s best open-source Maths package saw a new release at the end of May.  I haven’t had chance to download it yet but the breadth and depth of new features is astonishing.  Check out the comprehensive feature-tour here (note to commercial software developers I have bugged in the past – when I say that I’d like a detailed change-log for your latest release, this is the kind if thing I mean).

Scanning through the changes I note that they have dropped Maxima for core symbolic manipulation in favour of a package called Pynac.   Apparently, one practical upshot of this is that finding the determinant of symbolic matrices can be up to 2500 times faster!  very impressive.  I wonder how this change will affect symbolic calculus (if at all).

Unfortunately,  it looks like a full, native Windows version is still some way off which is a shame because one of my jobs this month is to prepare a load of maths applications for deployment on my University’s teaching clusters which run Windows.  Putting SAGE next to Mathematica, MATLAB and Mathcad would have been cool and a great way to encourage undergraduate users at Manchester University to try out the software.

Finally, some eye candy shamelessly stolen from the SAGE website.  The plot below demonstrates the new implicit_plot3d function.