I occasionally get emails from researchers saying something like this<\/p>\n
‘My MATLAB code takes a week to run and the cleaner\/cat\/my husband keeps switching off my machine \u00a0before it’s completed — could you help me make the code go faster please so that I can get my results in between these events’<\/em><\/p>\n While I am more than happy to try to optimise the code in question, what these users really need is some sort of checkpointing scheme. Checkpointing is also important for users of high performance computing systems that limit the length of each individual job.<\/p>\n The solution – Checkpointing (or ‘Assume that your job will frequently be killed’)<\/strong><\/p>\n The basic idea behind checkpointing is to periodically save your program’s state so that, if it is interrupted, it can start again where it left off rather than from the beginning. In order to demonstrate some of the principals involved, I’m going to need some code that’s sufficiently simple that it doesn’t cloud what I want to discuss. Let’s add up some numbers using a for-loop.<\/p>\n Using a for-loop to perform an addition like this is\u00a0something that I’d never usually suggest in MATLAB<\/em> but I’m using it here because it is so simple that it won’t get in the way of understanding the checkpointing code.<\/p>\n If you run this program in MATLAB, it will take about 100 seconds thanks to that pause statement which is acting as a proxy for some real work. Try interrupting it by pressing CTRL-C and then restart it. As you might expect, it will always start from the beginning:<\/p>\n This is no big deal when your calculation only takes 100 seconds but is going to be a major problem when the calculation represented by that pause statement becomes something like an hour rather than a second.<\/p>\n Let’s now look at a version of the above that makes use of checkpointing.<\/p>\n Before you run the above code, the checkpoint file checkpoint.mat<\/strong> does not exist and so the calculation starts from the beginning. After every iteration, a checkpoint file is created which contains every variable in the MATLAB workspace. If the program is restarted, it will find the checkpoint file and continue where it left off. Our code now deals with interruptions a lot more gracefully.<\/p>\n Note that we’ve had to change the program logic slightly. Our original loop counter was<\/p>\n In the check-pointed example, however, we’ve had to introduce the variable countmin<\/strong><\/p>\n This allows us to start the loop from whatever value of countmin was in our last checkpoint file. Such minor modifications are often necessary when converting code to use checkpointing and you should carefully check that the introduction of checkpointing does not introduce bugs in your code.<\/p>\n Don’t checkpoint too often<\/strong><\/p>\n The creation of even a small checkpoint file is a time consuming process. Consider our original addup code but without the pause command.<\/p>\n On my machine, this code takes 0.0046 seconds to execute. Compare this to the checkpointed version, again with the pause statement removed.<\/p>\n This checkpointed version takes 0.85 seconds to execute on the same machine — Over 180 times slower than the original! The problem is that the time it takes to checkpoint is long compared to the calculation time.<\/p>\n If we make a modification so that we only checkpoint every 25 iterations, code execution time comes down to 0.05 seconds:<\/p>\n Of course, the issue now is that we might lose more work if our program is interrupted between checkpoints. Additionally, in this particular case, the mod command used to decide whether or not to checkpoint is more expensive than simply performing the calculation but hopefully that isn’t going to be the case when working with real world calculations.<\/p>\n In practice, we have to work out a balance such that we checkpoint often enough so that we don’t stand to lose too much work but not so often that our program runs too slowly.<\/p>\n Checkpointing code that involves random numbers<\/strong><\/p>\n Extra care needs to be taken when running code that involves random numbers. Consider a modification of our checkpointed adding program that creates a sum of random numbers.<\/p>\n In the above, we set the seed of the random number generator to 0 at the beginning of the calculation. This ensures that we always get the same set of random numbers and allows us to get reproducible results. As such, the sum should always come out to be 52.799447 to the number of decimal places used in the program.<\/p>\n The above code has a subtle bug that you won’t find if your testing is confined to interrupting using CTRL-C and then restarting in an interactive session of MATLAB. Proceed that way, and you’ll get exactly the sum you’ll expect : 52.799447. \u00a0If, on the other hand, you test your code by doing the following<\/p>\n You’ll get a different result. This is not what we want!<\/p>\n The root cause of this problem is that we are not saving the state of the random number generator in our checkpoint file. Thus, when we restart MATLAB, all information concerning this state is lost. If we don’t restart MATLAB between interruptions, the state of the random number generator is safely tucked away behind the scenes.<\/p>\n Assume, for example, that you stop the calculation running after the third iteration. The random numbers you’d have consumed would be (to 4 d.p.)<\/p>\n 0.8147 Your checkpoint file will contain the variables mysum<\/strong>, count<\/strong> and countmin<\/strong> but will contain nothing about the state of the random number generator. In English, this state is something like ‘The next random number will be the 4th one in the sequence defined by a starting seed of 0.’<\/em><\/p>\n When we restart MATLAB, the default seed is 0 so we’ll be using the right sequence (since we explicitly set it to be 0 in our code) but we’ll be starting right from the beginning again. That is, the 4th,5th and 6th iterations of the summation will contain the first 3 numbers in the stream, thus double counting them, and so our checkpointing procedure will alter the results of the calculation.<\/p>\n In order to fix this, we need to additionally save the state of the random number generator when we save a checkpoint and also make correct use of this on restarting. Here’s the code<\/p>\n%addup.m\r\n%This is not the recommended way to sum integers in MATLAB -- we only use it here to keep things simple\r\n%This version does NOT use checkpointing\r\n\r\nmysum=0;\r\nfor count=1:100\r\n mysum = mysum + count;\r\n pause(1); %Let's pretend that this is a complicated calculation\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\n\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n>> addup\r\nCompleted iteration 1\r\nCompleted iteration 2\r\nCompleted iteration 3\r\nOperation terminated by user during addup (line 6)\r\n\r\n>> addup\r\nCompleted iteration 1\r\nCompleted iteration 2\r\nCompleted iteration 3\r\nOperation terminated by user during addup (line 6)<\/pre>\n
%addup_checkpoint.m\r\nif exist( 'checkpoint.mat','file' ) % If a checkpoint file exists, load it\r\n fprintf('Checkpoint file found - Loading\\n');\r\n load('checkpoint.mat')\r\n\r\nelse %otherwise, start from the beginning\r\n fprintf('No checkpoint file found - starting from beginning\\n');\r\n mysum=0;\r\n countmin=1;\r\nend\r\n\r\nfor count = countmin:100\r\n mysum = mysum + count;\r\n pause(1); %Let's pretend that this is a complicated calculation\r\n\r\n %save checkpoint\r\n countmin = count+1; %If we load this checkpoint, we want to start on the next iteration\r\n fprintf('Saving checkpoint\\n');\r\n save('checkpoint.mat');\r\n\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n>> addup_checkpoint\r\nNo checkpoint file found - starting from beginning\r\nSaving checkpoint\r\nCompleted iteration 1 \r\nSaving checkpoint\r\nCompleted iteration 2 \r\nSaving checkpoint\r\nCompleted iteration 3 \r\nOperation terminated by user during addup_checkpoint (line 16)\r\n\r\n>> addup_checkpoint\r\nCheckpoint file found - Loading\r\nSaving checkpoint\r\nCompleted iteration 4 \r\nSaving checkpoint\r\nCompleted iteration 5 \r\nSaving checkpoint\r\nCompleted iteration 6 \r\nOperation terminated by user during addup_checkpoint (line 16)<\/pre>\n
for count = 1:100<\/pre>\n
for count = countmin:100<\/pre>\n
%addup_nopause.m\r\n%This version does NOT use checkpointing\r\nmysum=0;\r\nfor count=1:100\r\n mysum = mysum + count;\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n%addup_checkpoint_nopause.m\r\n\r\nif exist( 'checkpoint.mat','file' ) % If a checkpoint file exists, load it\r\n fprintf('Checkpoint file found - Loading\\n');\r\n load('checkpoint.mat')\r\n\r\nelse %otherwise, start from the beginning\r\n fprintf('No checkpoint file found - starting from beginning\\n');\r\n mysum=0;\r\n countmin=1;\r\nend\r\n\r\nfor count = countmin:100\r\n mysum = mysum + count;\r\n\r\n %save checkpoint\r\n countmin = count+1; %If we load this checkpoint, we want to start on the next iteration\r\n fprintf('Saving checkpoint\\n');\r\n save('checkpoint.mat');\r\n\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n%Checkpoint every 25 iterations\r\n\r\nif exist( 'checkpoint.mat','file' ) % If a checkpoint file exists, load it\r\n fprintf('Checkpoint file found - Loading\\n');\r\n load('checkpoint.mat')\r\n\r\nelse %otherwise, start from the beginning\r\n fprintf('No checkpoint file found - starting from beginning\\n');\r\n mysum=0;\r\n countmin=1;\r\nend\r\n\r\nfor count = countmin:100\r\n mysum = mysum + count;\r\n countmin = count+1; %If we load this checkpoint, we want to start on the next iteration\r\n\r\n if mod(count,25)==0\r\n %save checkpoint \r\n fprintf('Saving checkpoint\\n');\r\n save('checkpoint.mat');\r\n end\r\n\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n%addup_checkpoint_rand.m\r\n%Adding random numbers the slow way, in order to demo checkpointing\r\n%This version has a bug\r\n\r\nif exist( 'checkpoint.mat','file' ) % If a checkpoint file exists, load it\r\n fprintf('Checkpoint file found - Loading\\n');\r\n load('checkpoint.mat')\r\n\r\nelse %otherwise, start from the beginning\r\n fprintf('No checkpoint file found - starting from beginning\\n');\r\n mysum=0;\r\n countmin=1;\r\n rng(0); %Seed the random number generator for reproducible results\r\nend\r\n\r\nfor count = countmin:100\r\n mysum = mysum + rand();\r\n countmin = count+1; %If we load this checkpoint, we want to start on the next iteration\r\n pause(1); %pretend this is a complicated calculation\r\n\r\n %save checkpoint\r\n fprintf('Saving checkpoint\\n');\r\n save('checkpoint.mat');\r\n\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n\n
\n0.9058
\n0.1270<\/p>\n%addup_checkpoint_rand_correct.m\r\n%Adding random numbers the slow way, in order to demo checkpointing\r\n\r\nif exist( 'checkpoint.mat','file' ) % If a checkpoint file exists, load it\r\n fprintf('Checkpoint file found - Loading\\n');\r\n load('checkpoint.mat')\r\n\r\n %use the saved RNG state\r\n stream = RandStream.getGlobalStream;\r\n stream.State = savedState;\r\n\r\nelse % otherwise, start from the beginning\r\n fprintf('No checkpoint file found - starting from beginning\\n');\r\n mysum=0;\r\n countmin=1;\r\n rng(0); %Seed the random number generator for reproducible results\r\nend\r\n\r\nfor count = countmin:100\r\n mysum = mysum + rand();\r\n countmin = count+1; %If we load this checkpoint, we want to start on the next iteration\r\n pause(1); %pretend this is a complicated calculation\r\n\r\n %save the state of the random number genertor\r\n stream = RandStream.getGlobalStream;\r\n savedState = stream.State;\r\n %save checkpoint\r\n fprintf('Saving checkpoint\\n');\r\n save('checkpoint.mat');\r\n\r\n fprintf('Completed iteration %d \\n',count);\r\nend\r\nfprintf('The sum is %f \\n',mysum);<\/pre>\n