Systematic CPU time comparisons

The matlab tic and toc command can be used to measure CPU-time of some computation. The tic-toc commands can unfortunately often give you misleading results.  This is due to the fact that your operating system is multitasking (several processes can run in the background),  variations in performance of your processor, and memory allocation issues. This problem occurs often for short CPU times.

Fortunately, the effect of this problem can be lessened by running your code several times in a systematic way and take the average.

Suppose we want to compare our own matrix multiply with the Matlab built-in version.

Put this in the file naivemult.m:

function z=naivemult(A,b)
z=zeros(size(A,1));
for i=1:size(A,1)
    for j=1:size(A,2);
        z(i)=z(i)+A(i,j)*b(j);
    end
end

Now you can run the code for several times for several sizes:

% Create vector samples of 10^k0 to 10^k1 with less points for
% higher numbers
%
k0=1.5; k1=2.5;
nv=round(logspace(k0,k1,50));

P=50; % number of samples

% Note: It is important to preallocate timing matrices, and not dynamically expand them
T1=zeros(length(nv),P);  
T2=zeros(length(nv),P);

for p=1:P
    for k=1:length(nv)
        n=nv(k);
        A=randn(n,n);
        b=randn(n,1);

        % Method one.
        tic
        z=A*b;
        T1(k,p)=toc;

        % Method two
        tic
        z=naivemult(A,b);
        T2(k,p)=toc;
    end
end

t1=mean(T1,2); # Plot the mean of the runs
t2=mean(T2,2);
clf;
plot(nv,t1,nv,t2);
legend('t1','t2');
xlabel('n');
ylabel('CPU time');

This will give you a plot like this.