Often I need to compare algorithms or software. This comparison often involves solution time and/or number of nodes in the branch and bound tree. In their 2002 article Elizabeth D., and Jorge J. Moré proposed a performance profile to be used to compare optimization software. This is a particularly nice way to compare systems, that tells more than other similar methods.

Generating the profile is not so difficult. Starting with a set of experiments and set of algorithms/software to compare, all you need to do is drawing a stairs (step) graph where the x-axis ($\tau$) represents the time divided into best solution time for each experiment (thus fastest algorithm always get a 1-unit value at 1) and y-axis represents percentage of experiments that are solved at the given $\tau$.

This was rather a short introduction. If you are new, check their article for more information.

For those who already know what they’re doing, I wrote a MATLAB script to make this process easy. I call the script oppa! It basically takes a matrix (say $X$) where rows represent experiments, columns represent algorithms and cells represents solution time (or any other performance measure). All you need to do is copying oppa.m file to your project folder and then call

and the results is more or less like this;

oppa! output

Github


You can find the project on Github. I’m planning to extend this script to other languages (Python and maybe C++) but you’re more than welcomed if you want to contribute.

Bugs, suggestions?


It would be great if you can post any bugs you noticed or suggest new features to it. I try to keep it simple so end-user can customize many things.

Next Steps


Well, first of all I should note that the script is not 100% complete. It takes algorithm names as an input, but I haven’t used them for plots. Colors, line styles could be user inputs (as I intended). As I said earlier, feel free to contribute.