warmup_stats

This system was originally developed as part of the Virtual Machine Warmup Blows Hot and Cold paper. It analyses VM performance data and outputs various statistics (e.g. if the VM warmed up or slowed down; how long it took to warmup; what (if any) the steady state performance is).

The system can be found at the warmup_stats repository.

Running benchmarks

High quality VM benchmarking is hard to achieve: it is easy to accidentally end up measuring something other than was intended. We wrote Krun to help make reliable VM measurement easier. Krun is not suitable for all purposes, but is probably the easiest way to obtain high quality benchmarking data.

A large-scale example

The Virtual Machine Warmup Blows Hot and Cold paper also serves as an example of a large-scale VM experiment. You can find that experiment at the warmup_experiment repository.

Generating output

The warmup_stats repository provides scripts for generating tables of benchmarking results (similar to those in the paper), plots, and 'diffs' between different versions of a VM. Results and diff tables can be rendered as either HTML (to integrate with a continuous integration system), or LaTeX / PDF. Plots can be produced in PDF format only.

Example results from warmup_stats

Hover over the table below to see a diff between these results and the same benchmarks run on a variant of the same VM.