Best Practices in Software Benchmarking 2016 (#bench16)

Wednesday April 20th 10:00 - 18:00

King's College London, Strand Campus (room K2.31)


For computer scientists and software engineers, benchmarking (evaluating the running time of a piece of software, or the performance of a piece of hardware) is a common method for evaluating new techniques. However, there is little agreement on how benchmarking should be carried out, how to control for confounding variables, how to analyse latency data, or how to aid the repeatability of experiments. This workshop will be a venue for computer scientists and research software engineers to discuss their current best practices and future directions.

On this page you can find an agenda, directions for travel, availability of wifi access for attendees, and information about our sponsors and organisers.

Registration is now closed.


Time Session Speaker
09:30 -- 10:00 Arrival and coffee
10:00 -- 10:05 Welcome Sarah Mount
10:05 -- 11:00 Keynote talk: Repeatability, Reproducibility and Rigor [slides] Jan Vitek
11:00 -- 11:30 Morning break with refreshments
11:30 -- 12:00 Why Aren't We Benchmarking Bioinformatics? [slides] Joe Parker
12:00 -- 12:30 Introduction to Time Series Data and Analysis [slides] Simon Taylor
12:30 -- 14:00 Lunch
14:00 -- 14:30 Evaluating Performance using Ratio of Execution Times [slides] Tomáš Kalibera
14:30 -- 15:00 The Role of Benchmarking in Buying a High-Performance Computer [slides] James Davenport
15:00 -- 15:45 Afternoon break with refreshments
15:45 -- 16:15 What Exactly do we Mean by VM Warmup? [slides] Edd Barrett
16:15 -- 16:45 Scalability in compiler development: how to get testing and optimization done in a reasonable time [slides] Jeremy Bennett
16:45 -- 17:00 Concluding discussion
18:00 -- late Dinner at Fire and Stone, 31/32 Maiden Lane, WC2E 7JS


King's College London is easily accessible from Central London. Full directions to Strand Campus can be found on the College website.

#bench16 will be held in room K2.31 (King's Building, 2nd floor, room 31) which is also called the Nash Lecture Theatre -- both names are printed on the door itself.

K2.31 will be clearly signposted from the main reception. If you have any problems finding the room, please asking the receptionists, who are friendly and knowledgeable about how to get around the campus.

WiFi access at King's

#bench16 attendees from UK academic institutions can use their Eduroam account during the event (SSID: eduroam). Visitors without an Eduroam account may use the free Cloud service (SSID: The Cloud).


We have been able to keep #bench16 free to attend, thanks to the generous sponsorship of the Software Sustainability Institute. The Software Sustainability Institute cultivates world-class research with software. The Institute is based at the universities of Edinburgh, Manchester, Southampton and Oxford.


If you have any questions at all about #bench16, please don't hesitate to contact us.

Organisers: Sarah MountLaurence Tratt
Twitter: @benchmarking16