MSRG Publication:: Introducing the Big Data Benchmarking Community

Introducing the Big Data Benchmarking Community

Chaitan Baru, Milind Bhandarkar, Raghunath Nambiar, Meikel Poess, and Tilmann Rabl.

6th Extremely Large Databases Conference.


The Workshop on Big Data Benchmarking (WBDB2012), held on May 8-9, 2012 in San Jose, CA, marked the first of a series of workshops aimed at developing industry-standard Big Data benchmarks. The workshop was attended by 60 invitees representing 45 different organizations both from industry and academia. Attendees were chosen based on their experience and expertise in one or more areas of Big Data, database systems, performance benchmarking, and big data applications. They agreed that there was both a pressing need and an opportunity for defining benchmarks to capture the end-to-end aspects of big data applications. In presentations and working sessions the workshop participants laid the foundation for future workshops by agreeing on key aspects of future Big Data benchmarks, e.g. the need to include metrics for performance as well as price/performance; the need to consider several costs, including total system cost, setup cost, and energy costs; and the need for an end-to-end benchmark serving the purposes of competitive marketing as well as product improvement. As a result of this meeting an informal “Big Data benchmarking community” has been formed, hosted by the San Diego Supercomputer Center, UC San Diego. Biweekly phone conferences are being used to keep this group engaged and to share information among members. Several benchmarking activities were started by members of this community. These range from simple examples of end-to-end benchmarks to complex components setups. Within our lightning talk and poster, we will introduce the Big Data Benchmarking Community and give an overview of current developments in Big Data benchmarking. The next Big Data Benchmarking workshops are currently planned for December 17-18, 2012 in Pune, India and June 2013 in Xian, China.


Tags: big data, benchmarking

Readers who enjoyed the above work, may also like the following:

  • Discussion of BigBench: A Proposed Industry Standard Performance Benchmark for Big Data.
    Chaitanya Baru, Milind Bhandarkar, Carlo Curino, Manuel Danisch, Michael Frank, Bhaskar Gowda, Hans-Arno Jacobsen, Huang Jie, Dileep Kumar, Raghunath Nambiar, Meikel Poess, Francois Raab, Tilmann Rabl, Nishkam Ravi, Kai Sachs, Saptak Sen, Lan Yi, and Choonhan Youn.
    In Sixth TPC Technology Conference on Performance Evaluation & Benchmarking, pages 44-63, 2014. Springer Berlin Heidelberg.
    Tags: bigbench, big data, benchmarking
  • BigBench Specification V0.1.
    Tilmann Rabl, Ahmad Ghazal, Minqing Hu, Alain Crolotte, Francois Raab, Meikel Poess, and Hans-Arno Jacobsen.
    In Proceedings of the 2012 Workshop on Big Data Benchmarking, pages 164-202, 2013.
    Tags: bigbench, big data, benchmarking
  • Big Data Generation.
    Tilmann Rabl and Hans-Arno Jacobsen.
    In Proceedings of the Workshop on Big Data Benchmarking, pages 20-27, 2013.
    Tags: pdgf, big data, benchmarking