DualTable: A Hybrid Storage Model for Update Optimization in Hive

Songlin Hu, Wantao Liu, Tilmann Rabl, Shuo Huang, Ying Liang, Zhang Xiao, Hans-Arno Jacobsen, Xubin Pei, and Jiye Wang.

In Proceedings of the 31st International Conference on Data Engineering, 2015.

Share

Abstract

Hive is the most mature and prevalent data warehouse tool providing SQL-like interface in the Hadoop ecosystem. It is successfully used in many Internet companies and shows its value for big data processing in traditional industries. However, enterprise big data processing systems as in Smart Grid applications usually come with complicated business logics and involve many data manipulation operations like updates and deletes. Hive cannot offer sufficient support for these while preserving high query performance. Hive using the Hadoop Distributed File System (HDFS) for storage cannot implement data manipulation efficiently and Hive on HBase suffers from poor query performance even though it can support faster data manipulation. There is a project based on Hive issue Hive-5317 to support update operations, but it has not been finished in Hive's latest version. Since this ACID compliant extension adopts same data storage format on HDFS, the update performance problem is not solved.

In this paper, we propose a hybrid storage model called DualTable, which combines the efficient streaming reads of HDFS and the random write capability of HBase. Hive on DualTable provides better data manipulation support and preserves query performance at the same time. Experiments on a TPC-H data set and on a real smart grid data set show that Hive on DualTable is up to 10 times faster than Hive when executing update and delete operations.

Download



Tags: big data, hadoop, dualtable


Readers who enjoyed the above work, may also like the following:
  • DGFIndex for Smart Grid: Enhancing Hive with a Cost-Effective Multidimensional Range Index.
    Yue Liu, Songlin Hu, Tilmann Rabl, Wantao Liu, Hans-Arno Jacobsen, Kaifeng Wu, and Jian Chen.
    Proceedings of the VLDB Endowment, 13(7)1496-1507, 2014.
    Tags: hadoop, hive, smart grid, nosql, dgfindex
  • Discussion of BigBench: A Proposed Industry Standard Performance Benchmark for Big Data.
    Chaitanya Baru, Milind Bhandarkar, Carlo Curino, Manuel Danisch, Michael Frank, Bhaskar Gowda, Hans-Arno Jacobsen, Huang Jie, Dileep Kumar, Raghunath Nambiar, Meikel Poess, Francois Raab, Tilmann Rabl, Nishkam Ravi, Kai Sachs, Saptak Sen, Lan Yi, and Choonhan Youn.
    In Sixth TPC Technology Conference on Performance Evaluation & Benchmarking, pages 44-63, 2014. Springer Berlin Heidelberg.
    Tags: bigbench, big data, benchmarking
  • CaSSanDra: An SSD Boosted Key-Value Store.
    Prashanth Menon, Tilmann Rabl, Mohammad Sadoghi, and Hans-Arno Jacobsen.
    In 30th IEEE International Conference on Data Engineering, pages 1162-1167, 2014.
    Tags: cassandra, big data, key-value store, nosql