Grand Challenge: High Performance Stream Queries in Scala

Dantong Song, Kaiwen Zhang, Tilmann Rabl, Prashanth Menon, and Hans-Arno Jacobsen.

In DEBS, 2015.

Share

Abstract

Traffic monitoring is an important stream processing application which is highly dynamic and requires aggregation of spatially colocated data. Inspired by this, the DEBS 2015 Grand Challenge uses publicly available taxi transportation information to compute online the most frequent routes and most profitable areas. In this paper, we describe our solution to the DEBS 2015 Grand Challenge written in Scala. Our large-scale solution employs Apache Spark, while our challenge implementation is highly specialized and can process events at a 10 ms latency and at a throughput of 114,000 events per second.

Download



Tags: grand challenge, spark, scala, taxi monitoring


Readers who enjoyed the above work, may also like the following:
  • Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing.
    Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, and Hans-Arno Jacobsen.
    In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, 2012.
    Best DEBS Challenge Award - Public Voting.
    Tags: event processing, grand challenge
  • DL-Store: A Distributed Hybrid OLTP and OLAP Data Processing Engine.
    Kaiwen Zhang, Mohammad Sadoghi, and Hans-Arno Jacobsen.
    In ICDCS Demos, 2016.
    Tags: olap/oltp, data store, distributed transactions, spark