MADES - A Multi-Layered, Adaptive, Distributed Event Store

Tilmann Rabl, Mohammad Sadoghi, Kaiwen Zhang, and Hans-Arno Jacobsen.

In Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems, 2013.


Application performance monitoring (APM) is shifting towards capturing and analyzing every event that arises in an enterprise infrastructure. Current APM systems, for example, make it possible to monitor enterprise applications at the granularity of tracing each method invocation (i.e., an event). Naturally, there is great interest in monitoring these events in real-time to react to system and application failures and in storing the captured information for an extended period of time to enable detailed system analysis, data analytics, and future auditing of trends in the historic data. However, the high insertion-rates (up to millions of events per second) and the purposely limited resource, a small fraction of all enterprise resources (i.e., 1-2% of the overall system resources), dedicated to APM are the key challenges for applying current data management solutions in this context. Emerging distributed key-value stores, often positioned to operate at this scale, induce additional storage overhead when dealing with relatively small data points (e.g., method invocation events) inserted at a rate of millions per second. Thus, they are not a promising solution for such an important class of workloads given APM’s highly constrained resource budget. In this paper, to address these shortcomings, we present Multilayered, Adaptive, Distributed Event Store (MADES): a massively distributed store for collecting, querying, and storing event data at a rate of millions of events per second.


Readers who enjoyed the above work, may also like the following:

  • Grand Challenge: The BlueBay Soccer Monitoring Engine.
    Hans-Arno Jacobsen, Kianoosh Mokhtarian, Tilmann Rabl, Mohammad Sadoghi, Reza Sherafat Kazemzadeh, Young Yoon, and Kaiwen Zhang.
    In The 7th ACM International Conference on Distributed Event-Based Systems, 2013.
    Acceptance rate 40%.
    Tags: event processing systems, live monitoring, soccer, content-based matching
  • Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing.
    Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, and Hans-Arno Jacobsen.
    In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, 2012.
    Best DEBS Challenge Award - Public Voting.
    Tags: event processing, grand challenge