Stream and Complex Event Processing in the Big Data Era

PhD Course 2019

This is the official page of the PhD course on Stream and Complex Event Processing in the Big Data Era offered by Politecnico di Milano in 2019.


Alessandro Margara, Emanuele Della Valle, and Gianpaolo Cugola

Short Description

An increasing number of distributed applications requires processing continuously flowing data from geographically distributed sources at unpredictable rate to obtain timely responses to complex queries. Examples of such applications come from the most disparate fields: Smart Cities, Social Media Analytics, Sensor Networks, Security, Intelligent Business management, Agile Enterprises, Robotics.

These requirements led to the development of a number of systems specifically designed to process information as a flow. In particular, two models emerged and are today competing: the data stream processing model [1] and the complex event processing model [2]. More recently, the community working on Semantic Web Technologies has proposed his own contribution to the area in the form of stream reasoning systems [3,5].

The course aims at presenting the state of the art of the field and the most recent research results in stream and complex event processing. A particular emphasis will be given to the research conducted at DEIB. The students will gain enough background on the topics to be able to use the tools made available by the academic and industrial community to solve prototypical problems. The exams will consist in reporting the experience in using the tools and in discussing the different trade-offs offered by them.


A – Background

  1. Reactive decisions matters!!!
  2. Taming velocity in Big Data
  3. Big Data processing engines
  4. Complex event processing abstractions

B – Discovering systems

  1. Apache Flink [Tutorial code]
  2. Esper EPL [Tutorial code]
  3. Kafka Streams / KSQL, Demo [Tutorial code]

C- Putting it all together for the final evaluation

  1. A practical scenario [Sensor data] [Metadata] [Oracle] [Match First-Half Second-Half]
  2. Experience report (students)


  • Mon 04/03 10:00–13:00, room 3B: Topic A.1 + A.2
  • Thu 07/03 10:00–13:00, room 2A: Topic A.3 + A.4
  • Mon 11/03 10:00–13:00, room 2A: Topic B.1
  • Wed 13/03 10:00–13:00, room 2A: Topic B.2
  • Mon 18/03 10:00–13:00, room 3B: Topic B.3
  • Thu 21/03 10:00–13:00, room 3B: Topic B.4
  • Mon 25/03 10:00–13:00, room 3B: Topic C.1
  • June/July (2):Topic C.2


[1] Brian Babcock, Shivnath Babu, Mayur Datar, Rajeev Motwani, and Jennifer Widom. Models and issues in data stream systems. In PODS 2002: Proceedings of
the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 1-16, New York, NY, USA, 2002. ACM.

[2] David C. Luckham. The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley Longman
Publishing Co., Inc., Boston, MA, USA, 2001.

[3] Emanuele Della Valle, Stefano Ceri, Frank van Harmelen, Dieter Fensel: It’s a Streaming World! Reasoning upon Rapidly Changing Information. IEEE Intelligent Systems 24(6): 83-89 (2009)

[4] Gianpaolo Cugola and Alessandro Margara. Processing flows of information: From data stream to complex event processing. ACM Computing Surveys,
44(3):15, June 2012.

[5] Alessandro Margara, Jacopo Urbani, Frank van Harmelen, Henri E. Bal: Streaming the Web: Reasoning over dynamic data. J. Web Sem. 25: 24-44 (2014)


[7] Ben Stepford: Designing Event Driven Systems. Concepts and Patterns for Streaming Services with Apache Kafka. O’Reilly, 2018