Tech Tuesday with NIRA: Stateful Stream Processing with Apache Flink

Stateful Stream Processing with Apache Flink at NIRA

At NIRA we use Apache Flink to implement event-driven, distributed, fault tolerant, stateful real time stream processing. Stateful stream processing can be seen as an evolvement of classic stateful applications where compute and storage tiers are separated. An example would be an event driven application with its state located in a RDBMS/NoSQL database. With Apache Flink state is colocated and deeply integrated with the application itself. This opens up opportunities to drastically reduce the complexity of the application and as a result lowering the cost. Another key concept of stream processing is that, in contrast to batch processing, output/results are naturally propagated in near real time, which can be important for use cases with low latency requirements.

At NIRA the state of a typical Flink application consists of data collected over long periods of time (months). Typically the state consist of road characteristics such as roughness, obstacles, slipperiness etc. As an example, this data can be used to pinpoint problematic road sections with repeated slipperiness during certain weather conditions such as heavy rain.