Dataflow Dynamism via Programmable Deployment

The dataflow model is a powerful abstraction commonly used to build data management systems. While dataflow graphs are often assumed to be static, a growing number of dataflow-based applications -especially long-running streaming applications- require some form of runtime dynamism. In this paper we focus on an important class of dataflow dynamism which we refer to as conditional subflows. Within a dataflow graph, a conditional subflow is a subgraph whose presence and relevance is driven by events on the input stream(s). Example use cases include surveillance systems where e.g., a face detection event can trigger face recognition pipelines. Other examples of conditional subflows are elastic parallelization and load shedding. We propose Programmable Deployment Framework, a novel and flexible solution for conditional subflows. We have implemented this framework, integrated into a stream processing platform, and validated it using various use cases.