XStream: a Functional Language for Distributed Stream Processing in the Edge Cloud

Internet of Things (IoT) data and video traffic is forecasted to explode within the coming decade. Characteristic of such streams is that they are highly context-dependent and rarely consumed by humans. Traditional "big data" architectures are a bad fit for intelligently processing this large variety of streams as they usually require routing all the data to one or a few data centers for centralized storage and processing. In this paper, we explore an alternative architecture where stream processing pipelines are distributed across both "centralized" and "edge" clouds (that is, compute nodes physically closer to the network edge), with the goal of reducing the cost of data transportation and processing. Our main focus in this paper is on our platform's high-level query language named XStream. XStream programs describe continuous queries over data and media streams. These programs are compiled into distributed query plans and subsequently deployed across a decentralized compute infrastructure. The high-level nature of the language enables developers to efficiently process data and media streams in real-time with very low friction in terms of end-to-end deployment.