Talk by Rebecca Isaacs, MSR - Naiad: Iterative and incremental data-parallel computation

Speaker: Rebecca Isaacs, MSR

Title: Naiad: Iterative and incremental data-parallel computation

Abstract:

In this talk I will describe Naiad, an ongoing project at Microsoft Research Silicon Valley.  Naiad is a system for data-parallel computation that is intended to bridge the gap between high throughput batch processing systems (e.g. MapReduc e, Dryad/DryadLINQ, Hadoop/Hive) and low latency stream processors (e.g. System S, StreamIt), by supporting loops and incremental computation.  Our approach is to implement the underlying data-parallel operators not as computations on collections of records, but computations on differences between collections.  This "differential dataflow" model allows us to efficiently implement incremental input updates, as well as arbitrarily nested fixed-point iteration, prioritized execution, and compositions thereof.  Naiad is particularly well suited for queries over large-scale graphs such as social networks or road maps, and many problems with a natural dynamic programming solution.