Skip to main content

Dataflow Processing and Optimization on Grid and Cloud Infrastructures

Complex on-demand data retrieval and processing is a characteristic of several applications and combines the notions of querying & search, information filtering & retrieval, data transformation & analysis, and other data manipulations. Such rich tasks are typically represented by data processing graphs, having arbitrary data operators as nodes and their producer-consumer interactions as edges. Optimizing and executing such graphs on top of distributed architectures is critical for the success of the corresponding applications and presents several algorithmic and systemic challenges. This paper describes a system under development that offers such functionality on top of Ad-hoc Clusters, Grids, or Clouds. Operators may be user defined, so their algebraic and other properties as well as those of the data they produce are specified in associated profiles. Optimization is based on these profiles, must satisfy a variety of objectives and constraints, and takes into account the particular characteristics of the underlying architecture, mapping high-level dataflow semantics to flexible runtime structures. The paper highlights the key components of the system and outlines the major directions of its development.

Manolis Tsangaris, George Kakaletris, Herald Kllapi, George Papanikos, Fragiskos Pentaris, Paul Polydoras, Eva Sitaridi, Vasilis Stoumpos, Yannis Ioannidis, "Dataflow Processing and Optimization on Grid and Cloud Infrastructures ", IEEE Data Eng. Bull. 32(1), Nov 2009: 67-74, 2009
Published at
IEEE Data Eng. Bull. 32(1), Nov 2009: 67-74
Related research area
No related research area
Related Organizations
No related organizations