Dataflow Processing and Optimization on Grid and Cloud Infrastructures
Authors: 
Manolis Tsangaris
Authors: 
George Kakaletris
Authors: 
Herald Kllapi
Authors: 
George Papanikos
Authors: 
Fragiskos Pentaris
Authors: 
Paul Polydoras
Authors: 
Eva Sitaridi
Authors: 
Vasilis Stoumpos
Authors: 
Yannis Ioannidis
Date published: 
2009
Published In: 
IEEE Data Eng. Bull. 32(1), Nov 2009: 67-74
Type: 
Journal Article
Abstract: 

Complex on-demand data retrieval and processing is a characteristic of several applications and combines the notions of querying & search, information filtering & retrieval, data transformation & analysis, and other data manipulations. Such rich tasks are typically represented by data processing graphs, having arbitrary data operators as nodes and their producer-consumer interactions as edges. Optimizing and executing such graphs on top of distributed architectures is critical for the success of the corresponding applications and presents several algorithmic and systemic challenges. This paper describes a system under development that offers such functionality on top of Ad-hoc Clusters, Grids, or Clouds. Operators may be user defined, so their algebraic and other properties as well as those of the data they produce are specified in associated profiles. Optimization is based on these profiles, must satisfy a variety of objectives and constraints, and takes into account the particular characteristics of the underlying architecture, mapping high-level dataflow semantics to flexible runtime structures. The paper highlights the key components of the system and outlines the major directions of its development.

Related files: 

MaDgIK 2009-2016