Istituto di Scienza e Tecnologie dell'Informazione     
Bertolucci M., Carlini E., Dazzi P., Lulli A., Ricci L. Static and dynamic big data partitioning on Apache spark. In: ParCo 2015 - International Conference on Parallel Computing (Edinburgh, Scotland, 1-4 September 2015). Proceedings, pp. 489 - 498. Gerhard R. Joubert, Hugh Leather, Mark Parsons, Frans Peters, Mark Sawyer (eds.). (Advances in Parallel Computing, vol. 27 (e-book: Parallel Computing: on the road to exascale)). IOS Press, 2016.
Many of today's large datasets are organized as a graph. Due to their size it is often infeasible to process these graphs using a single machine. Therefore, many software frameworks and tools have been proposed to process graph on top of distributed infrastructures. This software is often bundled with generic data decomposition strategies that are not optimised for specific algorithms. In this paper we study how a specific data partitioning strategy affects the performances of graph algorithms executing on Apache Spark. To this end, we implemented different graph algorithms and we compared their performances using a naive partitioning solution against more elaborate strategies, both static and dynamic.
URL: http://ebooks.iospress.nl/publication/42687
DOI: 10.3233/978-1-61499-621-7-489
Subject Data partitioning
Distributed computing

Icona documento 1) Download Document PDF

Icona documento Open access Icona documento Restricted Icona documento Private


Per ulteriori informazioni, contattare: Librarian http://puma.isti.cnr.it

Valid HTML 4.0 Transitional