Deadline-based Workload Management for MapReduce Environments: Pieces of the Perfromance Puzzle.
Verma, Abhishek; Cherkasova, Ludmila; Kumar, Vijay S.; Campbell, Roy H.
Keyword(s): MapReduce; Hadoop; performance; resource allocation; job scheduling;
Abstract: Hadoop and the associated MapReduce paradigm have become the de facto platform for cost-effective analytics over "Big Data". There is an increasing number of MapReduce applications associated with live business intelligence that require completion time guarantees. In this work, we introduce and analyze a set of complementary mechanisms that enhance workload management decisions for processing MapReduce jobs with deadlines. The three mechanisms we consider are the following: 1) a policy for job ordering in the processing queue; 2) a mechanism for allocating a tailored number of map and reduce slots to each job with a completion time requirement; 3) a mechanism for allocating and deallocating (if necessary) spare resources in the system among the active jobs. We analyze the functionality and performance benefits of each mechanism via an extensive set of simulations over diverse workload sets. The proposed mechanisms form the integral pieces in the performance puzzle of automated workload management in MapReduce environments.
Additional Publication Information: Will appear in NOMS 2012: 13th IEEE/IFIP Network Operations and Management Symposium, Maui, Hawaii, USA, April, 16-20, 2012
External Posting Date: April 23, 2012 [Fulltext]. Approved for External Publication
Internal Posting Date: April 23, 2012 [Fulltext]