Play It Again, SimMR!
Verma, Abhishek; Cherkasova, Ludmila; Campbell, Roy H.
Keyword(s): Mapreduce; Hadoop; simulation; performance modeling; traces
Abstract: A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment SimMR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.
Additional Publication Information: Will appear in Proc. of the IEEE Cluster 2011 (Cluster'2011), Austin, Texas, USA, September 26-30, 2011.
External Posting Date: August 21, 2011 [Fulltext]. Approved for External Publication
Internal Posting Date: August 21, 2011 [Fulltext]