Improvement of task retrieval performance using AMGA in a large-scale virtual screening

In this paper, we address performance and scalability issues when AMGA is used as a metadata service for task retrieval in the WISDOM environment, and propose optimization techniques to deal with the issues. First, to deal with the performance problem due to the communication overhead caused by the need for jobs to call a series of AMGA operations in order for them to retrieve a task from the AMGA server in the WISDOM environment, we propose a new AMGA operation which allows jobs deployed on the Grid to retrieve a task in a single operation instead of calling series of existing AMGA operations. According to the performance study that we have done, the throughput of task retrieval using the new AMGA operation can be as much as 70 times higher than the throughput of using the existing AMGA operations. Second, to address the scalability problem when thousands of jobs running have access to the single AMGA server concurrently in an attempt to grab available tasks, we propose the use of multiple AMGA servers for the purpose of task retrieval. Our test results demonstrate that throughput can be improved linearly in proportion to the number of AMGA servers set up for load balancing.