Realistic data concerning the characteristics of workflow-based Grid workloads is key to the adoption and the evolution of Grids, but is not readily available to scientists. To address this issue, in this chapter we present the characteristics of two long-term traces from the Austrian Grid, a Grid environment in which workflows are common.
We introduce a method for analyzing such traces, and then apply the method to the two Austrian Grid traces. The method identifies two broad classes of workflow characteristics, intrinsic and environment-related. Based on the observed values for the former, we devise six classes of workflows with distinct properties. The analysis of environment-related characteristics reveals that from the six classes several can be considered classes of ”problem-workflows”, which exhibit one or all of high variability of the work size of their tasks, high makespan, poor scalability, and higher than normal failure rate. Overall, we find that the workflow speedup is highly dependent on the system used for execution, and that the current task success rate requires more fault tolerance mechanisms, especially for large workflows.
We plan to extend our work with the analysis of other traces. In particular, we hope to find traces that include mostly production workflows submitted by real users. Based on these traces, we will be able to design a model for workflow-based Grid workloads. We conclude by addressing the whole Grid community with a request for making available their (workflow-based) Grid workload traces to other researchers.
Future research to improve the performance of the Montage workflow with ASKA- LON primarily addresses the inefficiencies discovered in the MontageBasic design. In order to improve the performance of multiple data transfers between activities we suggest inclusion of the GridFTP data-streaming feature into the ASKALON workflow execution engine. Data streaming allows multiple files to be transferred over the same data channel at the same time, which can be used to reduce the inefficiency caused by the file transfer initiation overhead. Another suggestion for improvement addresses the job submission strategy of ASKALON. Submission of a single job per activity is inefficient because of the scheduling overhead on the client side and the increased CPU load on the master node hosting the GRAM job managers. Future extensions to the enactment engine may include a feature that allows the submission of multiple identical activity instances of a parallel construct by means of a single RSL script.
For Grasil the calculation of the first 25000 galaxies has enabled the astrophysicists to determine the parameters responsible for different calculation times. These pa- rameters can be utilized by ASKALON’s performance prediction to assist in making scheduling decisions.
Calculating several hundreds of thousands of SEDs will also provide some interest- ing challenges, one of them being the management of disk usage, another the shere number of files involved in this task.
3.9 Summary
In this Chapter we presented the analysis of two years of workflow execution to iden- tify the workflow classes typical for scientific environments. Then we present de de- velopment of workflow versions two scientific applications in detail. Then three more applications used in our experiments are briefly presented. In the upcoming Chapter the workflows presented here will be used to evaluate the methods developed for this thesis.
Chapter 4
Architecture
With an increasing number of providers claiming to offer Cloud infrastructures, there is a lack in the community for a common terminology, accompanied by a clear defini- tion and classification of Cloud features. We first conduct in this chapter a survey on a selection of Cloud providers, and propose a taxonomy of eight important Cloud com- puting elements covering service type, resource deployment, hardware, runtime tun- ing, business model, middleware, and performance. We conclude that the provisioning of Service Level Agreements as utilities, of open and interoperable middleware so- lutions, as well as of sustained performance metrics for high-performance computing applications are three elements with the highest need of further community research.
Scientists have their Grid environments in place and can benefit by extending them with leased Cloud resources whenever needed. This paradigm shift opens new prob- lems that need to be analyzed, such as integration of this new resource class into exist- ing environments, applications on the resources and security. The virtualization over- heads for deployment and starting of a virtual machine image are new factors, which will need to be considered when choosing scheduling mechanisms. In section 4.3 we investigate the usability of compute Clouds to extend a Grid workflow middleware.
To validate our approach we run workflows on the Austrian Grid using the architec- ture extension presented in section 4.3.1 to validate the general usability of Clouds for scientific workflows.
4.1 Cloud Computing Survey
Scientific and business applications have an increasing demand for fast and scalable execution environments to deliver results for ever increasing problem sizes or concur- rent requests in a requested time frame. Faced with the problem of costly maintenance and rapid deprecation following the Moore’s law, companies and institutions oper- ating scientific and business applications prefer to rent modern resource capabilities from specialized hosting companies instead of buying their own hardware.
To automate the complex process of installing and running legacy applications on remote servers that requires manual intervention, the current trend is to virtualize the hardware and provide it as a service over the Internet which has coined a new term
called Cloud computing. In simple words, a Cloud computing provider can be re-
garded as a company that leases to its customers a number of reliable virtual resources (hardware or software of any kind) according to a certain business model.
Today, there is a growing number of providers on the market claiming to offer Cloud infrastructures, each of them describing its capabilities using own terminology, defini- tions, and goals. Despite gaining increased popularity, this lack of convergence makes the overall community effort sparse and uncoordinated. To address this deficiency, we perform in this chapter a survey on a number of Cloud computing providers with the goal of establishing a taxonomy that identifies a common terminology, architectural and functional similarities, as well as gaps for future research.
Our survey is based on an list of providers, selected based on the availability of technical information on the companies’ Web sites or generally in the Internet. Our analysis is based on data published by the surveyed companies in January 2009, which is constantly being updated, however, we believe this study builds a solid foundation for understanding the common characteristics and trends in current Cloud computing environments.
We continue this chapter with the Cloud taxonomy in Section 4.2 and then conclude in Section 4.6. During our survey, the reader is constantly referred to Table 4.2 which is at the end of Section 4.2.8.