Worker nodes, or worker instances, are servers which run worker processes. Worker processes are individual program threads running on a physical or virtual machine. The majority of instances in this pipeline, for the purpose of experimental control, are virtual machines hosted on the Amazon Web Service using the EC2 service. For an experiment all instances are usually set to be the same type as explained later in the Control System subsection in this chapter. Depending on the configuration of the instance it may be possible to run multiple worker threads which could result in an overall improvement in the amount of files processed by the instance. For simplicity, the worker process is not written to be multi-threaded, rather the operating system is instructed to run multiple copies of the same program which are kept in insolation from each other. Resource sharing is handled by the operating system.
As with the previous pipelines the acn-aphot.c program is run at the heart of the worker process. A control system implemented in Python wraps this program in a service which provides image data for cleaning. The core cleaning program was not modified for this pipeline.
In order for a server to join the global processing cloud it must first install an initialisation script which is run when the server is first started. This script call be installed into an existing server, or as in the case of this pipleine a reference virtual machine was constructed with this script installed and an amazon machine image (AMI) produced from which large number of virtual machine instance can be created.
This initialisation script will look for a science payload on a public web server then download and execute it. The payload is designed to clean the instance directory spaces, install scripts and utilities, and clone itself to create multiple programming processes if required and then begin whatever tasks are set by the worker utility script.
To ensure that all experiments started in a consistent manner all experimental worker instances were deleted and new instances were created from the reference AMI. The primary lifecycle of a worker instance is shown in Figure 5.7 and described below.
Start
Boot Instance
Run init.d Script Download Science Payload Install Pri- mary Worker Install Worker Thread N Install Worker Thread 2
Start Worker Start Worker N Start Worker 2
Stop
Figure 5.7: Flow Chart showing worker node initialisation during the boot up process
• Start The control system is responsible for starting the worker instances. A standard preconfigured AMI is used. In some experiments physical instances were also used.
• Boot Instance The standard Linux boot sequence brings the instance onto the network and mounts the local storage.
• Run init script The nimbus-worker init script is run which ensure that the server is cleaned of any previous experimental data.
• Download Science payload The science payload is downloaded, moved to the worker home directory on the server and installed. The work performed by a worker is detailed in the this package. This ensure that there is a flexibility to the system allowing for multiple types of processing to be performed depending on the payload. In most cases the worker will perform image cleaning and calibration of the FITS files, however any tasks are feasible. For the purpose of this pipeline the payload was altered to test network performance and web server performance by downloading files but not processing them, and by doubling the processing in an alternative payload to show the effects of a more CPU bound problem. • Install Worker thread 1-N The installed system will create the required number of inde-
pendent threads which are then run.
• Start worker thread. The workers are ready to process data and listen to the supervisor and command queue for the instruction to start downloading data to process.
• Stop/Delete The worker threads will sleep when there are no more messages to process, periodically checking for work. When an experiment is finished the control system may stop or terminate the running instance.
The flowchart for a running worker thread is shown in Figure 5.8. Each worker instance may have multiple worker threads all operating at the same time. All threads are isolated from each other however, with their own storage space and directory structure on the worker instance. The control system which operates this thread is a Python script inside which any science payload could potentially be inserted. The function of the worker thread is to listen to the sqs command queues for instructions on how to operated. Instructions such as start, stop, and sleep provide simple interfaces into the system. What to download and where to get it, is provided by the configurable message queue, and how to process it is determined by a central do_work script which then performs the required work. The output of the processing is then uploaded to a central web server and information on its location is pushed to a sqs based results queue.
5.3
Experimental Methodology
The function of the control system is to initiate all experiments and ensure that all systems are available and functioning correctly. It is important that experiments can be compared, and to do this the starting state must be consistent in all cases. The control system runs a Python script which tears down the experimental infrastructure, then rebuilds it before the start of the experiment. All systems must be accessible from the control system which resides on a virtual machine within the AWS cloud, running an Ubuntu instance on the EC2 service. A batch script contains the series of experiments to run, which in turn calls a script to start and experiment. The batch script contains a series of calls to the run-experiment.sh script, which takes a set of
Start Read Sup Queue Reg Worker Reboot Instance Read Cmd Queue Wait 60 Secs Stop Start Timer Clear Directories Read Worker Queue Msg Available Wait 60 Secs Download Files Call do_work Script Upload Results Delete Msg from Worker Queue No Sleep Stop Register Reboot
Figure 5.8: Worker control script managing the flow of work based on message reading status
parameters, shown in Table 5.4. The pipeline can also be left in a running state, which means that it will continue to process image files as they appear on queues one it has started, and new workers can be added or removed dynamically. The full workflow of the experimental run system run-experiment.sh is given in Figure 5.9. The experimental run script was required to ensure that the starting point of each experiment was consistent for each experimental iteration. A sample of the batch script used to run multiple experiments is shown in Appendix D.5.
Option Description
-a | -d | -n | -x Identify the combination of web servers to use in the experiment. -d uses the
DIT based web servers, -a indicates the use of the AWS based web servers, -n uses the heanet based web servers and -x indicates that all web servers should be used. For a web node to be used in an experiment it will advertise the files it has storage to the SQS worker queue.
Instances The number of instances of workers to run for an experiment less the mon-
itoring node which always runs. The maximum number of concurrent in- stances is set to 100 within these experiments. This limit required explicit permission from AWS Ireland to run instances within the Irish region.
Time The maximum amount of time in seconds that the worker nodes should
be allowed to run. In most cases experiments were set to 20 minutes. An experimental timer was set only when the EC2 instances were initiated and confirmed to be running.
Name The name of the experiment so that it can be identified.
Number The number of web servers to run per type. Webservers are configured in
pairs, so if a DIT web server is selected then either 1 or 2 will be allowed to run while 1-4 was the range allowable for AWS based web servers.
Size The size of the AQS EC2 instance to run. The parameter conforms to the
specific reference name that AWS uses for its instances. Most experiments used either the t1.micro the m1.large or in some cases the m3.2xlarge
Package Pay-
load
In addition to changing the experimental options above, the package used by a worker node can also be reconfigured. The most significant change is specifying the number of threads a worker instance is allowed to initiate when running. This ranged from 1 to 100. The number of files downloaded
by a worker before it begins processing the can also be specified. The
processing batch size for nearly all experiments was set to 10.
The experimental run script is a Unix bash script which call a series of Python scripts on remote system to control a specific experiment. The Python programming API used to control AWS services is BOTO, which provides access to primary services such as S3, EC2 and SQS. To ensure that an experiment starts consistently each time it is run efforts are made to ensure there is little or no system caching being performed by any system. The S3 buckets are emptied to ensure that no previous results are being counted for the experiment. Queues are deleted and recreated to eliminate the chance of old messages contaminating an experiment. All worker instances are created from a standard AMI instance and are deleted at the end of each experimental run.
The experimental pipeline performance has also been considered where possible to ensure that steps in the process for creation or tear down of the system are as efficient as possible. Some of the steps in the experiments are primarily focused on experimental integrity but are not essential to a production pipeline, such as queue deletion, and S3 result deletion. Where large number of entries are placed on queues a multi-threaded queue reader is employed to extract all messages. As previously discussed the logfile contains the largest number of entries within an experiment and as such the reader for this queue downloads messages via multiple threads. The process of recombining these are sorting data into a sequential stream is the final generate metrics step in the flow chart.
In preparation for the experiment a science payload is created and pushed to a web server as shown in Figure 5.10. The payload will include details on how many independent workers are to be created on the instance and what work each of these workers should perform and how it should perform that work. The science payload consists of the following components:
• Datafiles directory. The specific location to download batches of files.
• Results directory. The location where results files are held until they are uploaded to the S3 service.
• Masterfiles directory. Contains the mastervbias and the master flat images for use in image cleaning.
• bin directory. Contains the acn-aphot compiled utility used to clean image files and the funpack utility to uncompress images.
• scripts directory. Contains the do_work.sh script which performs the image uncompress and calls the acn-aphot program to perform cleaning, and the start-worker.sh script which installs the required number of worker threads and starts them.
• sql-reader.py. The central control script which monitors queue, downloads images, passes them to the do_work script, and uploaded results.
Start Terminate EC2 Instances Delete all queues
Empty S3 Buckets
Create all queues
Start Queue Monitor Write START to cmdq Write WORK to supervisor Connect to Storage Nodes Storage Node N Publish to SQS Storage Node1 Publish to SQS Start Worker Instances Start Worker N Start Worker 1
Wait for Timer to Expire Stop Queue Monitor Write STOP to cmdq Write STOP to supervisor Stop Worker Instances Stop Worker N Stop Worker 1
Read result queue
Read reg- ister queue Read log file
Generate Metrics
Stop
Start Update do_work Script Set number of workers Create Payload Push Payload to web server Webnode Stop
Figure 5.10: Flow Chart showing creation of payload for distribution to worker nodes