Salud, Ambiente y Reducción de la Vul- Vul-nerabilidad

During CSA06 the Tier-1 centers met the transfer rate goals. In the first week of the challenge using minimum bias events the total volume of data out of CERN did not amount to 150MB/s unless the datasets were subscribed to multiple sites. After the reconstruction rate was increased at the Tier-0 the transfer rate easily exceeded the 150MB/s target. The 30 day and 15 day averages are shown in Table 10. For the thirty day average all sites except two exceeded the goal rate and for the final 15 days all sites easily exceed the goal. Several sites doubled and tripled the goal rate during the final two weeks of high volume transfers.

The WLCG metric for availability this year is 90% for the Tier-1 sites. If we apply this to the Tier-1 participating in CSA06 transfers we have 6 of 7 Tier-1s reaching the availability goal.

Table 10: Transfer rates during CSA06 between CERN and Tier-1 centers and the number of outage days during the active challenge activities. In the MSS column the parentheses indicates the site either had scaling issues keeping up with the total rate to tape, or transferred only a portion of the data to tape.

Site Anticipated Rate (MB/s) last 30 day average last 15 day average Outage (Days) MSS used

ASGC 15MB/s 17MB/s 23MB/s 0 (Yes) CNAF 25MB/s 26MB/s 37MB/s 0 (Yes) FNAL 50MB/s 68MB/s 98MB/s 0 Yes GridKa 25MB/s 23MB/s 28MB/s 3 No IN2P3 25MB/s 23MB/s 34MB/s 1 Yes PIC 10MB/s 22MB/s 33MB/s 0 No RAL 10MB/s 23MB/s 33MB/s 2 Yes

The rate of data transferred averaged over 24 hours and the volume of data transferred in 24 hours are shown in Figures 35 and 36. The start of the transfers during the first week is visible on the left side of the plot as well as the transfers not reaching the target rate shown as a horizontal red bar. The twin peaks in excess of 300MB/s and 25TB of data moved correspond to the over-subscription of data. The bottom of the graph has indicators of the approximate Tier-0 reconstruction rate. Both the rate and the volume figures show clearly the point when the Tier-0 trigger rate was doubled to 100Hz. The daily average exceeded 350MB/s with more than 30TB moved. The hourly averages from CERN peaked at more than 650MB/s.

The transferrable volume plot shown in Figure 37 is an indicator of how well the sites are keeping up with the volume of data from the Tier-0 reconstruction farm. During the first three weeks of the challenge almost no backlog of files is accumulated by the Tier-1 centers. A hardware failure at IN2P3 resulted is a small accumulation. The

Figure 35: The rate of data transferred between the Tier-0 to the Tier-1 centers in MB per second.

additional data subscriptions leads to a spike in data to transfer, but is quickly cleared by the Tier-1 sites. The most significant volumes of data waiting for transfer come at the end of the challenge. During this time GridKa has performed a dCache storage upgrade that resulted in a large accumulation of data to transfer. CNAF suffered a file server problem that reduced the amount of available hardware. Additionally RAL turned off the import system for two days over a weekend to demonstrate the ability to recover from a service interruption. The Tier-1 issues combined with PhEDEx database connection interruptions under the heavy load of the final week of transfers to accumulate a backlog of approximately 50TB over the final days of the heavy challenge transfers. During this time CERN continued to serve data at 350MB/s on average.

Figure 37: The total volume of data waiting for transfer between the Tier-0 to the Tier-1 centers in TB per day. The CERN to Tier-1 transfer quality is shown in Figure 38. In CMS the transfer quality is defined as the number of times a transfer has to be attempted before it successfully completes. The link between two sites with 100% transfer quality would have had to attempt each transfer once, while a 10% transfer quality would indicate each transfer had to be attempted ten times to successfully complete. Most transfers eventually complete, having low transfer quality uses the transfer resources inefficiency and usually results in a low utilization of the network.

Figure 38: Transfer quality between CERN and Tier-1 centers over 30 days

The transfer quality plot compares very favorably to equivalent plots made during the spring. The CERN Castor2 storage element performed very stably throughout the challenge. There were two small configuration issues that were very promptly addressed by the experts. The Tier-1s also performed well throughout the challenge with

several 24 hour periods to specific Tier-1s with no transfer errors. The stability of the RAL SE before the transition to CASTOR2 can be seen at the left side of the plot, as well as the intentional downtime to demonstrate recovery on the right side of the plot. The IN2P3 hardware problems are visible during the first week and the GridKa dCache upgrade is clearly visible during the last week. Most of the other periods are solidly green. Both FNAL and PIC are above 70% efficient for every day of the challenge activities.

Tier-1 to Tier-1 transfers were considered to be beyond the scope of CSA06, though the dataflow exists in the CMS computing model. During CSA06 we had an opportunity to test Tier-1 to Tier-1 transfers while recovering from backlogs of data when the samples were subscribed to mulitple sites. PhEDEx is designed to take the data from source site where it can be efficiently transferred from. Figure 39 shows the total Tier-1 to Tier-1 transfers during CSA06. With 7 Tier-1s there are 84 permutations of Tier-1 to Tier-1 transfers, counting each direction separately. During CSA06 we successfully exercised about half of them.

Figure 39: Transfer rate between Tier-1 centers during CSA06