Síntesis y reflexiones finales

Cloud management platforms (CMP) are the suite of integrative software tools used to monitor and control cloud computing resources. They incorporate interfaces for self-service resource management and more-advanced offerings, like workload optimization, support for configuration of storage and network topology, shortly introduced at 1.4.3.2 section. Here, the two major open-source IaaS CMPs are used: OpenNebula and OpenStack.

〕 OpenNebula

OpenNebula is designed to help building simple and reliable, datacentre-like clouds on existing IT infrastructure in a cost-effective way. OpenNebula can be managed through a Web-based UI (Sunstone) that provides access to all features, yet it also has a CLI and a powerful RCP-XML API, essential to programmatically manage the cloud from other systems. Other main components of OpenNebula are the nodes, the image repository, the daemon, and the drivers. OpenNebula is structured as the classic cluster-like architecture with a front-end, where the API, Sunstone and other services are located, and a set of hypervisor-enabled cluster nodes, where VMs are instantiated. Such model is very convenient for our HPC cluster installations, as it permits to transfer compute nodes from the HPC to the cloud with a certain flexibility.

In terms of management, OpenNebula is organized as other CMPs: VM images > VM

templates > (VM services) > VM instances:

- VM images refer to the disks images containing either file systems or operating systems. Supported formats are QCOW2, RAW, and VMDK.

- VM templates correspond to YAML-like files defining the set of attributes required to compose a VM instance. Apart from specifying one or more VM disks, it includes information on CPUs, virtual CPUs, RAM memory, network interfaces with their gateway or IP, contextualization details, and many other attributes. Snippet 3.1 shows a simple VM template.

- VM services are multiple and varied, in fact, these are one of the differential traits among CMPs. OpenNebula offers self-provisioning tools for managing clusters, auto-scaling services (like OneFlow, below discussed), network definition tools, or VM authentication management, among others. However, the platform’s philosophy is offering good connectivity and interoperability capabilities while allowing new services implemented as add-ons.

- VM instances are the result of deploying a VM template, either manually or via one the advanced services.

- DESCRIPTION = "TADBIT FOR MUG" . CPU = "8" - CPU = "12" MEMORY = "189152" MEMORY_UNIT_COST = "MB" DISK = [ IMAGE = "mg-tool-tadbit" - TARGET = "hda" - DRIVER = "qcow2" ] FEATURES = [ ACPI = "yes" ] GRAPHICS = [ LISTEN = "0.0.0.0", TYPE = "VNC" ] HYPERVISOR = "kvm" NIC = [ NETWORK = "kvm-servers-mmb" - IP = "range" ] OS = [ ARCH = "x86_64", BOOT = "" ] - CONTEXT = [ - NETWORK = "YES", SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" ]

Snippet 3.1: Simple VM template for OpenNebula

Images, templates, instances, networks and other services, are all managed either through the Command Line Interface (CLI) or the Sunstone UI. Additionally, OpenNebula supports several cloud client interfaces, such as Amazon EC2, Google Cloud or vCloud, demonstrating being vendor-neutral. And importantly, the Open Cloud Computing Interface (OCCI) (based on draft 0.8) is natively supported, providing a standard endpoint to create, control and monitor VMs. Indeed, the analysis platforms here presented interact with OpenNebula via the OCCI connector.

OpenNebula supports XEN, KVM, and VMWare ESX hypervisors, yet our installations are all based on the popular KVM virtualization platform, operationally managed via the open- source virtualization library Libvirt. KVM is outperformed by purely bare-metal hypervisors like Xen, yet, it is widely used because of its simplicity and ease of use, running directly on Linux’s kernel and being delivered by default in most Linux distributions.

We have had access to two cloud infrastructures managed by OpenNebula, one at the Barcelona Supercomputing Center (BSC), the other at the Institute for Research in Biomedicine (IRB). Both are based on cluster-like architectures and correspond to two private on-premises deployments operated and managed by us from the respective institutions. Details on the underlying hardware are found in Table 3.1.

- The “INB Cloud” is hosted at the Barcelona Supercomputing Center (BSC) and installed on top of on a computational cluster It includes an externally accessible front-end which acts as a gateway for the internal cloud VLAN on HTTP(s), FTP(s) and SSH. A shared common storage system of some TBs is accessible to both, frontend and compute nodes. Accessed as a Network Attached Storage (NAS) on a high-speed network, disks export several NFS (v3) endpoints on top of a GPFS distributed file system.

- The “MMB Cloud” cloud is hosted by the Molecular Modeling and Bioinformatics Unit of the Institute for Research in Biomedicine (IRB). Likewise “INB Cloud”, OpenNebula manages a series of homogeneous nodes virtualized in KVM and the storage is also block-accessed over the network, yet in this case, the NAS protocol is CIF, while the distributed storage solution is based on NetApp.

Cloud Name Cores RAM Memory Storage System OpenNebula Project

INB Cloud (BSC)

4 x 12 cores Intel Xeon E5649

4 x 96 GB RAM Shared GPFS of 78TB (NFS) OpenNebula (v5.2.1) transPLANT MuG MMB Cloud (IRB) 3 x 58 cores Intel Xeon E5-2640

3 x 442 Gb RAM Shared disk 7.5TB (CIF)

OpenNebula (v4.4.1)

MuG Table 3.1: OpenNebula cloud infrastructures

〕 OpenStack

OpenStack is designed to control large pools of computing, storage, and networking infrastructure resources to create a massively scalable, flexible cloud platform of private and public IaaS architectures. It consists of a series of interrelated open source software projects that provide both, core services, and a catalog of pluggable advanced cloud tooling. A minimal OpenStack installation could include: Keystone (Authentication Service), Glance (Image Service), Nova (Compute Service), Neutron (Network Service), Horizon (Dashboard Service), Cinder (Block Storage) and Swift (Object Storage).

OpenStack offers a full ecosystem of RESTful APIs to programmatically control these services, and likewise OpenNebula, some non-proprietary interfaces are also supported, including the OCCI standard. The graphical UI is organized in “tenancies”, groups of users and resources that manage their own templates, images and networks. They even might have a separated pool of resources. In terms of virtualization, KVM, QEMU, Xen, LXC, VMware vSphere, Hyper-V are supported.

Part of the work presented here is underpinned by an OpenStack tenancy requested to the

Embassy Cloud. Hosted by the European Bioinformatics Institute at the European Molecular

Biology Laboratory (EMBL-EBI), the infrastructure is externally managed, unlike the above- mentioned infrastructures. The Embassy Cloud provides IaaS to multiple tenants (e.g. researchers, scientific projects and institutions, etc.) from the life science domain. EMBL- EBI features more than 4,000 cores and three petabytes of storage, with the interesting added value of being co-located with some of the most relevant European biomolecular databases (i.e. most of ELIXIR core data resources are physically hosted there, for example, ArrayExpress [9], PDB [5], etc.). So, even considering that the network cloud is logically isolated from EBI’s LAN, the mobilization of such reference data could be efficiently handled. Storage is also based on a shared NFS file system (under tenants’ petition), but some quota for object store is also offered (provided by OpenStack Cinder storage [202]). The following table details the specifications of our EMBASSY tenancy:

Name Cores Memory Storage System Network Project

EMBASSY Tenancy (EMBL-EBI)

16Gb 64 Gb RAM - 1Tb Object storage (CINDER) - NFS under request

- 2 floating public IPs - Internal VLAN

MuG Table 3.2: Cluster details of MuG development’s cloud