We empirically evaluate and discuss the performance of Weatherman in the case-study of a semi-production environment. We operate in the run-time analysis mode, which intercepts and analyzes operations, and we are interested in the performance of our analysis, in particular the application of the operations model and the information flow analysis.
The environment consists of 2 physical hosts and over 100 virtual machines. Weatherman itself runs in a Linux VM with 12 vCPUs, 12 GB RAM, and Java 1.7. We issue a variety of operations to the authorization proxy of Weatherman. These include the creation of virtual machines, virtual network interfaces, as well as virtual disks. Further, we change the VLAN identifier of a port group. This set of operations aligns with our subset of VMware operations (cf. Table 8.1) and covers all types of infrastructure resources as well as
0 100 200 300 400 500 600 700 Runtime [ms] VLAN ID update (Violation) VNic creation (Violation) VLAN ID update VDisk creation VNic creation
VM creation Graph SerializationGroove Startup
Initial InfoFlow Change Ops Adjust InfoFlow
Figure 8.6.: Time measurements for the analysis of a variety of operations, including two violating ones (the last two). We measure the times for the graph serialization, GROOVE start-up, initial and adjusting information flow analysis, as well as applying the change operation.
different kinds of operations. Further, we issue operations in a policy violating form to show how the analysis may stop early once a violation is found.
Fig. 8.6 shows the results of our performance evaluation with different operations on the y-axis, and the mean run-time measurement in milliseconds for 30 rounds on the x-axis. A first observation is that the majority of the analysis time for this environment is spent on serializing the graph model and initializing
GROOVE, which loads the grammar including the graph model. This is an implementation limitation
as GROOVE was not designed to be integrated into another application and requires to load a grammar from the filesystem. As part of the actual analysis, we observe that the times for the initial and adjusting information flow analyses are the dominant factors. Applying the operation to the graph model is almost negligible. Note that in the case of violating operations, the analysis can terminate early, i.e., not complete the adjusting information flow analysis, once a violation was found. Overall, the performance results for this environment, i.e., obtaining analysis results in under a second, are suitable for both run-time analysis and change planning.
8.5.2.1 Discussion on Scalability and Optimizations
We studied the scalability of Weatherman with a VMware infrastructure simulator, which is part of the official VMware vCenter server appliance. For a simulated environment with 1000 VMs, which resulted in a Realization model graph with 4121 vertices and 6140 edges, we obtained an overall analysis time of 253s for finding a violation in a UpdatePortGroup operation. This makes our approach suitable for the change plan analysis, but causes a long soft blocking for a run-time analysis. In a simulated environment with 10000 VMs (41201 vertices, 61400 edges) GROOVE ran out of available memory (12GB). In another case study reported by Smid and Rensink [SR13], GROOVE showed similar performance where in a case with 10000 elements GROOVE ran out of memory on a 10GB machine. A distributed variant of GROOVE with state compression has been proposed by Kant [Kan10], where an up to 52 times memory reduction has been achieved in one case.
In terms of performance comparison with the approach of Chapter 7, we observe for the real environments (size 100 respectively 150 VMs) a similar analysis time of 480ms for Weatherman respectively 476ms for
Cloud Radar in the initial mode in the case of finding a VLAN ID update violation. However, in terms of
scalability, Cloud Radar shows significant better performance for larger environments in particular in the
event-based mode.
We stress that establishing the models, methodology, and analysis system has been the primary focus of this work, and not providing an optimized and scalable analysis. We now outline multiple directions of optimizations and scalability improvements. A short-term optimization is to reduce the size of the Realization model graph by removing nodes of types that are not addressed by production rules of the
grammar. Possible long-term optimizations are to transform GROOVE graph grammars into native code (an approach employed by GrGen [GBG+06]), to exploit a parallel processing of production rules (in particular for rules with universal quantifier and the confluent simple information flow rules), and to leverage existing large-scale graph processing framework, which however are not yet aimed for graph transformations.
8.6 Summary
In this chapter, we tackle the problem of misconfigurations, insider attacks and resulting security failures in virtualized infrastructures. Our solution consists of a practical tool called Weatherman that employs a formal model of cloud management operations, an information flow analysis to determine isolation properties, and a policy verifier in order to proactively assess infrastructure changes with regard to their security impact. For instance, we are able to detect and mitigate changes that i) break the network isolation of tenants, ii) create virtual machines in the wrong location, and iii) introduce single point of failures. We offer the run-time enforcement of security policies as well as change planning for what-if analyses. While for concreteness we focus in this work on a particular practical system and goals, we believe that our work is a first step towards a general verification methodology for virtualized infrastructures. One key aspect of our approach is the use of graph rewriting, which offers an expressive and intuitive method for formalizing the operations, information flow analysis, as well as policies.