ORIGEN HISTORICO.

Autonomic computing applications rage from full system implementations [80, 95, 1, 182, 155, 27] to bespoke practical realisations of the MAPE-K components (monitor [2], plan [39], analyse [36, 6]). A detailed analysis of state of the art contributions is provided in 3.1, however, to paint a picture of the range and diversity of autonomic applications, the major problem domains where such technologies are applied are given below2.

• space exploration: NASA ANTS [88] supports collaboration between asteroid belts probes;

• distributed systems: load balancing and component repair [95], data collection frequency control [2], dynamic resource management [1, 182, 150, 147], network compartmentalisation and assembly [27], network parameter estimation [131, 148, 6], medical services in the cloud [4];

• electronic personal assistants: PC performance improvement via operating system’s processes management [108];

• web services configuration [124, 184];

• military and security-critical applications: hierarchical policy management [37];

• middleware [57, 117, 187];

• autonomic database management [3];

• data centre management [182].

Given the wide range and the increased diversity of autonomic applications, it is not surprising that a unified evaluation framework is yet to be formulated. The most common scenario is to test the performance of the autonomic system in the context of the application it was designed for. This does little in the way of enabling a rigorous performance comparison across a range of different systems. However, there exist a few proposals of general assessment criteria for autonomic systems that can be roughly classed as either qualitative or quantitative.

Qualitative Evaluation

Qualitative evaluation employs discrete, non-numeric scales for various system aspects [131]. IBM’s five levels of autonomicity (2.1) represent such a scale, used both in [88] and as one of the classification criteria in 3.1. Other qualitative metrics suggested in [131, 127, 113] are:

• architecture category: flat (horizontal) or hierarchical (vertical, where managers on a given level are coordinated by one situated on the level above),

• adaptation approach: the strategy the system employs to adapt to changes in its environment (can be policy based or utility function based),

2_{There is significant overlap between contributions: some papers should, in all fairness, be included in several categories.} This, as well as the abundance of evaluation criteria, makes it difficult to provide a clean classification of autonomic applications. A more thorough categorisation than that given here is attempted in chapter 3.

• learning ability: splits autonomic systems in two categories based on the presence or absence of a learning mechanism (there is a quantitative evaluation metric to capture learning efficiency, as shown in the next section),

• openness: autonomic services that are publicly available or with restricted access,

• evolvability: discriminates between autonomic systems that can be maintained and extended and those that cannot (a trademark of an evolvable system is the fact that its core elements are designed as middleware [131]),

• granularity: measures the modularity and coupling of system components [127] (either thick- grained or fine-grained),

• robustness: refers to a system’s capacity of avoiding/ recovering from failures [127],

• validation: differentiates between systems assessed via simulation, mathematical modelling or other qualitative/quantitative metrics, on the one hand, and systems without any associated evaluation data, on the other hand.

Quantitative Evaluation

• Probabilistic verification techniques measure how well systems meet quality requirements, such as “the probability of a successful attack must be lower than a given threshold”. The availability of a probabilistic model (e.g., based on Markov chains [34]) of the system is a prerequisite. This implies that system properties as well as quality requirements must be translated from informal language into a probabilistic temporal logic, with substantial computational effort [33]. ProProST [75], introduced to simplify the translation process, represents a collection of specification patterns to be used as building blocks and assembled to formulate arbitrarily complex probabilistic properties. A structured English grammar is also available to aid the translation of quality requirements and system properties from natural language into ProProST.

• Application specific metrics measure system performance in the context of the problem they were designed to solve. A list of such metrics [131] includes the quality of the autonomic response (e.g., the increase in the speed of response, the decrease in the overall number of faults over a given period of time, etc.), the cost of autonomy (the overhead introduced by the MAPE-K loop), the flexibility ratio (the rate of failure decrease divided by the overhead), the speed of the autonomic response (the total time necessary to adapt to change in the environment), the degree of proactivity (the number of unsupervised decisions made by the system).

• Generic metrics give an objective, application independent evaluation of (theoretically) any type of autonomic system. These metrics [131] may be relevant for comparing different autonomic systems on the condition that they were built to achieve similar goals. The quantitative evaluation criteria proposed in [131] have to do with the autonomic system’s efficiency at storing and manipulating its knowledge base. The coefficient of learning, cl, is meant to assess the system’s

capacity to learn from experience (stored in the knowledge base) in order to enhance future performance and is formally defined as follows:

cl =

max(1 −_E{NDE{D0}0_}, 0) + min( E{D} E{PD}, 1)

2 , (2.1)

where E{D} represents the average number of correct learning based decisions and E{PD} stands for the average number of “target achievers”, namely decisions with a significant impact on realising the system goal. E{D0} and E{ND0} are defined similarly, yet for incorrect decisions, with the latter representing the number of “target damagers”, i.e., decisions with a strong negative influence on the system (driving it away from a desirable state). Systems successful at learning from their experience will score close to 1 on the cl scale, meaning that out of all target achievers,

most were learning based decisions, whereas out of all target damagers, almost none were learning based decisions. Systems inefficient at learning from their knowledge base will have a cl value

close to 0. The efficiency of learning, el gives a numeric interpretation to the amount of progress

made by the system towards the target goal:

el=

K{PD}

K{PD} + K{ND0_}. (2.2)

Symbols PD and ND have the same significance as above, whereas K stands for “average percentage”. These last two metrics are used to define the learning index:

Il =

L

M· cl· el, (2.3)

where L represents the number of system parameters that can be learned and M stands for the total number of system management parameters. A system with a learning index of 0 either has no learnable parameters (L = 0) and/or is incapable of learning from knowledge (cl= 0) and/or makes

no target achieving decision (el = 0). In [131], such a system is termed closed adaptive system.

Open adaptive systems are situated at the opposite end of the spectrum, with an Il of 1. The

accuracy of awareness is a metric targeting the frequency of changes in the system’s knowledge base (preferably high, to reflect environment dynamics) as well as the overhead introduced by it (preferably low, to speed up execution). To assess the accuracy of awareness, [131] suggests taking several measurements including the time required to insert a new data sample, the trust of the update’s author (e.g., a credibility/reputation score) and the correctness of the knowledge base (the number of other data samples corroborating the newly added one).