The logging mechanism is the set of detectors, either hardware or software, that allows to reveal the occurrence of error events during the system operational phase. Figure 1.5 shows the causal relationship between faults and failures and clarifies the role of the detectors com- posing the logging mechanism. Once a fault is activated by a trigger (i.e., a specific sequence of inputs or condition of the execution environment that activates a fault) it generates one or more errors. Errors propagate through the various layers and/or nodes of the system, and might possibly fire the detectors. As a result, detectors will report the occurrence of the event in the form of entries in the log (Figure 1.5, event log). As discussed in Section 1.2.1, an error that reaches the service interface, causing a deviation from the correct service, is called failure. In order to be suitable to perform a failure analysis, the logging mech- anism should encompass only the errors that cause failures, i.e., Figure 1.5 (d).
Figure 1.5: Error logging mechanism: overview.
However, many situations can prevent the accurate reporting of the failures, thus making hard, if not impossible, to infer the information about their occurrence from the event log. For example, failures can go unreported in the log either because (i) no detector is able to catch the failure, i.e., Figure 1.5 (b), or (ii) even if the failure is detected, i.e., Figure 1.5 (a), there is no enough time to write an entry in the event log (e.g., in C/C++ programs, bad pointer manipulations can originate a crash of the OS process before any useful infor- mation is logged). Furthermore, event logs might report errors that did not cause a failure, i.e., Figure 1.5 (c): some error notifications that can be commonly observed in the log (e.g., unreleased files, pending sockets, null pointers) do not necessarily represent a failure indication, and have to be discarded before the failure analysis is performed: this task, if not supplemented by a detailed knowledge of the system, can distort analysis results.
Figure 1.6: Coverage of the event log.
Figure 1.6 provides a pictorial representation of the described scenarios, by highlight- ing the role of the event log at reporting failures. The ellipses represent the set of faults, the subset of activated faults, i.e., errors, and the subset of errors causing failures, respec- tively; the event log, represented by the ellipse with the dotted border, partially overlaps failures. Through the rest of the dissertation, unreported failures will denote failures that go undetected and/or unlogged by the logging mechanism, and thus escape the event log (as indicated in Figure 1.6). These failures are false negatives, i.e., a problem has occurred in the system but no trace can be found in the log; the failure is reported (or logged), oth- erwise. Similarly, reported errors (as indicated in Figure 1.6) will denote improper states of the system that do not cause a failure, but fire the logging mechanism by causing one or more entries to be written in the log; errors are named unreported, otherwise. Reported errors represent false positives to the failure analysis: even if they notify anomalous events in the log, no actual failure has occurred in the system.
1.4.1 Limitations
The implementation of the logging mechanism is currently a low-priority task that is left to the late stages of the system life cycle (e.g., coding). Key decisions about log production and management are taken by developers and programmers, each of them with his/her own programming skills, and deciding what and where to log with no systematic approach: as a result, entries in the log are often subjective [7] and unstructured [30]. In general, the logging mechanism is not envisioned at system design time.
The nature of the logging mechanism has serious limitations that compromise the ef- fective use of event logs for the failure analysis. Several works, such as [4, 7, 6], recognize that logs might lack any useful information to perform the analysis: as discussed, failures can go undetected by the logging mechanism. Unreported failures decrease the level of trust on log-based failure analysis, because they cause the overestimation of dependabil- ity attributes, e.g., availability, reduce the effectiveness of corrective actions performed by administrators, and lead to wrong insights into the behavior of the system.
As mentioned, the logging mechanism might detect errors that do not cause failures. Event logs are often conceived for debugging purposes rather than for failure analysis. For this reason, they contain many error indications on unreleased files or sockets, null pointers, and so on: these errors do not necessarily indicate that a failure has occurred. Figure 1.7 shows some example entries representing false failure indications. Examples have been extracted from the event log of Apache Web Server and TAO Open DDS, which will be discussed later in the dissertation. In all the cases the entries seem to report critical
1 - Example #1: Apache Web Server (ver 1.3.41) - 2
3 [ Tue May 26 0 0 : 0 5 : 5 3 2 0 0 9 ] [ error ] ( 9 ) Bad file d e s c r i p t o r : fcntl ( 4 2 0 , F_SETFD ,
4 F D _ C L O E X E C ) f a i l e d
5 [ Tue May 26 0 0 : 0 5 : 5 3 2 0 0 9 ] [ warn ] ( 9 ) Bad file d e s c r i p t o r : exec ( ) may not be safe
6
7 - Example #2: Apache Web Server (ver 1.3.41) -
8
9 [ Tue May 26 0 1 : 1 3 : 4 7 2 0 0 9 ] [ error ] [ client IP−ADDR ] r e q u e s t faile d : e r r o n e o u s
10 c h a r a c t e r s a f t e r p r o t o c o l s t r i n g : ∗
11 [ Tue May 26 0 1 : 1 3 : 4 9 2 0 0 9 ] [ error ] [ client IP−ADDR ] r e q u e s t faile d : e r r o n e o u s
12 c h a r a c t e r s a f t e r p r o t o c o l s t r i n g : ∗
13 [ Tue May 26 0 1 : 1 3 : 5 1 2 0 0 9 ] [ error ] [ client IP−ADDR ] r e q u e s t faile d : e r r o n e o u s
14 c h a r a c t e r s a f t e r p r o t o c o l s t r i n g : ∗
15
16 - Example #3: TAO Open DDS (ver 0.9) -
17
18 ( 2 8 5 7 0 | 3 0 7 7 0 1 6 4 9 6 ) E R R O R C a c h e d _ A l l o c a t o r _ W i t h _ O v e r f l o w : : f r e e b 5 d 1 3 f 1 c m o r e d e l e t e s
19 1 than al locs 0 to the heap
20 ( 2 8 5 7 0 | 3 0 7 7 0 1 6 4 9 6 ) E R R O R C a c h e d _ A l l o c a t o r _ W i t h _ O v e r f l o w : : f r e e 8 1 6 6 2 2 4 m o r e d e l e t e s
21 1 than al locs 0 to the heap
22 ( 2 8 5 7 0 | 3 0 7 7 0 1 6 4 9 6 ) E R R O R C a c h e d _ A l l o c a t o r _ W i t h _ O v e r f l o w : : f r e e b 5 d 0 b e b 4 m o r e d e l e t e s
23 1 than al locs 0 to the heap
Figure 1.7: Examples of false positives entries in the log.
conditions preventing the correct execution of the program. Errors involve OS and applica- tion data structures, e.g., files and buffers (Example #1 and #3, respectively), and client requests (Example #2). In particular, in the second example taken from Apache, the mes- sage in the log explicitly states that the HTTP request forwarded by the client has failed. Nevertheless, it has been observed that, in all the cases, the systems were able to provide correct service despite the notification in the log: in this cases, the information provided by the event log leads to erroneous conclusions.
The detection ability of the logging mechanism is of paramount importance to analysts, because, in most of cases, the event log is the only available source of data to achieve insights into the failure behavior of the system: misleading information in the log, i.e., in terms of missing or false failure notifications, can heavily distort the analysis and related results.
False failure notifications must be filtered out before the analysis; however, this task is hard when no further information, such as the ground truth (i.e., the knowledge of the actual failure behavior of the system), is available to supplement the content of existing logs.
The logging mechanism might report a single failure with multiple notifica- tions in the log. The nature of the workload and error propagation phenomena, due to natural interactions among system components, cause one or more error detectors to be fired many times and, in most of cases, over short time intervals during the operational phase [4, 22]. For example, the analysis of a supercomputing system revealed that, an in- put/output failure caused by a damaged disk sector affecting a storage node, lead to 142 “unknown partition table” entries in the event log: these entries spanned a timeframe of 28s. Similarly, it was observed a case where a single failure affecting MySQL DBMS caused 5,421,413 entries in the log.