Antonio Marquet, especialista drag queen

The problem of detection is the first step in performance diagnosis. We define detection as identification of significant deviations from baseline end-to-end performance. Suppose that we have a timeseries of active probing measurements from a sender S to a receiver R; we assume without loss of generality that the timestamps at S and R are taken at user

2

The interval durations can be adjusted based on the sampling rate for a path. We chose the durations based on the 10Hz sampling used in perfSONAR.

space. Under normal conditions, three invariants hold true for a timeseries of end-to-end measurements:

1. The end-to-end delays are close to (with some noise margin) the sum of propagation and transmission delays along the path,

2. No packets are lost, and

3. There is no reordering of packets (as received at R).

Invariant (1) assumes that the sampling process does not induce self-queueing effects; we address this in the detection pre-processing stage below.

Based on the invariants, we implement three types of performance problem detection:

delay, loss and reordering detection, depending on the baseline invariant that is violated.

At a high level, the detection logic divides the timeseries into non-overlapping back-to-back

windows of 5s duration3, and flags these windows as either “baseline”, or as “problem” (i.e., violating one or more invariants). The logic then aggregates “problem” windows into detection output.

Delay detection: Delay detection identifies significant deviations from baseline end- to-end delays. We use the delay invariant condition (1) above; in other words, under normal conditions, the distribution of delay samples in the window will be unimodal with most of the density concentrated close to the sum of propagation and transmission delays dminalong the path. If there is a deviation from the delay baseline, the delay distribution will either be multimodal with a low density around dmin, or will be unimodal with a large range of delays under the mode - depending on the “bin width” used.

We use a nonparametric kernel smoothing density estimate [111] to analyze modes in the delay distribution. A smooth density estimate will allow us to locate modes (local maxima), the start and end points of a mode (local minima), and the density inside a mode with a single pass on the density function. We use a Gaussian kernel; it gives us a smooth estimate since it is a continuous function. A kernel density estimator also requires a bandwidth

3

Algorithm 1 Delay detection logic.

Input: Time series of e2e one-way delays (ordered by send time): (ti, di); minimum prob- lem delay threshold τd, minimum problem duration threshold τl; and window merge gap thresholdτm.

Output: Set of disjoint time windows P in which there was a problem. 1. Initialize P ← ∅.

2. For each i: if packets i and i + 1 were sent close to each other (ti+1− ti < 100us), set di+1← di.

3. Divide timeseries into back-to-back windows Wj of duration 5s such that window Wj contains delays of all packets sent in [t0+ 5j, t0+ 5j + 5).

4. For delay sample in window Wj:

(a) Sample kernel density ˆfh for delays d in the sample range spaced by 100ms: ˆ fh(d) = 1 n X i

Kh(d − di), where Kh is a Gaussian kernel, and bandwidth h = 0.79_n0I.2, where I is the inter-quartile range of delays in Wj and n =| Wj |. (b) Compute the baseline: Traverse ˆfh to find the smallest-delay local maxima



db, ˆfh(b) 

, and the width (delay range) of the “hill” wb that contains this max- ima. When there is no problem, db is an estimate of the baseline delay. Suppose that the density of delays contained in this “hill” is αb.

(c) If αb < 0.4 or wb > 2ms, and if I > τd: add window Wj to P; set j → j + 1. 5. Aggregate: If windows Wj and Wk are in P, and if they are separated by less than

τm, delete them and add a single window encompassing Wj, Wj+1. . . Wkto P. Repeat until no merges are possible.

6. Delete windows from P that have a duration less than τl. 7. Return P.

estimate, which defines the variance of the Gaussian kernel and controls the “smoothness”. We use the Silverman’s rule of thumb to compute the bandwidth [111]. Algorithm 1 details our algorithm.

The delay detection algorithm works as follows. We pre-process the delay timeseries to: (1) account for delays induced by self-queueing effects by overwriting measured delays of probes sent less than 100us apart with that of the previous probe, and (2) check if the range of delays in the window is larger than an operator-defined delay threshold (5ms by default). The delay detection logic estimates the density function and computes the lowest

mode. The lowest mode is used as an estimate of the baseline delay for the window. The

detection module checks the density and width of the lowest mode to flag the window as “normal” or “problem”. The aggregation module merges “problem” windows separated by 5s or lower into one “problem” window. The delay detection module also keeps track of the previous window’s baseline for diagnosis of longer events.

The approach of detection by inferring a baseline and finding deviations from it has certain limitations. The baseline computation may sometimes be incorrect for long duration pathologies (e.g., a congestion event that creates a persistent backlog with low variability). In such cases, after processing a few windows, the baseline estimate would change to the new “level” of delays in the timeseries, although the new windows still include the performance problem. Hence, the new windows will not be diagnosed as a problem (until the delay timeseries start showing a baseline; for example, due to variability in the backlog). In practice, the aggregation module’s functionality of merging nearby problem windows may overcome this issue. Our focus in this work, however, is in detecting short-term events.

Loss detection: Loss detection identifies significant deviations from the baseline loss

invariant condition (2). Specifically, loss detection module flags a window as “problem” if

the window contains at least one lost packet. Lost packets are identified by missing sequence numbers when the agent reads the complete timeseries into memory. The aggregation module combines “problem” windows close to each other into a single “problem” window.

Reordering detection: Reordering detection identifies significant deviations from

baseline reordering invariant condition (3). The reordering module computes a reorder- ing metric R for each 5s window of sequence numbers in received order, {n1. . . nk}, based on the RD metric definition in RFC 5236 [59]. For each received packet i, it computes an

expected sequence number nexp,i; nexp is initialized to the lowest recorded sequence number in the timeseries, and is incremented with every received packet. If a sequence number i is lost, nexp skips the value i. A reorder sum is computed as (assume without loss of generality that the window starts with i = 1):

ηsum= k X

i=1

| ni− nexp,i |

The reordering module estimates the number of reordered packets in the window based on mismatch in the two sequence numbers:

η = k X i=1

I [ni6= nexp,i]

The reordering metric R for the window is defined as the ratio:

R =        (ηsum/η) η > 0 0 η = 0

Note that R is zero if there was no reordering, and R increases with the amount of reordering on the path. Our goal is not to estimate the number of reordered packets, but to get a quantitative measure of the extent of reordering for a window of packets. The reordering detection module flags a window as “problem” if R 6= 0 for that window. The aggregation module simply adds R to a “reordering timeseries”, which is used for reordering diagnosis. Sensitivity: Pythia provides a knob to the user to configure sensitivity of the detection module towards performance problems without defining any thresholds. This functionality reduces the number of events that the system reports to the user, while eliminating relatively

insignificant events. We use two heuristics to define sensitivity. First, the duration of the

event. A less sensitive detection module will only detect longer events. Second, the severity of the problem. This is defined in terms of the fraction of delays much higher than the baseline, the loss rate, or the value of the reordering metric (relative to a preset threshold) during the event.