Resource potential analysis. Suitable areas for ATES systems

Our basic aim is to provide complexity figures (perhaps in the O notation) in terms of the input size, and not as a function of any particular input. So far we have counted the maximum possible number of basic operations that need be executed by a program or function. As an example, consider the linear search algorithm. If the element x happens to be the first element in the array, the function linSearch returns after performing only few operations. The farther x can be located down the array, the bigger is the number of operations. Maximum possible effort is required, when x is not at all present in the array.

We argued that this maximum value is O(n). We call this the worst-case complexity of linear search.

There are situations where the worst-case complexity is not a good picture of the practical situation. On an average, a program may perform much better than what it does in the worst case. Average complexity refers to the complexity (time or space) of a program (or function) that pertains to a random input. It turns out that average complexities for some programs are markedly better than their worst-case complexities. There are even

examples where the worst-case complexity is exponential, whereas the average

complexity is a (low-degree) polynomial. Such an algorithm may take a huge amount of time in certain esoteric situations, but for most inputs we expect the program to terminate soon.

We provide a concrete example now: the quick sort algorithm. By partition we mean a partition function for an array of n integers with respect to the first element of the array as the pivot. One may use any one of the three implementations discussed above.

void quickSort ( int A[] , int n ) {

int i;

if (n <= 1) return;

i = partition(A,n); /* Partition with respect to A[0] */

quickSort(A,i); /* Recursively sort the left half excluding the pivot */

quickSort(&A[i+1],n-i-1); /* Recursively sort the right half */

}

Let T(n) denote the running time of quickSort for an array of n integers. The running time of the partition function is O(n). It then follows that:

T(n) <= T(i) + T(n-i-1) + cn + d

for some constants c and d and for some i depending on the input array A. The presence of i on the right side makes the analysis of the running time somewhat difficult. We cannot treat i as a constant for all recursive invocations. Still, some general assumptions lead to easily derivable closed-form formulas for T(n).

An algorithm like quick sort (or merge sort) is called a divide-and-conquer algorithm.

The idea is to break the input into two or more parts, recursively solve the problem on each part and subsequently combine the solutions for the different parts. For the quick

sort algorithm the first step (breaking the array into two subarrays) is the partition problem, whereas the combining stage after the return of the recursive calls involves doing nothing. For the merge sort, on the other hand, breaking the array is trivial -- just break it in two nearly equal halves. Combining the solutions involves the non-trivial merging process.

It follows intuitively that the smaller the size of each subproblem is, the easier it is to solve each subproblem. For any superlinear function f(n) the sum

f(k) + f(n-k-1) + g(n)

(with g(n) a linear function) is large when the breaking of n into k,n-k-1 is very skew, i.e., when one of the parts is very small and the other nearly equal to n. For example, take f(n) = n². Consider the function of a real variable x:

y = x² + (n-x-1)² + g(n)

Differentiation shows that the minimum value of y is attained at x = n/2 approximately.

The value of y increases as we move more and more away from this point in either direction.

So T(n) is maximized when i = 0 or n-1 in all recursive calls, for example, when the input array is already sorted either in the increasing or in the decreasing order. This situation yields the worst-case complexity of quick sort:

T(n) <= T(n-1) + T(0) + cn + d = T(n-1) + cn + d + 1

<= (T(n-2) + c(n-1) + d + 1) + cn + d + 1 = T(n-2) + c[n + (n-1)] + 2d + 2

<= T(n-3) + c[n + (n-1) + (n-2)] + 3d + 3 <= ...

<= T(0) + c[n + (n-1) + (n-2) + ... + 1] + nd + n = cn(n-1)/2 + nd + n + 1,

which is O(n²), i.e., the worst-case time complexity of quick sort is quadratic.

But what about its average complexity? Or a better question is how to characterize an average case here. The basic idea of partitioning is to choose a pivot and subsequently break the array in two halves, the lesser mortals stay on one side, the greater mortals on the other. A randomly chosen pivot is expected to be somewhere near the middle of the eventual sorted sequence. If the input array A is assumed to be random, its first element A[0] is expected to be at a random location in the sorted sequence. If we assume that all the possible locations are equally likely, it is easy to check that the expected location of the pivot is near the middle of the sorted sequence. Thus the average case behavior of quick sort corresponds to

i = n-i-1 = n/2 approximately.

We than have:

T(n) <= 2T(n/2) + cn + d.

For simplicity let us assume that n is a power of 2, i.e., n = 2^t for some positive integer t. But then

T(n) = T(2^t)

<= 2T(2^t-1) + c2^t + d

<= 2(2T(2^t-2) + c2^t-1 + d) + c2^t + d = 2²T(2^t-2) + c(2^t+2^t) + (2+1)d <= 2³T(2^t-3) + c(2^t+2^t+2^t) + (2²+2+1)d <= ...

<= 2^tT(2⁰) + ct2^t + (2^t-1+2^t-2+...+2+1)d = 2^t + ct2^t + (2^t-1)d

= cnlog2n + n(d+1) - d.

The first term in the last expression dominates over the other terms and consequently the average complexity of quick sort is O(nlog n).

Recall that bubble sort has a time complexity of O(n²). The situation does not improve even if we assume an average scenario, since we anyway have to make O(n²)

comparisons in the nested loop. Insertion and selection sorts attain the same complexity figure. With quick sort, the worst-case complexity is equally poor. But in practice a random array tends to follow the average behavior more closely than the worst-case behavior. That is reasonable improvement over quadratic time. The quick sort algorithm turns out to be one of the practically fastest general-purpose comparison-based sorting algorithm.

We will soon see that even the worst-case complexity of merge sort is O(nlog n). It is an interesting theoretical result that a comparison-based sorting algorithm cannot run in time faster than O(nlog n). Both quick sort and merge sort achieve this lower bound, the first on an average, the second always. Historically, this realization provided a massive impetus to promote and exploit recursion. Tony Hoare invented quick sort and popularized recursion. We cannot think of a modern compiler without this facility.

Also, do you see the significance of the coinage divide-and-conquer?

We illustrated above that recursion made the poly-time Fibonacci routine exponentially slower. That's the darker side of recursion. Quick sort and merge sort highlight the brighter side. When it is your time to make a decision to accept or avoid recursion, what will you do? Analyze the complexity and then decide.