In this experiment, all tasks occur simultaneously. We fix the number of sensors in the field and vary the number of tasks from 10 to 100. Each sensor has a cost, chosen uniformly at random from [0, 1], which does not depend on the task it is assigned to. This can represent the sensor’s actual cost in real money, the energy spent to operate the sensor or, e.g., a value indicating the risk of discovery if the sensor is activated. Each task has a budget, drawn from a uniform distribution with an average of 3 in the first experiment and varying from 1 to 10 in the second. As mentioned in Sec- tion 3.3.3, the static problem can be modeled as a combinatorial auction. The general CASS algorithm is not computationally tractable in our scenarios as the number of bids grows exponentially. Even for smaller scenarios its performance was inferior to the greedy and MRGAP algorithms. So, in the following results we only show the CASS algorithm with single-minded bidders which is more tractable. In the following
results, we show the average of 20 runs, which we found experimentally were neces- sary in order to get significative comparable results. In particular, by running each of the experiments 20 times we ensured a low standard deviation error (i.e. around ±2%).
10 20 30 40 50 60 70 80 90 100 Number of Missions 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
Profit (fraction of maximum)
Optimal Fractional MRGAP Greedy
Single-Minded CASS
Number of Tasks
Figure 3.2: Fraction of maximum profit achieved (250 nodes)
10 20 30 40 50 60 70 80 90 100 Number of Missions 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
Profit (fraction of maximum)
Optimal Fractional MRGAP Greedy
Single-Minded CASS
Number of Tasks
Figure 3.3: Fraction of maximum profit achieved (500 nodes)
The first series of results show the fraction of the maximum task profits (i.e., the sum of all task profits) achieved by the different schemes. We show the profits for the greedy algorithm, MRGAP, single-minded CASS, and an upper bound on the optimal value,
3.4 Performance Evaluation 74
running on two classes of sensor networks, sparse (250 nodes) and dense (500 nodes) (Figures 3.2 and 3.3, respectively). The upper bound on the optimal is obtained by solving the LP relaxation of program P, in which all decision variables are allowed to take on fractional values in the range [0, 1], and the profit is simply fractional based on satisfaction fraction; i.e., pjyjfor task Tj with no attention paid to the threshold T . The
actual optimal value will be lower than the fractional one.
The MRGAP scheme, which again can be implemented in a distributed fashion, achieves higher profits in all cases than does the greedy scheme, which is strictly centralized (be- cause tasks have to be ordered in terms of profit). The difference, however, is not very large. The single-minded CASS algorithm achieves lower profits in both settings. We note that with 500 nodes the network achieves higher profits, which is to be expected.
10 20 30 40 50 60 70 80 90 100 Number of Missions 0.2 0.25 0.3 0.35 0.4 0.45
Fraction of Budget Spent
MRGAP Greedy
Single-Minded CASS
Number of Tasks
Figure 3.4: Fraction of spent budget (250 nodes)
Figure 3.4 shows the fraction of the total budget each scheme spent to acquire the sensing resources assigned to it, in a network with 250 nodes. The MRGAP scheme achieves more profit than the greedy algorithm and spends a modest amount of addi- tional resources. The single-minded CASS algorithm spends the smallest fraction of the budget. The fraction of remaining budget is significant (more than 60% in almost all cases), which suggests either that successful tasks had higher budgets than they
1 2 3 4 5 6 7 8 9 10 Average Budget 0.4 0.5 0.6 0.7 0.8 0.9 1
Profit (fraction of maximum)
Optimal Fractional MRGAP Greedy
Single-Minded CASS
Figure 3.5: Varying the average budget (250 nodes)
could spend on available sensors or that unsuccessful tasks had lower budgets than necessary, preventing them from reaching the success threshold, and so their budgets were not spent. When the number of tasks is large, the unspent budget can be attrib- uted to the fact that there simply were not enough sensors due to high competition between tasks. We found that with 250 sensors and 30 tasks, on average about 82% of the tasks succeed and the remaining 18% fail. The successful tasks spend only 50% of their budget. Among the failed tasks, 50% fail because there are not enough resources for the tasks to reach the success threshold. In this case, on average, the available sensors provide only 24% of the demand, whereas 50% is needed for a task to be suc- cessful. The other 50% of the tasks fail because their budgets do not allow them to allocate enough resources to reach the success threshold. When the number of tasks is increased to 60, we found that on average 74% of the tasks succeed and the remaining 26% fail. Again, the successful tasks spend only 50% of their budget but we found that a higher percentage of the failed tasks, 67%, fail because there is not enough resources, i.e. due to competition.
We performed another set of experiments, in which the number of tasks was fixed at 50 and the average task budget varied from 1 to 10. Figure 3.5 shows the results for
3.4 Performance Evaluation 76
a network with 250 sensors. We observe that the achieved profit initially increases rapidly with budget size but slows as the influence shifts from budget limitations to competition between tasks. We observe the same pattern: MRGAP achieves highest profits followed closely by the greedy algorithm. The single-minded CASS algorithm comes in third.
Figures 3.6 and 3.7 show the same results when 500 nodes are deployed in the same area. We notice that more of the budget is spent in this case. Since there are more sensors, tasks are able to spend more of their budget and reach higher profits than when 250 sensors are deployed. We also note that the achieved profits are higher when we vary the amount of budget given to tasks. Again, this is because there are more sensors on which the tasks’ budgets could be spent.
10 20 30 40 50 60 70 80 90 100 Number of Missions 0.25 0.3 0.35 0.4 0.45 0.5 0.55
Fraction of Budget Spent
MRGAP Greedy
Single-Minded CASS
Number of Tasks
Figure 3.6: Fraction of spent budget (500 nodes)
As noted in Section 3.3, the static MSTA problem is NP-Hard and therefore it is worth considering also the performance of the allocation mechanisms in terms of computa- tional time. In Figure 3.8, we show that both MRGAP and Greedy provide timely solutions to our static MSTA problem model, while instead the combinatorial auction approach (Single-Minded CASS) displays the worst computational-time performance. These benchmarks are consistent with the theoretical computational time of the al-
1 2 3 4 5 6 7 8 9 10 Average Budget 0.4 0.5 0.6 0.7 0.8 0.9 1
Profit (fraction of maximum)
Optimal Fractional MRGAP Greedy
Single-Minded CASS
Figure 3.7: Varying the average budget (500 nodes)
gorithms presented and were measured on a Macbook Pro with a 2.4 GHz Intel Core 2 Duo processor and 4GB 667 MHz DDR2 SDRAM, running Mac OS X Snow Leopard (version 10.6.8). It is important to note that in the implementation of these algorithms we did not focus on optimizing the code in terms of computational runtime, and there- fore implementation improvements may be possible although the general trend should not change. In these experiments, we fix the number of sensors on the field to 250 and we increase gradually the number of tasks from 10 to 100; locations of sensors and tasks on the field, profits, utility demands and budgets were randomly generated with the same parameters and distributions used in the first set of experiments (shown in Figure 3.2 and Figure 3.4). The time performance displayed by the greedy algorithm is the most desirable, being almost constant in the range of 10-20 ms. Instead, the computational time for MRGAP grows sub-linearly with the number of tasks on the field and falls in the range from 300 ms (10 tasks) to 1300 ms (100 tasks). Although MRGAP takes longer than Greedy to converge to a solution, its performance is still very desirable compared to the Single-Minded CASS approach, which stays within the range from 400 ms (10 tasks) to 3400 ms (100 tasks) and grows with a steeper inclina- tion. Note that the standard deviation for the time measurements taken is around ±7 ms