HPCC MSP HCR CGG 0 5 10 15 20 25 12 8 6 4 2 1 Co nverg ence T ime (10 3 s)
(a) Map A Convergence Time.
CGG CR MSP HPCC HCR 0 2 4 6 8 10 12 14 12 8 6 4 2 1 Convergence T ime (10 3 s)
(b) Map B Convergence Time.
HCR CR HPCC MSP CGG 0 1 2 3 4 5 6 7 8 12 8 6 4 2 1 Convergence T ime (10 3 s)
(c) Map C Convergence Time.
Figure 3.11: Average Convergence Time for each Algorithm with different team size.
fact, there is no apparent relation between performance and convergence time. Differences between the approaches are more marked with low number of robots as well as with different environment connectivity. Nevertheless, global trends can be observed. For instance, convergence time generally drops when team size increases from 8 to 12 robots.
In order to verify the significance of the problem’s parameters tested in the experiments, Analysis of Variance (ANOVA) [Scheff´e, 1959] was applied to mea- sure quantitatively the group’s variable effect. ANOVA is a powerful statistical
technique which enables the comparison between parameters of more than two populations. From the analysis of the total dispersion present in a data set, it allows us to identify the source of the variations that led to that dispersion and evaluate the contribution of each factor, determining whether a significant relation exists between variables. The experiments presented in this work consider three factors (algorithm, team size and connectivity), thus the analysis of variance used to study their effects is called a three-way ANOVA.
Linear models were considered, assuming that the probability distribution of the response is normal, mutually independent and homoscedastic, i.e., the variance of the data inside the groups is equivalent. The ANOVA fundamental test, which makes use of the F-statistics distribution usually with 95% of confidence bounds, verifies the significance of the factors by checking the group’s variable effect (αi):
H0 : α1= α2= ... = αI = 0, (3.4)
H1 : ∃αi 6= 0, (3.5)
where H0 is the null hypothesis and H1 is the alternative hypothesis. If the proba- bility of the null hypothesis is near 0, a main effect is present due to the associated factor, meaning that the result is statistically significant. Arbitrarily high n-way ANOVA divides the total variation, given by the deviation of all observations from the global mean, into variations given by different factors and residual variation (or error). Also, if the model has a non-additive effect, variations given by the interaction of factors are also considered. These variations are calculated through sums of squares.
In this work, a test using the F-statistics distribution with 95% of confidence bounds and a model with first-order interaction effects between pairs of factors was adopted. This model explains 99.75% of the variation of the results. The ANOVA table, presented in Table 3.6, illustrates the model used.
3.4. Evaluation and Discussion 73
Table 3.6: ANOVA Table.
Source SS dof M S F P rob > F
Algorithm 64953.125 4 16238.281 14.603 0 Team size 12763618.172 5 2552723.634 2295.671 0 Connectivity 2574860.148 2 1287430.074 1157.789 0 Algorithm*Team size 36814.014 20 1840.701 1.655 0.0891 Algorithm*Connectivity 120574.630 8 15071.829 13.554 0 Team size*Connectivity 1279680.351 10 127968.035 115.082 0 Error 42254.968 38 1111.973 - - Total 17234127.010 87 - - -
The only factor that presents no relevant significance is the algorithm*team size interaction, seeing as the null hypothesis was accepted:
F –testAlg∗T S =M SAlg∗T S
M SE =
1840.701
1111.973 = 1.655, (3.6) 1.655 < F20,38(α = 0.05) ' 1.85. (3.7)
In fact, a clear indication of the low significance of this interaction is given by the IG values of Tables 3.3-3.5, which do not differ much when each of the associated columns are compared as a whole. In addition, analyzing the individual factors, it can be seen that the influence of team size and connectivity in the results is greater than in the algorithm case. As a consequence, the interaction factor between team size and connectivity is the most significant interaction. Figure 3.12a depict the approximately additive effect of interaction team size-algorithm and the non-additive effect of the interaction team size-connectivity is presented in Figure 3.12b.
A more complete model could eventually be obtained by considering second- order interaction of the three factors. However, this interaction is only responsible for the remaining 0.25% variation of the results.
These results are the natural evidence that performance relies heavily on the number of members in the team and the environment to patrol.
0 200 400 600 800 1000 1200 1400 1600 1800 2000 CR HCR HPCC CGG MSP Algorithm Team size = 1 Team size = 2 Team size = 4 Team size = 6 Team size = 8 Team size = 12 I𝓖
(a) Approximate additive effect of the algorithm*team size factor.
I𝓖 0 200 400 600 800 1000 1200 1400 1600 1800 2000 1 2 4 6 8 12 Team size Map A Map B Map C
(b) Non-additive effect of the team size*connectivity factor.
Figure 3.12: In a) the factors algorithm and team size present an approximate additive effect, which is clear by the near parallel curves. In this chart, the envi- ronment used was map A. In b) the factors team size and connectivity present a clear non-additive effect. In this chart, the algorithm used was HCR.
3.5 Summary
In this chapter, a study of the scalability and performance of five different multi- robot patrolling strategies was presented. This study is unprecedented in this field because it overcomes many limitations and simplifications of previous works by using: generic environments with different topological connectivity properties and weighted edges; realistic simulations that consider the robots’ dynamics; and is
3.5. Summary 75
based on the actual time in its performance metric instead of atomic iterations or simulation cycles. It was shown that different types of algorithms perform differently according to the environment and the number of robots running the patrol task. Consequently, the choice of a patrolling strategy for teams of multiple robots should take into consideration these two important parameters.
Quantitative analysis through three-way ANOVA was used to understand the significance of the different factors involved in the problem. The results presented were somehow unforeseen, given that the patrolling algorithms proved to be much less significant as a factor than team size and connectivity of the environment, which represents an important conclusion to this field.
Therefore, upcoming work should be guided towards approaches that ensure scalability and are appropriate, or perhaps can adapt, to all kinds of environment in order to improve the team’s performance and minimize interference between robots.
Beyond covering the benchmarking objective with realistic assumptions and the objective of evaluation of the scalability of diverse strategies in different topolo- gies, conducting simulation experiments in well-known platforms like ROS and Stage and making the code used in these simulations publicly available, repre- sents a step forward, towards the objective of presenting a standard environment for researchers all over the world to implement different strategies and precisely compare with previously existing ones.
In the next chapter, the focus is shifted so as to validate multi-robot exper- imentation in real-world scenarios, using a distributed coordination architecture and aiming to minimize the interference that arises in teams with large numbers of robots, as well as studying robustness against robot failures and communication errors.