ECUACIÓN GENERAL DE LA ENERGÍA

This section illustrates the assessment of the performance of GOAC taking advantage of OGATE for performing in- tensive automated tests for generating and executing plans

1

DALA software simulator courtesy of Felix Ingrand and Lavindra De Silva from LAAS-CNRS.

the previous section. Namely, OGATE is exploited to as- sess the capability of the GOACsystem to successfully con- trol the platform in the given domain but also to measure how different degrees of uncertainty affect the control be- haviors. To objectively analyze this, the methodology pre- sented above is exploited to characterize and compare the performance of the GOAC system when modifying the be- haviors of the deliberative component. In this regard, the as- sessment objectives and the scenarios over which test the GOACsystem are to be defined as a first step.

Here, the evaluation objective is to analyze the perfor- mance of the GOAC system focusing on how the two dif- ferent planning policies of the deliberative component affect the controller performance. In particular, in order to assess the implementation of a sense-plan-act cycle in the GOAC

controller, the evaluation is to focus on the time spent by the deliberative reactor while elaborating sensor data, gen- erating plans and dispatching commands to the Command Dispatcher. In this regard, the performance analysis should be performed considering different problems and execution scenarios in order to assess how they affect the system from a high level perspective.

Then, different planning/execution scenarios are consid- ered by varying the complexity of the robotic planning prob- lem dimensions and the execution conditions. In particu- lar, the following elements are considered: (1) Plan Length. Problem instances are considered with an increasing num- ber of requested pictures (from 1 to 3). (2) Plan Flexibility. For each uncontrollable activity (i.e., robot and PTU move- ments as well as camera and communication tasks), a min- imal duration is set, but temporal flexibility on activity ter- mination is considered, i.e., the end of each activity presents a tolerance ranging from 0 to 10 seconds. This interval rep- resents the degree of temporal flexibility/uncertainty intro- duced in the system. (3) Plan Choices. A number of visibil- ity windows spanning from 1 to 4 is considered as increasing opportunities to communicate picture contents. Increasing the number of communication opportunities raises the com- plexity of the planning problem with a combinatorial effect. In general, among all the generated problems instances, the ones with higher number of required pictures, higher tempo- ral flexibility, and higher number of visibility windows result as the hardest.

To complete the evaluation design step, metrics are to be defined according to the evaluation objectives, i.e., assess the sense-plan-act cycle. In this regard, the metrics to be analyzed are the following: State processing time: the time required by the deliberative reactor to analyze the observa- tions collected from the Command Dispatcher (sense phase). Deliberation time: the time spent by the deliberative reactor to generate a solution plan for the considered goals (plan phase). Operational time: the time spent by the deliberative reactor to dispatch commands for the other reactor to ac- complish its own high-level goals. For the above metrics, the following ranges (in seconds) have been considered: [0, 4] for the state processing time; [0, 10] for the deliberation timeand; [0, 20] for the operational time. The ranges for the

metrics have been obtained analyzing the results of differ- ent executions of the GOAC architecture in the considered scenarios. In the evaluation, all the metrics have the same weights. Finally, in order to evaluate how the different sce- narios described above affect the GOAC performance, the quadrants of the circular charts have been set to present the three metrics considering an increasing number of commu- nications opportunities.

In this work, we focus our attention on the metrics defined above but, in general, OGATE allows also the definition of additional metrics such as number of generated goals, mean processing time per goal, number of observations received and processed, etc. that can be considered within the evalu- ation process with no additional costs but their definition in the system.

In order to perform the tests execution, a suitable GOAC

plugin for OGATE has been implemented and adapted in order to monitor and store all the relevant performance in- formation from the internal components of the controller (again, see (Mu˜noz et al. 2014) for further details). Then, OGATE has been exploited to (i) generate the considered scenarios, (ii) carry out all the different controller executions and (iii) collect performance data from the GOAC. For each execution setting, 10 runs have been performed and average values for the defined metrics are reported.

After the collection of performance information in all the considered scenarios, OGATE is able to generates a report containing a wide set of charts corresponding to different control configurations, planning problem instances and ex- ecution settings. Figure 3 shows an excerpt (related to the 3 pictures scenarios) of the charts provided by the OGATE report.

Assessing the information shown in the reports, a first straightforward evidence that can be elicited observing the charts in Figure 3 is that the controller execution is not com- pleted in all the considered scenarios (see Fig. 3-d-e-f in the case of 3 and 4 communication windows). In some cases, even though the deliberative module is able to generate a valid plan, the GOAC controller fails in properly complet- ing its execution. After some further analysis of that specific scenarios, an issue related to a dynamic controllability prob- lem (Morris and Muscettola 2005) in the execution of the corresponding plans has been identified.

For what concerns the planning policy comparison, data in Fig. 3 shows that the all goals policy is unable to solve the scenarios with 3 pictures and 3 communication windows for all flexibilities and the ones with 5 and 10 seconds flexi- bility and 4 communication windows. Meanwhile, the single goalpolicy can solve all scenarios, experiencing execution failures with 3 pictures, 5 seconds flexibility and 4 commu- nication windows, and also with 10 seconds flexibility and 3 and 4 communication windows. It is possible to observe that, the all goals policy usually has the best operational time val- ues while for the state processing time the better scores cor- responds to the single goal policy. This is a consequence of how the deliberative reactor dispatches the goals: in the sin- gle goalpolicy, it dispatches goals one by one, increasing the time spent in this task (operational time), while the all goals policydo it just once. In opposition, the single goal

policy requires only to check the states for only one goal (state processing time), managing a shorter list of expected states, while the all goals policy must manage a list with the expected states for all pictures. Focusing on the central area, we can observe the average execution time in the circular bar. The time is expressed as seconds/degree, and computed considering both the correct executions and the failing ones (those who reach a timeout of 5 minutes). In the center is presented the Global Score that summarizes the metrics into a single value. Considering only this value we can see that increasing the flexibility, the performance of the system de- creases for both planning policies. Also, the single goal pol- icy shows a significative better GS than the all goals policy. Further investigating the values for the three metrics in the 3 pictures scenarios, additional consideration can be in- ferred. First, focusing on the deliberation time, both policies have a (close to) constant time for each number of pictures. Anyway, the single goal policy requires less time to gener- ate the plans, being remarkable the increment of the required time for 3 pictures with the all goals policy. Considering the difference between the time spent to generate the plan for 1, 2 and 3 pictures employing the single goal policy, we observe that the time is not proportional to the number of pictures (the average values for 1, 2 and 3 pictures are 0.61, 1.32 and 2.06 seconds respectively), which is an indication of the possible presence of an anomaly behavior (as every picture is planned independently, it is expected to take the same time for planning). A further investigation has been then performed considering the single goal policy in a sce- nario with 0 seconds flexibility and with 2 communications windows aiming to acquire 5 pictures. A temporal profile has been generated and it is presented in fig. 4 where the X-axis represent the tick count and the Y-axis is the time measured in seconds, and the filled area is the time spent de- liberation for each picture. The temporal profile is also part of the data generated by OGATE, and allows to observe that the single goal policy presents an important issue when con- sidering an incremental number of goals. In fact, it fails in obtaining the plan for the last picture in this scenario as it requires an increasing amount of time for processing addi- tional requested pictures. Then, it has been possible to figure out that while processing additional goals, the OMPSplan- ner performs some checks about past constraints, which are not relevant for the current planning but strongly increase the time spent in planning.

Figure 4:Temporal profiling for the single goal policy in a sce- nario with 5 pictures.

(a) Single goal with 0 secs flexibility. (b) Single goal with 5 secs flexibility. (c) Single goal with 10 secs flexibility.

(d) All goals with 0 secs flexibility. (e) All goals with 5 secs flexibility. (f) All goals with 10 secs flexibility.

Figure 3:GOACevaluation for 3 pictures with different planning policies and temporal flexibilities of the deliberative component. formance can be performed by taking advantage of the in-

formation provided by the OGATE report. Due to space is- sue, we report here only a quick overview of the main re- sults. For 1 and 2 communication windows, all the scenarios are correctly completed with both the planning policies. For 3 communications windows, the all goals policy is able to complete all scenarios for 2 pictures but it fails to solve the 3 pictures scenarios. The single goal policy solves all sce- narios for both 2 and 3 pictures (except 3 pictures and 10 secs flexibility) where it fails to execute the scenario 30% times. With 4 communications windows, the all goals pol- icy has a worse performance: it solves only 30% and 60% of the scenarios for 2 pictures with 5 and 10 seconds flex- ibility respectively, and 100% for the scenario with 3 pic- tures without flexibility. The single goal policy solves all scenarios without flexibility; a half in the case of flexibility 5, and, with flexibility 10, it achieves a successful ratio of 90% and 70% for 2 and 3 pictures respectively. As a final consideration, a general observation is that the exploitation of the single goal policy in the GOACarchitecture seems to be more suitable to address all the considered scenarios but it is affected by the issue shown in Fig. 4. On the other hand, the all goals policy allows to overcome such issue but does not provide good performance when facing scenarios with more than two communication windows. So, according to the gathered results, a suitable trade off for deploying the

GOACarchitecture seems to be the configuration with single goalpolicy when considering a short look-ahead for accept- ing new picture requests.

Finally, It is worth underscoring how performing the same empirical evaluation without the OGATE support consti- tutes a significant effort in terms of coding work, customi- sation of specific metrics to the considered control archi- tecture, collection of performance information and genera- tion of synthetic reports. By using OGATE, the main ef- fort required is related to the implementation of the plugin software for the specific autonomous controller. Thus, the OGATE framework constitutes an off-the-shelf tool capable of performing in automated manner a significant amount of work: that includes the scenarios definition, the tests execu- tion, the collection of information and the report generation.