Técnicas e instrumentos de recopilación de datos, validez y confiabilidad

The application of the Roth-Erev reinforcement learning algorithm for choosing the best bid option is presented in this sub-section.

3.6.2.1. Context

The main reasoning behind a reinforcement learning algorithm is that the tendency to perform an action should be strengthened, or reinforced, if it produces favourable results and weakened if it produces unfavourable results.

The main principle behind a reinforcement learning algorithm is that, not only are choices that were successful in the past more likely to be employed in the future, but similar choices will be employed more often as well. This is referred to as law of effect [Erev and Roth, 1998].

Roth and Erev [Roth and Erev, 1995] take this law of effect principle, widely accepted in the psychological learning literature, as the basic starting point in their search for a robust model of individual learning, i.e. the understanding of how people learn individually to behave in games with multiple strategically interacting players.

Roth and Erev argue that, in such contexts, the law of effect is not sufficiently able to support the learning process;

therefore they have also included in their studies an additional learning principle, also widely accepted in the psychological learning literature, which they refer to as the power law of practice [Jing et al., 2009]. This principle is based on the latter principle, which suggests that learning curves initially tend to be abrupt, and after some time they flatten out.

In order to experiment on each of these learning principles’ responsibility over a person’s learning process, Roth and Erev have developed a reinforcement learning algorithm, named as the Roth-Erev algorithm [Roth and Erev, 1995].

This algorithm’s way of testing such principles uses an experimentation or recency parameter, which defines the weight that the past experience will have on a subject’s learning process [Nicolaisen et al., 2001]. Roth and Erev show that this algorithm is able to successfully track the observed intermediate-term behaviour of human subjects over a wide variety of multiagent repeated games with unique equilibrium achievable, using stage-game strategies.

3.6.2.2. Implementation

Following the principles proposed by Roth and Erev [Roth and Erev, 1995], the Roth-Erev Reinforcement Learning Algorithm has been implemented to support the reinforcement learning algorithm’s purpose in the scope of ALBidS.

The differences from this methodology to the Simple Reinforcement Learning Algorithm concern the inclusion of the recency parameter, which defines the importance of the past experience to the evolution of the learning process. This way, taking advantage on this parameter, it is possible to define if, for each case, it will be attributed a higher importance for the recent events, adapting faster to the changing environment; or on the other hand, if the

accumulated past experience, and the achieved results in the past, will be the most influent factor in defining the confidence in an action’s performance [Vale et al., 2011a].

The main procedure behind this algorithm is very similar to the Simple Reinforcement Learning Algorithm’s.

The reinforcement calculation is done in the same way (as presented in equation (3.2)); and similarly, the same value of confidence in assigned for each strategy (action) at the moment of initialization. As the simulation progresses the values of confidence are updated for each context, through the application of the strategies’

performance based reinforcement. The application of this reinforcement, in the Roth-Erev Reinforcement Learning Algorithm’s case, concerning the recency parameter r, is done according to (3.3).

C_t+1=C_t× +r Rein f × −

(

)

As can be seen by the r parameter’s inclusion, which assumes values between 0 and 1, the higher this value is, the higher is the importance that will be given to the consideration of the previous days’ confidence value. On the other hand, if this value is small, the accumulated experience from the past days will vanish faster, and the new reinforcement for the current case will present a much higher influence over the new confidence value that is being updated.

The Roth-Erev Reinforcement Learning Algorithm also allows the attribution of a weight value for each strategy, reflecting the user’s preference for a strategy’s importance to the system.

3.6.2.3. Experimental Findings

The testing of the Roth-Erev Reinforcement Learning Algorithm’s performance includes the presentation of two simulations, concerning different values of the recency parameter, so that conclusion can be taken about the influence of this parameter, and the consequent variation in terms of results. Each of the tests considers a different simulation for fifteen consecutive days, including three seller agents, for whom the results are analyzed, and seven buyer agents. The simulations were performed considering solely the first period of the day, in order to ease their results comprehension. The proposed bids from the considered seller agents in both simulations are presented in Figure 3.26.

In the first test the recency parameter is assigned a value of 0.2, a small value to enhance the influence of recent events. Regarding the second simulation, the recency parameter assumes a value of 0.8, a large value, so that the past experience is taken into higher consideration for this case. Figure 3.27 presents the confidence values for each strategy throughout both simulations. Note that, similarly to the Simple Reinforcement Learning Algorithm’s case, the confidence value reflects the confidence in a strategy’s failure, so the smaller this value is, the better the proposal is expected to be.

a)

b)

Figure 3.27 – Seller agents’ failure confidence values for a recency parameter’s value of: a) 0.2, b) 0.8.

Analysing Figure 3.27 it is visible that for a higher importance for recent events (Figure 3.27 a)), the variations of the confidence values are highly abrupt. A strategy presenting the best confidence value for one day, may present the worst in the following, in case of its results being ineffective for a single case. Regarding Figure 3.27 b), concerning the case where the higher importance is attributed to the past experience, the confidence values’

evolution is much smoother, since a good or bad result in one day does not present such an higher influence over the confidence value’s update.

Figure 3.28 presents the results, in incomes, achieved by ALBidS use of the strategy that presented better results (Average 1), in both simulations.

a)

b)

Figure 3.28 – Average 1’s achieved incomes for a recency parameter’s value of: a) 0.2, b) 0.8.

These results are obvious in what regards the difference in achieved results by the Average 1 strategy. The higher volatility in the confidence values for the first simulation reflects a higher uncertainty in what regards the achieved incomes. This uncertainty is smothered by the considering of a higher importance for past experience, which resulted in a higher consistency of results for this strategy, for this case.

The balance between past experience and recent results is an important issue to be dealt with when defining the preferences for the execution of the Roth-Erev Reinforcement Learning Algorithm. These tests showed that for this case the past experience’s higher weight proved to be more advantageous; however, this is a fact that may depend on the circumstances and on the context of the market negotiations.