OTROS PASIVOS FINANCIEROS

This section introduces three different strategies to change the weights of each heuristic and heuristic modifier during the timetable construction. As reported in Chapter 3, the heuristics are changed during the timetable construction by alternating the graph colouring heuristics in the list. The alternation of graph colouring heuristics is based on the improvement to the solution quality. Due to the different number of violated examinations detected during examination assignment when using different heuristics, it is advantageous to change dynamically the examination weights during the examinations assignment. In this case, the number of violated examinations could be varied and different examination ordering could be obtained when a different heuristic modifier is incorporated. To simplify implementation, the weight combinations are divided into four main groups related to the contribution of current heuristics, namely, highLD, highSD, highHM and balance. The weight combinations of each group are illustrated in Table 4.3 and there are determined adhocly. The three strategies introduced in this section adaptively change the weight values for each heuristic and are discussed in the following subsection. It can be noted that these strategies are employed only with the combination of LDSDHM tested with the dynamic heuristic modifier and top-window approach.

Table 4.3: The grouping of different weight combinations for LDSDHM

Group Weight Combination

highLD (0.8, 0.1, 0.1), (0.7, 0.2, 0.1), (0.7, 0.1, 0.2) highSD (0.1, 0.8, 0.1), (0.1, 0.7, 0.2), (0.2, 0.7, 0.1) highHM (0.1, 0.1, 0.8), (0.1, 0.2, 0.7), (0.2, 0.1, 0.7) balance (0.3, 0.3, 0.4), (0.4, 0.3, 0.3), (0.3, 0.4, 0.3)

4.1.4.1 Dynamic Weights

This strategy changes the weight combinations based on the cost of each examination assignment. If the assignment of an examination incurred some penalty cost, the weight combination is changed in order to find a lower penalty cost than the current examina-tion. Thus, the new examination ordering based on the chosen weight combination is obtained. A new difficult examination is chosen and evaluated whether or not it should be accepted for the time-slot assignment based on penalty cost. The implemented strat-egy is described in Algorithm 9 below. Let G be the set of heuristic groups. During the

iteration, the process is started by choosing the weight combination randomly within HighSD. The weight from the HighSD is chosen because, as explained in Chapter 3, the saturation degree could give better ordering compared with other graph colouring heuristics. It can be observed that the weight combination from the HighSD group still utilised the normalised value of the largest degree heuristic and heuristic modifier but with a smaller contribution.

The penalty cost is calculated for an examination found to have the best time-slot. If the best found time-slot incurred some penalty cost, then the weight combination is changed by choosing randomly from the next successive heuristic group in the list in G.

At this stage, only the dif f iculty scorei(t) of examinations within the top-window size is calculated and ordering is performed only for examinations within the top-window size in order to reduce the running time. After calculating the normalised value of examination i and reordering of examinations, the examination with the highest dif f iculty score is taken for further analysis. The process continues until zero penalty cost is found.

Once all G has been visited and still no zero penalty cost is found, then the size of the top-window is doubled and the weight changes are repeated once more within the heuristic group in G. If still no zero penalty cost is found, then the examination with the lowest penalty is chosen and is assigned to the best time-slot. The time-slot assignment is implemented as described in Chapter 3 where the least penalty time-slot is chosen for the time-slot assignment. If more than one least penalty time-slot is found then they are chosen randomly.

4.1.4.2 Linear Weights

The strategy is to automatically change the weight value linearly as the number of iterations is increased. This strategy involves only the weight changes of SD and HM.

As the process starts, the weight of SD is set to the highest value, while the weight of HM is set to the lowest. On the other hand, the weight value of LD remains constant throughout the iterations. The aim is to change only the weight of SD and HM as both heuristics make significant changes to the examinations ordering. The reason for setting the weight of HM to the lowest is because, at the beginning of the iteration, the HM value is zero and no examinations are considered cannot be scheduled. The HM value contributes no significant changes to examination ordering at that time. As the number of iterations increased, the HM value of some examinations might increase due to the possibility of cannot be scheduled during the iterations. Moreover, our previous study showed that dynamic ordering of saturation degree can contribute to good examination ordering and it is the reason why the weight of SD is set as the highest weight at the start of iterations. As shown in Algorithm 10, at each successive k ∗ counter iterations,

Algorithm 9 Dynamic change of weights during the timetable construction G = {HighLD, HighSD, HighHM, Balance}

for t = 1 to number of iterations do

Choose weight combination randomly from the group of HighSD for i = 1 to number of examinations do

for j = i to number of examinations do

Calculate the normalise value of chosen heuristics: choosemax(heurmodNj(t)), HeuristicN_LD,j, HeuristicN_SD,j with weight value

Calculate the dif f iculty scorej(t) for each examination according to the cho-sen heuristics using equation 4.1

end for

Sort(dif f iculty scorei(t)) in decreasing order

Calculate penalty-cost for assigning i to the best time-slot if penalty-cost of i > 0 then

Change to next group of weight and choose weight combinations randomly Calculate the normalise value for examinations within top-window size and order examinations

if All G has been applied and penalty cost of i > 0 then Increase the top-window size * 2

Calculate the normalise value for examinations within top-window size and order examinations

else

Choose the lowest penalty-cost and assigned i to lowest penalty time-slot end if

else

Assign i to the best time-slot end if

if i cannot be scheduled then

Increase and modify heuristic modifier of i (refer Subsection 3.1.2) end if

Update the Saturation degree end for

Evaluate solution, store if it is the best found so far end for

the weight values of SD, wSD and HM, wHM are simultaneously changed. The weight value of SD, w_SD is decreased by 0.05 while the weight value of HM, w_HM is increased by 0.05. Timetables are constructed and evaluated for k time before proceeding to the next weight changes. The process continues until the weight of SD, wSD reaches the minimum value and the weight of HM, w_HM reaches the maximum value.

4.1.4.3 Reinforcement Learning

Reinforcement learning is a learning mechanism that interacts with the behaviour of the environment by giving negative and positive rewards based on the performance of an environment and it is achieved by training a learning agent (Sutton and Barto,

Algorithm 10 Linear change of weights during the timetable construction Set the weight combination w_LD = 0.1, w_SD = 0.85 and w_HM = 0.05 Set counter = 1

if Linear increment of wHM then

Set k = (numberof iterations/m) where, m = the number of weight changes that occurred during the iteration in order to achieve the highest value of wHM

end if

for t = 1 to number of iterations do //Linear weight changes

if t = k ∗ counter then

change the weight value with 0.05 decrement to w_SD and 0.05 increment to w_HM Increase counter by one

end if

for i = 1 to number of examinations do for j = i to number of examinations do

Calculate the normalise value of chosen heuristics: choosemax(heurmodN_j(t)), HeuristicNLD,j, HeuristicNSD,j with weight value

Calculate the dif f iculty scorej(t) for each examination according to the cho-sen heuristics using equation 4.1

end for

Sort(dif f iculty scorei(t)) in a decreasing order if i can be scheduled then

Schedule i in the time-slot with the least penalty

In the case of the availability of multiple time-slots with the same penalty, choose one randomly

else

Increase and modify heuristic modifier of i (refer Subsection 3.1.2) end if

Update the Saturation degree end for

Evaluate solution, store if it is the best found so far end for

1998). The reinforcement learning method involved three main concepts namely the environment, the actions and the reward. The environment allows the learning agent to behave at a defined time, the actions indicates the desirable state of an environment and the reward gives feedback of agent’s action in the environment. The approach was successfully implemented in scheduling problems such as nurse rostering (Burke et al., 2003d), parallel machines job shop scheduling (Martinez et al., 2010), wake-up schedul-ing (Mihaylov et al., 2010), resource constrained project schedulschedul-ing (Gersmann and Hammer, 2003), examination timetabling ( ¨Ozcan et al., 2010) and course timetabling (Obit et al., 2009). Burke et al. (2003d)) used a hyper-heuristic method incorporated with tabu search as high level heuristic and a number of low level heuristic that op-pose each other based on the rules motivated by reinforcement learning principal, while study by Martinez et al. (2010) introduced value iteration strategy and policy iteration

strategy as their reinforcement learning method implemented on the parallel machines job shop scheduling problem. In a wake-up scheduling problem, Mihaylov et al. (2010) presented a decentralized reinforcement learning algorithm. Meanwhile, Gersmann and Hammer (2003) implemented a reinforcement learning approach to resource constrained project scheduling using rout-algorithm combined with support vector machine. Study by ¨Ozcan et al. (2010) chose the low-level heuristics based on a reward and punishment scheme known as ‘utility value’ where this learning mechanism employed an adaptation scheme that was based on the move acceptance by a remembrance mechanism i.e. by remembering or forgetting of the utility value. The forgetting mechanism creates the upper and lower bounds of the utility value. This strategy was combined with the great deluge algorithm as a move acceptance criterion. In a course timetabling problem, Obit et al. (2009) employed a reinforcement learning method with a non-linear great deluge acceptance criteria as a strategy to choose low level heuristics within the hyper-heuristic framework. For further details on the application of reinforcement learning within the hyper-heuristic approach, see Section 2.2.5.

In this study, the strategy uses a reinforcement learning mechanism to punish and reward each weight combination by giving them scores. As illustrated in Algorithm 11, the punishment is based on the performance of each weight combination. If the chosen weight combination can improve the current best solution, then a reward is given by increasing the score by one. On the other hand, if no improvement occurs then a penalty is given to the chosen weight combination by decreasing the score by one. At first, all the score values for each weight combination are initialised as 5 and are decreased or increased at each iteration until they reach the ceiling level. The score value of each weight combination never exceed the value of 10 and the value remains constant even though more improvement occurred in successive iterations. However, the score value of weight combination never less than 0 if no improvement occurs in successive iterations.

The next weight combination to be used in the next iteration is chosen based on the best weight combination that has the best score value. In the event that there is more than one best score, then the best score is chosen randomly.