The overall idea of an option [Sutton et al., 1999] is simple: to extend the choices available to the agent to also include temporally extended actions that encode “extra” domain knowledge. In its elementary form, an option denotes a fixed policy that the agent follows for some period. The option’s policy may be thought of as providing a local, domain dependent strategy, in a wider context. In that sense, options may be used to specify subgoals that are to be achieved during learning.
In the options framework each state s ∈ S is associated with a set of options O, synony- mous with the action set Asin the standard reinforcement learning setting. The set O
may also supply the primitive actions Asspecified as single-step options. Each option
o ∈ O is further defined by an input set I ⊆ S, and the option is considered to apply in state s only if s ∈ I. If option o is selected is state s, then actions are executed according to the programmed policy π : S × As → [0, 1] until the option terminates
stochastically according to a termination condition β : S → [0, 1]. Using an equivalent time-extended reward function R(s, o) and state transition function P (s0|s, o),Sutton et al.[1999] define a generalised form of the Bellman optimality equation VO∗(s) over a given option set O that reduces to the standard form (see Equation7.2.1) if all options are single-step options. As such, the options framework provides a generalisation for the standard definition of actions in reinforcement learning.
Sutton et al.[1999] use a grid world example to illustrate the options idea. Consider the case where an agent is situated in a building with interconnected rooms with a single door connecting adjoining room as shown in Figure7.2. Primitive actions allow the agent to move to an adjacent cell in either direction in a stochastic manner, i.e., with some probability the move action fails and the agent actually moves in a different direction.
The task is to learn to navigate from anywhere within one room to anywhere within another room, such as from the cell marked × to the cell marked G. Clearly, the best way to achieve this is to take the door marked D. One way to impart this information to the agent is through an option (o1) that applies in all states that correspond to the
agent being in the south-west room, and if selected, applies a policy that directs the agent towards the state that corresponds to being at cell D. In this way, while in the south-west room and given a goal G, the agent follows the local policy specified by o1. On reaching cell D, the option o1 terminates and is no longer applicable, at which
point the agent can choose from the remaining single-step options (that correspond to the primitive actions) in order to reach G. Were the goal to navigate to cell H instead, one could specify a second option o2that applies in the south-east room and directs the
agent to the interconnecting door to the north, and so no.
The motivation behind the options framework is to augment rather than reduce the set of available actions, such as in the grid world example where the sub-task of navigating between rooms is “reusable” across all individual tasks that require navigation from a cell in the first room to another in the second. While this means that the option set O is larger than the action set A of the initial MDP, it should be noted that the added options capture a sequence of primitive actions that make sense for the domain as specified by the expert. So, executing a relevant option is generally more advantageous compared to trying other (possible meaningless) combinations of primitive actions for the same period. Further, if the option set is specified such that it does not contain every possible single-step option, then the search space for the problem is significantly reduced. Such a specification by the expert that reduces the state space is also the motivation behind Hierarchical Abstract Machines (HAMs) that we discuss next.
The options framework allows an expert to supply procedural domain knowledge to the learner in the form of policies for sub-tasks. In this way, the options framework is
closely related to our learning framework, and an option is comparable to the procedural know-how encoded in a BDI agent’s plan library. Further, the initial set I for a given option is comparable to the context condition of a given plan: both specify the condi- tions under which the procedure applies. In our case, however, this set is not known upfront (indeed we are trying to learn it), whereas knowledge of the set I is necessary in the options framework.
Another important difference between the two approaches is in their treatment of sub- goals. In the BDI learning framework, procedural knowledge is encoded as a rich hi- erarchy of (sub)goals and plans to handle them. In the options framework, subgoals are not represented explicitly but may be viewed as states that are desirable during the execution of an option: the value of such states being encoded in subgoal specific re- ward functions. The idea is that the agent may wish to bring about certain desired states (subgoals) during the execution of an option.
This separation of concerns using a hierarchy of (sub)goals in the BDI system allows us to learn to resolve such subgoals independent of any higher level plan in which they may be used. In the options framework however, one cannot accommodate a rich hier- archy of subgoals (by specifying options in terms of options for instance), and learning over subgoals (i.e., learning inside options) is significantly more restricted. Overall, options are intended to resemble actions as much as possible (hence the name tempo- rally extended actions), in order to inherit and benefit from the rich formal theory of reinforcement learning.