Enfoque Epistemológico

Systems

There is much to be learned from the application of Game Theory and Decision Theory to Computer Science, and specifically to algorithms involved in multi-agent systems. The interaction, within the same environment, of many agents is complicated when they must each understand what the other agents are doing. As more agents are considered, and more detail about each agent is factored in, understanding their

interaction leads to larger and less manageable state spaces. This leads quickly to in-computability using normal algorithms and approaches where the system attempts to focus on all available information monolithically. One approach to resolving this large state-space search problem was proposed in (Soumya Paul and R. Ramanujam,

2010) whereby the researchers suggest a theoretical framework using imitation. In this work, they state that in large games there are optimizers (those agents who, due to superior resources, have a better understanding of the state of the game and thus perform optimally) that can be exploited by imitators (those agents with more limited resources that choose to imitate the optimizers rather than innovate). They incorporate principles derived from game theory to explain this interaction within the context of repeated normal form games. Ramanujam et al. propose using a finite graph with infinite moves, a small set of known strategies, and turn based progression (each player takes a turn rather than simultaneous moves being allowed). Starting from the foundation laid in (Soumya Paul and R. Ramanujam, 2010), this research contributes a framework that moves their work from the theoretical to the practical. Their proposed mini-game will be realized using small robots as the players that must follow the same rules. This will lead to a larger scale solution that emulates an increasingly complex environment within a simulation system. In the end, their proposal states that, given enough time, the imitators can improve their performance (though not to the level of the optimizers) using fewer resources. This should result in the robots playing the game with improved performance (higher overall reward). This initial insight led to the strategy-based systems implemented in

the proposed framework. While this reference work focuses on a small, single-agent team, theoretical environment, the proposed framework works in large, multi-agent, multi-team, practical environments.

Their simple mini-game, as dictated in (Soumya Paul and R. Ramanujam,2010), will be expanded in three ways: first, allowing simultaneous moves; second, co-located players (more than one player in a given location as opposed to the exclusivity of each location in the original proposal); third, finite length for the game. This first change adds uncertainty due to simultaneity and requires probabilistic modeling. The second change adds complexity to the algorithmic requirements because the state space is not narrowed down by spaces being held by other players. The third change adjusts the time scale allotted for realizing what other players are doing, imitating them, and increasing performance. As Ramanujam et al. proposed their game it is unbounded in length. Unbounded length allows more time for the slow gains realized by the imitators to pay off. In a finite game (which could also represent a single episode of a longer, perhaps infinite game) the time for the payoff to factor in is greatly reduced. As a result, a different representation scheme is required for strategies.

To understand the evolution of the proposed strategy inference (utilized in this research and in the final experiments), it is necessary to bridge the gap from some theoretical background work (Soumya Paul and R. Ramanujam,2010,2011a,b), to an applied system that can be constructed, run, and evaluated. Many of the issues found with those early theoretical models are described here and the reason for abandoning

this approach and shifting to the final, novel, proposed approach is also elucidated here.

For single-agent systems, moving from the theoretical to the generic, each agent’s strategy must be considered and compared. In this manner the focus agent’s strategy,

σ1, can be compared to the most likely strategies of the other agents. The steps of this strategy inference are shown in Table 1.1.

Step Techniques

Develop a strategy Observation, Expert System, Coach Choose optimal move independently Pre-determined policy from strategy Choose most likely moves of other agents FSA and Belief Network

Match best possible move Re-examine policy and select Table 1.1: Steps and Techniques for Strategy Inference

1.10.2

Specification

In an effort to help the reader, the evolution of the concepts of strategies and policies is given next. This specification uses some similar terminology and symbology with the final product (the effort of this research), but it is not the same. The explanation follows within each section. In summary, the original concept, as communicated in the reference papers, was sufficient for a technical discussion, but not sufficient for a real-world system. The reference system has some issues that prevent it from being implemented in practice. However, to aid the research community, this concept is presented in its original form and using the reference notation. Once this is relayed,

explained, and discussed, the new concepts, formalism, and notation are introduced (introduced generally here, formally presented in Chapter 3).

A formalization of strategy, presented in (Soumya Paul and R. Ramanujam,2010), consolidates this type of strategic interaction. This paper is written using the game- theoretic definition of strategy, so it is similar in practice to a policy, but the intention is larger than this standard definition. This research seeks to move their work from the theoretical and policy-bound to the practical and strategy-bound. In this case, the game is considered as a turn-based game of unbounded duration. Rather than give a simple reference, the paper’s main points are included here in a paraphrase with some interpretation.

The game in (Soumya Paul and R. Ramanujam,2010) is mapped as a finite graph. For any positive integer n, let [n] = 1, .., n.

Definition 1 Let n ∈ _N, n > 1. An n-player game arena is a directed graph

G = (V1, . . . , Vn, A, E), where Vi are finite sets of game positions with Vi∩Vj =∅ for

i 6= j, V = ∪i∈[n]Vi, A is a finite set of moves, and E ⊆ (V ×A× V) is the move relation that satisfies the following condition:

1. For every v, v1, v2 ∈ V and a, b ∈ A, if(v, a, v1) ∈ E and (v, b, v2) ∈ E then

a6=b.

2. For every v ∈V, there exists a∈A and v0 ∈V such that (v, a, v0)∈E.

When an initial position v0 ∈ V is specified, we call (G, v0) an initialized arena or just an arena.

However, there are issues with this theoretical foundation when the game is moved to a real-world simulation. Namely, no models are offered for implementation, there are no probabilistic transitions, and only one player can be in any particular state at one time. Since this research seeks to investigate systems with probabilistic progression with strong modeling of multi-agent teams, such limitations require another formulation of the problem.

This research creates an arena defined as this graph G with the players (the agents) as the vertices and the possible moves as the edges. This definition allows the definition of the neighbors of the players to be those who are located along the connected edges of the graph. These neighbors,vE, are the singularly located so that there are no other players at this same vertex. Additionally, the edges denote those actions, or moves, which are possible for the player at this state and each is available to this player as long as it is enabled (i.e., vertex v0 is unoccupied). These moves can then be chained together as a sequence of moves along this finite graph which will be called a play, labeled as vi

ai+1

→ vi+1 for i ∈ N. Since there is no ‘game board’ in the scenario being used in this research this series of moves is a projection into a space of possible and inter-related moves. This complex scenario of moves is really just a mapping of enabled moves for a player that do not overlap their own teammates moves nor create a co-location with a player (i.e., choosing the identical unique action) from any team. It is important to note that there may be identical moves that are allowable, such as two players claiming rewards from a pool of rewards, and moves that are not allowable, such as two players trying to claim the same reward.

For example, for two players i and j, their possible moves are vi ={1,2,3, ..., n} and vj = {1,2,3, ..., m}, respectively. The sets of moves represent the action taken by a player at each timestep t ∈ T, and there is always a ‘no move’ option. In this manner, there is a move made by each player at each time step, even if it is a ‘no move’. Thus, the moves are aligned within the vector of moves such that vi[a1] and

vj[a1] occur contemporaneously. The gameGconstrains the set of moves that enforces the rules of the game (i.e., the set of policies that govern the environment, which in itself can be thought of as another implicit player (Soumya Paul and R. Ramanujam,

2011a) whose strategy must be considered) and ensures that vi[an] 6=vj[am],∀t ∈T. Similarly, there are alwaysa∈As.t. ∀v ∈V ∃a0, so there are no absorbing states in the game from which a player cannot return. Each of these players is implementing a strategy, σ1and σ2, respectively. These strategies are composed of policies,πi and πj, where π1..n ∈ Πi and π1..m ∈ Πj. Therefore vi and vj are following policies parceled out to them by their strategies.