Conclusiones parciales.

The field most closely related to the research described in this thesis is program-

ming by example (PBE), also known as programming by demonstration (PBD); the

pioneers of the field being Henry Liebermann [Lieberman, 2000], Daniel Halbert Halbert [1984], Mathias Bauer [Bauer et al., 2001], Brad Myers [Myers et al., 1995], Francesmary Modugno [Modugno, 1996] and Richard McDaniel [McDaniel, 2001].

2.6.1 PBE: Theory and Models

In PBE programming is not done by the agent but by the user and the process

is in fact a form of inductive heuristic where the user’s paradigm contains hidden

states[McDaniel, 2001]. The userreplicatesan internal mental process which is then

transcribed and examples are extracted. Part of that process is hidden, implying a

partially observable Markov process [Smallwood and Sondik, 1973] from which the

agent is programmed. Smith et al describe it as thecreator approach [Smith et al.,

2001] where rules are inferred by observing changes from before to after. They

describe the process, and assert that: 7 http://artificial-intelligence.silk.co/page/PreAct 8_{http://sw.opencyc.org} 9 http://opencog.org/ 10_{http://cognitive-ai.com/page2/page2.html}

”Each inferred rule represents an arbitrary number of primitive opera- tions, or statements in other languages.” [Smith et al., 2001, 10]

Smith et al declare the need and use of a representation, the process of observing changes in the user examples from which they infer rules. They expand on the need for a symbolic representation basing much of their work on the psychologist Jerome Bruner who focused on the cognitive development of children and its relation

to education. According to Bruner there are three stages of development: enactive,

iconic and symbolic [Bruner, 2009]. Inspired from Bruner Smith et al propose that

the use of Symbolism governed by rules or laws, is a Fregean representation; they

named it the Runer’s approach, a Symbolic-enactive representation based on UI

interactions and visual, iconic and symbolic representations.

McDaniel describes passive observation asthe passive watcher in an argument

against observation; he asserts that the passive watcher cannot request from the

demonstrator the hidden states and that the object (e.g., paradigm or teaching

material) may not be useful to the agent [McDaniel, 2001], without however proving it.

An action-focused approach is the one taken by Bauyer et al, who viewed

the task of replicating the learnt process as a sequence of actions [Bauer et al., 2001, 100–102]. Although he did not expand much on the idea he named the processes ”recipes” indexed by atask library in the agent’s memory. Each recipe uses action estimation based on a monotonic probabilistic training approach [Bauer et al., 2001, 101], shown in (2.28).

EU(a, u, n) = [ass(wa)−ass(wcurr)]·Pu(a)−annoy(u, n). (2.28)

The formula describes the expected use of an action a, for user u with an

already executed number of actions n. The descriptorsass(wa) and ass(wcurr) are

the assessments of the bestpolicy, before and after executing the actiona. The prob-

abilityPu(a) denotes userucarrying out actionaand functionannoy approximates

the user’s characteristics growing monotonically as the training increases. This is an

interactive approach using a UI and it penalises the agent when requesting actions from the user.

Bauer et al further describe the issue of sharing knowledge between the

user and the agent. This is a fundamental approach which relates to how agents are trained, the partial observability or hidden states, and how all aspects overlap (shown in Figure 2.14).

Figure 2.14: Knowledge sharing by agent and user

knowledge with the trainee (e.g., the agent). This dramatisation is quite accurate;

since the agentsubjectively experiences (e.g., qualia) the components shown above,

those are often hidden, undeclared, invisible, implied or even unknown.

- Structural knowledge relates to theinternal properties of the domain.

- Procedural knowledge relates to understanding part or the entirety of the processes involved.

- Visual and semantic knowledge relates to knowing andunderstandingrelations

and ontologies.

- Domain knowledge, isspecific knowledge (e.g., in the case of NLP,syntax and

grammar).

As evident from Figure 2.14, those categories often overlap; in some cases they can be acquired by other means (e.g., a lexicon or an external process), and in

other cases they may beinferred.

Most of the PBE research uses simple inference, heuristics, rules and in- ductive logic [Lieberman, 2000; Myers and McDaniel, 2001; Modugno, 1996; Myers et al., 1995; Myers and Zanden, 1992; Zanden and Myers, 1990]. Machine learning was absent from PBE research, with the only exceptions being [King et al., 1992] and [Liebermann et al., 2001] albeit using rudimentary models. It wasn’t until 2013 that researchers from UCLA, the Weizmann Institute and Microsoft publicised work

[Menon et al., 2013] that combines the two fields; in this case, rules, probabilities

2.6.2 PBE: Application Domains and Criticism

The majority of PBE focused on user interfaces (UI); most of the PBE research used rule-based approaches in order to interact with the user and extract some form

of aprogram [Lieberman, 2000; Myers and McDaniel, 2001; Modugno, 1996; Myers

et al., 1995; Myers and Zanden, 1992; Zanden and Myers, 1990]. A few others were interested in automating information acquisition tasks [Bauer et al., 2001], mostly by transcribing interactions to queries. Bauer criticises PBE as having been overly simplistic and admits it was not designed with AI in mind (or by AI researchers) in

what he callslevel of intelligence [Bauer et al., 2001, 49]. Extracting rule definitions

from multiple examples was considered a novelty [Liebermann et al., 2001] and it

was applied in tasks such astext categorisation.

In hindsight, PBE took a minimalistic approach to enabling user-agent in- teraction, mapping of knowledge, action prediction (or suggestion); it was in fact

focusing onprogramming and not learning. The early stage of UI and the graphical

UI of the OSes at the time (Microsoft Windows 95, 98, 2000, NT) in combination with the fact that not as many people used computers back then as nowadays, may have contributed to the sunset of PBE. The web was just starting to appear, and personal computers were not as widespread as they are today.

PBE at that stage took atoy-problem approach and simplified the underlying

mechanisms. Whereas complexity of an agent is a different topic, in the case of PBE it may have been detrimental to the abilities of the agents. No machine learning

was used11, few (if any) AI-related models or algorithms were employed; let us not

forget that its focal point was extracting a sequence from the UI being replicated for the UI or operating system.

However, PBE set the foundations for much of what isimitation learning in

AI today: it described the example extraction process as a MDP, it emphasised on

the need for multiple examples, it showcased how visual generalisation (and visual UI interactions with agents) can be flexible and easy to use [Amant et al., 2001] and that PBE can be used for domain-independent tasks or use spatial representation for languages [Paynter and Witten, 2001]. Furthermore and perhaps most important

it reinforced the notion of using Symbolic representation and requiring a temporal

structure for procedures, and including the use of Semantics, and domain attributes. 11

With the aforementioned exceptions of King [King et al., 1992] and Liebermann [Liebermann et al., 2001]

2.6.3 PBE: Differences from Imitation Learning

Evidently PBE was a subset oflearning by example as it only focused on interfaces,

automation and programming. There are many similarities with the work presented in this thesis, in fact we drew inspiration from Lieberman, McDaniel and Amant,

however imitation in AI is more than merely programming. PBE sets the basis on

using a visual interaction for creating examples, the need to use multiple examples, the use of inference in transcribing user paradigms to an MDP, and raises the issue ofpartial observability in the MDP/example.

Terminology issues aside (programming by example versus learning by ex-

ample) the major difference is that in Icarus engine (a cognitive agent) I focus on

learning which is enabled bydeep reinforcement learning in combination with other techniques; there is no focus on the UI, and the user-UI interaction is of little interest to the field of AI.