FUNCIONAMIENTO DEL EQUIPO.

The concept of learning continually from experience has always been present in artificial intelli- gence and robotics since their birth [Turing, 1950;Weng,2001]. However, it is only at the end of the 20th century that it has began to be explored more systematically. Within the machine learning community, the lifelong learning paradigm has been popularized around 1995 byThrun

[1996]; Thrun and Mitchell [1995] and Ring [1994]. Since then it has been researched in four main areas. Here we give a brief history of the CL research in each of these areas.

Continual Supervised Learning. Thrun [1996] was one of the first to study continual learning within a supervised context, where each previous or new task aims at recognizing a particular concept using binary classification. Several CL techniques were then proposed in the contexts of memory-based learning and artificial neural networks. The neural networks approach was improved by Silver and Mercer [2002]; Silver and Poirier [2004]. Ruvolo and Eaton [2013b] proposed an“Efficient lifelong learning algorithm” (ELLA) to improve the multi-task learning method proposed byKumar and Daume III[2012]. Here the learning tasks are independent from each other and a regularization strategy based on theFisher Informationwas firstly introduced.

Ruvolo and Eaton[2013a], however, were among the first who considered ELLA also in an active task selection setting. Cheng, Hao and Fang, Hao and Ostendorf [2015] further proposed a continual learning technique in the context ofNa¨ıve Bayesian classification. A more theoretical study of continual learning was firstly accomplished by Pentina and Lampert[2015] within the PAC-learning framework.

Continual Unsupervised Learning. While intuitively better suited for unsupervised learning, continual learning research in this area have not been extensive and mainly focused on topic modeling and information extraction. Chen and Liu[2014a,b] and Wang et al.[2016] proposed several continual topic modeling techniques that extract knowledge from topics produced within many previously encountered tasks and use it to help generate better topics in the new tasks. Liu et al.[1999] proposed a continual learning approach based on recommendation for information extraction in the context of opinion mining. Shu et al. [2016], instead, proposed a continual relaxation labeling method to solve a unsupervised classification problem.

Continual Semi-Supervised Learning. The work in this area is well represented by theNever- Ending Language Learner (NELL) system byCarlson et al.[2010];Mitchell et al. [1998], which has been reading the Web continuously for information extraction and learning since January 2010, and it has accumulated millions of entities and relations.

Continual Reinforcement Learning. Mitchell and Thrun[1993] first proposed some CL algorithms for robot learning which tried to capture the invariant knowledge about each individual task.

Tanaka and Yamamura [1997] treated each environment as a task for continual learning. Ring

[1994] proposed a continual learning agent that aims to gradually solve complicated tasks by learning easy tasks first withing an extensive and general approach to continual reinforcement learning. Wilson et al.[2007] proposed a hierarchical Bayesian continual reinforcement learning method in the framework of Markov Decision Process (MDP). Fern´andez and Veloso [2013], instead, specifically worked on policy reuse in a multi-task setting. A nonlinear feedback policy that generalizes across multiple tasks was proposed in [Deisenroth et al., 2014]. Ammar et al.

Chapter 1. Introduction 16

[2014] proposed, instead, a policy gradient efficient continual learning algorithm following the idea presented with ELLA [Ruvolo and Eaton, 2013b]. This work was further enhanced with cross-domain continual reinforcement learning byAmmar et al.[2015a] and with constraints for safe continual reinforcement learning [Ammar et al.,2015b].

Continual Learning techniques working in other areas also exist. For example, Kapoor and Horvitz [2009] studied predictive user modeling under continual learning, and worked on man- aging and using user feedback with the help of CL.Silver et al.[2013] wrote a survey of continual learning trying to encourage more researchers to work in this area.

As we can see, although continual or incremental learning has been proposed for more than 20 years, research in the area has not been extensive. At this point a question normally arise: why continual learning despite its intuitiveness and naturalness is becoming a solid interest of the machine learning and AI community only now? As we already hinted in the introduction, there were more complex and fundamental problems to solve before the deep learning revolution and a number of additional constraints:

• Lack of a systemic approach: machine learning research for the past 20 years has focused on statistical and algorithmic approaches on simple tasks. CL typically needs a systems approach that combines multiple components and learning algorithms. Moreover, contin- ual learning greatly complicate training and evaluation procedures. Disentangling“static” learning performance from continual learning side effects was important for the very incre- mental nature of the research in this area.

• Limited amount of data and computational power: digital data are a luxury of the 21th century. Before the big data revolution collecting, processing data was a daunting task. Moreover, the limited amount of compute power available at the time, did not allow com- plex and expensive algorithmic solution to run effectively, especially in a continual learning setting which undoubtedly makes learning more complex dealing with multiple task at the same time and incorporating the concept of time into the learning process.

• Manual engineered features and had hoc solution: Before early 2000 and early works on representation learning creating a machine learning system would mean to handcraft fea- tures and finding had-hoc solution which may differ significantly depending on the task or domain [Russell and Norvig, 2016]. Having a general algorithm for a more systemic approach seemed for a long time a very distant goal.

• Focus on supervised learning: creating labeled data is probably the slowest and the most expensive step in most machine learning systems. This is why learning continuously has been for a long time not a viable and practical option.

The relaxation of these constraints thanks to recent advancements and results in machine learning research as well and the rapid technological progress witnessed in the last 20 years, have opened the door for starting tackling more complex problems like learning continually.

In the following chapters we will focus on recent continual learning developments in the context of deep learning, as generally known from 2012. For a more detailed description of many other

Chapter 1. Introduction 17

classic approaches to continual learning with shallow architectures please refer to [Chen and Liu,

2018].