ELEMENTOS DE UN JUICIO SOBRE DERECHOS DE AUTOR

In spite of the wide range of EMLCs already proposed in the literature, many of them do not consider some characteristics of the data when building the mod- els. For instance, the performance of ECC directly relies on the chain ordering, but it selects the chains randomly, not considering any of the characteristics of the data. Similarly, RAkEL selects subsets of labels to build each member, but they are randomly created without considering for example the relationship among these labels. On the other hand, some of the EMLCs are still not able to deal with these special characteristics of the multi-labeled data, as EBR, which is not able to model the dependencies among labels.

Aiming to improve the predictive performance of state-of-the-art EMLCs, we propose an evolutionary approach for the automatic generation of EMLCs, called EME, where the characteristics of the data are taken into account in the building phase. The EMLC generated by EME is based on the same idea as RAkEL, i.e., each base classifier is focused on modeling a small subset ofklabels. In this way, it is able to model the relationship among groups of labels, but drastically reducing the com- plexity of each of them in scenarios with a high dimensionality of the output space. However, while RAkEL selects thosek-labelsets just randomly, and thus it does not ensure that all labels are being considered or the number of times that each of them appears in the ensemble, in EME thek-labelsets are selected by looking for an op- timal structure of the ensemble, and also ensuring that all labels appear a similar

number of times.

In EME, each individual of the population in the EA represents an EMLC. These individuals are initially randomly created, but as the generations go by, they evolve towards more promising combinations of classifiers into the ensemble. Individuals are evaluated not only by considering their predictive performance, but also taking into account the number of times that each label appears in the ensemble. In this way, we look for EMLCs that are not only accurate but that also that do not neglect some labels regardless of their frequency or ability to be predicted, thus trying to improve their generalization ability.

On the other hand, we propose a mutation operator for the EA where the re- lationships among labels are considered. For this purpose, for a given classifier inside the ensemble, labels that are more related with labels already appearing in the classifier have more chance to mutate and therefore to be included in the classi- fier than those that are not related with them. However, all labels still have a small chance to mutate, but we bias the evolution to the achievement of EMLCs dealing with subsets of labels that are related among them.

So, both RAkEL and EME deal with all the main problems of MLC: I) they reduce the dimensionality of the output space in each base classifier, II) thus leading to a reduction of the label imbalance, and III) being able to model the label relationships in each base classifier. Nonetheless, while RAkEL does not consider neither the relationship among labels nor the imbalance in its building phase, EME does.

EME is based upon a generational elitist algorithm, so it ensures that the best individual in the last generation is also the best individual so far in the EA process. Therefore, as each individual represents an entire ensemble, the best individual is returned at the end of the evolution.

In the experimental study, we first conduct a preliminary study to select the best parameters for the evolutionary algorithm, involving four datasets that are then not used in the comparison with state-of-the-art methods (in order to not to bias the final results). Later, we compare EME not only to other standard MLC methods but also to state-of-the-art EMLCs. Compared to EMLCs, it is demonstrated that EME has a better and more consistent performance in overall, being the best method in

one metric and also being the only method that did not perform significantly worse than the best method in any case. Tables with full results are available online1_.

Furthermore, EME has a better overall performance than RAkEL in four out of five evaluation metrics, and it also performs significantly better than RAkEL in two of the metrics. That proves that the fact of evolving thek-labelsets towards more promising combinations of labels instead of just selecting them randomly lead EME to achieve a better predictive performance.

Following, we present the paper associated with this chapter of the thesis[J5].

Title An evolutionary approach to build ensembles of multi-label classifiers

Authors Jose M. Moyano, Eva L. Gibaja, Krzysztof J. Cios,

Sebastián Ventura

Journal Information Fusion

Volume 50

Pages 168 - 180

Year 2019

DOI 10.1016/j.inffus.2018.11.013

IF (JCR 2018) 10.716

Category Computer Science - Artificial Intelligence

Position 3/133 (Q1)

ContentslistsavailableatScienceDirect

InformationFusion

journalhomepage:www.elsevier.com/locate/inffus

An

evolutionary

approach

to

build

ensembles

ofmulti-label

classifiers

JoseM.Moyanoa,e_{, EvaL.Gibaja}a,e_{,KrzysztofJ.Cios}b,c_{,SebastiánVentura}a,d,e,∗

a_{Department of Computer Science and Numerical Analysis, University of Córdoba, Córdoba, Spain} b_{Department of Computer Science, Virginia Commonwealth University, Richmond, VA 23284, USA} c_{Polish Academy of Sciences, Institute of Theoretical and Applied Informatics, Gliwice, Poland} d_{Faculty of Computing and Information Technology, King Abdulaziz University, Saudi Arabia}

e_{Knowledge Discovery and Intelligent Systems in Biomedicine Laboratory, Maimonides Biomedical Research Institute of Córdoba, Spain}

a r t i c l e i n f o Keywords: Multi-labelclassification Ensemble Evolutionaryalgorithm a bs t r a c t

Inrecentyears,themulti-labelclassificationtaskhasgainedtheattentionofthescientificcommunitygivenits abilitytosolveproblemswhereeachoftheinstancesofthedatasetmaybeassociatedwithseveralclasslabelsat thesametimeinsteadofjustone.Themainproblemstodealwithinmulti-labelclassificationaretheimbalance, therelationshipsamongthelabels,andthehighcomplexityoftheoutputspace.Alargenumberofmethods formulti-labelclassificationhasbeenproposed,butalthoughtheyaimedtodealwithoneormanyofthese problems,mostofthemdidnottakeintoaccountthesecharacteristicsofthedataintheirbuildingphase.In thispaperwepresentanevolutionaryalgorithmforautomaticgenerationofensemblesofmulti-labelclassifiers bytacklingthethreepreviouslymentionedproblems,calledEvolutionaryMulti-labelEnsemble(EME).Each multi-labelclassifierisfocusedonasmallsubsetofthelabels,stillconsideringtherelationshipsamongthem butavoidingthehighcomplexityoftheoutputspace.Further,thealgorithmautomaticallydesignstheensemble evaluatingbothitspredictiveperformanceandthenumberoftimesthateachlabelappearsintheensemble, sothatinimbalanceddatasetsinfrequentlabelsarenotignored.Forthispurpose,wealsoproposedanovel mutationoperatorthatconsiderstherelationshipamonglabels,lookingforindividualswherethelabelsare morerelated.EMEwascomparedtootherstate-of-the-artalgorithmsformulti-labelclassificationoverasetof fourteenmulti-labeldatasetsandusingfiveevaluationmeasures.Theexperimentalstudywascarriedoutintwo parts,firstcomparingEMEtoclassicmulti-labelclassificationmethods,andsecondcomparingEMEtoother ensemble-basedmethodsinmulti-labelclassification.EMEperformedsignificantlybetterthantherestofclassic methodsinthreeoutoffiveevaluationmeasures.Ontheotherhand,EMEperformedthebestinonemeasurein thesecondexperimentanditwastheonlyonethatdidnotperformsignificantlyworsethanthecontrolalgorithm inanymeasure.TheseresultsshowedthatEMEachievedabetterandmoreconsistentperformancethantherest ofthestate-of-the-artmethodsinMLC.

1. Introduction

Inrecentyears,theMulti-LabelClassification(MLC)taskhasgained theattentionofthescientificcommunitygivenitsabilitytosolveprob- lemswhereeachoftheinstancesmaybeassociatedtoseveralclassla- belsatthesametime,insteadofjustone.Letbe ={𝜆₁,𝜆₂,…,𝜆_𝑞}the setofqdifferentbinarylabels(withq>2),andthesetofminstances, eachcomposedbydinputfeatures;letusdefinethemulti-labelclassi- ficationtaskaslearningamappingfromanexample𝒙𝑖∈toasetof labels𝒚_𝑖⊆ .Labelsinthesetyiarecalledrelevantlabels,andtherest

(𝒚𝑖)arecalledirrelevant.Agreatdealofreal-worldproblemshavebeen successfullysolvedthankstotheapplicationofMLC,suchassocialnet- worksmining,whereeachusercouldbesubscribedtoseveralgroupsof interest[1];multimediaannotation,whereeachimageormultimedia itemcould beassociated toseveral classlabels[2]; andtextcatego-

∗_{Correspondingauthor.}

E-mailaddress:[email protected](S.Ventura).

rization,whereeachdocumentcouldbecategorizedin severaltopics simultaneously[3];amongothers.

ThemostchallengingproblemsinMLCaredealingwiththeimbal- anceofthedata[4],modelingcompounddependenciesamongthela- bels[5],andthepossiblehighdimensionalityoftheoutputspace[6]. Inmanyproblemsthelabelsdonotappearwiththesamefrequencyin thedataset,withsome labelsappearinginmostoftheinstancesand otherthatarebarelypresent,appearinginafewinstances.Thismight leadtoanimbalanceddatasetwherethefrequentlabelscouldbemuch betterpredictedthantheinfrequentones,asthereisverylittleinfor- mationabouttheinfrequentlabels.Besides,labelsarenotusuallyinde- pendentbuttendtoberelatedtoeachother,wherealabelmayappear morefrequentlywithsomelabelsthanwithothers.Thefactof mod- eling,orlackof,compounddependenciesamonglabelshasadecisive effectnotonlyonthepredictiveperformanceofthemodelbutalsoonits

https://doi.org/10.1016/j.inffus.2018.11.013

Received12April2018;Receivedinrevisedform8November2018;Accepted25November2018 Availableonline26November2018

complexity.Thecomplexityofthemodelisalsousuallyrelatedtothe sizeoftheoutputspace.Thegreaterthenumberoflabels,thehigher thecomplexityofthemodel,whichcanmaketheproblemintractable. Inordertotrytoovercometheseproblems,severalmethodologies have beenproposedin theliterature.Forexample,PrunedSets(PS)

[7]wasproposedinordertoreducetheimbalanceinthefinalproblem. Besides,toovercometheproblemofmodellingthecompounddepen- denciesamonglabels,ClassifierChains(CC)[5]consideredtherelation- shipamongdifferentbinarymethodsthatoriginallydidnottakeinto account.Fortheoutputdimensionalityproblem,RAndomk-labELsets (RAkEL)[8]dividedthelabelspaceintosmallersubsets,resultingin lesscomplexoutputspaces.Furthermore,thecontinuousstreamofin- putdataisagrowingprobleminmanydataminingtasks,andithas beenalsosuccessfullyaddressedinMLC[9,10].Manyoftheseproposed methodswerebasedonthecombinationofseveralclassifiers.However, inMLConlythosemethodsthatcombineseveralclassifierswhichare abletodealwithmulti-labeldataareconsideredasEnsemblesofMulti- LabelClassifiers(EMLCs)[11].Ontheotherhand,besidestacklingthe aforementionedproblems,ensemblesusuallyperformbetterthansingle classifiers.Oneofthewaystoobtainanensemblethatoutperformeach of theindividualsclassifiersistocombinea setof diverseclassifiers

[12,13].Despitethisfact,manyoftheproposedensemblemethodsin

theliteraturegeneratediversityonlybyrandomsamplingofattributes, instances,orlabelsforeachclassifier,butnotensuringthattheentire ensembleisdiverseenough.

Inthispaper,weproposeanevolutionaryapproachfortheautomatic generationofensemblesofdiverseandcompetitivemulti-labelclassi- fiers.Thealgorithm,calledEvolutionaryMulti-LabelEnsemble(EME), takes intoaccount characteristicsof themulti-labeldatasuchasthe relationshipsamongthelabels,imbalanceofthedata,andcomplexity oftheoutputspace.Theensembleisbasedonprojectionsofthelabel space,consideringinthis waytherelationshipsamongthelabelsbut alsoreducingthecomputationalcostincaseswheretheoutputspaceis complex.Thesesubsetsoflabelsarenotonlyrandomlyselectedbutalso theyevolvewiththegenerationsoftheevolutionaryalgorithm,looking forthecombinationsthatperformthebest.Also,anovelmutationop- eratoris proposed,so thatitconsiders therelationshipamonglabels favouringmorerelatedcombinationsoflabels.Further,EMEtakesinto accountallthelabelsapproximatelythesamenumberoftimesinthe ensemble,regardlessoftheirfrequencyoritseasetobepredicted;so thattheimbalanceofthedataisconsideredandtheinfrequentlabels arenotignored.Forthat,thefitnessfunctiontakesintoaccountboth thepredictiveperformanceofthemodelandthenumberoftimesthat eachlabelisconsideredintheensemble.Finally,thediversityofthe ensembleisnottakenintoaccountexplicitly,buttheensemblesevolve selectingtheirclassifiersbasedontheiroverallperformance.

The experimental study carried out over fourteen multi-label datasetscompared EMEwith classicstate-of-the-artmethodsin MLC andalsootherEMLCsusingfiveevaluationmeasures.Thefirstexper- imentdeterminedthatEMEperformedsignificantlybetterthanclassic MLCmethodsinthreeofthefiveevaluationmeasures.Inthesecondex- periment,EMEachievedthebestperformanceinonlyonemeasure,but itwastheonlyalgorithmthatdidnotperformsignificantlyworsethan anyofthecontrolalgorithmforanyevaluationmeasures.Theseresults showedthatEMEachievedabetterandmoreconsistentperformance thantherestofthestate-of-the-artmethodsinMLC.

Therestofthearticleisorganizedasfollows:Section2includesre- latedworkinmulti-labelclassification,Section3describestheproposed evolutionaryalgorithm,Section4presentstheexperimentalstudyand

Section5presentsanddiscussestheresults.Finally,Section6endswith

conclusions.

2. Relatedwork

The traditionalsingle-labelclassificationtask aimstopredict the classorgroupassociatedtoeachoftheinstancesdescribedbyasetor

inputfeatures.Eachoftheinstancesisclassifiedinjustoneclassfroma previouslydefinedsetofclasses.However,inMLC,eachinstancemaybe labeledwithmorethanoneoftheqclasslabelssimultaneously.Given asetofqpredefinedlabels={𝜆1,𝜆2,…,𝜆𝑞},thesubsetof relevant labelsassociatedwitheachoftheinstancescanbeviewedasabinary vector𝒚={0,1}𝑞whereeachelementis1ifthelabelisrelevantand 0otherwise.Inthisway,thegoalofMLCistopredict,foranunseen instance,abipartitionincludingits setsofrelevant(𝒚̂)andirrelevant labels(𝒚̂).

SeveralmethodsforMLChavebeenproposedintheliterature,aim- ingtohandlewiththethreemainproblemsinMLC,suchastheimbal- anceoftheoutputspace, therelationshipamonglabelsandthehigh dimensionalityoftheoutputspace.Thesemethodsarecategorizedinto threemaingroups:problemtransformation,algorithmadaptation,and EMLCs[14,15].

Problem transformation methods transform the multi-label prob- lemintooneormoresingle-labelproblems.Theseproblemsarethen solvedbyusingtraditionalsingle-labelclassificationmethods.Forease ofunderstanding,schemesofthemaintransformationsarepresentedin

Fig.1.BinaryRelevance(BR)[16]decomposesthemulti-labelproblem intoqindependentbinarysingle-labelproblems,thenbuildingqinde- pendentbinaryclassifiers,oneforeachlabel.BRissimpleandintuitive,

Fig. 1. MainproblemtransformationsinMLC.ForPS,labelsetsappearingless than2timesareprunedandreintroducedwithmostfrequentsubsets.

butthefactofconsideringthelabelsindependentlymakesitunableto modelthecompounddependenciesamongthelabels.BRdonotdeal withanyofthepreviouslydescribedproblemsinMLC.Inordertoover- comethelabelindependenceassumptionofBR,ClassifierChain(CC)

[5]generatesqbinaryclassifiersbutlinkedinsuchawaythateachbi- naryclassifieralsoincludesthelabelpredictionsofpreviousclassifiers inthechainasadditionalinputfeatures.InthiswayandunlikeBR,CC isabletomodeltherelationshipsamongthelabelswithoutintroduc- ingmorecomplexity.However,althoughitdealwiththerelationship amonglabels,itdoesnotconsiderthem,oranyothercharacteristicsof thedatatoselectthechain.Sincetheorderofthechainhasadetermi- nanteffectonitsperformance,otherapproacheshavebeenproposedto selectthebestchainordering[17,18].

LabelPowerset(LP)[19]transformsthemulti-labelproblemintoa multi-classproblem,creatinganewclassforeachdistinctcombination oflabels,calledlabelset,thatappearsinthedataset.Thismethodisable tostronglymodeltherelationshipsamongthelabels,butitscomplex- itygrowsexponentiallywiththenumberoflabels;itisalsonotable topredict alabelset thatdoesnot appear inthetraining set.There- fore,althoughitisabletohandlewiththerelationshipamonglabels, LPgreatlyincreasesthedimensionalityoftheoutputspace,aswellas itsimbalance.PrunedSets(PS)[7]triestoreducethecomplexityofLP, focusingonmostimportantcombinationsoflabelsbypruninginstances withlessfrequentlabelsets.Tocompensateforthislossofinformation, itreintroducestheprunedinstanceswithamorefrequentsubsetofla- bels.Thus,PSconsiderstheimbalanceofLP’soutputspacetoreduceits dimensionalityandcomplexity.ChiDep[20]createsgroupsofdepen- dentlabelsbasedonthe𝜒2_{testforlabelsdependenciesidentification.} ForeachgroupofdependentlabelsitbuildsaLPclassifier,whilefor eachsinglelabelwhichisnotinanygroupitbuildsabinaryclassifier. ChiDeptriestoreducethedisadvantagesoftheindependenceassump- tionofthebinarymethodsandallowsforsimplerLPmethods.Besides, ChiDepconsiderstherelationshipamonggroupoflabelsandthedimen- sionalityoftheoutputspaceinbuildingphase,thereforebeingableto reducetheimbalanceineachmodelifthegroupsaresmall.

Themethodsinthealgorithmadaptationgroupadaptorextendex-