PROW: algorithm A Pairwise with constRaints, Order and Weight The Journal of Systems and Software

ContentslistsavailableatScienceDirect

The Journal of Systems and Software

jo u r n al h om e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / j s s

PROW: A Pairwise algorithm with constRaints, Order and Weight

Beatriz Pérez Lamancha

^a,∗

, Macario Polo

^b

, Mario Piattini

^b

aSoftwareTestingCentre,RepublicUniversity,Montevideo,Uruguay

bAlarcosResearchGroup,Castilla-LaManchaUniversity,CiudadReal,Spain

a r t i c l e i n f o

Articlehistory:

Received28November2012 Receivedinrevisedform23July2014 Accepted2August2014

Availableonline16September2014

Keywords:

Softwaretesting Combinatorialtesting

a b s t r a c t

Testingsystemswithmanyvariablesand/orvaluesisoftenquiteexpensiveduetothehugenumberof possiblecombinationstobetested.Thereareseveralcriteriaavailabletocombinetestdataandproduce scalabletestsuites.Oneofthemispairwise.Withthepairwisecriterion,eachpairofvaluesofanytwo parametersisincludedinatleastonetestcase.Althoughthisisawidely-usedcoveragecriterion,two maincharacteristicsimproveconsiderablypairwise:constraintshandlingandprioritisation.

Thispaperpresentsanalgorithmandatool.Thealgorithm(calledPROW:PairwisewithconstRaints, OrderandWeight)handlesconstraintsandprioritisationforpairwisecoverage.ThetoolcalledCTWeb addsfunctionalitiestoexecutePROWindifferentcontexts,oneofthemisproductsamplinginSoft- wareProductLinesviaimportingfeaturemodels.SoftwareProductLine(SPL)developmentisarecent paradigm,whereafamilyofsoftwaresystemsisconstructedbymeansofthereuseofasetofcommon functionalitiesandsomevariablefunctionalities.AnessentialartefactofaSPListhefeaturemodel,which showsthefeaturesofferedbytheproductline,jointlywiththerelationships(includesandexcludes) amongthem.PairwisetestingcouldbeusedtoobtaintheproductsamplingtotestinaSPL,usingfea- turesaspairwiseparameters.Inthiscontext,theconstrainthandlingbecomesessential.Asadifference withrespecttoothertools,CTWebdoesnotrequireSATsolvers.

ThispaperdescribesthePROWalgorithm,alsoanalysingitscomplexityandefficiency.TheCTWeb toolispresented,includingtwoexamplesofthePROWapplicationtotworealenvironments:thefirst correspondstothemigrationofthesubsystemoftransactionsprocessingofacreditcardmanagement systemfromAS400toOraclewith.NET;thesecondappliesboththealgorithmandthetooltoaSPLthat monitorsandcontrolssomeparametersoftheloadintrucks.

©2014ElsevierInc.Allrightsreserved.

1. Introduction

Softwaredevelopmentsuffersfromtheimpossibilityofexhaus- tivetesting.Thus,researchersstrivetofindabalancebetweentest suitesizeandcoverage(e.g.theabilitytofindfaultsinthesystem undertest).Differenttestgenerationtechniquesproducedifferent testsuitesaimedatreachingacertaintestdatacoveragecriterion (i.e.usingeachtestdatumatleastonce,usingallpairsofvalues, etc.).

Combinationstrategies(alsocalledCombinatorialInteraction Testing,CIT)areatypeoftestcaseselectionmethodwheretest casesarecreatedbythecombinationof“interestingvalues”,which havebeenpreviouslyidentifiedbythetester.

Theinputisasetofparameters,eachwithsomeelements(val- ues).Theoutputisasetofcombinations,eachonecomposedby

∗ Correspondingauthor.

E-mailaddresses:bperez@fing.edu.uy(B.PérezLamancha), [email protected](M.Polo),[email protected](M.Piattini).

oneelementofeachinputset(Grindaletal.,2005).Forinstance, t-wise strategies ensurethatall combinationsof anyt parame- ters (input sets)are includedin at leastone test case (Grindal etal., 2005).Pairwiseis aparticularcase oft-wise, wheret=2.

Inthis case,thecoveragecriterionis thatallthepairsbetween thevaluesofanytwoparametersareincludedinatleastonetest case.

Fig.1showsthepossibleconfigurationelementsofawebappli- cationthatcouldbeexecutedwithdifferentoptionsforOperating System,Browser,WordprocessorandDataBase.Atestsuitethat achievepairwisecoverageforthisexampleisshowninFig.2,which containsallthepossiblecombinationsbetweenpairsofvaluesfor OperatingSystem,Browser,WordprocessorandDataBase.How- ever,thistest suitedoesnothave intoaccounttherelationship betweenpairsof values,and canleadtotheinclusion ofmany testcasesthatcontainsemanticallymeaninglesscombinationsof test data. For theconfigurations in Fig.2, the casescombining LinuxwithIExplorer or LinuxwithMicrosoftWord(test cases1, 12, 13 and 16)make nosense.If the unfeasibletest cases are directly removedfrom thesuite,then there will beinteresting http://dx.doi.org/10.1016/j.jss.2014.08.005

0164-1212/©2014ElsevierInc.Allrightsreserved.

Fig.1. Configurationparameters.

Fig.2. Testcasesobtainedusingpairwisecriterion.

pairs which will remainuntested: iftest case 1 (Linux, Firefox, MicrosoftWord)isremovedduetotheincompatibilityofparam- eters1and3,thenthepair(Firefox,MicrosoftWord),whichislegal andshouldbetested,wouldbeoutsidethetestableconfigurations.

Then,thefurtherremovalofinvalidtestcasesisnotanefficient solution, since valid pairs will be also removed from the test suite.

Detectingthesecombinationsinadvancerequiresresortingto thesemanticsoftheinvolvedparameters,i.e.whattheparameters standfor.Thetoolsmustnotfocusstrictlyonprovidinganalgorith- micsolutiontothemathematicalproblemofcombinatorialtesting, alsoaccountforothercomplementaryfeatures,whicharerather importantinordertomakethesetoolsreallyusefulinpractice,such astheabilitytohandleconstraintsontheinputdomains(Calvagna andGargantini,2008;Grindaletal.,2005).

Arecentsoftwaredevelopmentparadigmwherecombination testingisbeingappliedisSoftwareProductLine(SPL).ASPLis

“asetofsoftware-intensivesystems,sharingacommon,managed setoffeaturesthatsatisfythespecificneedsofaparticularmar- ketsegmentormissionandthataredevelopedfromacommon setofcoreassetsina prescribedway”(ClementsandNorthrop, 2001).ProductsinanSPLshareasetofcharacteristics(common- alities)anddifferinanumberofvariationpoints,whichrepresent thevariabilitiesoftheproducts.

VariabilityisacentralconceptinSPLdevelopment.Itallows forthegenerationofdifferentproductsinthelinebyreusingcore assets.Variabilityis captured throughfeatures. Afeature is an incrementinprogramfunctionalitythatcustomersusetodistin- guishoneSPLproductfromanother(Kangetal.,1990).Afeaturecan beaspecificrequirement,aselectionamongoptionaloralternative requirements,orcanberelatedtocertainproductimplementation orcharacteristics(Griss,2000).Severaldocumentedexamplesof SPLexists,¹forexampletheNokiaSPLwhereseveralmobilephones shareasetofcommoncharacteristics:inthiscaseafeaturecanbe

1 ProductLineHallofFame:http://splc.net/fame.html.

thelanguage,andthefeaturevariantscanbe:Spanish,French,Chi- nese,amongothers.Featuremodelsareusedtorelatefeaturesto eachotherinvariousways,showingsub-features,alternativefea- tures,optionalfeatures,dependentfeaturesorconflictingfeatures (Griss,2000).

OneofthemainproblemsinSPLtestingistheselectionofthe productstotest.Froma featuremodel,severalproductscanbe built,andthepossiblecombinationsoffeaturestoobtainproducts can make it impossible to test all of them. One possiblesolu- tionis obtainingaProductsampling settoSPLtesting.To this end,combinatorialtestingstrategiescanbeused,takingthefea- turesasparametersandthefeaturevariantsasparametersvalues.

Moreinformationcanbeobtainedfromafeaturemodel:wealso wouldgetallthepossiblerelationshipsbetweenfeatures(includes, excludes)and beable torepresent this withthecombinatorial technique.Theserelationshipsbetweenfeaturesrestrictsthepos- siblecombinationofproducts,forexample,ifafeature“excludes”

anotherfeature,thenalltheproductsthatcombinesthevariantsof thesefeaturesaremeaningless,therelationshipsinafeaturemodel canbehandledautomaticallyusingconstrainthandling.Duethe relationshipsinafeaturemodeldefinestheproductsthatcanbe generatedintheSPL,thefunctionalityofconstraintshandlingthat couldbedesirableincombinatorialtestingalgorithmsinSPLtesting becomesessential.

Thecontributionofthisarticleistwofold:first,itpresentsan algorithm called PROW (Pairwisewith constRaints, Orderand Weight)forgeneratingpairwisetestsuites poweredbydomain semanticsandknowledge,whicharecapturedthrough:

• Constraints:Thealgorithmmakesitpossibletoexcludeallthe pairsbetweenparametervaluesthataresemanticallymeaning- less.

• Weight assignment: The algorithm is alsocapable of includ- ingthosepairswhichrequiremorefrequenttesting,assigning aweighttoeach pairbetweenparametervalues,thisvalueis referredinthispaperas“pairweight”.Withthis,thetestercan specifythemostimportantpairsfromatestingpointofview.

• Orderingfortestcaseprioritisation:Thetestsuitegenerated bythealgorithmisordered usingthesummation ofthe“pair weights”involvedinthetestcase.Theorderedtestsuitecanbe usedtotestplanning,sincethemostimportanttestcasesmustbe testedearly.Thisisespeciallyusefulinregressiontesting,when thetimefortestingisnotsufficienttoexecutethecompletetest suite.

ThePROWalgorithmhasbeenimplementedinapubliclyavailable webtool,underaGNUlicensecalledCTWeb.Thisisthesecond contribution:CTWebtoolprovidesfeaturesthatallowtheuseof PROWefficientlyandalsoforproductsamplinginSPL.Someofthe functionalitiesthatprovideforCTWebtoimprovePROWinclude:

• Productsampling:Allowstoloadafeaturemodelandautomat- icallyprocessesthefeaturesandtheirrelationshipstoobtainthe parameterstoexecutePROW.Theresultisthesetofproductsto betestedintheSPL.

• Settingthebasetestsuite:Testcasesobtainedwithcombinato- rialtestingmaynotexactlybethemostwidelyusedinreality, evenifweightisassignedtoprioritisethem.Thisfunctionality allowsthetestertodefineasetoftestcasesasthebasisonwhich torunPROW.Thisbasetestsuiteisuploadedinatab-separated fileandCTWebmarkallpairscoveredbythebasetestsuiteas visitedpriortorunningthealgorithmPROW.Thisfunctionality canbeusedinboth,combinatorialtestingandproductsampling.

Theresultisasetoftestcasesthatsatisfiespairwisecoverage andcontainsthebasetestsuiteasasubset.

• Uploadingparameters file: For combinatorialtesting, CTWeb allowstouploada tab-separated file,where thetester speci- fiestheparameters,itsvaluesandtheconstraintsandassociated weightbetweenpairs.ThisfileistheinputtoPROWtoobtainthe testcasesforcombinatorialtesting.

AfteranoverviewofcombinationtestingstrategiesinSection2,the PROWalgorithmisdetailedandpresentedinSection3.Section3.4 presentstheorderanalysisandtheevaluationofthePROWalgo- rithm,showingthesmoothdegradationofPROWalongthenumber ofparametersandparametervalues,andtheefficiencygainscom- paredwithotherapproaches.Section4includesadescriptionof CTWebandthefunctionalitythatallowsbothproductsamplingin SPLandpoweringofcombinatorialtesting.Twocasestudiesare presented,firstinSection5,acasestudywherecombinatorialtest datawasusedtotestabankingsysteminatestingconsultancyand theninSection6,asensormonitoringSPLisusedasacasestudyto obtainproductsampling.Section7discussessomeworksrelated toconstrainthandlingandtestcaseprioritisationincombinatorial testingandproductsamplinginSPL.Finally,conclusionsandfuture workarepresented.

2. Combinationtestingstrategies

ThecompletenessofthecoverageinCombinatorialInteraction Testing(CIT)isuptothedesigner,whodetermineshowtheparam- etervaluesarecombinedandused:Each-usecoverage(alsoknown as1-wise)indicatesthateachtestvalueisincludedinatleastone testcaseinthetestsuite.

Pairwisecoverage(alsoknownas2-wise)requiresthatevery possiblepairofvaluesofanytwoparametersbeincludedinsome testcase.t-wiseisageneralisationofpairwisewhereeverypos- siblecombinationofinterestingvaluesoftparametersbeatleast includedinonetestcase(Grindaletal.,2005).

Pairwisereliesonthesystem’spronenesstoexhibitfaultswith theinteractionsofconcretepairsofvalues,offeringagoodrela- tionbetweenthetest suitesize and itscapabilityfor revealing faults.Duetothecostofexecutingandre-executingtestsuites, itisimportanttokeepthetestsuitesizewithinreasonablelimits.

SincetheproblemofgeneratingminimumpairwisetestsetsisNP- complete(LeiandTai,1998),differentresearchershavedeveloped strategiestogeneratenear-minimumpairwisetestsets.Different testgenerationstrategieshavebeenpublishedforpairwisetest- ing(seeGrindaletal.,2005;Cohenetal.,2007;Nie andLeung, 2011).

Table1liststhemostimportantCITalgorithmsinthelitera- ture.Forthecategorycolumn,weusethetaxonomyfromCohen etal. (2007), whoclassifyCIT in:Algebraic(mathematicaltech- niquesareused,theseconstructionsarefastandproducesmall coveringarraysefficiently,althoughtheyarenotgeneral),Greedy (constructsasetoftestswhereeachtestcoversasmanyuncovered combinationsaspossible)andHeuristicsearch(whichstartfroma preexistingtestsetandthenappliesaseriesoftransformationsto thesetuntilitcoversallthecombinations)(NieandLeung,2011).

Foreachalgorithm,Table1shows:itsname,thebibliographic reference, the category, algorithmavailability (indicates whether the pseudocode was publishedor not),test size table (most of thealgorithmsinclude atest sizetablecomparingthetestsize obtainedagainstotheralgorithms),timetable(indicateswhether thealgorithm includesa tablewiththeexecution time for dif- ferent testinputs),constraints(indicates whetherthealgorithm considersthetreatmentofconstraintsbetweenparameters),pri- oritisation(indicates whetherthealgorithm considers priorities betweenparametervalues,strength(referstothestrengthofthe algorithm,i.e.,ifitresolvespairwiseort-wisecoverage),license(if

thealgorithmisincludedinatooldevelopedbytheauthors,this columnindicatesthelicensetypeforitsdistribution)andplatform (indicatestheexecutionplatformforthetool).LastrowinTable1 showsthedataforthePROWalgorithmpresentedinthispaper.

Thetablealsoshowsthatgreedyalgorithmsarethemostpopular strategyforCIT.

Therearetwoclassesofgreedymethods:thefirstisone-row- at-a-time,basedontheAutomaticEfficientTestCaseGenerator (AETG).Algorithm 1 showsits pseudocode.Basically,each iter- ation startswithanemptycombination, whichis progressively completedwiththeinclusionofthevaluewhichappearsinthe mostunvisitedpairs.Iftheselectedtestvalueproceedsfromtheith parameter,itisplacedattheithpositionofthecombination,which inthiswaystayspartiallyfilled.Theremainingvaluesareselected takingintoaccountboth(1)theircompatibilitywiththepreviously selectedvaluesand(2)thenumberofunvisitedpairstheyvisit.

Asanexample,considertestingthewebapplicationwhosecon- figurationparametersareshowninFig.1.Thetesterwishestotest thesystembehaviourdependingontheoperatingsystem,browser, wordprocessoranddatabase(seeFig.1).AETGproducesthe16test casesappearinginFig.2.

ThesecondclassofgreedymethodsisbasedontheInParameter Order(IPO)algorithm(LeiandTai,1998),whichbeginsbygener- atingallt-setsforthefirsttfactorsandthenincrementallyexpands thesolution,bothhorizontallyandverticallyusingheuristicsuntil thearrayiscomplete(NieandLeung,2011).

Algorithm1. AETGalgorithm(takenfromCohenetal.,1996) Assumeasystemwithktestparametersandthattheithparam- eterhasl_idifferentvalues.Assumethatwehavealreadyselected rtestcases.Weselectther+1byfirstgeneratingMdifferentcan- didatetestcasesandthenchoosingonethatcoversthemostnew pairs.

Eachcandidate testcase is selectedby thefollowinggreedy algorithm:

1.Chooseaparameterfandavaluelforfsuchthatthatparameter valueappearsinthegreatestnumberofuncoveredpairs.

2.Letf₁=f.Thenchoosearandomorderfortheremainingparam- eters.Then,wehaveanorderforallkparametersf₁,...,f_k. 3.Assumethatvalueshavebeenselectedforparametersf1,...,f_j.

For1≤i≤j,lettheselectedvalueforf_ibecalledvi.Then,choose avaluevj+1 forf_j+1asfollows.Foreachpossiblevaluevforf_j, findthenumberofnewpairsinthesetofpairs{fj+1=vand f_i=vi,1≤i≤j}.Then,letvj+1beoneofthevaluesthatappeared inthegreatestnumberofnewpairs.

Notethat,inthisstep,eachparametervalueisconsideredonly onceforinclusioninacandidatetestcase.Also,thatwhenchoos- ingavalueforparameterf_j+1,thepossiblevaluesarecompared withonlythejvaluesalreadychosenforparametersf₁,...,f_j. 2.1. Theconstraintproblem

Insoftwaretesting,somecombinationsofparametervaluesare frequentlyinvalid.If thepairwisealgorithm doesnot providea solutionfortheconstraintsbetweenparameters,thetestermust manually review theresults obtainedfrom pairwise strategies, since there aremany situations where thegenerated testsuite containsmeaninglesspairs.

NieandLeung(2011)recentlypresentedasurveyaboutcom- binatorial testing. They argue that theexistence of constraints increasesthedifficultyinapplyingcombinatorialtestingdueto:

(1) Several existing test generation methods cannot deal with constraints,andignorethem,inspiteofthefactthatignoringcon- straintsmayleadtothegenerationoftestconfigurationsthatare invalid,ineffectivetestplanningandwastedtestingeffort.(2)Itis

Table1

ExistingCITtechniques.

Technique Reference Category Algorithm Size

table Time table

Constraints Prioritisation Strength License Platform

AETG Cohenetal.(1997)

andLottetal.(2005)

Greedy Y Y N Y N t-wise Proprietary Web

PICT Czerwonka(2006) Greedy Y Y N Y Y t-wise Freeware Windows

Comm.line

IPO/PairTest LeiandTai(1998) Greedy Y Y Y N N 2-wise Not

available

ATGT Calvagnaand

Gargantini(2008)

Algebraic Y Y N Y N 2-wise Freeware JavaApp

Simulated Annealing/CASA

Cohenetal.(2003) andGarvinetal.

(2009)

Heuristic Y Y Y Y N t-wise Open

Source (GNU)

Comm.line

TestCover/CATS Sherwood(1994) Greedy, Algebraic

N Y N Y N 2-wise Proprietary Web

DDA BryceandColbourn

(2006,2007,2009)

Greedy Y Y Y Y Y t-wise Not

available

CTS Hartman(2005) Algebraic Y Y Y Y N t-wise Hasbeen

retiredby IBM

TConfig Williams(2002) Algebraic Y Y Y N N t-wise Freeware JavaApp

TCG TungandAldiwan

(2000)

Greedy Y N N Y N 2-wise Freeware Windows

EXACT YanandZhang

(2008)

Algebraic Y Y Y N N t-wise Freeware Linux

IPOG/ACTS/FireEye Leietal.(2008) Greedy Y N Y Y N t-wise Freeware JavaApp

PROW Greedy Y Y Y Y Y 2-wise Open

Source (GNU)

Web

difficulttodesignageneralalgorithmfortestgenerationwithdue considerationofconstraints.(3)Evenasmallnumberofconstraints maygiverisetoanenormousnumberofinvalidconfigurations.

Whenthegenerated testsuitecontainsmanyinvalidtestcases, thiswillcausealossofcombinationcoverage.(4)Complicatedcon- straintsmayexistintheSUT,andmultipleconstraintscaninteract toproduceadditionalimplicitconstraints.Itisbothtimeconsum- ingandhighlyerrorpronetomanuallydealwithconstraintsintest suitegeneration(NieandLeung,2011).

Cohenetal.(2007)discussthatmanagingconstraintsinexisting algorithmsforCIT exhibitsatleast oneofthefollowing limita- tions:itignorestheconstraintsaltogether,requiringtheuserto explicitlydefineallillegalconfigurations;itattemptstobiastest generationtoavoidconstraints“ifpossible”,butdoesnotguarantee theavoidanceofillegalconfigurations;itmentionsconstraintsasa straightforwardengineeringextensiontobesolvedlater;orituses aproprietary(unpublished)methodthatcannotbereproducedby theresearchcommunity.

Itisnoteworthythatconstraintshandlingbringsaboutachange inthenumberoftestcasesobtainedforpairwisecoverage.Depend- ingontherestrictionsthat aredefined, moreor lesstest cases maybeobtained,comparingwiththoseobtainedwithouttaking intoaccountconstraints.Fortheconfigurationexample,ifthepairs (Firefox,MicrosoftWord)and(Firefox,SQL)arenotpossible,thenthe firsttestcaseinFig.2mustbechanged.Thistestcasevisits6pairs, 2ofwhichweredeleted.Theremaining4pairscannotbecovered withasingletestcase,sincethepairs(Linux,Firefox)and(Linux, SQL)cannotbeinthesametestcasebecause(Firefox,SQL)has beeneliminated,thenweneedtwotestcasesinsteadofone.

Thus,properhandlingofconstraintsisakeyissueincombinato- rialtesting.Wesolvethisproblemwithapairwisealgorithmcalled PROW,agreedyalgorithmthatsupplementsFig.1withinforma- tionaboutconstrainedcombinationswhichshouldnotbeincluded inthetestsuite(e.g.combiningLinuxwithIExplorerorLinuxwith MicrosoftWord),sothatthealgorithmskipsthesecombinations.

AmoredetailedanalysisoftheexistentCTmethodsandtoolsis presentedinSection7.

3. ThePROWalgorithm:Pairwisecoveragewith constRaints,OrderandWeight

PROWalgorithm(PairwisewithconstRaints,OrderandWeight) isagreedyheuristic,buildingone-test-a-timethatlocallyoptimises thesolution.PROWisagreedyalgorithmpartiallybasedonAETG, empoweredwithdomainsemantics.Specifically,parametercon- straintsandweightsforpopularconfigurationsareintroducedto avoidundesiredpairsandhandlerepetitivevaluetesting,respec- tively.Another significantdifferenceisthat thePROWsolution isdeterministic,i.e.,twoexecutionswiththesameinputreturn thesameoutput.Obviously,thisallows theregenerationofthe same test cases later on, but is specially interesting to gener- atemoreorlesssimilartestsuiteswhenconstraints,weightsor newparametersorfeaturesareaddedorchanged.So,thediffer- encesbetweenPROWandAETGareenoughsignificant:besides requiringadditionalinputs(namelyconstraintsandweights),the methodfor selecting thenexttest case and thestop condition arealsodifferent,sinceittakesintoaccountvaluesincompatibil- itiesandtheimportanceofeachnewtestcasethroughitstotal weight.

Ingeneral,PROWstrivestoovercomesomeoftheshortcomings inpairwiseapproaches,namely:

• inter-parameterconstraintsarenotconsidered.Thiscanleadto unrealisticvaluecombinationsbeingchecked(e.g.LinuxwithIEx- plorer).

• repetitivevaluetesting.IntheexampleshowninFig.2,Linuxis testedin8testcaseswhileWindowsandMacaretested4times each.Sincethisrepetitionisinevitable,itispreferabletoselect thosevaluesthataccountforthemostpopularsettings:e.g.if themostcommonconfigurationswillbebasedonWindowsand Chrome,thenthetesterwillbeinterestedinincludingthispairin moretestcases(ifs/hecanselectoneoftwopairswiththesame numberofvisits,probablys/hewillprefertoinclude(Windows, Chrome)insteadof(Linux,Opera)).

This section describes preconditions, pseudocode and post- conditions for PROW algorithm, using as runningexample the configurationsinFig.1.

3.1. Preconditions

PROW takes into account the elimination of pairs and the weightassignedtoeachpair:beforeexecutingthealgorithm,the undesiredpairsaredeletedandthepairweightsassigned.These preconditionsaredescribedinAlgorithm3.

Algorithm2. PROW:auxiliaryfunctions LetSbethesetofparameters,S=



S1...Sn



.EachparameterSi

isanorderedsetofvalues,withk≥1variableforeachparameter S_i.

LetP=



S_i×S_j|i=1...n−1,j=i+1...n



theCartesianprod- uctofS,representingallthepairsbetweenparametersvalues.Let P^⊆PtheresultofdeletingtheundesiredpairsinP.

Thefollowingsfunctionsareusedinthealgorithm:

• The weight of each pair is defined by a function weight: P^→R,weight(p)=w,p∈P^andw_R,wrepresentstheweight assignedtothepairpinP^.

• Thenumberoftimesthatapairisvisitedbythealgorithmis definedbythefunctionvisits:P^→N|visits(p)=n.Atthebegin- ningofthealgorithmvisits(p)=0,∀∈P^.

• The number of pairs in that each value for each parameter appearsinP^isrepresentedbytheremainsfunction:S×N→N.At thebeginningofthealgorithmremains(S_i,vt)=



n

j=1



#S_j x=1r_t



, wherert=1when{∃^p∈P^|p∈S_i×S_j,p=(vt,vx)orp=(vx,vt)}, elsert=0.remainsrepresents,foreachvalueineachparameter, thenumberofpairsthatthealgorithmhasnotcoveredyet.This functionhasintoaccountthedeletedpairs.

• TS represents the test suite, TS=



TS₁...TSm



. The weight of the test case TS_i can then be calculated as the sum- mation of the weight of the pairs covered by the test case,thefunctionis:weight_testcase:TS→R|weight_testcase(TS_i)=



n−1 t=1



j=n

j=t+1weight(s_it,s_ij).

• Theamountofpairsvisitedbyatestcaset=t₁...t_n,thatwerenot visitedbyprevioustestcases.Pairs_visited(t)=



n−1

i=1



n

j=2visited_ij, wherevisited_ij=1ifvisits(ti,tj)=0andvisited_ij=0ifvisits(ti,tj)>0.

Fig.3indicatesthepreconditionsfor thesampleproblemin Fig.1.Constraintsbetweenpairsaremarkedasremovableandpri- oritisationbetweenpairsissetinthe“Sel.factor”column,which representstheweightassigned.Theweightscanbesetbyhandor begiveninthedescriptivefileofthesystemundertest.

3.2. Pseudocode

Algorithm3showsthepseudocodeofPROW.Auxiliaryfunc- tionsusedaredescribedinAlgorithm2.

MainconceptsinvolvedinPROWare:

1.Thenexttestvaluetobeincludedinthenexttestcase:thedecision aboutwhetheracombinationisincludedinorexcludedfrom thefinaltestsuiteconsiders:(a)thatthepairsselectedhave notbeenremoved,(b)thecombinationthatvisitsthehighest numberofunvisitedpairs,and(c)ifmorethanonecombination visitsthehighestnumberofpairs,choosethecombinationwith thehighestweight.

2.Thestopcondition:TheoriginalAETGalgorithmstopswhenall pairsinthepairsethavebeenvisitedonce.Whensomepairsare deleted,pairsthatareunreachablemayexist.Inthiscase,PROW stopswhenthecombinationselecteddoesnotvisitanyunvisited

pair.Ifvisits(p)=0,thenthepairpisunreachableandthereis nocombinationwiththeremainingparametersthatachievesa validtestcaseusingthepairp.Ifallthepairsarereachable,then PROWstopswhentherearenomoreunvisitedpairsinP^.

Let TS=



TS1...TSm



, test suite obtainedusing the PROW algorithmwhereTS_i=



s_i1∈S₁,...s_in∈Sn



representstheith testcase.

LetPairs_i=



(s_it,s_ir)|t=1...n−1,r=2...n



,where sitand s_ir∈TS_i.Pairs_iarethepairscoveredbythetestcaseTS_i.Atthe endof thealgorithmthefunctionvisits:P^→N|visits(p)=n, whichrepresentsthenumberoftimesthatthepairpisvisited byallthetestcases,canbe:

(a)visits(p)>0,thenthepairpisvisitedinatleastonetestcase (b)visits(p)=0,thenthepairpisunreachableandthereisno combination oftheremainingparametersthat achievesa validtestcaseusingthepairp.Inthiscase,itdoesnotexist atestcasec=



s_i1∈S1,...s_in∈Sn



wherePairs_visited(c)>0 andccontainsthepairp.

3.Orderedtestsuite:Thealgorithmordersthetestcasesaccording tothesummationoftheweightofthepairsinvolvedinthetest case.Itallowstesterstoknowthemostvaluabletestcasestobe executed.Ifthetimescheduledfortestingtasksisnotsufficient torunallthetestcases,theycanbeexecutedaccordingtothis order.

Algorithm3. PROWalgorithm Preconditions:

LetSbethesetofparameters.EachparameterS_iisanordered setofvalues,withk≥1variableforeachparameterS_i.

LetP=



S_i×S_j|i=1...n−1,j=i+1...n



theCartesianprod- uctofS,representingallthepairsbetweenparametersvalues.Let P^⊆PtheresultofdeletingtheundesiredpairsinP.

ForeachpairpP^,theweightisassignedbythetester,being zerotheweightbydefect.

Input:S,P’

Steps:

1.TS={};TSrepresentsthetestsuite

2.c={};crepresentsthenexttestcasetobeaddedtoTS 3.continue:=true

4.While((∃p∈P^|visits(p)=0)andcontinue

(a)Initialisetest casecputtingthevaluevj inthepositioni, wherevjisthevaluethatappearsinmoreunvisitedpairs,i.e., v_jmaximisesthefunction:Maxⁿ_i=1(Max^#S_j=1ⁱ(remains(S_i,v_j)).

(b) ForeachremainingparameterS_k={v1...vt},k=1...n,k /=i i.maxPairs:=Pairs_visited(c),maxPairsstores thepairs cov- ered by thetest case c up tonow and uncovered by previoustestcases.

ii.maxWeight:=weighttestcase(c), stores the summation of theweightofthepairscoveredbythetestcasecupto now.

iii.maxRemains:=0,stores thevaluethatappearsinmore unvisitedpairs

iv.Foreachvaluev_i∈S_k,letcibethecombinationcwiththe valueviassignedfortheparameterk,c_iiscandidatetobe cifmeets:

A.∃ pair p=(v_j,v_i)∈P^,∀ parameterj assigned previ- ouslyinc_i

B.if (maxPairs<Pairs(c_i)) then {maxPairs=Pairs(c_i);

maxRemains:=remains(S_k,v_i);ctemp:=c_i}

C.else if ((maxPairs=Pairs(c_i)) and (maxRemains≤remains(S_k,vi))) then { if (maxWeight<weighttestcase(c_i))thenctemp:=c_i} v.endfor

vi.c:=ctemp}

Fig.3.PairtablesgeneratedusingPROWalgorithm.

(c)endfor

(d)ifPairs_visited(c)>0then i.TS:=TS∪{c}

ii.updatethenumberofvisitsforthepairscoveredbyc(and returnedlaterbythefunctionvisits

iii.updatethenumbersofpairsthatremainsuncoveredsub- tractingthepairscoveredbyeachvalueassignedinthe testcasec(thesevaluesarereturnedlaterbythefunction remains

(e)else

i.continue:=false 5.endWhile

Output:TS

Astep-by-steptrackingofthePROWalgorithmforthesample problemisdepictedinFig.4.Foreachparametervalue(O.S.={Linux, Windows,Mac},...),thecorrespondingcellrepresentsthenumber of unvisited pairs in which that value appears in the iteration denotedbythefirstcolumn.Thisvalueisreturnedbythefunc- tionremainsin thealgorithm pseudocode.The firsttest case is initialisedbysettingthevaluewhichvisitsthemostunvisited,non- removedpairsinthecorrespondingposition,forexample,atthe beginning of the algorithm (row 1), remains(OperatingSystem,

Linux)=8 because Linux appears in eight pairs of values.

ThealgorithmstartswithatestsuiteTS={}andatestcasec={}.

Instep4(a)(correspondingtothefirstrow),SQLandOracleappear inelevenpairs,thealgorithmselectsSQLand,then,c={-,-,-,SQL}.

Step4(b)completestheotherparameters.ForOperatingSystem, theselectedvalueisWindows,becauseitistheonewiththemost unvisitedpairs(nine).Now,c={Windows,-,-,SQL}.

For Browser, both Firefox and Chrome visit nine pairs. The algorithm selectsFirefoxbecauseit maximises thetotalweight, being now c={Windows, Firefox,-,SQL}. The same occurs for WordProcessor, OpenOffice, Feng Officeand Google Docs which havenine pairseach. Thealgorithmselects OpenOfficebecause this value maximises the weight of the combination, which is: weight(Windows, Firefox) + weight(Windows, Open Office) + weight(Windows,SQL)+weight(Firefox,OpenOffice)+weight(Firefox, SQL)+weight(OpenOffice,SQL)=0.8+0.5+0+0+0+0=1.3.Finally c={Windows,Firefox,OpenOffice,SQL}.

Beforestartingthenextiteration,PROWupdatesthenumberof unvisitedpairsthatremainforeachvalue(secondrowinFig.4).

Thesecond iterationselects Oracleas themostunvisited value, includesitinthecurrenttest caseand completesitwithLinux, ChromeandFengOffice.Thistestcaseremainsas(Linux,Chrome,Feng Office,Oracle),whoseweightis1.1.

Fig.4. PROWalgorithmexecutedstepbystep.

At this point is important to note that the algorithm must findthevalueineachparameterthatappearsinthemostunvis- itedpairs,alwaystakingintoaccountthatthepairsbetweenthe selected values must exist. In iteration 9 for example,Microsoft Wordhasthemostunvisitedpairs (4).PROWincludesitinthe initialcombination(-,-,MicrosoftWord,-).ForOperatingSystem, Macisthevaluewiththegreatestvalue,butitcannotbepickedup sincethepair(Mac,MicrosoftWord)wasdeleted.Therefore,PROW selectsWindowsandcompletesthecombinationwith(Windows, Chrome,MicrosoftWord,SQL),whoseweightis1.3.

Toillustratethestopcondition,considerthatinthe16throw, thepair(Opera,MicrosoftWord)remainsunvisitedbecauseitisnot possibletofindacombinationofparametervaluesthatcontains thispair.Apartiallybuilttestcaseis{-,Opera,MicrosoftWord,-}.

ThepossiblevaluesforO.S.are:Linux,WindowsandMac,butboth LinuxandMacareincompatiblewithMicrosoftWord.Thus,only WindowscouldbeselectedwithrespecttoWord.ButWindowsis incompatiblewithOpera.Then,bytheselfdefinitionofthesystem (andalthoughitisnotexplicitlyspecified),itisimpossibletofinda combinationcontaining(Opera,MicrosoftWord),soremainingthis pairunvisitedattheendofthealgorithm.Thealgorithmstopswhen therearenomoreselectabletestcasesandallvalidpairshavebeen visited,suchasinthisexample.

3.3. Postconditions

OncethePROWalgorithmhasbeenexecuted,itsresultsfulfil thepostconditionsdescribedinAlgorithm4.Inalgorithmswithout constraintshandling(forexampleinAETG),attheendofthealgo- rithm,eachpairwasvisitedatleastonce.InthecaseofPROW,due totheconstrainthandlingandtheweight,thepairsinP^canhave beenvisitedonce,morethanonce(thisisthecaseforthepairswith moreweightassignedbythetester)orzerotimes(thisisthecase withunreachablepairs).

Algorithm4. PROWpostconditions Postconditions:

LetTS=



TS1...TSm



,testsuiteobtainedusingthePROWalgo- rithm where TS_i=



s_i1∈S₁,...s_in∈Sn



represents the ith test case.EachtestcaseTS_icovers(n*(n−1)/2)pairs.

Let Pairs_i=



(s_it,s_ir)|t=1...n−1,r=2...n



, where s_it and s_ir∈TS_i.Pairs_i arethepairscovered bythetestcase TS_i.Theset ofallthepairscoveredbythetestsuiteTSisPairs=U^m_i=1Pairs_i.

Theweightofthetestcase TSicanthenbecalculatedasthe summationoftheweightofthepairscoveredbythetestcase,the functionis:

weight_testcase:TS→R|weight_testcase(TS_i)=



n−1 t=1

j=n



j=t+1

weight(s_it,s_ij).

The test cases resulting are ordered using the weighttestcase

function.Attheendofthealgorithmthefunctionvisits:P^→N| visits(p)=n,whichrepresentsthenumberoftimesthatthepairp isvisitedbyallthetestcases,canbe:

1.visits(p)=1,thenthepairpisvisitedonceforthePROWalgorithm 2.visits(p)>1, then the pairp is visited in more than one test

case.LetTS_i=



s_i1∈S₁,u,...v,s_in∈S_n



beoneofthetestcases wherethepair(u,v)appears.If(x∈S_i,w∈S_j)exists,wherethe ucanbesubstitutedbyxinS_iandvcanbesubstitutedbywinS_i, remainingTS_iasavalidtestcase,thenweight(u,v)≥weight(x,w).

3.visits(p)=0,thenthepairpisunreachablethereisnocombina- tionoftheremainingparametersthatachievesavalidtestcase usingthepairp.

Fig.5showsthetestcasesgeneratedbyPROW,orderedbytheir weight.Thetestercanleavetheweightforoneormorepairsunas- signed;inthatcasetheweightisconsidered0.

3.4. PROWevaluation

ThissectionevaluatesPROWinthefollowingsaspects:com- plexityanalysis,testsizeandtestexecutiontime.

3.4.1. Complexityanalysis

Forthecomplexityanalysis,weconsiderthenumberofparam- eters.Eachparameterhasanumberofvalueswhichareusually different.Toestimatethecomplexityweuseanarbitrarilylarge inputn(n→∞),takingnasboththenumberofparametersandthe parameterwiththemostvalues.

Step4isawhileloopwhichbuildsthetestsuite.Thefirststepin thewhileloopis4(a)wherethevaluethatvisitsthemostunvisited pairsisselectedasthefirstvalue.Thissearchiscarriedoutthrough therowsofthetableinFig.4,whichhasn²columns.Thusthisstep hasanorderofn².Step4(b)completestestcasec.Observethat step4(b)considers,first,thatthepairsinthecombinationexist;

Fig.5.OrderedtestsuiteusingPROWalgorithm.

second,thatthecombinationvisitsthemostunvisitedpairs;and third,thattheweightofthecombinationismaximum.Step4(b)is themostcomplexwithacomplexityorderofO(n⁶).

Thecomplexityfortheentirealgorithmdependsonthetimes thatthewhileloop(step4)isexecuted.Forthat,westudiedthe worstcase.Theworstcaseoccurswhenthealgorithmcalculates onetestcaseforeachunvisitedpair.SincethenumberofpairsTa- blesis(n^*(n−1)/2),andn²isthenumberofpairsineachtable,the complexityofthewhiledecision isn²*(n^*(n−1)/2)=(n⁴−n³)/2.

Thus, in the worst case the while construct must be executed (n⁴−n³)/2times.

Withinthewhileloop, thestepwiththemostcomplexity is step3(b)withO(n⁶).ThewhiledecisionorderisO(n⁴*n⁶).Then thewhileloophasO(n¹⁰)intheworstcase.TheorderofthePROW algorithmisthenO(n¹⁰),apolynomialorder.

3.4.2. Testsizeanalysis

Forthetestsizeanalysis,PROWiscomparedwithexistingalgo- rithmsforCIT.FirstwecomparePROWwithoutconstraints(i.e., withoutexcludinganypairorassigningweights).PROWwascre- atedtohandleconstraints,butthecomparisonhelpstoseethat PROWworks fine withoutconstraints. Fig.6 extends the table presentedinCzerwonka(2006),presentingtheresultingtestsize fordifferentinputsofparametersandvaluesforeachtool.These inputsarecommonlyusedtocompareCITalgorithms.Theinputis describedasNumvaluesNumparameters,indicatingthatthesamenum- berofvaluesisrepeatedforthenumberofparameters.Inthefirst row,forexample,theinputisfourparameterswiththreevalues each.ReferencestothetoolscanbefoundinTable1.Thelastcol- umninFig.6showsthetestsizeforPROW,withtheresultsshowing thatthecomparisonwithPROWdependsontheinputandthealgo- rithm.Asitis seenthetest sizesoftheotheralgorithmsarein generalsightlylessthanPROW,butforsomeinputsPROWisbet- ter.Note,however,thatthesedataareonlytohaveanideaofthe PROWbehaviour,sincePROWisdesignedtobeusedinpresence ofconstraintsand/orweights.

RegardingPROWwithconstraints,thefollowingtoolshandle constraints(seeTable1):AETG,PICT,ATGT,TestCover,DDA,EXACT, CASA,CTS,TCG,ACTS.Someofthesetoolsarenotavailable,andthe existingworkthatdealswithconstraintsdoesnotcomparethetest sizewithpreviousalgorithms.Therefore,comparingPROWwith theothersisverydifficultbecauseitdependsnotonlyonthetest inputsizeandthenumberofdeletedpairs,butalsoontherelation betweenthedeletedpairs.ToshowtheperformanceofPROWwith respecttootheralgorithms,weusePICT,TestCoverACTSandCASA.

Fig.7showsthecomparisonfordifferenttestinputs.Thefirst rowcorrespondstotheConfigurationsexampleinFig.1.Theothers rowscorrespondtoexamplespublishedintheliterature,showing thereference.

ConsideringPROWwithprioritisation,duethefactthatthe weightisusedinPROWforrepetitivetesting(asexplainedinSec- tion3),thetestsizeisthesameforPROWwithoutconstraintsor weightsorforPROWwithoutconstraintsandwithweights.The sameoccurswithPROWwithconstraints.Then,Figs.6and7are alsovalidforPROWwithweightsassigned.

Fig.6.Testsizecomparison.

Fig.7.Testsizecomparisonwithconstraints.

Fig.8. PROWexecutiontimes(inseconds).

Fig.9.Executiontimeinfunctionofdeletedpairs.

3.4.3. Executiontimeanalysis

Withtheaimofevaluatingtheexecutiontime,welaunchedthe algorithmagainstseveralsetsofparameters(from2to49)witha variablenumberofvalues(from2to29).Fig.8showstheexecution times(inseconds)ofthealgorithmfordifferentconfigurationsof parametersandvalues.Thisstudywascarriedoutonacommon laptopwithaCPUIntelCore2Duoat2.39GHzand3.49GBofRAM memory.Wedo notinclude comparisonswithotheralgorithms becausenotallofthemareavailableand,thosewhichare,runon differentplatforms,whatshouldleadtonon-extrapolableresults (forexample:PICTrunswithcommandlineofWindows;TestCover runsonawebapplication).

As we noted when the analysis order was presented, the execution time depends on both variables included in Fig. 8.

Note that the time has a stronger dependence on the num- berofvalues thanonthenumber ofparameters.In conclusion, we can say that the algorithm obtains test cases in a reason- able time even in uncommon situations, such as when using 30 parameterswith50 values each.In that case, thealgorithm takes469s.Inmostlikelycases,with5–20parametersand5–20 values each, PROWtakes between0.01 and 10s to obtainthe result.

ItisalsointerestingtoknowtheperformanceofPROWwhen differentnumbersofpairsaredeleted.Fig.9showsthetimeexe- cutionandthetestsizeobtainedwhentheparametershave10, 14,18,22,26 and30values each.Takingintoaccountthecon- straints,between0 and90%ofthepairsweredeleted.Asmore pairsare deleted,fewertestcases aregenerated byPROW, but moretimeisrequiredtoobtainthetestsuite.Thisisduetothe factthatPROWmustsearchforthesuitablecombinationtoful- filtherequirementsofdeletedpairs,whichiscostly.Eveninthe leastfavourablesituation(30values,90%deletedpairs),thetime is10.33s.

4. ImplementationofPROWinCTWeb

CTWeb² isawebapplicationforcombinatorialtestingwhich implementsdifferentalgorithmsfortestcasegeneration.CTWeb assiststhetesterintheuseofPROW,fortwodifferentpurposes:

combinatorialtestingandproductsampling.

4.1. CTWebforcombinatorialtesting

Inthiscase,CTWebassiststhetestdesignerinobtainingmean- ingfulcoveragetestswiththePROWalgorithm.Theprocessfollows thesesteps:

1.Introducetestparametersandtheirvalues:Thedatacanbe manuallyintroducedorbyuploadingatab-separatedvaluesfile.

Fig.10showsthemainpageoftheapplication,withthedata presentedinthepreviousexample.Thedesignerselectstheradio buttoncorrespondingtothePROWalgorithmandpressesthe Executionbutton.CTWeboutputsatablewithallthepossible combinations(seeFig.3).

2.Setconstraints:Thisisanoptionalstep,wherethetestercan deleteanyunrealisticcombinations.InFig.3thetesterremoves (Linux,IExplorer),(Linux,MicrosoftWord),(Mac,IExplorer),(Mac, MicrosoftWord)and(Windows,Opera).Theconstraintscouldbe alsouploadedinthetab-separated filewhere theparameters andvaluesaredefined.

3.Setweights:Thisisanoptionalstep,wherethetesterassigns theweight.Fig.3illustratesthispoint:theSel.Factorcolumn indicatestheweightfor eachpair.Thedesigner setsthefol- lowing combinations at the top witha commonness weight

2http://alarcosj.esi.uclm.es/CombTestWeb/combinatorial.jsp.

Fig.10.CTWebandthePROWalgorithm.

of1:(Linux,Firefox),(Windows,Explorer),(Mac,Firefox),(Linux, Open),(Windows,MicrosoftWord), (Mac,Open)and(IExplorer, MicrosoftWord).Theweightassignedbetweenpairscouldbe alsouploadedinthetab-separatedfilewheretheparameters, valuesandconstraintsaredefined.

4.Setbasetestsuite:Thisisanoptionalstep,wherethetestercan uploadafilewiththetestcasesthatareusedasabaseforPROW.

Thisfunctionalitycanbeappliedforcombinatorialtestingorfor productsamplinginSPL.Section6.2explainsthisfunctionality.

5.RunPROW:Thetesterpresses againtheExecutebuttonand CTWebrunsthePROWalgorithmwiththeinformationentered bythetester.Theoutputisthetestsuiteorderedaccordingtothe weightofthetestcases.Fig.5showsthe15testcasesobtained fortheconfigurationsexample.Section5showstheapplication ofPROWtoarealsystem.

4.2. CTWebforproductsamplinginSPL

Inthiscase,CTWebhelpsthetestertoobtaintheproductsamp- lingsetofallpossibledefinedinafeaturemodel;thestepsinCTWeb are:

1.Uploadfeaturemodel:AfeaturemodelrepresentingtheSPL is uploaded. The feature model is represented using a UML Model, that modelsthe OrthogonalVariability Model (OVM) (Pohl,2005).AUMLprofileforOVMwaspreviouslydefinedfor usinPérez Lamanchaand Polo(2010).CTWebautomatically processesthefeaturemodelintwosteps:first,thefeaturesare definedasparameters,itsvariantsasitsvaluesandtherelation- shipsbetweenthefeatureanditsvariantsareprocessed.These relationshipsadd newvalues to theparameters. The second stepprocessestherelationshipsbetweenfeaturesorbetween variantsbelongingtodifferentfeatures.Theserelationshipscon- straintsthecombinationbetweenparameters.

2.Setweights:Thisisanoptionalstep,wherethetesterassigns theweightstothecombinationbetweenvariants.

3.Setbasetestsuite:Thisisanoptionalstep,wherethetestercan uploadafilewiththetestcasesthatareusedasbaseforPROW.

Thisfunctionalitycanbeappliedforcombinatorialtestingorfor productsamplinginSPL.Section6.2explainsthisfunctionality.

4.RunPROW:Thetesterpresses againtheExecutebuttonand CTWebrunsthePROWalgorithmwiththeinformationprovided bythe tester.The outputis theproduct samplingtest suite.

Section6describestheapplicationofCTWebandPROWinan industrialSPL.

5. Casestudyforcombinatorialtesting:creditcard managementsystem

Recently, we collaborated with a testing consultancyin the SoftwareTestingCentre(CentrodeEnsayosdeSoftware,CES)³ in Uruguay.

Thesystemundertestwasacreditcardmanagementsystem developedtoprocessthecreditcardoperationsofdifferentbanks.

Theprojectcorrespondstoa migrationfromanAS400platform withRPGtoa.NETapplicationwithanOracledatabase,sothatthe testinginvolvedthemigrationofdatabetweenplatforms.Oneof thecriticalsubsystemstotestwas“Processingtransactions”:when aclientmakesapaymentbycreditcard,thetransactionsendsa messagetoaserver,codifiedundertheISO8583standard(Finan- cial transaction card originated messages — Interchange message specifications).Dependingonthecardstatus,associatedaccount, currency,etc.,thesubsystemundertestprocessesthetransaction andmayauthoriseorrejecttheoperation,aswellastakingother kindsofdecisions.Inthissubsystem,asetof21parameterswas identified (Fig. 11shows the namesofthe parameters andthe numberofvaluesineach).

Takingintoaccountthe21parameters,themaximumnumberof possiblecombinationsinthesubsystemundertestis990,904,320.

Forillustration,Fig.12showsthevaluesof12ofthe21existing parameters.

5.1. Consideringneitherconstraintsnorweights

TheapplicationofPROWtothesetofparametersinFig.11with- outtakingintoaccounttheconstraintsproduces99testcases,37of whichcontainunfeasiblepairs.Sinceeachtestcaseisatupleof21 valuesproceedingfromthe21parameters,thesearchforunfeasi- bletestcasesiscostlyanderror-prone.Moreover,validpairscould besetoutsidethetestsuiteiftheircontainertestcasesaredirectly removed.Fig.13showsthefirst10testcasesgeneratedbyPROW.

5.2. Consideringconstraintsbetweenpairs

Testingsystemswithmanyvariablesand/orvaluesisoftenquite expensive,duetothehugenumberofpossiblecombinationsto betested.Inordertoproduceanexample,weonlyfocusonthe constraintsbetweenthreeparameters:

3Software Testing Centre (Centro de Ensayos de Software, CES), http://www.ces.com.uy/.

Fig.11.Parametersandnumberofvalues.

• Transactioncurrency:representsthecurrencyusedinthetrans- action.

• Accountcurrency:representsthecurrencyoftheaccountcorre- spondingtothecreditcard.

• Location:indicateswhetherthetransactionwasmadeinthecard holder’scountryorabroad.

Therestrictionsbetweentheseparametersare:

• IftheAccountcurrencyispeso,thentheTransactioncurrencycan- notbeeuroorother.

• Thecustomerinthecountrycanonlypaywiththenationalcur- rency(peso)ordollars,andcannotpayabroadwithpesos.Then thesepairsbetweenTransactioncurrencyandTransactionlocation aredeleted:(peso,abroad),(other,local),(euro,local).

TheshadowedcellsinFig.13correspondtounfeasiblepairs.Test case2containsatransactioncurrencyinpesoswiththelocation abroad.Thistestcaseisunfeasibleandneedstobemodified.The sameoccurs withtest cases3, 6and 7.In thisexample,taking onlyintoaccountrestrictionsbetween3parameters,37%ofthe test casesneed tobemodified. Whenwe takeintoaccount all therestrictionsbetweenalltheparameters,thenumberofmod- ifiedtestcasesgrowsanditisverydifficulttomakethechanges manually.

TotakeintoaccounttheconstraintswithPROW,thetester makesapreviousselectionofundesiredpairsinordertogenerate thecombinations.PROWresolvesthisproblemgenerating116test cases,17testcasesmorethanwithoutanyconstraints.Thisdiffer- enceisduetotherestrictionsbetweenpairs(deletedpairs).Fig.14 showsthefirst10testcasesgeneratedusingPROW:allofthem arevalidandcanbeexecutedimmediately.PROWtakes670msto generatethe116testcases.

5.3. Consideringconstraintsandweightsbetweenpairs

Moreover,thetestercanassignaweighttothosepairsthatare moreworthwhiletotest.Thefollowingweightswereassigned:

• For transactions inside the country, it is more impor- tant to test transactions in pesos. For transactions abroad, it is more important to test them in dollars. Then the pairs betweenTransaction currency and Transaction Location have:weight(peso,local)=2,weight(dollar,local)=1weight(peso, local)=2, weight(dollar, abroad)=1, weight(euro, abroad)=0.5, weight(other,abroad)=0.5.

• The most important combinations are the local transactions.

Furthermore, the tester assigns more importance to testing the dollar transactions over other currencies for payments by credit card in foreign countries. The interaction between

Fig.12. Someparametersandthevaluesofthe“ProcessingTransaction”subsystem.

Fig.13.First10testcasesgeneratedusingthePROWalgorithmwithoutconstraints.

the Transaction location and whether the credit card opera- tes abroad or not is also important. The weights assigned are: weight(abroadyes, local)=1, weight(abroadno, local)=2, weight(abroadyes,abroad)=0.5,weight(abroadno,abroad)=1.

Withtheseweightsassigned,eachcombination(testcase)willbe returnedtothetesterinagivenorder,whichwilldependonthe weightofthepairsinthecombination.Thus,thealgorithmprovides aprioritisedlistofcombinations,whichallowstheprioritisedexe- cutionofthosetestcaseswithmoreimportance.Acharacteristic

ofpairwisetechniquesisthat,dependingonthenumberofvalues intheparameters,therewillbepairswhichwillappearmorethan once.Sometimes,differentvaluescanbeselectedforanewtest case.Fromthetester’spointofview,thebesttestcasesarethose thattestthemostimportantcombinations.Forthebankingexam- ple,withoutassigningweight,24transactions(20%)takelocalas theTransactionLocation.Whentheweightisassignedasdescribed above,87outof112transactionsarelocal.

Fig.15showsthetestcasesobtainedbymeansofPROWwith restrictionsandweights,usingthevaluesfortheweightandthe

Fig.14.First10testcasesgeneratedusingthePROWalgorithmwithconstraints.

Fig.15.First10testcasesgeneratedusingPROWwithconstraintsandweights.

restrictionsdescribedabove.Lastcolumnshowsthetotalweight calculatedforeachtestcase;thisweightiscalculatedbyaddingthe weightofeachpairinthetestcaseasdefinedbythetester.Thetest casesare,moreover,orderedusingthetotalweight.Bothduring theimplantationofthenewsystemasinitsfuturemaintenance, testerswillbeabletoexecutethemostimportanttestcasesfirst, accordingtotheorderedtestsuite.

Intheactualcasestudy,thereweremuchmoreconstraintsthat thosepresentedin theexample,and weremuch morecomplex sincetheyinvolvedalmostalltheparameters.Onesingularcase isaspecialbank(labelled“BankX”inFig.11)whichonlyopera- tesabroad,indollars,withacertainvalueinthe“Type”parameter, etc.Inordertodothemanagingoftheserestrictionsmoreeasily, wedecidedtosplittheinputfileintotwofiles:onewiththecon- cretevaluesforBankX,andanotherwiththegeneralconstraints applicabletotheremainingbanks.AlsointhespecialBankXfile therewererestrictionsamongpairs,whichalsoservedasamotiva- tiontoapplyPROW.Thetesterspositivelyvaluedthehelpprovided byPROW,whichmadetheirworkinthisprojecteasier.Inprevious testingprojectswithpairsexclusions,thetaskofreviewingthetest toremoveinvalidcombinationswasdifficultandtime-consuming, andPROWrelievedthemofthesetasks,acceleratingthetestdesign stepandlettingthemanticipatethetestexecutionstep.Notethat, besidesremovingtheinvalidpairsofatestsuitegotwithausual pairwisealgorithm,thetestermustaddnewtestcasestovisitthe validpairsthatarevisitedbytheinvalidtestcases.

6. UsingPROWforproductsamplinginSPL

Aswepointedout intheIntroduction,combinatorialtesting strategies arealsorequired inother contexts,suchasSoftware ProductLines(SPLs).Atahighlevel,aSPLcanbedescribedwith afeaturemodel,thatdescribesthefeaturesinthelineandhow theycanbecombinedinordertogetproducts.Thenotationinthe featuremodelrestrictsthepresenceofcertainfeaturesin some products,whatisaexcellentcontextfortheapplicationofPROW.

AsacasestudyweusetheSensorsSystemSPL,thatallowsmon- itoringandcontrollingtheconditionsoftheloadtransportedina truck.Alongthetruckthereareseveralnodes,andeachnodehas upto5sensorstomeasurethetemperature,pressure,humidity andconcentrationsofCH₄andCH₆gases.Viabluetooth,thenodes periodicallysendthesensordatatoasmartphoneinfrontofthe driver.ThisSPLhas8featuresandmorethan20variants.Fig.16 showsitsfeaturemodel,therectanglesrepresentfeatures,optional featuresaredepictedusingawhitecircleabovetherectangleand mandatoryfeaturesaredepictedusingablackcircle.Noticethat thefeaturesarenottotallyindependent.Forinstance,“F1requires F2”denotesthatnoproductcanexistthatexhibitsfeatureF1with- outF2(e.g.,BasicmonitoringrequiresBluetooth).Followingthe featuresarebrieflyexplained:

• Sensorsystem:Thesensorsusedtomonitorthetruckcanbeof threetypes:GPS,InertialsystemorSensornode.

• Sensorsdevice:Indicatesthepossiblesensorsinthetruck,the variantscanbe:Temperature,Pressure,CO,CH4,C6H6,Radiation, Humidity,etc.

• Positioning:Indicatesthepositionofthetruck.Itcanbecalcu- latedusingaGPSortheInertialSysteminthetruck.

• Bluetooth:Indicatesthewayinwhichthesensorsdataissend.

• MobilePhone:Indicateswhetherthemobilemonitoringusesa basicoratom-tomsystem.

• MobileO.S:IndicatestheO.S.ofthemobile,itcanbe:Windows Mobile,AndroidorSymbian.

• MobileScreen:Indicateswhetherthemobilehastouchscreenor not.

• AlertsCommunication:Indicatesthewayinthatthealertsare communicatedtotheuser.Itcanbe:sms,mmsore-mail.

Forthisfeaturemodel,upto18432productscouldbegenerated.

Building and testing such number is impossible in a reason- able time. CTWeb and PROWhelp to reduce thetesting effort

Fig.16.FeaturemodelforSensorsSystemSPL.

obtainingaproductsamplingset.Nextsubsectionexplainsthese stepsindetailfortheSensorSystemSPL.

6.1. Uploadingandprocessingthefeaturemodel

ThefirststepisUploadingtheFeatureModel.Oncethefeature modelisloadedinCTWeb,itisautomaticallyprocessed.Inprevi- ouswork(PérezLamanchaandPolo,2010),thewayinwhichthe featuremodelisloadedandprocessedinCTWebwasdescribed.

ThissectionexplainsbrieflyhowCTWebprocessestheSensorSys- temfeature model. This processing is madein two automated steps:

1Creatingparametersanditsvalues:Thissteptakesintoaccount thefeature,itsvariantsandtherelationshipsbetweenthem.The basicrelationshipsbetweenafeatureanditsvariantscanbe:

• Optional: optional variants are included in some products (Pohl,2005).InFig.17(a),theBluetoothfeatureisoptional.For thisfeature,twovaluesmustbetakenintoaccount:whenthe featureispartoftheproductandwhennot.Thevaluesforthis parameterare: BluetoothyesandBluetoothno.MobileScreen

Fig.17.Somefeaturesandhowtheyareconvertedintoparameters.

(Fig.17(b))isalsooptionalandhastwovariants,bothofthem optionaltoo.Thevaluesforthisparameterare:touch,notouch andMobileScreenno.Thislastparameteris addedtospecify whenthefeatureisnotincludedinaproduct.

• Mandatory:mandatoryvariantsmustbeselectedforanappli- cationif,andonlyif,theassociatedvariationpointispartofthe application(Pohl,2005).InFig.17(c),theAlertCommunication featureismandatoryandhasthreevariants:mmsande-mail (whichareoptional)andsms(whichismandatoryand,thus, mustbepresentinallthevaluesofthisparameter).Then,the valuesforAlertCommunicationare:sms,smsandmms,sms ande-mail.

• Alternative: The alternative choicegroups a setof variants thatare relatedthrough anoptionalvariability dependency tothesamevariationpoint.Alternativedefinestherangefor theamountofoptionalvariantstobeselectedforthisgroup (Pohl,2005).Fig.17(d)showstheoptionalMobilePhonefeature, whichhastwooptionalvariants:BasicMonitoringandTom- tom.Thisfeaturealsohasan alternativerelationshipwhich statesthatthisfeaturehaseither1or2variants.Thevaluesfor MobilePhoneareobtainedcombiningthepossiblevariants,i.e.

BasicMonitoring,Tom-Tom,BasicMonitoringandTom-tom.

AlsothevalueMobilePhonenoisaddedbecausethefeatureis optional.

Fig.18showsthe8parametersobtained,showinginredthe valuesthatdonotappearinthefeaturemodelandthatwere addedduetherelationshipsbetweenthefeatureanditsvariants.

2Setconstraintsbetweenvalues:Thisstepprocessestherela- tionshipsbetweenfeatures,theserelationshipscanbe:requires orexcludes.Fordetailedinformationabouthowtotreatallthe relationships,consult(PérezLamanchaandPolo,2010).Fig.16 shows therelationshipsthatoccurs in SensorSystemfeature model,theserelationshipsare:

• Featurerequiresfeature:Fig.19describesthissituation,where theMobilePhone featurerequiresthefeatureMobile Screen.

ThismeansthattheproductthatcombinesavariantofMobile PhonewithoutavariantofMobileScreenisnotpossible.Inthis casethepairsbetweenBasicMonitoringandMobileScreenno andbetweenTom-tomandMobileScreennoaredeletedfrom thepairstablebetweenthefeaturesMobilePhoneandMobile Screen.

• Variant requires feature: Fig. 20(a) describesthis situation, wherethevariantBasicMonitoringrequiresthefeatureBlue- tooth. This means that the product that combines Basic

Fig.18.SetofparametersforSensorSystemSPL.

Fig.19.Featurerequiresfeaturerelationships.

Monitoringwithout Bluetoothis not viable. In this case the pairscontainingBasicMonitoringandBluetoothnoaredeleted fromthepairstablebetweenthefeaturesMobilePhone and Bluetooth.ThesameoccurswiththevariantSensorNodethat requiresthefeatureSensorsDevice(seeFig.20(b)).

• Variantrequiresvariant:Fig.21describesthissituation,where thevariantBasicMonitoringrequiresthefeatureSensorNode.

ThismeansthattheproductthatcombinesBasicMonitoring withoutSensorNodeisnotviable.Inthiscasethepairscon- taining Basic Monitoringand not containingSensor Node are deletedfromthepairstablebetweenthefeaturesSensorSystem andMobilePhone.

• Variantexcludesvariant:Fig.22describesthissituation,where thevariantWindowsMobileexcludesthevariantNoTouchin thefeatureMobile Screen.Thismeansthat theproductthat

combinesWindowsMobilewiththevariantNoTouchofMobile Screenisnotpossible.InthiscasethepairbetweenWindows MobileandNoTouchisdeletedfromthepairstablebetween thefeaturesMobileO.S.andMobileScreen.

Once the feature model is uploaded and automatically processed,CTWebshowstheparameters,itsvaluesandthepairs tablewiththepairsthatwereremoved.

6.2. Settingthebasetestsuite

This functionality is particularly important in SPL testing becauseeachtestcaseobtainedfromPROWisaproductthatwill beassembledandtested.Sincethisiscostlyandtimeconsuming,it ispreferabletoassemblethemostfrequentproducts.Thisfeature allowstoenteralistoftheproductsthatwillbetestedanyway

Fig.20.Variantrequiresfeaturerelationships.

Fig.21.Variantrequiresvariantrelationships.

Fig.22.Variantrequiresvariantrelationships.

becausetheyareimportantproducts.Theseproductsareuploaded intoCTWeb.Thetoolmarksasvisitedthepairstheyinclude.So, PROWtakesthesetestcasesasthebasisfortherestofthetest suite.Fig.23shows4productsthatarethemostcommonlybuilt fortheSensorSystemSPL.

6.3. RunPROWforproductsamplinginSPL

Oncethetesterssetthepreferences,thePROWalgorithmcan beexecuted,andasaresult,47productsareobtained.PROWtakes 63msinobtainthetestsuite.These47productsaretheproduct samplingsetthatrepresentstheproductstotestinSensorSystem SPL.Fig.24shows10outof47productstobetestedforSensor SystemLPS.For example,thefirstproducthasa Sensorsystem withGPS,InertialsystemandSensorNode,themobilephoneuses theBasicMonitoringandtheTom-tom,theSensorDevicemeas- uresRadiation,themobilescreenisNoTouch,hasBluetooth,uses AndroidasmobileO.S.,thepositioningiscalculatedusingtheGPS andthealertsarecommunicatedsendingsmsandane-mail.None oftheproductsdefined inFig.23 appear inthe finaltest suite

obtained.ThereasonforthatisthatPROWoptimisesthetestsuite tocoverallthepairsbetweenvalues,minimisingthetestsuitesize.

Usingthebasetestsuite:Ifa basetest suitewasuploadto CTWeb,thepairsinvolvedineachtestcasethatbelongstothebase testsuitearemarkedasvisited.Then,PROWisexecuted.Inthis case,thetestcasesincludedinthebasetestsuitearecopiedtothe outputtestsuite.Theothertestcasesarebuilttocoverpairwiseand minimisingthetestsuitesize.Obviously,duetothefactthatthe basetestsuitemaynotbetheminimumtestsetthatcoversthese pairs,theoutputtestsuiteobtainedbyPROWisequalorgrater thantheobtainedwithoutusingabasetestsuite.Forthebasetest suiteshowninFig.23,53productsareneededtoobtainpairwise coverage.

7. Relatedworks

Table1summarisedthemain algorithmsinCIT. Inthis sec- tionwepresentrelatedworksdescribingsolutionsfortwoaspects ofcombinatorialtesting:theconstraintproblemandtheprioriti- sationproblem.

Fig.23. BasetestsuiteforSensorSystemSPL.