An oracle is usually unavailable to scientists as they try to implement new ideas. To mitigate this problem to some extent, I propose an idea to replicate the MATLAB code in
R or Java. These replications can be considered as oracles. There are two important uses for this type of oracle creation. First, in rare cases inbuilt MATLAB functions used in the code may exhibit round off or acknowledged errors. Second, if any MATLAB code is changed or updated for any reason, then the MATLAB code may be unit tested using these oracles to check whether the modifications or updates have changed the original behavior.
For all the 8 functions mentioned in Chapter 5 and for 1 function from the LUCY package, oracles are created using either R or Java. The 8 functions are unit tested using the oracles created. For example, sphereFnet is one of the 8 functions and is shown below along with its oracles.
MATLAB Code for sphereFnet:
function out = sphereFnet(v, m, d, g, p, u)
Re = p*v*d/u; cd = calcCd(Re); out = m*g - cd*0.5*p*v^2*pi*d^2/4; function cd = calcCd(Re) data = load('sphereCd.dat'); if Re < 0
error('Negative Re value encountered') elseif Re <= 2e4
cd = 24/Re + 6/(1+sqrt(Re)) + 0.4; elseif Re <= 3.99e6
else cd = 0.1810;
end
Oracle for sphereFnet using R:
require("pracma",quietly=TRUE); sphereFnet <- function(v, m, d, g, p, u) { Re <- p*v*d/u; cd <- calcCd(Re); #round(cd, 16) out <- m*g - cd*0.5*p*v^2*pi*d^2/4; output <- format(out,digits = 22) cat(output) } calcCd <- function(Re) { x <- c(20000, 38200, 73000, 144000, 220000, 258000, 293000, 319000, 341000, 350000, 365000, 384000, 417000, 462000, 548000, 743000, 1190000, 2050000, 3220000, 3990000) y <- c(0.4430, 0.4900, 0.5030, 0.5150, 0.5080, 0.4940, 0.4740, 0.4360, 0.3740, 0.2580, 0.1260, 0.0836, 0.0692, 0.0655, 0.0726, 0.0888, 0.1230, 0.1570, 0.1740, 0.1810)
if (Re < 0 && !(is.nan(Re))) {
error('Negative Re value encountered') }
else if (Re <= 2e4 && !(is.nan(Re))) { cd <- 24/Re + 6/(1+sqrt(Re)) + 0.4; }
else if (Re <= 3.99e6 && !(is.nan(Re))) { cd <- interp1(x, y, Re, method = "linear"); }
else
cd <- 0.1810; }
Oracle for sphereFnet using Java Apache Commons:
import org.apache.commons.math3.analysis.UnivariateFunction; import org.apache.commons.math3.analysis.interpolation.LinearInterpolator; import org.apache.commons.math3.analysis.interpolation.UnivariateInterpolator; import java.io.FileNotFoundException; import java.io.IOException; import java.lang.Math; import java.math.BigDecimal; public class sphereFnet {
double[] x,y;
public void getvalues(){ x= new double[20]; x[0] = 20000; x[1] = 38200; x[2] = 73000; x[3] =144000; x[4] =220000; x[5] =258000; x[6] =293000; x[7] =319000; x[8] =341000; x[9] =350000;
x[10] =365000; x[11] =384000; x[12] =417000; x[13] =462000; x[14] =548000; x[15] =743000; x[16] =1190000; x[17] =2050000; x[18] =3220000; x[19] =3990000; y = new double[20]; y[0] = 0.4430; y[1] = 0.4900; y[2] = 0.5030; y[3] = 0.5150; y[4] = 0.5080; y[5] = 0.4940; y[6] = 0.4740; y[7] = 0.4360; y[8] = 0.3740; y[9] = 0.2580; y[10] =0.1260; y[11] =0.0836; y[12] =0.0692; y[13] =0.0655; y[14] =0.0726; y[15] =0.0888;
y[16] =0.1230; y[17] =0.1570; y[18] =0.1740; y[19] =0.1810; }
sphereFnet(double v, double m,double d,double g,double p,double u) throws FileNotFoundException, IOException {
double Re, cd, out,cr ;
System.getProperty("sun.arch.data.model"); Re = p*v*d/u;
cd = calcCD(Re);
out = m*g - cd * 0.5 * p * Math.pow(v,2) * 3.141592653589793 * Math.pow(d, 2)/4; System.out.println(String.format("%.22f",out));
//System.out.println(BigDecimal.valueOf(out)); }
private double calcCD(double Re) throws FileNotFoundException, IOException{ getvalues();
double twentythousand = 20000; double thirtyninelakhs = 3990000; double cd = 0;
if (Re < 0)
System.out.println("Negative Re value encountered"); else if (Re <= twentythousand){
cd = 24/Re + 6/(1+Math.sqrt(Re)) + 0.4; //constant replaced: 24 -> 23 }
else if (Re <= thirtyninelakhs){
UnivariateInterpolator i=new LinearInterpolator(); UnivariateFunction uf=i.interpolate(x,y);
cd = uf.value(Re); } else cd = 0.1810; return cd; }
public static void main(String[] args) throws FileNotFoundException, IOException { // TODO Auto-generated method stub
double v,m,d,g,p,u;
v = Double.parseDouble(args[0]) ; m = Double.parseDouble(args[1]); d = Double.parseDouble(args[2]); g = Double.parseDouble(args[3]); p = Double.parseDouble(args[4]); u = Double.parseDouble(args[5]); sphereFnet sp = new sphereFnet(v,m,d,g,p,u);
} }
4.2.4.1 Implementation of Unit Testing
Unit testing is done on all 8 functions and on 1 function from the LUCY package. The replications in R or Java act as oracles as said before, and the set of test cases developed using our proposed technique are taken as the test suite for unit testing. Unit testing is performed using the MATLAB unit testing framework. For every function a Unittesting script is created such that, when string ‘java’ is given by the user then it unit tests the MATLAB function with the Java oracle. The Unittesting script uses the R oracle when string ‘R’ is given as input.
(SphereFnetUnittesting). If the SphereFnetUnittesting script is called by the user with input string as ‘java’ then each test case is sent one by one to sphereFnetjavaapache script. The sphereFnetjavaapache script will call the java oracle which is a jar file and fetches the result. The results from the oracle and MATLAB function are compared and the unit test result (pass or fail), is sent back to SphereFnetUnittesting. This process repeats for all the test cases and the count of total unit tests passed is printed. The code used for unit testing can be seen below.
SphereFnetUnittesting Script:
function SphereFnetUnittesting(platform)
load('Genetic_sphereFnet_main.mat'); relRcount = 0;
for row = 1 : size(TPop.sphereFnet,1) spherev = TPop.sphereFnet{row,1}{1}; spherem = TPop.sphereFnet{row,1}{2}; sphered = TPop.sphereFnet{row,1}{3}; sphereg = TPop.sphereFnet{row,1}{4}; spherep = TPop.sphereFnet{row,1}{5}; sphereu = TPop.sphereFnet{row,1}{6}; disp('---') disp('---'); fprintf('evaluating SphereFnet testcase no %d',row); disp(' ');
if platform == 'R'
disp(' '); test1 = sphereFnetRscript; test1.spherev = spherev; test1.spherem = spherem; test1.sphered = sphered; test1.sphereg = sphereg; test1.spherep = spherep; test1.sphereu = sphereu; testresult = run(test1); end if platform == 'java'
fprintf('evaluating java SphereFnet testcase no %d',row); disp(' '); test1 = sphereFnetjavaapache; test1.spherev = spherev; test1.spherem = spherem; test1.sphered = sphered; test1.sphereg = sphereg; test1.spherep = spherep; test1.sphereu = sphereu; testresult = run(test1); end if testresult.Passed == 1 relRcount = relRcount + 1;
fprintf('testcase no %d is passed ',row); disp(' ');
disp('---'); disp('---END---')
disp(' '); disp(' '); end
end
fprintf('No of unit tests passed %d\n',relRcount); end
Unit testing MATLAB code using Java oracle:
classdef sphereFnetjavaapache < matlab.unittest.TestCase %UNTITLED2 Summary of this class goes here % Detailed explanation goes here
properties spherev spherem sphered sphereg spherep sphereu end methods (Test) function testsphereFnet(testcase) v = testcase.spherev; m = testcase.spherem; d = testcase.sphered; g = testcase.sphereg;
p = testcase.spherep; u = testcase.sphereu;
initdir = pwd; % store the current director
initpath = path; % store current path so it can be restored targetdir = '/home/subhash/Desktop';
javaargument = 'java -jar sphereFnet.jar '; path(initdir, path);
actualSolution = sphereFnet(v, m, d, g, p, u); sprintf('actual solution is %16.f',actualSolution)
cd(targetdir);
callstring = [javaargument num2str(v,64) ' ' num2str(m,64) ' ' num2str(d,64) ' ' num2str(g,64) ' ' num2str(p,64) ' ' num2str(u,64)];
[status,cmdout] = unix(callstring); resultfromjava = str2double(cmdout); sprintf('Java result is %16.f',resultfromjava) tempjavaexpectresult = (sprintf('%.4f',resultfromjava)); javaexpectresult = str2double(tempjavaexpectresult); tempactualresult = (sprintf('%.4f',actualSolution)); actualresult = str2double(tempactualresult); cd(initdir); path(initpath); testcase.assertEqual(actualSolution,resultfromjava,'RelTol',10^-10);
end
end end
Unit testing MATLAB code using R oracle:
classdef sphereFnetRscript < matlab.unittest.TestCase
%SPHEREFNETRSCRIPT Summary of this class goes here % Detailed explanation goes here
properties spherev spherem sphered sphereg spherep sphereu end methods (Test) function testsphereFnet(testcase) v = testcase.spherev; m = testcase.spherem; d = testcase.sphered; g = testcase.sphereg; p = testcase.spherep; u = testcase.sphereu;
initpath = path; % store current path so it can be restored targetdir = '/home/subhash/Desktop/Thesis_Matmute/Rscripts'; Rargument = 'Rscript sphereFnetR.R ';
path(initdir, path);
actualSolution = sphereFnet(v, m, d, g, p, u); sprintf('actual result is %16.f',actualSolution) cd(targetdir);
callstring = [Rargument num2str(v,64) ' ' num2str(m,64) ' ' num2str(d,64) ' ' num2str(g,64) ' ' num2str(p,64) ' ' num2str(u,64)]
[status,cmdout] = unix(callstring); resultfromR = str2double(cmdout); sprintf('R result is %16.f',resultfromR)
Rexpectresult = str2double(sprintf('%.4f',resultfromR)); actualresult = str2double(sprintf('%.4f',actualSolution)); cd(initdir); path(initpath); testcase.assertEqual(actualSolution,resultfromR,'RelTol',10^-10); end end end
5 Results
To evaluate the automated test generation process, automated test suites were created for the 8 functions used by Hook[16] in Mutation Sensitivity Testing. Hook took these 8 functions from an introductory scientific computing course. Hook created popperian test cases (TPop), random test cases (Trnd), and also a combination of both popperian and random test cases (Tcmb). Hook manually analysed each function and came up with good test suites. The (Tcmb) test suite gave good results for the 8 functions in Hook’s thesis. If we compare the automated test suites (Tauto) with Hook’s test suites, then we can evaluate the quality of the automated test suites. Now let us see the results for each of the function.
sphereFnet
This function takes 6 inputs. They are velocity of sphere, mass of sphere, diameter of sphere, gravity, density of fluid, and viscosity of fluid. Using these inputs, sphereFnet computes the net force on the sphere when it is falling into a liquid.
The sphereFnet detection graph for Hook’s test suite can be seen in Graph 5 and the detection obtained from the auto generated test suite can be seen in Graph 6. By carefully comparing the graphs the following observations are made:
1. The graph moved to the right for Tauto test suite, this indicates that mutants are detected at high relative error. Also 100 % mutants are detected at high relative error when compared to Hook’s graph.
2. Hook’s test suites discovered 197 mutants out of 217 and the rest of 20 mutants are either deemed equivalent or terminal failure mutants, whereas Tauto test suite detected 195 out of 217. Fewer mutant detections for Tauto only happened for the sphereFnet function and in the rest of the functions, mutation detections are more. The reason for this might be that Hook added manually a test case after examining a difficult to detect mutant.
Graph 6: sphereFnet detection graph generated using auto generated test suite
simpson
This function takes 4 inputs: a mathematical function, upper bound of integration, lower bound of integration, and the number of panels for integration. Using these inputs simpson integrates the mathematical function and returns an approximate value.
The simpson detection graph for Hook’s test suite can be seen in Graph 7 and the detection obtained from the auto generated test suite can be seen in Graph 8. By carefully comparing the graphs the following observations are made:
1. The detection score scale for the graphs are different. The detection scale of Tauto graph is very high when compared to Hook’s graph. It can also be clearly observed that mutants got detected at high relative error by Tauto. The 100% mutants detection point is also moved to the right in the Tauto graph.
2. Hook’s test suites discovered 133 mutants out of 181. The Tauto test suite detected 136 mutants out of 181. The 3 extra mutants detected by Tauto are considered as equivalent or terminal failure mutants by Hook’s test suites.
Graph 7: simpson detection graph generated using Hook’s test suite
powerit
This function takes 2 inputs: an input matrix and the number of iterations to conduct. Using these inputs, powerit returns an approximate value of the largest eigenvalue of the input matrix.
The powerit detection graph for Hook’s test suite can be seen in Graph 9 and the detection obtained from the auto generated test suite can be seen in Graph 10. By carefully comparing the graphs the following observations are made:
1. The detection score scale for the graphs are different. The detection scale of the Tauto graph is very high when compared to Hook’s graph. The 100% mutation detection is at a lower relative error for Tauto when compared to Hook’s test suites. This is because more mutants are detected by Tauto (20) than Hook’s test suites (14). Apart from the 100% mutants detection point, the rest of the graph moved towards to the right for Tauto which is a good sign.
2. Hook’s test suites discovered 14 mutants out of 32. The Tauto test suite detected 20 mutants out of 32. The 6 extra mutants detected by Tauto are considered as equivalent or terminal failure mutants by Hook’s test suites.
Graph 9: powerit detection graph generated using Hook’s test suite
odeRK4
This function takes 4 inputs. They are a mathematical function, stopping value of the variable, step size for the variable, and initial condition of the dependant variable. The odeRK4 function returns a vector containing a numerical solution at each step of the independent variable.
The odeRK4 detection graph for Hook’s test suite can be seen in Graph 11 and detection obtained from the auto generated test suite can be seen in Graph 12. By carefully comparing the graphs the following observations are made:
1. The detection score scale for the graphs are different. The detection scale of the Tauto graph is very high when compared to Hook’s graph. It can also be clearly observed that mutants got detected at high relative error for the Tauto test suite. The 100% mutants detection point is also moved to the right in the Tauto graph.
2. Hook’s test suites discovered 362 mutants out of 424. The Tauto test suite detected 375 mutants out of 424. The 13 extra mutants detected by Tauto are considered as equivalent or terminal failure mutants by Hook’s test suites. The rise in the number of mutants detected is a good indication of the increase in quality of the test cases.
Graph 11: odeRK4 detection graph generated using Hook’s test suite
nwtsqrt
This function takes 2 inputs: a non-negative real number x and an initialization value for Newton's method iterations. The nwtsqrt function returns the square root value of x.
The nwtsqrt detection graph for Hook’s test suite can be seen in Graph 13 and the detection obtained from the auto generated test suite can be seen in Graph 14. By carefully comparing the graphs the following observations are made:
1. There is no major difference in the graphs of Tauto and Hook’s test suites. The Tauto graph is close to the Tcmb graph. However, the Tauto graph is generated for 48 mutants whereas Hook’s graph is generated for 47 mutants.
2. Hook’s test suites discovered 47 mutants out of 65. The Tauto test suite detected 48 mutants out of 65. Tauto detected one more mutant than Hook’s test suites.
Graph 13: nwtsqrt detection graph generated using Hook’s test suite
GEPiv
This function solves the system Ax=b, where the first input is A (coefficient matrix), and the second input is b (right-hand side vector). The GEPiv function returns a vector x such that Ax=b is satisfied.
The GEPiv detection graph for Hook’s test suite can be seen in Graph 15 and the detection obtained from the auto generated test suite can be seen in Graph 16. By carefully comparing the graphs the following observations are made:
1. The detection score scale for the graphs are different. The detection scale of the Tauto graph is very high when compared to Hook’s graph. It can also be clearly observed that mutants got detected at high relative error for the Tauto test suite.
2. There is no difference in the number of mutants detected. Both Hook’s test suites and Tauto detected 176 mutants out of 225.
Graph 15: GEPiv detection graph generated using Hook’s test suite
gaussQuad
This function takes 5 inputs: a mathematical function, upper bound of integration, lower bound of integration, the number of panels for integration, and the number of nodes in each panel. Using these inputs, gaussQuad integrates the mathematical function and returns an approximate value.
The gaussQuad detection graph for Hook’s test suite can be seen in Graph 17 and the detection obtained from the auto generated test suite can be seen in Graph 18. By carefully comparing the graphs the following observations are made:
1. The detection score scale for the graphs are different. The detection scale of the Tauto graph is very high when compared to Hook’s graph. It can also be clearly observed that mutants got detected at high relative error. The 100% mutants detection point is also moved to the right in the Tauto graph.
2. Hook’s test suites discovered 200 mutants out of 266. The Tauto test suite detected 203 mutants out of 266. 3 extra mutants are detected by Tauto.
Graph 17: gaussQuad detection graph generated using Hook’s test suite
binSearch
This function takes 2 inputs: a vector X with increasing order of real values, and the real value r that is to be located. The binSearch function gives the index value of r in vector X.
The binSearch detection graph for Hook’s test suite can be seen in Graph 19 and the detection obtained from the auto generated test suite can be seen in Graph 20. By carefully comparing the graphs the following observations are made:
1. There is not much difference in the mutants detected between Tauto and Hook’s test suites. This might be because of the output that binSearch returns. The binSearch function returns an index value in the range of 0 to 25. High relative errors cannot be obtained, for such values.
2. Hook’s test suites discovered 60 mutants out of 82. The Tauto test suite detected 61 mutants out of 82. 1 extra mutant is detected by Tauto.
Graph 19: binSearch detection graph generated using Hook’s test suite