E.R.D ERNESTO APARICIO JARAMILLO

The expected test results are theError Log for each test case, and theSummary Report.

The Error Log

The Error Log is a text file containing error reports generated while running each test case. The Error Log’s content is structured into three parts:

(1) Action Taken on the Annotated IDF: This part describes the attribute instance(s) removed from the annotated IDF to form the test case. We used this information to determine whether the error(s) detected (and reported) were in agreement with what we expected. (2) IDF to XML Conversion Errors: This part contains all the errors that occurred while

converting the test case to an XML document. We used this error report to locate and correct bugs in the IDF to XML conversion module.

3.3. TestInfrastructure forIDFs 37 Test Case 1 Test Case 2 . . Test Case n TS 1 Test Case 1 Test Case 2 . . Test Case n

Automated Testing Tool

Error Logs Summary Report Annotated IDF Test Case 1 Test Case 2 . . Test Case n TS 2 TS 3 Execution Module

Generate Error Log for each test case

johar.idf package  Convert Test Case to XML document

 Validate the XML document against the XSD

 Check for violations against other constraints

Test Case

Error report

(3) Errors detected in the Test Case: This part contains all XML validation errors (produced while checking the XML document against the XSD), and violations against other constraints specified in the IDF Format Specification document. These errors are linked to invalid entries in the test case. We used this error report to determine whether all the invalid entries in the test case were detected and reported.

The error logs for the test cases in Figures 3.4 and 3.6 are shown in Figures 3.11 and 3.12 respectively.

Figure 3.11: Error Log for Test Case 1 in TS 1. [No error is detected]

3.3. TestInfrastructure forIDFs 39

The Summary Report

The summary report, shown in Figure 3.13, presents the overall test result at a glance. From this report, we can easily determine the total number of test cases generated, the number of test cases that passed or failed, and the test cases that failed (if any) for each test suite.

Figure 3.13: Summary Report of the tests

Determining test cases that passed or failed

Recall from subsection 3.3.1 that TS 1 contains valid test cases which should not produce error when executed; while TS 2 and TS 3 both contain invalid test cases that should each produce only one error (or related errors in some cases) when executed. Thus, we determine passed and failed test cases as follows:

• For each test case in TS 1,

If test case’s error log contains no error, then Flag test case as "Passed"

Otherwise,

Flag test case as "Failed" Next

• For each test case in TS 2,

If test case’s error log contains no error, then Flag test case as "Failed"

Otherwise,

Flag test case as "Passed" Next

• For each test case in TS 3,

If test case’s error log contains no error, then Flag test case as "Failed"

Otherwise,

Flag test case as "Passed" Next

For example, the valid test cases in Figures 3.4 and 3.5passedbecause their error logs (one of which is shown in Figure 3.11) contain no error message. Moreover, the invalid test cases in Figures 3.6, 3.7, 3.8, and 3.9 also passed since their error logs (one of which is shown in Figure 3.12) each contain an error message.

In conclusion, Figure 3.13 is a demonstration of the automated testing tool. The testing tool accepts the annotated IDF in Figure 3.3 as input, and then proceeds to generate and validate the valid and invalid test cases. The summary report shows that 7 valid test cases are in TS 1 (two of which are shown in Figures 3.4 and 3.5), 7 invalid test cases are in TS 2 (two of which are shown in Figures 3.6 and 3.7), and 2 invalid test cases are in TS 3 (as shown in Figures 3.8 and 3.9). The report further reveals that all the test cases in TS 1, TS 2, and TS 3 passed the tests.

In order to achieve our test objective which is to determine whether all errors in an IDF are detected and reported, we wrote different annotated IDFs and fed each of them to the automated testing tool as input. Prior to commencing the tests, we checked the annotated IDFs to make sure that they (collectively) cover all the attributes an IDF can contain, as defined in the IDF Format Specification document. After running all the test cases generated from the annotated IDFs, the automated testing tool revealed several failed test cases. The Error Log of each failed test case contained between 1 to 10 faults (or errors) which are linked to bugs in

thejohar.idfpackage and/or the XML Schema document (johar.xsd). We resolved all the

bugs and repeated the tests to verify whether all the test cases would pass. As expected, all the test cases passed.

The latest source files of the Johar framework are available in our online repository2_.

Chapter 4

The

Star

Interface Interpreter

In this chapter, we discuss a new interface interpreter which is based on the new version of Johar. This interface interpreter is called Star, and its overall purpose is to present WIMP (Windows, Icons, Menus, and Pointers) graphical user interfaces to users. Starsupports most of the common interaction styles evident in GUIs, such as menu selection, data entry and selection, table browsing, wizard-style interaction, customization of widgets (i.e. addition, rearrangement and deletion of widgets), and so on.

Star itself is not an accessible application, but a means through which application developers present their apps to users. Most application developers will preview and test their apps using Star in order to ascertain that their apps are working as expected and that all functionality or features have been implemented. Thus, successful development of Star is very important to the overall success of the Johar project. We therefore focused on Staras the interface interpreter with the highest priority for implementation. Other interface interpreters, including prototypes providing accessibility to some groups of disabled users, will be discussed in Chapter 5.

This chapter covers the specification and design of Star, as well as its implementation and demonstration.

4.1

Requirements Specification of

Star

GUI

In this section, we describe the features available in a Stargraphical user interface. A more technical description of these features is presented in the Star GUI Specification document1.

1_{Available in Appendix E of this thesis.}