Francisco J. Almeida-Martínez
Serie de Informes Técnicos DLSI1-URJC
1. Introduction ...3
2. Description of the evaluation ...3
2.2. Experimental design and tasks...4
3.1. Instructors observations...5
3.2. Answers to questionnaires...5
VAST. Evaluation of the execution-visualization
Francisco J. Almeida-Mart´ınez, Jaime Urquiza-Fuentes
Universidad Rey Juan Carlos
Departamento de Lenguajes y Sistemas Inform´aticos I c/ Tulipn s/n, 28933 Madrid, Spain
Abstract. This document describe an observational and usability evalu-ation of VAST in the course 2008-2009. This evaluevalu-ation was performed af-ter implementing the first functional integration calledexecution-visualization. The results of this evaluation show that students like VAST but it is necessary to add new features as the second functional integration,
From the results of previous evaluations , we plan the necessity to adapt the process of visualizations generation to the real process of parser generation. Due to this, we planned two functionality integrations. One of this integration is the
execution-visualization. This functionality was added to VAST, so the user can
edit the input stream and run the parser from the own user interface. Thanks to this, the user can generate new visualizations without effort and observe how the parser behave for different input streams. The objective of this evaluation is observe the usability of this new integration.
Description of the evaluation
In this section we describe the evaluation. We refer to the participants, the experiment’s design, the tasks performed during the session, the protocol and the results.
In this evaluation participated5 studentsof the subjectCompilers and
Inter-preters of the Rey Juan Carlos University in the 2008-2009 course. The
2.2 Experimental design and tasks
This evaluation was designed as an usability experiment plus an observational study. In this evaluation the students were divided in two groups. For this task we used a class exercise as a pretest of knowledge (see appendix 5).
The tasks performed during the evaluation consisted in two exercises (see appendix 5). From the lexical and syntax specifications given, the students had to annotate, generate the parser and executed it, using the new functional in-tegration, execution-visualization of VAST. When the groups were created, the students had to annotate the lexical or the syntax specification. The objective of this task was to take note of the difficulties of each part.
The dependent variables were the students’ opinion about four different as-pects: the ease of use, quality, satisfaction and learning help. The experiment lasted 2 hours (one class). During this time the students had to use VAST’s documentation. At the end of the session we asked the students for answering an opinion questionnaire about the tool, including the new integration.
Two weeks before the evaluation we performed a pretest of knowledge, which was done by only two students. For this reason we did a random division of the participants being impossible to balance both groups. One week before the evaluation we presented VAST and how to work with the tool. This lasted about 30 minutes, during the rest of the session students worked on the class exercises. In the table 1 we show the protocol followed in this evaluation.
Group 1 Group 2
Pretest of knowledge Introduction to VAST
Lexical annotation 1 Syntax annotation 1 Syntax annotation 2 Lexical annotation 2
Answers to questionnaries
Table 1.Protocol followed in this evaluation
3.1 Instructor’s observations
During the evaluation we observed how the students performed some of the tasks of the test. Some students did changes in the visual representation of the syntax tree using the configuration tool of VAST. Others were more interested on the changes of the user’s interface (distribution of views). In general, students preferred the used of the floating view instead of using the horizontal/vertical distribution. It was interesting that students made a high use of the parser’s stack and the automatic animation of the syntax tree building process.
Furthermore, we observed that students required the
3.2 Answers to questionnaires
The results of the questionnaires are focus in the values of the dependent vari-ables: ease to use, general quality of the tool, student satisfaction and learning support.
Aspect % of answers between 4/5
Ease to use 100%(5/5) Quality of the tool 100% (5/5) Student satisfaction 80%(4/5)
Learning support 80%(4/5)
Table 2.Summary of questionnaires answers
In table 2 we show a summary of the results obtained in the opinion ques-tionnaires filled after the evaluation by students.
In the questionnaire we also included open questions in order to make possible students to express their opinion easily. As positive aspect students pointed out the visualization of the syntax tree, its building process, the ease of use and the user interface. As posible improvements they mentioned the interaction with the syntax tree, the automatic lexical/syntax annotation and the exportation of animations.
The results obtained show that the development of VAST is centered on the students.
as hard and monotonous. Because of this the general satisfaction ofVASTapiis lower. Finally, the learning improvement also receive a worse mark. One student gave a 3 to the use of the stack.
As result of this evaluation we plan to implement the another integration
This project is supported by project TIN2008- 04103/TSI of the Spanish Min-istry of Science and Innovation.
Exercise used for the pretest
WWW stands forWorld Wide Web, a global net based on the architecture to transfer hypertext files between machines connected to the internet. To make posible the localization of a hypertext file on the internet it has been defined a language which sentences are called URL (Uniform Resource Locator). The URL allows to specify the communication protocol, the machine and the file. For example, the following URLftp://apolo.uma.es/dist/doc/drol.ps shows that we want to obtain the file/dist/doc/drol.psfrom the machineapolo.uma.esusing the protocolftp. For this exercise we only consider that the WWW only admits the following protocolos:http, ftp andgopher. The name of the machine can be written in two ways:
– In symbol notation, as in the previous example, using the machine’s name and (optionally) the domains, separated by the point character, for example:
– Using decimal notation, specifying the four numbers of the internet direction, for example: 18.104.22.168.
The file’s name of the URL address is any correct name of a valid file in UNIX including the complete path (using only alphanumerics characters, the point and the slash to separate directories), for example: index.html or/dist/doc/drol.ps. In a URL reference the protocol is optional, so by default it asumes thehttp pro-tocol. Also the file’s name is optional, so it uses by defaultindex.html. For exam-ple, the following address are valid and equivalent:apolo.uma.es, 22.214.171.124,
Using the tools JFlex and Cup, build a parser to recognize the URL address. A possible grammar could be:
S::=U RI S|U RI
U RI::=P ROT OCOLO://N OM BRE M AQU IN A U RI2|N OM BRE M AQU IN A U RI2
P ROT OCOLO::=F T P|HT T P|GOP HER
DIRECCION IP::=DIGIT O . DIGIT O . DIGIT O . DIGIT O DOM IN IO::=IDEN T|IDEN T . DOM IN IO
DIRECCION F ICHERO::=/IDEN T /N OM BRE F ICHERO|/IDEN T DIRECCION F ICHERO N OM BRE F ICHERO::=IDEN T . IDEN T
Exercises for the evaluation
1. Exercise 1: Given the following grammar to declare procedures/functions:
S::=T Y P E ident(LIST P ARAM)SEN T LIST T Y P E::=boolean|integer|f loat
LIST P ARAM::=T Y P E ident|T Y P E LIST P ARAM, ident SEN T LIST::=SEN T|SEN T LIST; SEN T
SEN T::=ident = EXP;
EXP::=EXP + T|T T::=T ∗ F|F F::=ident|(EXP)|cte
(a) Build an animation using VAST. The animation must show the whole declaration of a function with N parameters and at least one assignment instruction.
(b) Modify the input stream to include four assignment instructions. 2. Exercise 2: Given the following grammar to declare SQL queries:
LQU ERY ::=LQU ERY ;QU ERY|QU ERY
QU ERY ::=select LID in LIST T ABLAS where LIST CON D LID::=LID, ident|ident
LIST T ABLAS::=LIST T ABLAS ident|ident LIST CON D::=F OP LIST CON D|F OP F OP::=and|or|nand|xor
(a) Build an animation using VAST. The animation must show the whole declaration at least one SQL query with at least two conditions. (b) Modify the input stream to include two SQL queries with at least two
conditions each one.