Solución propuesta

Having the list of features to implement and the summary of the features available in each of the analyzed CBA systems, the next step was to decide whether or not one of the existing systems could be used as the basis for the desired one. To support the decision, three tables were created: table5.9, table5.10 and table5.11. They present the features to implement with, respectively, high, medium and low priority already available in the existing CBA systems.

CBA System

Feature CourseMark

er

BOSS xLx Mooshak RoboPr

of

Submit! GAME DOMjudge F1.1 Exams Yes Yes Yes Yes Yes Yes Yes Yes

F1.2 Tests Yes Yes Yes Yes Yes Yes Yes Yes

F2.1 Start time Yes Yes Yes Yes Yes Yes Yes Yes

F2.2 Deadline Yes Yes Yes Yes Yes Yes Yes Yes

F2.3 Accepted programming languages No No No Yes No No No Yes

F2.4 Tolerance time No No No No No No No No

F2.5 Duration No No No No No No No No

F3.1 Text files Yes Yes No Yes Yes Yes Yes Yes

F3.2 Unit tests No Yes Yes No No No No No

F4.1 Weight for assessment grade No Yes Yes Yes No Yes No No

F5.1 Mechanism for uploading solutions Yes Yes Yes Yes Yes Yes Yes Yes

F6.1 Toggle immediate/non-immediate feedback No No No No No No No Yes

F6.2 Regulation of the level of feedback Yes No No No No No No No

F7.1 Consult grades statistics Yes No No No No Yes No No

F7.2 Manual alteration of assessment results No Yes No No No Yes No No

F7.3 Student history Yes Yes Yes Yes No Yes Yes Yes

F7.4 Student notification Yes Yes No No No No No No

F8.1 Plagiarism detection Yes Yes No No Yes No Yes No

Total 11 12 8 9 7 10 8 9

CBA System

Feature CourseMark

er

BOSS xLx Mooshak RoboPr

of

Submit! GAME DOMjudge F1.3 Exercises Yes Yes Yes Yes Yes Yes Yes Yes

F2.6 Max number of submissions Yes No No No No No No No

F4.2 Static analysis of code quality Yes Yes No No No Yes Yes No

F4.3 Max runtime Yes No No Yes No No No Yes

F4.4 Max memory No No No Yes No No No No

F5.2 Skeleton mechanism Yes No No No No No No No

F6.3 Personalized feedback messages No No No No No No No No

F7.5 Sending of e-mails No No No No No No No No

F7.6 Alteration of submitted code No Yes No No No No No No

F8.2 Q&A system Yes No No Yes No No No Yes

F8.3 Exercise Database No No No No No No No No

Total 6 3 1 4 1 2 2 3

Table 5.10: Features with medium priority

CBA System

Feature CourseMark

er

BOSS xLx Mooshak RoboPr

of

Submit! GAME DOMjudge F2.7 Duration of each individual question No No No No No No No No

F3.3 Regular expressions No No No No No No No No

F4.5 Distinguish grades based on solving time No No No No No No No No

F4.6 Penalization of common mistakes No No No No No No No No

F7.7 Exporting of assessment results No No No No No No No No

F7.8 Environment for manual testing of subm. No No No No No No No No

F8.4 Submissions ranking No No No Yes No No No Yes

Total 0 0 0 1 0 0 0 1

Table 5.11: Features with low priority

The tables show that the existing systems already implement a significant amount of high priority features and a few of the medium and low priority ones. Therefore, it was considered to be advantageous to reuse one system instead of implementing a new system from scratch.

By inspecting table 5.9 it is possible to conclude that the CourseMarker and BOSS systems are the ones that have a higher number of already implemented features. How- ever, CourseMarker was ruled out since it is a paid system and cannot be extended with

the features that it lacks. The BOSS system was also ruled out as a viable alternative for the base system, despite being an open-source project. After installing and testing BOSS, it was noted that the core functionality of the system - the automatic assessment of code - was not as mature and user-friendly as desired.

Since the maturity of the automatic assessment of code components seemed to be a problem of the systems developed specifically to be used at Universities, the 2 systems created for programming contests, Mooshak and DOMjudge, were considered to be the more viable alternatives.

5.2.1 Mooshak versus DOMjudge

Mooshak and DOMjudge are both open source, already implement 9 out of 18 of the high priority features and 1 out of 7 of the low priority features. In terms of medium priority features, Mooshak has one more than DOMjudge. By analysing both systems’ source code and documentation, the following observations were drawn:

• DOMjudge uses a MySQL database for keeping contest and submission data, while Mooshak uses the file system;

• DOMjudge has, in its architecture, a clear separation of the components responsible for running the contest and the components responsible for automatically assessing the submitted solutions. There are even two different programs that can (and should) be executed on different machines;

• DOMjudge has more documentation available, including detailed Administrator, Jury and Team manuals.

Based on this analysis, DOMjudge was chosen over Mooshak as the base system. Since, for the desired CBA system, the features related to the running of contests are ir- relevant, the fact that DOMjudge makes a clear distinction between those features and the automatic assessment of code, allows for a more straightforward creation of an abstraction layer responsible for the communication with external platforms. This abstract layer can directly access DOMjudge’s database and be independent of DOMjudge’s source code. The components responsible for the automatic assessment of code have an autonomous behavior, use the same database, but execute on different machines.

Besides the robust automatic assessment of code component, DOMjudge has also proved to be a very safe system, being used with success in some important programming contests. Furthermore, and as referred in section 2.8.1, it allows for the definition of validatorsthat may be useful for handling data that do not fit within the standard scheme of fixed test data.