The systems that we are going test – NavEx, CUMULATE, and PERSEUS – were not designed solely for the experiments. NavEx has been used in various classroom studies since 2004;see for example (Brusilovsky et al., 2004). CUMULATE in its current implementation was used as a primary user modeling server since 2005 (Brusilovsky et al., 2005). PERSEUS was deployed to be used in class as early as Fall 2007 (Yudelson & Brusilovsky, 2008).
For the experiments, we changed the original programming code of the production systems. That was done in order to factor out some of the overhead processing not related to the adaptation process and as a result to make the performance testing less biased and the comparison of the PERSEUS and NavEx systems fairer. Some of the changes were implemented ubiquitously across all systems involved. Others were specific to particular realizations of
adaptation techniques. In this section we provide a detailed account for all the changes made in the code.
4.2.1.1 Data Collection (Logging)
In PERSEUS, CUMULATE, and the previous version of the load generator (Yudelson et al., 2007; Zadorozhny et al., 2008), logging was originally done in the database. Every transaction was saved as a row in a special table. During the normal mode of operation that does not seem to be problematic. However, during load testing the database communication overhead becomes a tangible factor influencing the performance of the system. This is why the logging mechanism was changed for simple tab-separated files. The structure of the logs remained unchanged: A row is a transaction and a tab-separated column is a field. NavEx had no logging implemented originally. Tab-separated file logging, identical to that of PERSEUS, was added.
Logging procedures in PERSEUS’s concept-based adaptation service and NavEx were different in the first and second parts of the experiments. In the second part of the experiments, both systems, in addition to logging request data, were recording the time it takes to perform particular operations (file i/o, data initialization, making the adaptation decision, etc.). Table 3 is a summary of all data logged by all systems involved in the experiment.
Besides logging, several other aspects of the systems’ code were changed for the experiments to make the comparison unbiased. Changes to each of the systems are discussed below
Table 3. Summary of logged experimental data System Experi -ments Logged data Load Generator all
request ID*, start timestamp†, finish timestamp, request duration, response size, response status¶ (6 fields)
PERSEUS 1
request ID, start timestamp, finish timestamp, request duration; user model query for concept knowledge start, finish and duration; user model query for resource progress start, finish and duration; response status (11 fields)
NavEx 1 same as above
CUMULATE all
request ID, request type (concept knowledge or resource progress)._start timestamp, finish timestamp, request duration, response status (6 fields)
PERSEUS 2
request ID and durations of… request, file input/output, data input and initialization, user model query for concept knowledge, user model query for resource progress, uploading user model values, production of the adaptation decision (8 fields)
NavEx 2 same as above
* request ID encodes load parameters (delay between requests and request size), name of the tested systems and adaptation technique, and a unique number of the request in the experiment. † start and finish timestamps correspond to the span of the request lifecycle in the target system. Load generator creates requests, while other systems register events of requests received/sent. ¶ response status – OK, if success, error stack trace otherwise.
4.2.1.2 PERSEUS
PERSEUS is the system that was changed the most. The most significant change is related to adaptation model initialization. In the production version of PERSEUS, the snippet of the hyperspace needing adaptation is passed by a URL reference. Using this reference, PERSEUS retrieves the link structure of the hyperspace snippet via an additional HTTP request to the content manager (portal). NavEx, in contrast, has the hyperspace structure cached locally and does not make additional HTTP calls. To make the comparison of PERSEUS and NavEx more fair and to avoid the manifestation of a C10K problem (Kegel, 2010), the structure of hyperspace has been cached locally for PERSEUS and made accessible in the form of RDF files – the format identical to the one used by the production version of PERSEUS.
In the case of the social navigation support and concept-based navigation support services, the cache files contain the link structures of the referred hyperspace snippets requiring adaptation. In the case of the topic-based navigation support service, where the hyperspace snippet is a set of folders containing resources, additional files with each representing structures of every folder are cached. Also, as mentioned above, database request logging was substituted with file-based logging to reduce computational overhead.
4.2.1.3 NavEx
NavEx has been altered in several ways. The original version, upon every invocation of NavEx, sent an acknowledgement to CUMULATE that the interface had been loaded; code related to this report has been purged. NavEx was a state-full system that kept user data in session variables as the user was working with the system. User data was cleared when a session timed out. In the experimental version, NavEx was made stateless to be more comparable with PERSEUS: No
user information was kept in session variables between requests. As was mentioned above, file- based logging was added to NavEx. Also, the order of operations was changed to mirror the implementation of concept-based adaptation in PERSEUS. The output HTML code was simplified for compactness.
4.2.1.4 Load Generator
The load generator is virtually identical to the one used in our previous work on performance evaluation and the pre-study (Yudelson & Brusilovsky, 2008; Yudelson et al., 2007; Zadorozhny et al., 2008), with the exception of the change from database logging to file-based logging. The only change in CUMULATE used for the experiments is file-based request logging.