Following the approach in Figure 4-2, the two systems are tested to validate the design and implementation of the mechanisms. The results can be used to make revisions and improvement of the mechanisms.
The systems have been tested by a team involved in the project and over fifty external users who were paid. The testing was conducted in both laboratory and online environments. The former allows controlling factors on the client side (e.g. computer, web browser, activities and scalability), while the latter mimics a real-world situation where people conduct procurement transactions online. The users were asked to record errors and difficulties when using the systems and to provide comments and suggestions related to the systems and process at the end of the testing.
The systems were tested and improved on three levels following the approach:
(1) Market mechanisms (the core models), which are directly related to the rules and parameters in the design and implementation of MARA and MBMN;
(2) Auxiliary models and tools (the supplementary models), which support the users in constructing, generating and comparing alternatives;
(3) User interface (the representation or views), which focuses on the layout and format of information displayed on user’s screen.
generation was implemented and tested. The parameters with regard to the number of sets and the increment were manipulated, through which the minimum value of these parameters was identified. Different rules on information revelation were also considered and controlled in order to support various auction settings. Moreover, the closing rules were improved with regard to both the auction deadline and the number of available bids and bidders. The testing and revisions resulted in a set of parameters for different auctions at the mechanism level.
Table 4-4. Summary of system testing
Imaras Imbins
Phase 1 Phase 2 Phase 1 Phase 2
Layer 1:
Mechanism Algorithm (rules and parameters) Round duration, closing rules Information exchange Information exchange Layer 2:
Auxiliary models and tools
Bid construction
and comparison Bid generation Offer construction and comparison Offer generation Layer 3:
User interface Activity-oriented design, navigation
Information-oriented design, server pushing, visualization
Activity-oriented
design, navigation Information-oriented design, server pushing, visualization Environment Laboratory Laboratory, online Laboratory Laboratory, online Problem with 125
alternatives with 216 alternatives with 216 alternatives with 3375 alternatives Participants Project team Project team
and students Project team and students Project team and students Results Poor process
tracing Unexpected termination of auction, generated unacceptable bids
Poor process tracing Message not fully displayed
Revision Information-
oriented design More flexible closing rules, bid generation incorporated with limit-sets and break- even points Information- oriented design, negotiation update page Full message display
The bid construction, generation and comparison functions were tested within different auction settings. They were perceived as very useful in making efficient bids. Based on user’s feedback, the bid generation was dynamically incorporated within the limit-sets, i.e. only permissible bids are generated. Moreover, the break-even points of bidders are also indicated when bidders construct and generate alternative bids.
At the user interface level, an initial activity-oriented design was changed to an information- oriented design. The former decomposes the auction process into activities and the users can select
and undertake these activities. This design is simple and easy to use but requires users to navigate between web pages. The latter integrates the available and relevant information on the main page (i.e. the screen shown in Figure 4-6), which allows the users to make decisions and take actions in one place. As multi-attribute auctions are information intensive, the information-oriented design was recommended. Moreover, besides the list of bids and alternatives in tables, a visualization tool was tested and improved to present the bidding process.
In Imbins testing, information exchange at the mechanism level has been a major concern. The method of information exchange was revised so that the buyer can send both offers and messages to one or more suppliers. This maintains the flexibility of negotiation mechanisms and also allows the buyers to control the negotiation process.
Both buyers and suppliers are provided with analytical support (e.g. construction and generation of alternative offers), communication support (e.g. construction of messages) and visualization support (e.g. offer-history graph). In addition, taking into account the buyers’ effort in processing offers and messages from more than one supplier, additional support has been provided to present the most recent offers and messages from each supplier in one place. This dynamically and simultaneously updates the negotiation status with all suppliers. Once the buyer decides to reply to one supplier, the system navigates to another screen but keeps the selected supplier information. This has been considered useful in tracing the negotiation process.
Similar to Imaras, an information-based design was followed in Imbins. The main screen shown in Figure 4-7 contains the recent offers and/or messages for making counter-offers or sending messages with argumentation. The negotiation history page was initially focused on the exchanged offers, while the content of messages was only partially displayed. Based on users’ suggestions, an additional table was added to show the full messages.
In addition to the testing and improvement on the above three layers, the systems have been tested with different cases and different numbers of suppliers in each transaction for the
consideration of usability and scalability. A dynamic calculator was added in the preparation stage to represent the participants’ preferences. A stress test also resulted in replacing the database server and decomposing several modules into smaller components.
Furthermore, users also suggested providing: (1) more detailed and clear instructions to guide them in using the system and conducting the transaction; and (2) a demonstration of the system and process to familiarize users with the systems and tasks. Thus, a video demonstration was created to present the key system features and the procurement process (see the demo at Imaras 2011; Imbins 2011). The demo illustrates the systems and their use within different scenarios. A short quiz was also added to help users in reviewing the system and tasks. The results indicate positive effects of these improvements and the users provided positive comments on the system, process and experience.