To evaluate the MUS-NET platform, especially the main transcription interface (the only semi-automatic module), we held an online contest with participants recruited from a general crowd of people. The contest asked participants to take transcription tasks from the platform and complete them on site. The participants competed with each other for more and better transcriptions during a given time constraint.
14.5.1 Contest Setup
The online contest lasted for two continuous hours, but could be taken any time at one’s convenience. Opening instructions informed the participant of the goal and procedure of the contest, which could be read before and reviewed during the contest. The participant began by clicking the start button located at the end of the instruction page, which took her to the profile page and started the timer.
During the contest, every participant was expected to stay on site and accomplish as many transcription tasks as possible. There are two webpages—profile page and music transcription interface—that a participant could land on and switch between. The profile page (Figure 14.8) is the main portal for taking new tasks. It also displays the identity of the participant and her status in the contest. A participant’s status lists a pool of pending tasks, a separate pool of submitted tasks, and another pool of skipped tasks. A pending task is either a new or an incomplete transcription that is saved for later changes; a submitted task is the final version for grading; a skipped task typically indicates challenges due to a participant’s knowledge or the platform’s capacity, which is (strategically) skipped to save time. While both submitted and skipped tasks do not allow further editing, one can start/continue working on a pending task, which takes the participant to the music transcription interface.
Alex
Pending Submitted Skipped
Add Task
Work-No Title Composer Genre Period Key Page-No
Op. 72 Fidel… Beeth… Opera… Roman… 8 RC. 38… Fanta… Coper… Fanta… Renais… 1
BACK HOME GALLERY ALEX
Figure 14.8: Contestant’s profile page: the main portal for getting new tasks and checking status.
The transcription interface (Figure 14.6) is the main site for working on a task. It is equipped with a small tutorial widget located on the navigation bar. The tutorial is a brief introduction of the tools (and their shortcuts if any) on the interface but no music theory or ways/strategies for task completion. It is not required and the participant may or may not spend time on reading the tutorial or part of it; most of the provided functionality can also be explored by simply playing around with the tools. The participant has full freedom in managing her time to save, to submit, and to skip a task, as well as to go back to the profile page for more tasks.
The contest ends when time is up. All participants are invited to complete a post-contest survey to collect demographics as well as comments based on their experience. After the contest, participants can log into MUS-NET again to review what they did in the contest, but no changes can be made.
14.5.2 Evaluation Metric
A participant’s performance in the contest is evaluated by the speed and accuracy of her transcription. The evaluation metric is made clear to the participant before the contest. We designate a set of basic music elements including notes, accidentals, dots, ties, clefs, key signatures, time signatures, and the like, each of which is worth one point. We also give bonus points for correct music notations that require inference based on music context. For instance, recognizing an incomplete measure (anacrusis) or figuring out the implicit time signature in the middle of a piece is worth more than one point. An expert grader sums
Education number In college 15 Bachelor 15 Master 11 PhD 2 Others 4
Prefer not to say 3
Profession number Engineering 18 Arts 11 Business 8 Science 5 Humanities 4 Others 4
Table 14.1: Contestants’ eduction levels and professions.
5 1 2 3 # of submission # o f participan ts <1 1-2 2-3 3-4 >4 (x100) score # o f participan ts 27 13 8 2 12 10 10 8 10
Figure 14.9: Contestants’ overall performances measured by the quantity (left) and quality (right) of their contributions.
points to obtain the participant’s final score, which is a number in the range of [0,∞).
14.5.3 Results
We report results from 50 contestants (excluding 27 out of 77 total participants due to lack of survey) with diverse education and profession. The balanced distributions indicate that this group of participants is a good representative of the general crowd (Table 14.1).
We analyze participant performance based on both the quantity and quality of their contributions. The frequency distributions of the overall submission rate and performance score are given in Figure 14.9. On average, participants submitted 1.7 tasks and achieved scores of 262.3.
Results are encouraging for a couple of reasons. First, we see the platform is intuitive to learn and easy to contribute to. Within the two-hour constraint, every participant had to independently explore the platform and figure out her own way/strategy to compete. We purposefully did not provide either training or thorough documentation, having an interface that is intuitive and self-explanatory to participants. The provided tutorial only included a list of tools and shortcuts, whose role was to provide information but not detailed instruc-
tion. Thus, a participant’s performance reflects a combination of her learning ability as well as improved proficiency within the time limit. The fact that the majority of the participants completed a decent amount of work shows that the learning curve for MUS-NET’s contrib- utor is much less steep than that for many extant notation programs (which provide pages of documentation and may require years of training to become proficient).
Second, we see no zero contribution. Everyone can make a contribution no matter big or small. We had participants that submitted many incomplete tasks (e.g. one submitted 5 tasks and a high score of 855) and participants that submitted only a few but nearly complete tasks (e.g. another submitted only 2 tasks but also a high score of 600). Since contributors are eventually collaboratively achieving better and better tasks possibly based on other’s work, accepting partial contribution allows more contributors to be involved. This is in stark contrast to many extant digital sheet music repositories, where the submission has to be complete, a 0/1 contribution in a non-collaborative process.
Further, given the importance of partial contribution, we demonstrated the efficiency of incremental contribution. An incremental contribution refers to a task that is performed from some existing work rather than from scratch. In the contest, existing work refers to sources from the source generator, yet in general, the sources can also be other contributors’ work. Among all submissions with timestamps (timestamps were dropped if completion time is less than 3 seconds), 20% are completed from provided sources with average score 179.3 and average completion time 12.5 min; the remaining 80% are completed from scratch with average score 148.1 and average completion time 42.3 min. This shows that incremental contribution is important in the self-improving cycle of work-correct-validate, since it not only makes contribution easy, but also yields increased accuracy.
Lastly, we report that the average grading/validation time is 8.2 min per task, which is less than transcription and correction time. This emphasizes the point of less work but increased reliability that is manifest in the self-improving mechanism of work-correct-validate.