MIRMaid : an interface for a content based Music Information Retrieval test-bed

I would also like to thank past and present members of the Advanced Information Management laboratory at UCT for their support and other individuals who acted as patient test subjects. I would also like to thank past and present students from the Masters in Information Technology cohort for their help, in particular Samuel King and Victor Katoma, for their support and friendship.

Problem Statement

The Solution

It also proposes to automatically test different combinations of test metrics for the same recovery task and the same data set against each other without having to do it manually. This project also gives suggestions on how modules from different frameworks can be combined and different combinations of modules can be tested against each other.

Thesis Structure

The future work chapter also explores the option of interfacing to an International Music Information Retrieval Laboratory (IMIRSEL) [Downie, 2003] but not as a rival to larger and well-established evaluation frameworks such as M2K, as this interface is more geared towards handling small specialized repositories. The last chapter gives some suggestions on how the interface can be extended in various ways.

• Overview
• Data Mining and Musical Digital Libraries
• Musical Digital Libraries
• Test-beds
• The Structure of Digital Audio Data
• Sampling
• Quantisation noise and resolution
• Psychoacoustics, sound perception and music cognition
• Basic Signal Processing operations
• Fundamental Frequency estimation
• Event Detection and Windowing
• Feature extraction
• Matching
• Transcription Models
• Summary

The raw audio data is stored in a file that specifies other information such as the data format and the resolution/bitrate of the sampled audio. The overwriting methodology and the methods used for the extracted data are directly dependent on the original signal format and the method that will be used to match the data.

• Overview
• Information Retrieval Frameworks
• CLAM
• M2K
• MARSYAS
• MUSART
• Music processing languages
• Matab
• Octave
• Labwindows
• Nyquist
• External Libraries
• Machine learning libraries
• Music processing libraries
• Visualisation
• Other Tools
• Sphinx-3
• WEKA
• Summary

It is designed to be easily decoupled from the rest of the CLAM framework. Matlab is a high-level language for technical computing and provides an interactive environment for developing algorithms, data processing, data analysis, nL",eric cOO1pLtation and building graphical, user interfaces [Mathematics, no category]_.

• Overview
• Structure
• Test-bed Structure
• The repositories
• Modules
• The Interface
• Design Goals
• Interface Elements
• Interface development
• Strategic positioning of the framework
• Interviews
• Improvements on the interface
• Description of development tools used
• Summary

The import process relies on the user to provide accurate information about the input and output format of the modules they are importing into the testbed. Modules can be ordered on the network in any order, provided that the input format of the current module corresponds to the output format of the previous module. The rest of this chapter and the next two chapters will be devoted to a discussion of interface development.

This was done to facilitate discussion about the interface and to provide a concrete visual representation of the prospective system for both expert and novice users. This interface was basic and contained only broad elements and vague representations of the elements that would eventually be represented in the interface. The group interviews were in the context of an informal discussion in which the sco;Je of the study.

We then went through the interface and discussed attributes such as navigation, layout and browsing. One of the big changes to the interface was that order added to the navigation by adding nLmbers to the interface. 'fe'ent panels of the interface. Predictive defaults have been placed in editable slots to make it easier for potential users of the interface to answer the most frequently asked questions.

Then there was a discussion about how the interface was developed and the interview process that helped improve the layout and structure of the interface. The design of the interface evaluation system will be discussed in the next chapter.

• Overview
• Population Selection
• Experiment Design
• The Tasks
• Arranging outcomes from the interface
• Questionnaire Design
• Subject profile
• Interface and task based questions
• Justification for not using time measurement as an evaluation tool
• User observation
• Confounding variables
• Summary

The experiment tested how effective the layout and overall logical structure of the interface were. The experiment was designed to confirm that all users in the sample set were able to perform the task they were set to perform. The tasks were selected in a way that would make the most sense to novice users of the interface.

Typically, the wizard observes the user's actions from another space and manipulates the responses to user actions in real time. The purpose of the questionnaire was to find out if the interface is successful in being both user-friendly and usable. One of the factors considered in deciding the length of the questionnaire was the cumulative time it 43 .. would take users to complete both tasks.

Task-based and subjective measurement of the interface allows adequate measurement of the interface. Before the main user experiments, a small pilot study was conducted to identify any obvious usability or technical issues and thus reduce confounding factors in the experiments. The questionnaire was then discussed and the reasons behind some of the questions in the questionnaire were discussed.

• Overview
• Sample population analysis
• Results Summary
• Discrepancies between results from task based and interface based statements
• Evaluation of design results
• Measurement of design goals
• Problems with the MIRMaid interface
• Navigation
• General layout and operation logic
• Controlling query options
• Finding and Loading Sound Clips
• Summary

There is an interesting relationship between the responses to most of the interface-based statements in general and the task-based statements. Among expert users, there were two testers who did not get stuck in the interface. All novices got stuck somewhere in the interface except for two test subjects who disagreed with the statement.

The design goals against which this interface was measured were outlined at the beginning of the interface creation process. The pop-up screen appeared in the upper right corner of the screen, forcing subjects to return to previous elements in the interface. One factor that contributed to this confusion is that terminology and naming differed across the interface and task instructions.

Checking query options is the most consistent problem revealed during the experimentation process by the test subjects' interaction with the interface. The logical layout of the interface also created many problems, even though all areas of the interface were numbered. The user experimentation process also revealed a few gaps in the navigation and program flow of the interface.

Overview

This could be important because it would communicate to users where they are in the process and their location in relation to wider systems. The interface can be improved by minimizing the testbed's dependency on importing externally generated modules by adding functionality for modules to be written in MATLAB within the interface. This will allow the interface to fit into a broader architecture for a large-scale MIRlMDL test and development environment, as described in the MIRIMOL evaluation project white paper collection [Downie, 2003].

It would also be useful if the interface could allow users to switch between different assessment modes or be able to convert a question asked in one way to another, just by providing additional information. The framework can support the export of successful combinations of modules that have been evaluated on the testbed. The interface can be changed so that it can be indirectly connected to each other through the workspace.

The interface could produce specifications for systems from the output returned for various tests. The interface can easily be extended to accept different types of data in addition to audio content. Having different kinds of data in the testbed could also make it possible to test different content-based and text-based information retrieval methods against each other.

Future testing of the test-bed and the interface

One consequence of preprocessing is that the results returned from the preprocessed collection can be biased, due to the error that will be introduced when resampling and converting the clips from their original format. The framework can be improved by more gracefully handling music clips with different sampling rates, reflecting the differently formatted data present in different repositories, similar to how it would be in the real world. The second way would be to modify the interface so that it feeds into M2K and users are asked to fill out a web questionnaire.

Support can be added to the interface for creating benchmarks for evaluating multiple algorithms for a retrieval task. This can be improved by adding test collections in addition to data collections in the interface. This will also allow different metrics to be used for evaluating different retrieval strategies for a specific information retrieval task.

This will allow the evaluation of different combinations of transformations and retrieval strategies, but each time you use a different way to test the effectiveness, after which you can set up a matrix to compare different retrieval strategies against each other. This could be used to create an evolutionary tournament for different test metrics on specific retrieval tasks. Different combinations could be tested on several criteria, and then the different combinations and variations were tested through an evolutionary game.

Summary

Support can also be added to allow users to run related queries and compare them to those they have already run or compare results from different evaluation modes.

Conclusion

It was revealed that the discrepancy was partly due to an extra step omitted in the task instructions, and partly due to a mismatch between the numbering on the task instructions and the numbering on the interface. Interruptions in the workflow of the interface were also seen to be problematic and should be avoided. It included suggestions on how the interface could be improved and expanded in various ways.

This included suggestions for extending the interface's functionality with tools for: 1) writing modules within the interface; The scope of the interface can be expanded by accepting other types of data in addition to audio content. The user experiments are intended to test the relevance of the interface I built, to see how easily transformations can be applied through the interface, and to see if the interface works properly.

The other use of the experiments is to see how you interact with the interface by performing simple functional tasks. After completing the tasks, you will be required to fill out a questionnaire in which you will document your experiences with the interface and with the tasks you have passed. If there's anything you'd like to change about the interface, what would it be?