Calidad de imagen en equipos convencionales

In this chapter, we have reviewed works relevant to this thesis in several areas. First, we have seen common techniques for feature extraction from MoCap data. Next, we have reviewed different mapping paradigms that have been used for DMI design. Then, we

22_{According to the authors, the decision of using horizontal movement is based on previous explorative}

tests asking non-conductors to perform conducting gestures. This is something we have not found elsewhere in the literature, nor something we have observed in our own observation studies of the sort (see Chapter4).

23_{https://www.leapmotion.com/} 24_{http://www.gestrument.com/}

2.6 Conclusions have reviewed studies that computationally analyze conducting movements. Finally, we made a comprehensive review of DMIs based on the conductor metaphor.

Having done this review, we can identify some opportunities for research with DMIs based on the conductor metaphor.

Regarding the conclusions from existing studies analyzing conducting movements, we identify two problems with respect to their applicability in the design of DMIs. First, most of these studies are performed under conditions other than those found in a per- formance. This is desirable in some cases if what is desired is to have a controlled environment where a particular phenomenon is observed. However, when designing a DMI based on the conductor metaphor, we must keep in mind that the idea that the average user has about what a conductor does (and, therefore, what she will tend to imitate) is associated with what she can see in a performance. For this reason, it is desirable to observe conductors in actual performances, although this implies that the observed effects should be treated with greater caution. Second, most of these studies have been performed for areas other than the design of DMIs. We also think it is ade- quate to perform this analysis with such use case in mind. For this, we must consider what parameters can be controlled in a DMI based on this metaphor and observe the performance looking for causal relations between the movements of the director and these parameters.

Regarding the design of DMIs based on the conductor metaphor, we believe that the progress in motion-sound mapping paradigms offers new opportunities for exploration. We have seen how the vast majority of the revised DMIs offer important improvements in many aspects (more precise control of the tempo, extension of the dimensions under control, etc.). However, in a scenario where users may have different expectations about how to interact with the system or where they may even wish to develop their own style, this aspect has not been explored in depth. Recent motion-sound mapping paradigms encourage to involve the user in the design of the DMI. Following this idea, we believe that the case of the conductor metaphor offers a good use case for investigating the adaptation to user-specific styles or specificities.

Finally, it is difficult to find cases where, even when the instrument has been designed with the goal to attract new audiences to classical music, the extent to which this objective is achieved is discussed. Accordingly, in a context such as that of the PHENICX project in which this thesis is developed, it is important that this is not left aside. For this reason, we believe that it is relevant to design experiences where not only this goal is key but also where, accordingly, its success in this matter is thoroughly evaluated.

Chapter 3

Learning from real performances

Music conducting is a highly complex musical art. It involves many different aspects. Among the many tasks for which conductors are responsible, leading the ensemble using gestures during musical performance is probably the most characteristic and prominent one. In this context, conductors use gestures to establish the tempo, indicate entries to different sections in the orchestra (or voices in a choir) or convey expressive intentions that get reflected in variations in, for example, dynamics and articulation. Although the goal of this work is not to build a realistic model for virtual conducting targeted to professional conductors or students, a better understanding of the causal relationships between conducting gestures execution and the resulting performance informs the design of interactive systems based on the conductor metaphor. In this context, we made an interview with professional conductors and students to identify which specific aspects of a performance could be observed and automatically analyzed. Motivated by the conclusions of this interview, we recorded and analyzed specific parts of a conductor performance focusing on two aspects: the musicians’ synchronization with conductor’s gestures and the relationship between some body movement descriptors and the overall loudness of the performance. This analysis provides the basis for the subsequent parts of this thesis in terms of the methods to estimate movement-music synchronization and estimate correlations between body movement and loudness.

3.1 Introduction

There are roughly two strategies to follow when designing interactive applications based on the imitation of an already established activity as orchestral music conducting. The first one is to build a system that replicates as closely as possible the real scenario. This is useful in cases where the application is meant to be used as a teaching or ed- ucational tool. In the case of conducting, we could think of applications for different

contexts. One could replicate the performance scenario and would consist on a virtual orchestra that interprets and responds to the gestures of the conductor in the same way a real orchestra does. Another application could replicate a teacher during a lesson. In this case, the system should be able to properly identify the correctness of performed gestures and to provide coherent instructions accordingly. These applications could be particularly useful for conductors (professional or students). Obviously, designing this kind of applications requires an extensive knowledge of the replicated scenario and thus can be significantly improved by studying the scenario to be replicated in situ.

The second strategy consists on taking the real activity just as a metaphorical inspiration. This is, the interaction is designed taking elements from the original activity but in a way that does not try to maximize realism. This is is the case of our work, where we look for strategies to make virtual conducting applications more intuitive and appealing to non-conductors and classical music outsiders. Here, there is total flexibility regarding how close or far we want to be from the actual conducting scenario. For example, a virtual conducting application can consist on a DMI where the user performs just one of the tasks that a real conductor does (e.g. indicating tempo). In this sense, it is possible to decide whether to use knowledge from the replicated activity or not. For instance, in the case of indicating tempo, there are different options. One would be to observe and analyze how real conductors communicate tempo and how orchestras react and to replicate this accurately in the system. Another one would be to predefine a simple rule to trigger beats. And yet another one would be to allow the user to define how she wants to indicate tempo.

As we advanced in theIntroduction, this work has been developed, within the PHENICX project, at ESMUC, the Catalan Higher School of Music. Thanks to this, we had the chance to work with professional conductors in different scenarios. We attended conducting lessons, rehearsals and concerts. The availability of commercial contactless unobtrusive motion capture (MoCap) devices such as the Microsoft Kinect allowed us to take recordings of conductors without affecting their performance with an easy setup. An analytical approach of the motion-sound relationships taking place in real performances can help to identify the body movement descriptors that are more useful in the design of DMIs based on the conductor metaphor. Considering that the device used for the recordings is suitable to be used in interactive applications, we ensure that what we learn about these descriptors is applicable.