El informe de impacto de género del programa presupuestario

Figure 7.2 gives an overview of the main dialogue platform components. Rectangular boxes depict software components, elliptic boxes components in the physical world, e.g., sensors, devices, or actuators. The black arrows visualise the main data flow of a stan- dard interaction. Coloured arrows represent the flow of information independent from the main data flow. All components of SiAM-dp can be attributed to one of four layers (in Figure 7.2 layers are distinguished by their colour):

Environment Layer:

The environment layer contains all modules that directly constitute the interface to the physical environment. For the interaction between human and computer it comprises input devices and output devices that exchange information with the users. Fur- thermore, it includes those components that build an interface to the cyber-physical environment of the current domain, e.g., a smart home or a vehicle. These components either observe changes in the environment with sensors or directly manipulate it by actuators.

It is intended that components of this layer are highly adaptable and extendable in order to develop dialogue applications for a wide range of various domains and interaction modalities. Therefore, the platform and programming language for the implementation of a module in this layer is not restricted to a specific type and the connection for communication between these modules and the core system supports a variety of different protocols, to mention some of them: TCP, REST, SIP, Apache Thrift, or the direct implementation in an OSGi bundle.

Core Layer:

The core components contain the main functionality of a dialogue application and the architecture corresponds to reference architectures well known from other multimodal dialogue system architectures, e.g., from SmartKom (Herzog and Reithinger, 2006). For all modules in the core layer SiAM-dp provides implementations with strategies that are based on the declarative models which are introduced in Section 6. These concepts are designed with respect to requirements that were derived from many years of experience in multimodal dialogue system research (Schehl et al., 2008; Porta et al., 2009a; Neßelrath and Porta, 2011; Bergweiler et al., 2010).

However, the topics for multimodal dialogue interaction are widely scattered and can have their origins in diverse areas such as research in HCI/CHI, computational linguis- tics, cognitive science, usability, or knowledge management. In order to provide a high flexibility and the option of customisation, all core modules are encapsulated in their own

7.1 SiAM-dp Architecture 153

OSGi bundles and can be arbitrarily replaced by modules that address a very specific research issue in a more adequate way.

The black arrows in the figure represent the data flow of a standard interaction. A component that is origin of an arrow provides information to the component the arrow is pointing to. Normally an interaction starts with an input from the environment layer, passes through the components of the core layer, and ends up with a response to the environment again. Nevertheless, it is feasible that when initiative is taken by the system the interaction starts in the dialogue management.

First, the input from sensors and input modalities is further processed in unimodal interpreters for devices and sensors, which generate a semantic representation of the input and hypotheses about the users’ intentions. This step is necessary in order to make the input machine-understandable and allow further processing steps in the fol-

Figure 7.2 – The main architecture of the SiAM dialogue platform. The black arrows visualise the data flow of a standard interaction. The resource and context management, and backend components are orthogonally integrated and can be applied by all core components.

lowing multimodal fusion and discourse resolution component (see Section 7.9). It merges multimodal input from distinct modalities and complements missing and am- biguous meaning with information from the discourse context.

The main application logic is located in the dialogue manager (see Section 7.3) that reacts to user input and events in the physical environment. The dialogue manager is responsible for the flow of conversation with the user and generates the output of a dialogue application. If not enough information is available for performing a task, a clarification dialogue with the user may be triggered in order to retrieve additional input. Other tasks of the dialogue manager are retrieving information from backend services, which give answers to users’ questions, and the control of actuators in the environment. In all cases it is an approved approach to give feedback to the user. In the dialogue manager component this is done on an abstract level using concepts for the semantic representation of communicative functions as defined in Section 5.5. So the dialogue manager decides which information is given to the user but not how and where it is presented.

The latter is realised in the presentation planning and multimodal fission (see Section 7.10) component that decides which devices are involved in output presentation and what the presentation looks like. Finally, the modality specific output generators produce concrete device and technology specific representations of the output that are presented to the user by applying the output devices in the physical environment. Ad- ditionally, actuator controllers generate commands in order to control actuators that change the state of the physical environment.

All modules of the core layer have access to support modules that are located in the resource and context management layer and the backend layer. These components are orthogonally integrated into the architecture by utilising the OSGi service concept. This allows to register the components to the OSGi service registry and makes them available to every other OSGi component. Thus, for every component it is possible to retrieve information necessary for dialogue processing that is provided by the support modules. Furthermore, core components can update resource and context knowledge or execute calls on external services for information retrieval and transfer.

Resource, Context, and Knowledge Management Layer:

The resource, context, and knowledge management layer comprises all components that keep track of the current system and context states. This includes a project manager (see Section 7.4) that manages and distributes the dialogue application specification models. The device manager (see Section 7.6.1) monitors connected devices and joins them to the dialogue application if the application supports them. Furthermore, it pro- vides valuable information to the presentation planning module. The session manager (see Section 7.6) collects information about the active sessions, participating users, and used devices. The content represented on graphical user interfaces is managed by the