In this section, the quality and limitations of this research are assessed regarding the reliability and validity of the included studies. This research was explorative in nature and primarily utilized
qualitative data gathered during five case studies. The research questions reflect the practical
approach in the studies, most of which were conducted in close collaboration with practitioners from companies, including real end-users and products on the market.
Reliability refers to the question whether the results are repeatable, i.e., if subsequent
researchers could arrive at the same insights by following the same steps again (Dubois & Gibbert, 2010; Denzin & Lincoln, 1994). Limitations in terms of reliability lie in the descriptions of the conducted studies. Although the studies have been documented by the researchers who conducted them, the publications may have limitations in some of their descriptions. For example, the exact results of the evaluation studies in Study I were not reported due reasons of space and confidentiality issues. In general, the reliability was enhanced by carefully describing the study participants, utilized methods, and procedures for data collection and analysis in each publication. For instance, when presenting the guidelines for developing and evaluating visual data analytics tools for logged usage data, references are provided to the empirical findings during the study or from related literature to justify the proposed guidelines.
Validity addresses how consistent the conclusions from the research are. The rigor of field
research is commonly assessed with construct validity, internal validity, external validity, and ecological validity (Dubois & Gibbert, 2010).
Construct validity refers to the quality of the study in investigating what it claims to investigate,
i.e., how successfully the research procedure leads to accurate observations of reality (Dubois & Gibbert, 2010; Denzin & Lincoln, 1994). Triangulation, i.e., studying the phenomena from different views by utilizing different data collection methods and data sources (Denzin & Lincoln, 1994), can improve the construct validity in case studies (Dubois & Gibbert, 2010). The RQ1 (“How can user experience goals be defined and communicated among stakeholders in product development?”) was explored in two studies, Study III and IV. Study III included survey
responses from researchers representing nine different design cases, i.e. data sources, while Study
IV contributed especially to the provided examples of UX goals, based on 53 survey responses
and previous literature. Alternative data collection methods, such as interviews, could have improved the construct validity in Study III by providing a more comprehensive view of the design cases. Regarding RQ2a (“What kinds of perceptions do product development practitioners have about the usefulness of long-term user experience evaluation?”); only one company was involved in Study I, where the usefulness of long-term UX evaluation results was assessed. However, Study I included three separate evaluation studies with different products and perceptions of employees working in different roles in the company, therefore improving the quality of the study. Next, findings related to the feasibility of specific methods and tools for long-term UX evaluation (RQ2b: “How can user experience evaluation methods and tools support the long-term user experience evaluation in product development?”) are limited in a sense that they are derived from single case studies during Studies I, II and V. Another limitation in answering RQ2b is that the implications concerning the feasibility of the used UX evaluation methods are based on experiences of the researchers, and not observed when actual development teams in companies were utilizing these methods. However, the studies were conducted in close collaboration with practitioners in companies, with representative users from their customer base. In addition, more than one researcher participated in each study, therefore diminishing the possibility of a single researcher’s dominating view. Finally, regarding RQ3 (“How can the utilization of usage data logging be supported in product development?”), the results to support usage data logging in product development, e.g., guidelines, although limited to a single case study, were derived from practitioners in different roles over several data collection points (Study
V). Furthermore, multiple evaluation methods, as proposed by the MILC approach (Shneiderman
& Plaisant, 2006), were utilized in Study V to improve validity of the data.
Internal validity refers to causality between the collected data and the results, i.e., how well
the logical reasoning can defend the research conclusions (Yin, 2003; Dubois & Gibbert, 2010). However, the studies included in this thesis were primarily explorative and did not aim to prove causality, as would be the case in, e.g., experiments (Mayo, 1996). When reporting implications from the research, they were supported with examples from the collected data, e.g., quotes from the participants’ responses or descriptions of events during the studies, to clarify the reasoning. Apart from the employee surveys in Study I, two or more researchers always participated in data analysis, therefore decreasing threats to validity. Furthermore, the Experience Goal Elicitation Process model (Study III), as a main theoretical contribution of the thesis, was iteratively developed from three initial versions by collecting feedback from study participants who had conducted experience design cases in practice. However, the evaluation and further development of the resulting model are concerns for future research.
External validity is concerned with the generalizability of the findings from the study context
into other settings. As case studies do not allow statistical generalization, analytical generalization is recommended instead, where generalizations are made from empirical observations to theory (Yin, 2003). In Study III, implications from the survey responses regarding nine different design
cases were utilized in design of the Experience Goal Elicitation Process model. Using web survey as a data collection method may have posed some limitations to the richness of the data, and interviews with each participant might have provided more elaborate comments regarding the utilization of UX goals in each case study. However, all the participants had an opportunity to comment on the initial process models and the instructions for defining and evaluating UX goals, allowing the researchers to consider different viewpoints when defining the final versions.
Study I included perceptions of product development practitioners in one company
developing digital sports products, posing a limitation to the generalizability of the results. However, the external validity was increased by conducting the employee surveys in three different long-term UX evaluation studies and including practitioners with different roles in the company. Although the perceived usefulness of the long-term UX evaluation results reflected views of practitioners only in the studied domain, it is argued that most of the identified aspects regarding the usefulness of UX evaluation results were general enough to, for example, motivate the long-term UX evaluations of other digital, commercial products. However, the actual utilization of long-term UX evaluation results in industry contexts is something that should be studied further, also in other contexts, as the results in this topic were limited due the small number of respondents in the follow-up survey (see Section 4.2.1).
Considering the external validity of Study V, contributions related to supporting the utilization of usage data logging (RQ3) are derived from a case study with a flexible manufacturing systems supplier company and are therefore most applicable in similar contexts. However, some of the guidelines (see Section 4.3.2) include similar findings from related research in other domains, therefore encouraging the generalization of specific guidelines into the broader setting. In practice, development teams utilizing any design guidelines should be critical in what is applicable in their current product development context.
Ecological validity refers to the relevance of the research findings in the real world. The data
during this research work was collected in a real-life context, including real companies and design cases, products on the market, employees, and actual or potential customers or end-users. The case studies were conducted in a real-world setting in close collaboration with practitioners from industry, and UX evaluations included users utilizing the products in natural use contexts over time. Therefore, the ecological validity of the research is considered to be high.