141
4.5 Reliability and Validity Considerations
Rigour attainment is an important aspect of any scholarly research. This section reviews the key steps taken to minimise possible sources of bias during data collection and analysis in both key phases of this study. Specifically, Section 4.5.1 considers reliability and validity measures of Phase 1 (in-depth interviews, Study 1) and Section 4.5.2 reviews reliability and validity measures of Phase 2 (Studies 2-4).
4.5.1 Phase 1: Qualitative Research
Although trustworthiness of qualitative research is often questioned from the quantitative perspective on threats to reliability and validity, established frameworks for ensuring rigour in qualitative research have been in existence for many years (Shenton, 2004). While these frameworks use different terminology, perhaps in seeking to distance qualitative research from the positivist paradigm, the core concern at the root of these frameworks remains: ensuring that a particular study represents features of the phenomena it seeks to theorise, explain or describe (Hammersley, 1992; Long and Johnson, 2000; Shenton, 2004). Merriam (1995) asserts that another important consideration in assessing the worth of a qualitative study should be the paradigm within which a particular study is conceived and conducted. Finally, from a cross-cultural and cross-national perspective on validity, assessment of whether the phenomena under study are context-specific or transferrable across contexts is essential (Yaprak, 2003).
Healey and Perry (2000) offer a set of comprehensive criteria for judging qualitative research within the realism paradigm that draw from a range of techniques discussed below:18
18 The fifth criterion of ontological appropriateness also identified by Healey and Perry (2000) is addressed in Section 4.2.1
142 1) Contingent validity (in preference to internal validity);
2) Value-cognizant approach to discerning reality (in preference to objectivity);
3) Methodological trustworthiness (in preference to reliability);
4) Analytic generalisation and construct validity (in preference to external validity or generalisability).
Although initially developed for case study research, these criteria have been adopted by researchers as a benchmark of judging qualitative studies in other realism research contexts (Golafshani, 2003; Krauss, 2005; Bollingtoft, 2007). Sections 4.5.1.1 and 4.5.1.2 below detail how these criteria were addressed in the context of this study.
4.5.1.1 Contingent Validity and Value Cognizance
Contingent validity. As in case with internal validity, contingent validity is concerned with internal coherence of the study in assessing that it measures what it intends to measure. In addition, addressing contingent validity entails assessing the findings a study draws for stability over contexts. Following recommendations set out by Healey and Perry (2000), following steps were taken to meet the contingent validity criteria in qualitative data collection and analysis:
1) Theoretical and literal replication. This refers to ensuring that the information is obtained from appropriate, data-rich sources using in-depth questions and that the contexts of the study participants are adequately described. As detailed in Section 4.2.3, contexts of both country sites where this study was conducted were considered carefully. Furthermore, adoption of the same theory-based sampling frame in both country sites ensured selection of information-rich and cross-country comparable cases. While positivist stance on qualitative enquiry recommends random sampling approaches (i.e. Lincoln and Guba, 1985), purposeful sampling is advantageous in that it captures the central shared themes that cut across variation among individual cases (Patton, 1990).
2) In-depth exploration of a phenomenon. By choosing in-depth interviews, an established technique for generating culturally contextualised insights (Miller
143 and Glassner, 1997), researcher was able to obtain thick and rich descriptions of the phenomena without imposing any constraints on participants’ discourses.
Value cognizance. Unlike positivism that is value-free (i.e. assumes that reality is a universal ‘out-there’) and interpretivism that is value-laden (i.e. assumes that reality is a subjective construction of each individual), realism research is required to be value-aware. This perspective encompasses discerning reality through exploration of each participant’s perception as a window to reality that, when triangulated with other perceptions, facilitates emergence of the reality picture. As recommended by Healey and Perry (2000) the researcher followed a series of key steps to ensure credible interpretation of the study’s findings, specifically:
1) Multiple interviews were conducted following the same interview protocol that started with a series of broad questions followed by probes to elicit participants’
perceptions of cultural phenomena in question and views on whether and how individual participants relate to the cultures they encounter;
2) Analysis was triangulated by following a step-wise data analysis approach where individual cases were analysed separately first, then considered on country level to identify core shared meanings, and only subsequently a combined analysis of both country datasets discerned the expressions and perceptions emerged across country sites. This accounted for facilitating consistent expressions of the focal phenomena to emerge on both emic and etic levels, as recommended by Douglas and Craig (2001) and for conducting validity checks in the qualitative stage of mixed method research, as recommended by Creswell and Plano Clark (2011).
3) Regular debriefing sessions were held with the director of studies and peers in both country sites, to discuss the relevance and appropriateness of interview procedures and to scrutinise the interpretation of findings.
4) Finally, to meet general criteria for credible qualitative research, several tactics were employed to ensure honesty from participants. Specifically, participants were informed about the broad aims of the study and were given opportunity to
144 refuse from participating. In addition, participants’ anonymity was protected by use of pseudonyms, and participants were made aware of their rights to withdraw from the study at any point during the interview, and in the course of a cooling off period. These steps assured that data were obtained only through voluntary contribution from participants. An introductory section to the interview was designed to establish a good rapport with each participant, and in encouraging participants to elaborate on their answers researcher utilised neutral phrases so that not to create impressions of approval or disapproval that could potentially result in bias of the answers offered by the participants (Mack et al., 2005).
4.5.1.2 Methodological Trustworthiness, Analytic Generalisation and Construct Validity
Methodological trustworthiness. This criterion is closely related to the general principles of research reliability. Healy and Perry (2000) do not elaborate much on this criterion definition, referring the researchers to follow principles of qualitative research dependability introduced by Lincoln and Guba (1985) that requires assuring a consistent research design and detailed report of the data collection and analysis processes.
Following this requirement, Section 4.3.1.2 comprehensively details how the consistency of data collection and analysis techniques were addressed in this study. In particular, following a consistent interview protocol in all interviews, verifying translated documents with local experts, audio-recording the interviews (with participants’ consent), detailing the analysis procedure and providing an example of how data coding was approached demonstrates steps taken to ensure consistency of the study. In addition, as recommended by Healy and Perry (2000), findings reported in Chapter 5 offer quotations from participant discourses (see Section 5.2.1) and also present the data in summary tables where relevant (see Section 5.2.2).
Analytic generalisation and construct validity. These criteria relate to objectivity of theory-building in realism paradigm. A core realism principle is that theory should be built first and confirmed or disconfirmed before testing its’ generalisability to a
145 population (Miles and Huberman, 1994). In this regard, Healey and Perry (2000) stress that qualitative study design should be underpinned by prior theories, to define constructs in question that are subsequently triangulated with the data. In this regard, conceptual definitions of the constructs of Local, Global and Foreign cultures and conceptual model of Consumer Multiculturation were established through rigorous inter-disciplinary review of the literature on acculturation, cross-cultural psychology and culture-informed consumption to ground conceptualisation in extant knowledge. In addition, the conceptual model was presented at several peer forums to obtain feedback and recommendations and has also been submitted and published in a form of a conceptual paper in a peer reviewed journal (see Kipnis et al., 2014). The obtained reviews and recommendations uncovered additional useful strands of the literature that informed refinement of construct definitions presented in Chapter 3 (p:60) As reported in Chapter 5 (p:151), these definitions informed data collection and analysis to ensure that emergent findings are adequate expressions of the constructs in question observable in both country sites. In particular, the coding procedure minimised ambiguity in categorisation of cultural meanings emerged from participant discourses, particularly serving to clearly delineate between foreign and global cultural meanings which was one of the key aims of the study. Finally, some of phase 1 findings were written up in conference papers and presented at peer reviewed conferences (see Kipnis, Emontspool and Broderick, 2012; Emontspool, Kipnis and Broderick, 2013).
It is important to stress that because qualitative findings are derived from small number of individuals, limitations of their applicability to wider populations need to be acknowledged (Shenton, 2004). In the context of this study, the boundaries of sampling frame and the number of the participants were clearly conveyed (see Table 4-5) and the data analysis strategy (see Section 4.3.2) specifies that participants were treated as variants of countries’ social settings rather than indicators of general trends among wider population (Crouch and McKenzie, 2006). Furthermore, discovery of rich and thick construct expressions from different perspectives served as an important informant of the quantitative theory-testing phase 2, in particular adding rigour to deriving and refining construct measures. Reliability and validity considerations of phase 2 of the study are discussed next.
146