Methodology must be developed carefully in cross-cultural research, as this type of research brings with it a range of methodological issues that would not arise in monocultural studies, including those relating to translation, equivalence of measurement, sampling, data analysis and presentation (Matsumoto and van de Vijver, 2011). Bias and lack of equivalence in studies can affect the way cross-cultural differences are interpreted. A good cross-cultural study needs to keep these in mind in addition to the aim of the research, the use of contextual factors and the cultural distance of participants. Cultural distance refers to the extent of difference between the cultures being studied. When groups have a larger difference, it is more likely that significant cross-cultural differences will be found. However, uncontrolled variables are also more likely to influence the results and offer alternative non-cultural explanations for the differences. Therefore, when groups with a large cultural difference are studied, bias and inequivalence are likely to cause problems. In order to measure cultural difference in a study, participants should be assessed for the difference they feel towards a set of other countries, or towards different ethnic groups and the dominant culture (Matsumoto and van de Vijver, 2011). The inclusion of contextual factors as variables in a study is also important, as they help to explain cultural differences, increase the validity of a study and diminish the influence of bias and inequivalence by providing information about their importance to the group. Contextual factors are variables that describe the characteristics of
74
participants and their culture, such as socio-economic status and religious affiliation (Poortinga and van de Vijver, 2009). Finally, the aim of a study will also impact cross- cultural research methodology. Exploratory studies that aim to increase understanding of cultural differences through highlighting similarities and differences between groups, are less likely to be prone to bias and inequivalence than studies that are hypothesis-testing in nature and attempt to advance a theory. Of course, hypothesis-testing studies offer more information about the causes of difference and, thus, a trade-off exists that must be considered when conducting cross-cultural research (Matsumoto and van de Vijver, 2011).
Ethnic and Language Matching
The identity of the researcher may have an important impact on the outcome of social research, particularly when studying ethnic groups. To improve responses from participants, it has been argued that ethnic and language matching should occur at all stages of the research process, from design to analysis (Grewal and Ritchie, 2006). Achieving complete ethnic matching is very difficult, considering the complex nature of ethnicity. Therefore, matching can be thought of on a scale from not matched to perfectly matched (Grewal and Ritchie, 2006). A researcher would then lie somewhere along the scale, depending on the extent to which they are matched.
There is evidence that the impact of ethnic matching is both productive and potentially obstructive (Zinn, 1979; May, 1993; Phoenix, 1994; Egharevba, 2001). The main advantage of ethnic matching is a shared identity, which allows the researcher to be considered an insider during the interview and allows for rich data collection. Rapport and trust between an ethnically matched researcher and participant may develop more rapidly and thus increase depth of response. Elam et al. (1999) examined how the content of the interview or the sensitivity of questions may be influenced by ethnic matching. Sensitive issues, such as racism, sexuality and religion, were found to be discussed more openly with ethnically matched interviewers. Further, a researcher from a similar ethnic group to the participants being studied may offer expertise and insight to the study design and may help in the recruitment of participants. Grewal and Ritchie (2006) discussed the disadvantages of ethnic matching. Participants may assume that researchers who share a common ethnicity already know the answers to questions. Further, respondents might not open up to those researchers,
75
as they may be afraid of being judged or gaining a reputation, particularly in small communities. Researchers, similarly, may not fully probe respondents on issues with which they are familiar, and may make assumptions about the respondent.
Language matching, too, is important across all stages of research, including design, data collection and data analysis. Shared language aids communication, and allows the respondent to fully present their views (Hughes et al., 1995; Bradby, 2001; 2002; Fallon and Brown, 2002). However, in some cases, the impact of using a shared language may be less positive. Respondents may consider the status of a researcher interviewing in a language other than English to be lower, and subsequently judge their performance negatively (Grewal and Ritchie 2006). Thus, in both ethnic and language matching, pros and cons exist. Yet there seems to be a general consensus that the benefits of ethnic and language matching outweigh the disadvantages.
Equivalence in Cross-cultural Research
Cross-cultural studies use comparison between groups at their core. In order to effectively measure group differences, methodological issues associated with equivalence and bias should be considered during the design and analysis phase of the study. Equivalence refers to the degree of comparability of measurement outcomes (Matsumoto and van de Vijver, 2011), while bias occurs when participants from different cultures do not give the exact same meaning to a measurement instrument (van de Vijver and Leung, 2011). The two concepts are directly linked in any study; bias reduces the equivalence of measurement outcomes across cultures, and only those instruments that are free from bias will carry the same meaning to participants within and across cultures.
There are different types of equivalence, all of which are important to take into account when conducting any study. These include construct equivalence, structural equivalence, measurement unit equivalence and full score equivalence (scalar equivalence).
76 Construct Equivalence
Before any comparison between cultural groups can take place, it is essential for researchers to check that the constructs being measured are equivalent. Equivalence occurs when each of the groups being studied share the same meaning for the construct being measured. In some cases, construct equivalence can never be met (e.g. where culturally specific psychological constructs exist). An example of this is seen in the case of Amok, which is present among some East Asian men (particularly among Malay men), and is a sociopathic syndrome characterised by a short period of hostile and destructive behaviour often triggered by an insult. Under Amok, men show automatic behaviours and feelings of persecution and, afterwards, are left exhausted and with no memory of the incident having occurred. Although aggression among young men is a universal trait, the various aspects associated with Amok mean it is highly culturally specific to this group (van de Vijver and Leung, 2011). In other cases, construct inequivalence occurs when comparisons are made between constructs that do not show evidence of comparability across cultures. An example of this is found in research on achievement bias. In a Western setting, achievement is shown to be an individualistic quality; however, ‘individualism’ is not always replicated in other cultures. In fact, McClellend et al. (1985), in their study of achievement motivation, were criticised for not taking contextual and cultural factors into account. Studies have shown that in some non- Western societies, achievement is more socially orientated and associated with the collective (Doi, 1982; Kagan and Knight, 1981; van di Vijver, 2009). In Chinese culture, for example, actively aiming for self-betterment and putting oneself first are not universally positive traits (Winter, 1996). Instead, their culture values a more socially driven approach, with individuals aiming to meet the expectations of important people and groups, such as families or peers. Therefore, researchers must consider different understandings of constructs across different groups to avoid construct inequivalence.
Structural Equivalence
When a measurement instrument assesses the same construct across cultural groups, structural equivalence is said to occur (van de Vijver et al., 2009). The instrument should show similar patterns of relationships to theoretical variables or, in other words, similar factor structures in all groups (van de Vijver and Leung, 2011). Various personality tests, such as
77
the five-factor model of personality (McCrae and Costa, 1997), have been assessed for structural equivalence. Functional equivalence is a special case of structural equivalence that occurs when the instrument measuring a particular construct, such as depression, measures the same construct in each cultural group with high validity, showing similar patterns of convergence and divergence. In other words, the measure is correlated with other related measures of the construct, and not correlated with non-related measures of the construct (van de Vijver and Leung, 2011).
Measurement Unit Equivalence
Measurement unit equivalence occurs when instruments have the same measurement units in their measurement scales across cultures, but with different origins (van de Vijver et al., 2009). A non-psychological example of this is temperature measurement and the Celsius and Kelvin scales. Measurement unit equivalence is present in instruments with scale intervals or ratio-level scores, which maintain the relative scores of individuals in each cultural group (i.e. the same ranking), irrespective of sources of bias that differentially shift scores of the different cultural groups as a whole (van de Vijver and Leung, 2011). An example of a source of bias is stimulus familiarity, which can affect scores on questionnaires in some cultures more than others, but individuals from a cultural group may be affected in a similar way. Measurement unit equivalence should be assessed in order to achieve clear understanding of the results of between-group comparison using such instruments. This can be achieved through the use of statistical methods such as factor analysis. For a construct to have measurement unit equivalence, factor loadings for measurement items should be equal across groups. This implies the questionnaire or test item is understood in a similar way across cultures (Kankaras and Moors, 2010).
Full Score Equivalence
Direct cross-cultural comparisons of measures should ideally take place only when an instrument shows full score equivalence. When measures have the same ratio or scale unit of measurement and the same scale origin in the cultural groups being studied, full score
78
equivalence is present and the mean scores between groups can be directly compared (van de Vijver and Leung, 2011).
Causes of Inequivalence in Cross-cultural Research
Bias refers to factors that arise during cross-group comparison and affect the outcome of measurements between groups. The effects of bias should be accounted for and minimised when measurement instruments are used cross-culturally. Examples of bias include construct bias, method bias and item bias.
Construct Bias
Construct bias can cause construct inequivalence, and occurs when a particular construct is studied across groups, despite having a different meaning to the various cultural groups being studied.
Method Bias
Method bias refers to bias that results from methodological and procedural aspects of a study. It can be of different types, including administration bias, sample bias and instrument bias (van de Vijver et al. 2009).
Administration Bias
Administration bias takes place at the time of test administration and can occur due to a range of factors, such as different test environments, use of novel equipment for the participants, unclear instructions for participants, unclear guidelines for researchers and communication issues.
79 Sample Bias
Sample bias relates to differences in sample characteristics that can affect results. It is important to take these into account when groups from different cultures are being studied, to account for background variables. For example, factors such as socio-economic status and level of education, if not matched between groups, can result in such bias. Therefore, matching cultural samples is very important, as is assessment of contextual factors that are likely to explain difference.
Instrument Bias
Instrument bias occurs when there is a problem with the characteristics of instruments, causing a cross-cultural bias. This can take place when participants have different levels of experience and familiarity with test material, such as instructions.
Item Bias
This type of bias occurs at the item level and may cause full score inequivalence (Holland and Wainer, 1993). Item bias occurs when participants with the same level of understanding of a particular construct (for example, participants with the same intelligence) have different mean scores on the item because of different cross-cultural backgrounds (van de Vijver and Leung, 2011). An example of how this can occur is through poor translation, wording of items or adaptation of items. Items may not be fully understood by specific cultural groups, as they might see them as unclear or irrelevant.
Methods of Countering Bias
The literature that examines bias in cross-cultural research has focused on various approaches to countering bias in measurement. Two of these methods are the judgmental approach and the statistical approach (van de Vijver et al. 2009). The judgmental approach uses the knowledge of cultural experts to identify items that may cause bias. Using ethnically matched researchers to help in the design and analysis stage is one example of this. The statistical
80
approach uses statistical techniques to reduce bias. Depending on the measurement method, sample size and number of groups being studied, different techniques are selected. Often, the statistical approach is used at the data analysis stage in cross-cultural studies. Techniques such as exploratory factor analysis and confirmatory factory analysis are used to assess whether different types of equivalence, such as structural equivalence and measurement unit equivalence, have been achieved across the cultural groups being studied (van de Vijver and Leung, 2011). Van de Vijver el al. (2009) suggest that a combined approach of statistical and judgment methods during research is the most effective.
The type of bias should inform the techniques used to counter it. The presence of construct bias, for example, may be assessed statistically by techniques used to compare data structures between groups, such as comparing factor structures. The existence of construct bias indicates dissimilarity in the construct being measured across cultures and would result in a need to reassess the measures being used in the study (van de Vijver and Leung, 2011). Culturally tailored measures developed within the cultural groups being studied reduce construct bias by ensuring participants from each group understand the context being measured. Cultural decentering can also be applied (Tanzer, Gittler and Ellis, 1995). This technique involves trying to ensure that all groups being studied understand the items being measured by removing cultural details from stimuli (van de Vijver and Leung, 2011).
Method bias can be reduced by intensively training administrators, preparing thorough manuals for assessment and administration across cultural groups, having straightforward instructions and matching samples by balancing them according to demographic and contextual variables. Analysing response styles is another technique for minimising method bias. This includes checking to see how participants from different cultures answer questions (e.g. whether they select end points of a response scale (extreme-response style) or if they agree with questions irrespective of the question (acquiescence bias)). The choice of response style may be associated with characteristics in the specific culture, such as sincerity (Johnson et al., 2005).
Item bias can be minimised by ensuring that items have been correctly translated and are used in an appropriate context. A combination of the judgment approach, in which specialists in the group being studied assess the suitability of the items being used, as well as statistical
81
measures, such as differential item functioning analysis, can be used to combat this type of bias (van de Vijver and Leung, 2011).
Methods Used to Reduce Bias in the Present Study
A combination of judgmental and statistical approaches for countering construct, method and item bias was used across all phases of the study. In order to check for construct inequivalence, the measures were piloted to assess relevance and shared meaning for the different ethnic groups.34
A great deal of effort was made to minimise the effects of method bias. Administration bias was reduced by the further development of an already detailed manual for administration and coding of interview data, which had clear instructions and guidelines for each section of the interview. The researcher was trained in the administration and scoring of the interview. Representatives from the British Indian, British Pakistani and non- immigrant White ethnic groups were interviewed and asked to describe what they understood each question to mean. These representatives were selected through convenience sampling and came from a range of socio-economic statuses. Due to the use of universal measures between groups, cultural decentering was then applied to many of the interview questions and questionnaire items, if meanings were found to be unclear. Both construct bias and item bias were reduced through the use of the judgmental approach, which relies on cultural expertise on the groups being studied. This was achieved through the ethnic matching of the principle researcher (who was of British South Asian origin) to British Indian and British Pakistani groups. Ethnic matching also facilitated the recruitment of participants and increased the validity of the information obtained from the British Indian and British Pakistani groups. Further, it ensured that particular topics in the interview, such as racism, were approached in a culturally sensitive way. Item bias was reduced by the use of questionnaires, which had been used in previous studies across different cultural groups so that the items had previously been checked for relevance and applicability. An example of this is the Strengths and Difficulties Questionnaire (SDQ), used to measure children’s psychological adjustment, which has been applied across a wide range of ethnic groups and has shown clinical utility and good psychometric properties at the item level and as a whole (see Woerner et al., 2004).
34
According to the procedures outlined by Bernstein et al., 2005; Krishnakumar et al., 2004; van de Vijver and Tanzer, 2004
82
Interrater reliability analyses conducted during the data collection stage further ensured administration bias was minimised. As all of the mothers were British-born and English was their native language, the need for language matching was minimised. However, in some sections of the interview, certain group-specific cultural terms were used to refer to specific practices and beliefs. An example of this was the use of the word biraderi (occupational caste) for the British Pakistani group, which was used to understand more about the background of participants.
Every attempt was made to reduce sample bias. The balanced nature of the sample with respect to the key socio-demographic variable of educational level ensured that sample bias was reduced. The sample was also matched for other socio-demographic variables, including child’s age and child’s educational level. The population census for the London area was used to help with matching. Details were obtained of the educational levels of Indian, Pakistani and White populations living in London.35
Finally, cultural distance was minimised in the sample being studied, as all participants were born in the United Kingdom and lived in multicultural areas. An acculturation scale that measured how the British Indian and British Pakistani groups felt towards the dominant culture was included to find out more about cultural distance. Also, the non-immigrant White group was asked how they felt about the British Indian and British Pakistani groups during the interview. In this way, any cross-cultural differences were more likely to be a result of actual ethnic differences rather than confounding variables.
Based on these levels, corresponding grids of educational levels for the three ethnic groups were derived for the study. These detailed the number of participants at each educational level that would be required to create a matched sample. The inclusion and measurement of many contextual variables in the study design also helped obtain a clear picture of the sample. In addition, to ensure participants