DESCRIPCIÓN DE LA EXPERIENCIA Antecedentes

The meaning of quality in social science research

The concept of quality in research is interpreted as what makes research essentially reli- able and valid (Bowling, 2002). The concepts of reliability and validity are multi- faceted. Cohen et al. (2000) argued that validity and reliability are not all-or-nothing characteristics of the research instrument, and that threats to validity and reliability can- not be eliminated completely. However, the researcher strives to maximise the validity and reliability of his findings. This section discusses and considers validity and reliabil- ity issues in relation to this research.

It is commonly suggested that both reliability and validity criteria carry connotations of measurement, which is generally considered to be a major preoccupation with quantita- tive research and less so with qualitative research. However, there has been some dis- cussion among qualitative researchers about an alternative set of criteria that would be more relevant to qualitative research. These alternative criteria in question are identified as issues related to trustworthiness and authenticity (Lincoln and Guba, 2000). This re- search adopted a mixed-method approach using both quantitative and qualitative meth- ods. Therefore, consideration was given to validity and reliability issues in both ap- proaches.

Validity

Validity is the degree to which a measure of a concept obtained by a method truly re- flects the concept that it is denoting (Benton, 1995). Hence, it is the degree to which an instrument measures what it is supposed to be measuring (Carter, 1995; Bryman, 2004; Polit and Hungler, 1993). In qualitative terms, Janesick (2000:393) defined validity as having to do with description and explanation, and whether the explanation fits the de- scription. In other words: “is the explanation credible?” It was also defined to be con- cerned with the appropriateness, meaningfulness and usefulness of the inferences that are generated from research based on data collected (Fraenkel and Wallen, 2000). Sev- eral kinds of validity measures were identified in the literature. Those relevant to this current research are discussed below.

Internal validity / credibility

Cohen et al. (2000) described internal validity as a measure which seeks to demonstrate that the explanation of a particular event, issue or set of data which a piece of research provides, can actually be sustained by the data. The authors argued that in some degree this concerns accuracy; that is, the findings must accurately describe the phenomena be- ing researched. In order to enhance the internal validity of the findings, young people in the focus groups were presented with the main findings of the survey and asked to air their views on them. The level of convergence between the findings from the survey and young people’s agreement or disagreement with the inferences was considered as a gauge of internal validity of the findings from the survey.

Furthermore, throughout the focus group discussions, every time a particular theme or issue was discussed, the conclusions reached were communicated back to the partici- pants to seek their agreement with the researcher’s understanding of the participants’ views. The latter technique is often referred to as respondent validation. Peer debriefing of the findings was also employed to enhance the internal validity of this research, where the researcher exposed himself to disinterested peers in a manner akin to cross examination, in order to test the honesty, working hypothesis and to identify the next steps in the research.

External validity / transferability

External validity concerns whether the results of a study can be generalized to settings or samples beyond the specific research context in which it was conducted (Polit and Hungler, 1993). One aspect of a study’s external validity concerns the adequacy of the sampling design. Every effort was made to employ the most favourable sampling tech- niques towards the survey and obtain a representative sample of the Maltese young peo- ple population. However, the overall non-response rate could have demeaned the gener- alisability of the results obtained from the quantitative survey findings. This is further discussed in the final chapter.

Lincoln and Guba (2000) suggest the criterion of transferability to parallel with external validity in qualitative research. They argued that whether qualitative findings hold in some other context is an empirical issue. In qualitative research, it is not the researcher’s

task to provide an index of transferability. Rather, they suggest that readers should de- termine whether transferability of findings from one piece of research to another situa- tion is possible, and the extent to which it is possible. In this respect, transferability re- quires thick description of data. Therefore, every effort was employed to provide suffi- ciently clear, detailed and in-depth description of the qualitative data obtained, as well as the process by which it was obtained, so that the reader can determine the extent of transferability of the data from the focus group discussions to the wider population.

Ecological validity

Ecological validity is concerned with the question of whether social scientific findings are applicable to people’s everyday natural social setting. The fundamental premise of qualitative research, which often involves a naturalistic stance, is that the researcher de- liberately avoids manipulating variables or conditions, making sure that the situations in the research occur naturally (Cohen et al., 2000). Every effort was made to collect data in as naturally occurring situations and environments, as opposed to fabricated, artificial ones. Focus group discussions were conducted within the same classroom set-up where young people usually discuss topics related to sexuality and relationships at school. Every attempt was made to stay mindful of the approach that was typically being used during the focus groups, to be less directive and carefully word questions to ensure they did not imply a right or best answer to give.

Content / face validity

Cohen et al. (2000) argued that to demonstrate content validity the research instrument must show that it fairly and comprehensively covers the domain or items that it intends to cover. Content validity is of particular relevance to researchers designing tests of knowledge in a specific content area, as was the case in the questionnaire survey of this study. There are no objective methods for ensuring the adequate content coverage of an instrument. Face validity is, therefore, an essentially intuitive process (Polit and Hun- gler, 1993). In this current research, peer professionals in the field were called on to act as judges and analyse the items in the questionnaire to determine whether on the face of it the measure seemed to reflect the concept concerned.

Construct validity

Cohen et al. (2000) argued that in qualitative research construct validity must demon- strate that the categories the researcher uses are meaningful to the participants them- selves and reflect the way in which the participants actually experience the situations. In other words, that the researcher sees the situation through the participants’ eyes. In this respect, a conscious effort was made during the focus groups to enter into the partici- pants’ shoes and experience the concepts discussed as much as possible in the same way that young people do. This was very challenging indeed. Matters arising during the fo- cus groups as a result of the participants’ discussion were taken up and probed further. This was attempted by ending the discussion on every specific topic with the following: “So, just to make sure that I am understanding you well - but please correct me if you believe I misunderstood you: you seem to feel / believe that ….”.

Triangulation

An emerging trend in social science research over the last few decades has been the in- creased blending of methods. Two or more qualitative or quantitative methods are used sequentially or simultaneously, provided the analysis is kept separate and the methods are not muddled, to determine how far they arrive at convergent findings (Mertens, 1998). This is believed to enhance the degree of confidence the researcher can place in the data collected (Wallen and Fraenkel, 2000). The latter approach to investigation is referred to as triangulation, or methodological pluralism (Tones and Tilford, 2001). Denzin and Lincoln (1994) pointed out that triangulation has also come to mean con- vergence among researchers, that is, agreement between field notes of one investigator and observations of another. But generally the term triangulation in the literature has been reserved for those specific occasions when researchers seek to check the validity of their findings by cross-checking them with another method. Denzin (1989) identified different triangulation strategies in his extended view of the multi-method approach, some of which were adopted in this research and are discussed below.

Data triangulation

Data relevant to the overall purpose of the thesis were gathered from both a quantitative survey and qualitative focus group discussions. Different data sources were tapped, us-

ing different sampling strategies, but maintaining the same foci to obtain diverse views from young people having different experiences of learning about sex, sexuality and re- lationships. Pupils from single-sex and mixed-sex schools participated in both stages of the study, while pupils from state schools, junior lyceums, church schools, and inde- pendent schools were invited to participate in the survey. Therefore, the data were col- lected from different groups at one point in time - otherwise known as a ‘panel study’.

Another way by which data triangulation was employed in this research was by compar- ing learning needs identified by the researcher through the interpretation of the survey data (such as age at first intercourse, contraception use, condom use) with young peo- ple’s own perception of need through the focus group discussions. In other words, iden- tifying young people’s learning needs by comparing the normative (the researcher’s) with young people’s (felt/expressed) interpretation of need.

Space triangulation

Data were gathered among pupils across Malta and Gozo and from different types of schools. Therefore, at least two sub-cultures of Maltese society (Maltese and Gozitan) were represented in the samples. Cohen et al. (2000) described space triangulation as an advantageous method to overcome the limitations of studies conducted within one cul- ture or subculture. Human behaviour is both culturally and sub-culturally bound. Find- ings that would have been obtained from studying a single group cannot be generalised.

Investigator triangulation

Sim and Sharp (1998) argued that individual observers and researchers have their own particular observational styles which are reflected in the data collected and the infer- ences made. Therefore, the use of more than one investigator to analyse the data en- hances the validity of data by counteracting the shortcomings of one investigator with the strengths of another. In this current research, colleagues specialists in health promo- tion and public health were called to analyse the survey data and share their interpreta- tions. Furthermore, a PGCE student with a shared interest in the topic but a different area of expertise and perspective participated as a second moderator in the focus groups and also reviewed the themes that emerged from the data following analysis.

Methodological triangulation

Denzin (1989) stated that ‘between methods’ triangulation is a check on validity em- bracing the notion of convergence between independent measures of the same issue. Thus the shortcomings of one technique or method will be compensated for by the strengths of the other. In this current research a number of objectives related to young people’s learning sources of knowledge about sex and sexuality (actual and preferred) were asked in both the questionnaire survey and the focus group discussions.

The potential benefits of triangulation to research

Triangulation raises researchers above their personal biases, partially overcoming the deficiencies of a single method (Denzin, 1994). Multi-methods produces different kinds of data on the same topic, allowing the researcher to understand the topic more com- pletely (Denscombe, 2003). Findings from one method can be checked against findings from another method. The strengths of one compensates for the weaknesses of the other (Brewer and Hunter, 1989). Therefore, triangulation is not a simple combination of dif- ferent kinds of data, but a strategy that attempts to relate them so as to counteract the threats to validity in each (Schwandt, 2001). Triangulation does not prove any particular method of data collection right, but gives some confidence that the meaning of the data has some consistency across methods and that the findings are not too closely tied up with a particular method used to collect the data. Thus, triangulation increases validity in research studies, enhancing confidence in ensuing research findings.

Triangulation also provides a better insight of the phenomenon. Denzin and Lincoln (1998) argued that the combination of multiple methods reveals the qualitative re- searcher’s attempt to ensure an in-depth understanding of the phenomena at hand. Tri- angulation helps to capture a more complete, holistic and a contextual portrayal of the topic under study adding breadth and depth to the findings. Foss and Ellefsen (2002) ar- gued that triangulation provides better insights to situations where established conven- tional approaches yielded an over-simplified or frequently distorted picture. They assert that combining different methods coming from different epistemological traditions has the potential to reveal new perspectives to the phenomenon under investigation which otherwise may have been neglected by the single-method approach.

The limitations of triangulation techniques

Triangulation is sometimes accused of subscribing to a naïve realism that implies that there can be a single definitive account of the social world (Foss and Elefsen, 2002). Writers aligned with constructionism argue that research findings should be seen as just one among many possible interpretations of social life. The multi-method approach can contribute to the overall quality of a research study by supporting the strength of inter- pretations and conclusions in qualitative research, but it does not make it faultless (Mertens, 1998). It is at best understood as a strategy that adds quality and rigour to any investigation. Furthermore, Sim and Sharp (1998) argued that triangulation is not a straight forward manner by which all aspects of validity are enhanced in research. Probably the most common variety of measurement validity enhanced is criterion- related validity. It is debatable how triangulation enhances other measurements.

Triangulation, as a means of validation, rests upon the assumption that the individual methods have different patterns of error associated with them, and that such errors will rule each other out. Therefore, a more accurate picture of the truth is supposed to emerge from combining the two data collection methods. However, it has also been ar- gued that the attempt to relate different kinds of data through triangulation of different methods is a challenging task as data derived through different methodologies are viewed as incommensurable (Sim and Sharp, 1998; Foss and Ellefsen, 2002).

Sim and Sharp (1998) argued that research combining methods should not be regarded as inherently superior to single-method research in all circumstances. Some research questions may be adequately answered by a single method. Thus, the appropriateness of the triangulation strategy is not self-evident, but depends upon the nature and scope of the particular research question. Within the context of this research, there was no means by which young people’s personal sexual behaviour could be explored other than the anonymous questionnaire survey. Therefore, not all objectives in this research could be triangulated. Also, obtaining similar findings from different sources does not in itself guarantee that the findings are valid. Both data sets could be equally incorrect. Further- more, Denzin and Lincoln (1994) argued that independent measures never converge fully. Sources can be inconsistent or even conflicting, with no easy means of resolution. Knowledge gained from triangulated approaches should not be seen as incontestable pools of knowledge, but as a different position on a continuum of knowledge (Foss and Ellefsen, 2002), not necessarily a test of validity (Denzin and Lincoln, 1998).

Reliability Issues

Reliability is concerned with the degree of consistency with which the instruments measures an attribute (Benton, 1995; Bryman, 2004). Seeking a good level of reliability means that the research instrument produces the same data time after time that it is used unless due to variations in what is being measured and not the volatile nature of the re- search instrument itself (Denscombe, 2003). The idea of reliability is then closely re- lated to another criterion– replicability, which is particularly related to quantitative re- search. Details of the setup and instructions given to teachers invigilating pupils com- pleting the questionnaires to ensure consistency have already been described above.

Replicability is more difficult to meet in qualitative research since the social setting cannot be ‘frozen’to make it replicable in the sense in which external reliability is usu- ally employed (Bryman, 2004). But Lincoln and Guba (2000) propose the idea of de- pendability as a parallel to reliability in qualitative research, achieved by adopting an ‘auditing’ approach. To this end, complete records were kept of the research process and a full account of the process has been given to serve as an audit trail. Furthermore, during the conduct of the focus group, the second moderator could establish how far proper procedures were being followed using a checklist (see Appendix 5).

Lincoln and Guba (2000) propose that one of the objectives of the auditors would be to establish confirmability which is concerned with ensuring that, while recognising that complete objectivity is impossible in social research, the researcher can show to have acted in good faith, and has not overtly allowed personal values or theoretical inclina- tions manifestly sway the conduct of the research and findings deriving from it. To this end, further down I am presenting an account of my own biography and affiliations while acknowledging what influences these could have brought to the research field. Therefore, the readers can act as auditors to this research.

Furthermore, the researcher and the second moderator conducted separate content analysis of the focus group transcripts and compared the decisions taken to categorise the items arising out of the discussion. The consistency of their judgements was sought (inter-observer consistency), which can also be a measure of reliability of the findings (Denscombe, 2003).

Reflexivity

The ‘self’ in qualitative analysis

There is a general acceptance among qualitative research practitioners that the re- searcher’s identity, values and beliefs, are inevitably an integral part of the design and interpretation of data (Stake, 1995; Coffey, 1999; Denscombe, 2003). This has created a major concern in social science research in that the researcher may prioritise his or her own subjectivity over the respondents’. Thus qualitative data can easily be produced by the way they are interpreted and used by the researcher, inducing bias in the findings. This is in stark contrast to the ambitions of positivistic researchers, who criticised quali- tative research for its subjective approach to empirical enquiry and challenged it, called it unscientific and one that degraded the reputation of the objective researcher.

Peshkin (1988) argued that while researchers cannot eliminate subjectivity, rather than fear it, they should be encouraged to formally and systematically seek out to monitor