The two fundamental requirements of a quantitative measurement are validity and reliability (Kember & Leung, 2009). The validity of a measurement describes whether it measures what it is supposed to measure. Reliability describes whether a measure produces similar results under consistent conditions. In mixed-methods research this issue is addressed with legitimation (Nastasi, Hitchcock, & Brown, 2010), which reflects how well the study is implemented, whether the implemented quantitative and qualitative elements complement each other and justify the made conclusions. In this section, first the validity of the used measurements is evaluated followed by issues of reliability and thirdly issues related to legitimation.
While the list of all the different ways of approaching validity is extensive (see Joy, 2007), in educational research Richardson (2009) emphasises the import- ance of face validity, criterion validity, discriminative validity and construct va- lidity. As this thesis used different measures in each study, which all were devel- oped earlier, the focus here is to cover the essential aspects of validity in each study. Face validity examines the wording or structure of items and whether the participants understand them in the way they are supposed to be understood. This was especially in focus in Study III, when implementing the use of mobile phones as a means for data gathering. Students were interviewed before and after follow- up periods to ask them how they perceived the measurement and whether they saw a graph drawn from their answers as accurately reflecting their experiences. Cri- terion validity describes whether the scores of a measurement correlate with scores obtained with another independent criterion, such as study success. This was ad- dressed in Study I where the main result was that those Theology students, who were committed to their study-related goals at the beginning of their studies, pro- gressed more rapidly in their studies. Discriminative validity describes the extent to which an instrument gives different scores for groups of participants who would be expected to differ from one another. This was addressed in Study I by analysing how students who evaluated their goals differently also had different contents for their study-related goals. Further on students with different approaches to learning
in Study IV also differed in how they saw their learning environment. Construct validity is evaluated by examining the relationships among the scores of compo- nents of a measurement: It was evaluated in each study separately. Its main conclu- sions are presented in the following section.
To evaluate the construct validity of the used measurements, both exploratory (EFA) and confirmatory factor analysis (CFA) were used in this thesis. The differ- ence between the two is that while EFA is used to uncover the underlying structure of variables, CFA can be used to examine a set of expected connections between variables. In other words, EFA is used to explore a structure, while CFA is used to examine a structure predefined by the researcher and its use has been suggested to increase the repeatability of used scales (Kember et al., 2004). In Study I EFA, a maximum likelihood factor analysis with a varimax rotation was used to explore the evaluations concerning study-related personal projects. The reason for using an exploratory approach is the nature of Personal Project Analysis (PPA), which al- though being widely used, has no fixed measurement, but rather is an approach adopted by each researcher for a specific purpose (Little, 2005). Thus, striving for confirming a structure was not sought and instead the focus was on exploring the structure. In Study II the CFA was conducted for the Higher Education Stress In- ventory (HESI) (Dahlin, 2007; Lonka et al., 2008), which has been used earlier, but not in a Finnish setting. CFA was therefore performed to confirm the structure obtained with a Swedish version.
In addition to validity, the reliability of the used instruments was analysed in each study. In Studies I (personal goals) and III (emotional experiences), Cron- bach’s alphas were used to evaluate the reliability of the used scales. In both stud- ies, the alphas were over .70, which reflects a good reliability. Although Cron- bach’s alpha gives a suitable lower boundary for internal consistency of a scale, it has been claimed that as a measure of reliability, it has many flaws, such as often giving a too low value (Sijtsma, 2009). Therefore in Studies II and IV more accu- rate measures of reliability were used. In Study II reliability was assessed using coefficient H (Hancock & Mueller, 2001), which measures the degree of replica- bility of a construct and is recommended to be used along with CFA. The construct reliability values ranged from 0.64 to 0.89 reflecting acceptable construct relia- bility. In Study IV General reliability (Tarkkonen & Vehkalahti, 2005) was used. It showed that the reliabilities of perceptions of the learning environment were at a good level (0.63–0.83), but concerning approaches to learning, surface approach the reliability measure was lower (0.56). In general the reliabilities of the used scales were acceptable.
In mixed-methods research the issues of validity are addressed with legitima- tion (Nastasi et al., 2010; Onwuegbuzie & Johnson, 2006), which address issues not associated with monomethod designs. As Study III represented most clearly, a mixed methods study, these issues are addressed according to the design of this study. Onwuegbuzie and Johnson (2006) address different types of legitimation, of which the three most crucial concerning this study are discussed here. Inside- outside legitimation reflects whether a study has been conducted by an insider in the population under study or an outsider researcher, because these two might have different views on the data. In this study, the first writer, who has been part of an
earlier similar study group, conducted the interviews. However, the analyses were done together with a teacher (Lasse Lipponen) of the study group, bringing to- gether two different standpoints. However, both were partly insiders and the view of a complete outsider was lacking from the interview analysis. Weakness minimi- zation legitimation reflects whether the strengths of the quantitative and qualitative methods are used to minimize the weaknesses of each approach. Although in Study III it could be argued that either the quantitative analysis or qualitative analysis could have been taken further to make them more fine-grained, the combining of the two to focus on experiences from two different data was best able to shed light on how students experienced the two different study periods. Thus, as such the study questions could be answered. Multiple validities legitimation refers to the extent to which all relevant research strategies are utilized and the research can be considered high on the multiple relevant “validities”. Validity of the quantitative approach issued with the CASS methodology using experience sampling with mo- bile phones enabled the minimization of a memorization error in the replies to queries. From the point of view of multiple validities, the two approaches also ad- vanced the development of the used methodology. As this project progressed, the methodology also evolved. For example, during the second year, feedback images from the CASS data were constructed immediately after the follow-up and they could be used in the post-interviews after the two-week follow-up period raising the validity of the interview as a stimulated recall situation. Together the quantita- tive and qualitative analyses revealed a richer picture than a monomethod setting would have.