The new LMS-TPACK survey containing 58 items (refer to item summary in Table 4) was administered via the UNISA server using LimeSurvey. An initial invitation, containing a hyperlink to the survey, was emailed to a target population comprising all UNISA educators on the Pretoria and Florida campuses. The cross-sectional survey (Creswell, 2012) made it possible to collect data at one time during September/October
66 2014 about educators’ current perceptions regarding their myUNISA integration knowledge.
Several open-source survey software packages are available for designing, gathering and analysing survey data. In this instance, LimeSurvey offered a relatively easy and convenient way for designing and hosting the online questionnaire as well as gathering and analysing the data. A major advantage was its compatibility with SPSS (Statistical Package for the Social Sciences). This made it possible for responses to be directly entered into and stored in the database and easily transferred and converted to numerical data for meaningful statistical analysis. Automated personalised feedback was provided on individual findings, giving subjects a general indication of the knowledge areas that might need to be developed. Even though feedback was provided, the report served as a mere reflection and was not used for research purposes. Initial responses were slow and so a reminder follow-up email was sent (see Appendix J).
The sum of at least 300 subjects was the target sample size, with the objective being “to eliminate subject variance as a significant problem” (DeVellis, 2003, p. 87). The final sample size was 332 (full responses). Since a large enough sample was obtained, statistical analysis was performed to confirm or refute validity and reliability for the new LMS-TPACK instrument.
4.9 DATA ANALYSIS
In an attempt to develop and validate a new reliable instrument for assessing ODL educators’ LMS-TPACK, this study addressed two research questions:
a) What are the constructs and underlying dimensions that need to be measured to ascertain LMS-TPACK?
b) Will the measuring instrument developed be valid and reliable for measuring the seven TPACK constructs described by Mishra and Koehler?
Firstly, TPACK theory and the literature review were used to establish the initial constructs and help clarify the underlying dimensions that emerged from the LMS- TPACK survey. To further strengthen the instrument's content and face validity, a focus group, expert review and pre-test were used to verify whether the underlying
67 dimensions described in the LMS-TPACK survey were indeed represented. If the latent dimensions were confirmed to be present in the instrument, the survey could possibly be used for the purposes of measuring ODL educators’ perceived LMS-TPACK. Subjects’ responses could be used to more accurately assess the strengths and weaknesses of existing professional staff development programmes and facilitate the alignment of training that can meet the needs and competences of individual educators as well as connect with the broader institutional operational requirements.
Secondly, since a standardised instrument was not being used, the self-report questionnaire was tested for evidence of validity and reliability. Different statistical techniques using SPSS Statistics 22 software were applied. EFA was used for testing the validity of all the constructs in the questionnaire. This method is employed to describe variability among observed variables in terms of a smaller number of unobserved variables called factors (constructs). In other words, by reducing the large number of items, the seven latent constructs underlying LMS-TPACK could be identified. Individual items of one construct had to load (or contribute) significantly onto that specific construct as in the questionnaire. Item analysis was performed for testing the reliability of each construct in the LMS-TPACK questionnaire. Cronbach’s alpha coefficients were calculated for each of the constructs as well as overall instrument reliability. The goal here was to establish whether the item related to with the particular construct for which it was intended. Items that failed to show significant relationships with the intended construct were then removed so as to attain a higher reliability coefficient. The following guiding procedures for validity and reliability testing as recommended by Williams, Onsman and Brown (2010) and Field (2013) was applied. See Figure 10. Each of these tests and their roles and functions will be elaborated upon in the next chapter (Chapter 5).
68
Figure 10: Guiding procedures for validity and reliability testing (adapted from Williams et al., 2010, and Field, 2013)
4.10 SUMMARY
The research approach and survey design were summarised in this chapter. The unit of analysis, the population, sample size and non-probability sampling method that were used to meet the research objectives, including the permissions needed for the study, were described. Since it was decided to make use of a web-based self-report questionnaire for quantitative data collection, the rationale for and scale development procedures to be followed in the construction of the test were presented. In conclusion, the data analysis and statistical techniques employed to test for instrument validity and reliability were described. Subsequently, the research results and findings will be described in Chapter 5.
69
CHAPTER FIVE
RESULTS AND FINDINGS
5.1 INTRODUCTION
This chapter presents the results and findings from the statistical analyses conducted during the development of the LMS-TPACK survey. Expert reviews were carried out prior to the survey pre-test. Descriptive statistics were used to describe the demographic variables and better understand the sample population. DeVellis (2003) sees factor analysis as an essential tool in scale development (p. 137). He states that a key function of factor analysis is to help the researcher determine the number of factors or constructs (latent variables) that underlie a set of items so that statistical techniques such as Cronbach’s alpha can be computed correctly. Moreover, factor analysis is able to provide insight into the nature of the latent variables underlying the set of items. Both EFA (principal axis factoring) and reliability estimates of the LMS-TPACK survey were performed to establish a basis for instrument validity and reliability.