Reliability refers to the degree of consistency or accuracy with which an instrument measures the attribute it is designed to measure (Drost, 2011; Noble & Smith, 2015). This means that if a study and its results are reliable, the same results would be obtained if the study were to be replicated by other researchers using the same method. However, it should be noted that while reliability in quantitative research refers to exact replicability of the processes and the same results, the essence of reliability for qualitative research lies with consistency (Babbie, 2012; Noble & Smith, 2015). In sum,
137
reliability is consistency of measurement or stability of measurement over a variety of conditions in which basically the same results should be obtained. In an effort to ascertain the validity and reliability of the research instruments before engaging them in the main study, the researcher conducted a pilot study. Reliability can be measured in different procedures which include: test-retest reliability, inter-rater reliability, and internal consistency reliability, among others.
3.7.2.1 Test-retest reliability
Test-retest reliability focuses on the stability of measurement and is determined by administering a test at two different points in time to the same individuals and determining the correlation or strength of association of the two sets of scores (Droessaert& van-Dinther, 2015). There are two necessary assumptions in test-retest reliability. The first is that the true score does not change between administrations. The second is that the time period between administrations is long enough to prevent learning, carry-over effects, or recall. In short, test-retest reliability is concerned with the repeatability of observations made on, or by individuals(Takaki, Taniguchi &Fujii, 2014).In this method, reliability can either be quantified as the canonical correlation between the two sets of repeated questions or be determined through considering repeated responses within a participant as longitudinal data and compute the intra-class correlation coefficient (ICC) based on a mixed-effects linear model (Vaz, Parsons, Passmore, Andreau& Falkner, 2013). In the current study, neither of these reliability tests was employed. However, Droessart and van-Dinther (2015) argue that there are generally no standard criteria for the interpretation of either canonical correlation or ICC, but a general guideline suggests that values of above 0.75 indicate good to excellent reliability and values below 0.75 correspond to poor to moderate reliability.
3.7.2.2 Inter-rater reliability
Inter-rater reliability is the measurement of the consistency between evaluators in the ordering or relative standing of performance ratings, regardless of the absolute value of each evaluator’s rating (Graham, Milanowski& Miller, 2012). In research, this type of reliability is meant to show the magnitude of agreement achieved between data coders (Hallgren, 2012). A significant number of authors concur that the most popular method used for testing inter-rater reliability is correlation. Correlation tests the relationship
138
between the scores of two raters, which can be achieved by reporting Pearson coefficient, and the relationship among the scores of all the raters can be tested using Cronbach’s alpha (Liao, Hunt & Chen, 2010).
3.7.2.3 Internal consistency reliability
This is a measure of how well the items on the instrument measure the same construct or idea. The internal consistency of an instrument indicates whether items on a test that are intended to measure the same construct produce consistent scores or are correlated (Tang, Cui & Babenko, 2014). In this study, the Cronbach’s alpha coefficient was calculated to measure internal consistency reliability of question items of the different sections in the two questionnaires (teachers’ and learners’ respectively).
The Cronbach’s alpha is reliability test score showing the ratio of true score variance and observed score variance (Ritter, 2010). It is usually expressed as a number between 0 and 1, and the acceptable values of alpha range from 0.70 to 0.95 (Tavakol & Dennick, 2011). Khalid,Khan and Mohd-Zain (2012) present detailed categories of Cronbach’s alpha reliability values as rules of the thumb, as 0.9 = excellent; 0.8 = good; 0.7 = acceptable; 0.6 = questionable; and 0.5 = poor or unacceptable. The Cronbach’s alpha is typically used when the researcher has several Likert-type items that are summed or averaged to make a composite score. It is used in such cases to determine whether items are consistent with one another in a test based on total or average point, to determine whether the items measure a hypothetical variable (Bindak, 2013). In short, the coefficient alpha means a prediction of correlation between two samples drawn randomly from the total items.
Cronbach’s alpha is convenient for calculatingbecause it requires only a single measurement given at one time. It is more practical than other reliability coefficients in case of limited time and resources. In the current study, there were five themes to examine teachers’ role in the promotion of CFS environments and Cronbach’s alpha was calculated using the SPSS for each theme or section of the questionnaire for teachers and learners. Pilot study responses were used to measure the internal consistency based on number of items per theme as illustrated in Table 3.6
139
Table 3-6: Cronbach’s alpha reliability of internal consistency for Likert-scale sections
SECTION Themes Type of
respondents
Cronbach’s Alpha
Number of items
B Teachers’ understanding of the CFS
concept
Teachers 0.7 23
C School-based support for teachers to
promote CFS environments
Teachers 0.8 22
D Teachers’ strategies to promote CFS
environments
Teachers 0.9 15
Learners 0.7 15
E Challenges faced by teachers in the
promotion of CFS environments
Teachers 0.9 19
Learners 0.7 10
F Implications of CFS for teacher
professional development
Teachers 0.8 16
Sections D and E of teachers’ questionnaire had excellent Cronbach’s alpha reliability value of 0.9 while sections C and F had a good reliability value of 0.8. Only section B had an acceptable reliability value of 0.7. The values ensured the researcher valid and credible data collection. For the learners’ questionnaire, the calculated reliability value for the two sections was an acceptable 0.7, which again ensured collection of reliable data.
3.7.2.4 Pre-testing research instruments
In order to ascertain the validity and reliability of the research instruments, a pre-test of the research instruments was undertaken before embarking on full scale data collection of the main study. A pilot study refers to a small scale test of the methods and procedures to be used on large scale (Hazzi & Maldaon, 2015). The main importance of a pilot study lies in improving the quality and the efficiency of the main study. In short, a pilot study can be used as a small version of a full-scale study or trial run in preparation for the main study. Pilot study is essentially a feasibility study of the tools and the process of implementing them (Kumur, 2011). In the current study, the pilot study was conducted to afford the researcher an opportunity to revisit the research instruments with the intention of ensuring their validity and reliability. The exercise also assisted in
140
improving the clarity and precision of the question items, numbering of the question items per section, noting of any omissions on the instruments, and identifying any logistical challenges that could negatively affect the main study.
The pilot study was conducted with 15 learners, 20 teachers, a school head, and 1 G & C focal person from a local government secondary school, who did not participate in the main study. Both teachers and learners managed to complete the questionnaires within the stipulated 30 and 40 minutes respectively. Most questions seemed clear and were easily understood, but it emerged that the CFS concept required some unpacking for learners in order for them to give relevant data which addressed the demands of the research questions. As a result, a preamble of the CFS concept was given on the questionnaire just after the instructions. In the actual study, teacher interviewees were asked to define the CFS concept as part of assessing their understanding of the concept. Besides having a preamble in the learners’ questionnaire, some question items in the teachers’ questionnaire were adjusted, rephrased or completely dropped. Improvements made on the research instruments are shown in Table 3. 7.
Table 3-7: Improvements on data collection instruments after pilot study Type of
instrument
Nature of correction effected
Item before pilot testing
Item improvement after pilot testing
Learners’ questionnaire
Preamble of the CFS concept
No preamble Child-friendly schools (CFS) concept is
an international educational intervention with a mandate to promote school curriculum implementation processes; conditions; policies; rules and regulations; teacher-pupil and pupil-pupil interactions that serve the best interest of the learners and uphold the rights of the child.
Teachers’ questionnaire
Question relevance
Comment on what you know about any one of the listed national policies
Choose any one of the listed national policies and show how it relates to the CFS concept.
141
questionnaire priority in the training
of CFS principles
ranking of CFS principles: SA; A ;U; D; SD Merging of questions 12 & 13 Q 12 rating of teachers’ knowledge of the CFS concept Q 13 rating the importance of the CFS
Q 11 rate the level of your understanding of each of the six CFS principles on a five point scale
Interview schedule for G & C teachers
Interview method Note-taking Voice recording
Interview schedule for school heads Interview schedule sections alignment Section D interview questions focusing on teachers’ strategies
Section D interview questions focusing on teachers’ challenges in the promotion of CFS environments