This study adapted the iterative process of a construct modelling approach (Wilson 2005) in the process of instrument development and validation. These elements were integrated into the three phases of the study. The construct modelling approach consists of four building blocks that were used iteratively but not necessarily in
sequence throughout the three study phases, as suggested in Table 4. The table shows these four building blocks were designed to integrate into the three phases of this study. These four building blocks are now spelled out in greater details.
3.3.1.1 Building block I - Construct mapping
According to Graziano and Raulin (2000), a construct “is an idea constructed by the researcher to explain events observed in a particular situation. Once formulated, constructs are used as if they are true to predict relationships between variables in situations that had not previously been observed” (p.419). This first building block is used to help the instrument developer to focus on the essential latent factors of what is to be measured, mapped on the construct map. The construct map visualised a consistent definition of the latent variable and ordered the item responses into a series of levels (Wilson, 2005). It can be considered the theoretical framework that guides the instrument’s design. The construct map of the study can be referred to in Figure 6.
The construct mapping was the initial step to determine the rationale for the chosen factors representing the construct of the SETSIS and the significance of the study. A rationale content analysis of the literature text and documents of the BoTP
curriculum were used in order to establish the construct mapping and to interpret the factors selected in measuring the construct.
The construct map describes the continuum of the construct from high to low self- efficacy in teaching using science inquiry skills. It was then used to guide the design and development of the instrument. The instrument was developed to describe and define the continuum with qualitatively distinct levels, so it will able to identify the levels of respondents on the construct. The construct map is the important first step in establishing the validity and reliability of an instrument (Brown and Wilson, 2011)
Table 4 Summary of research design of the study
Study Phase Building block Construct Modelling Activity Method Research Question(s) Literature review Variable interpretation Items pooling Experts review of the items Conducting interview for technical qualities of SETSPS. Focus group interview Establishing the responses to the SETSIS Descriptive and inferentian analysis Evaluate the SETSIS outcome with concurrent validation using i) Knowledge Test Assessment (TISP) ii) Teaching Practise assessment Correlation and Regrassion RQ5 Validating the factors contribution with the scales
interpretation
Factorial Analysis Pre-Test using paper & pencil
survey
Establising item mapping and administering the main research
for the SETSIS using
Main research using paper & pencil survey Phase 3 Validating and Evaluating the SETSIS IV Measurement model III Outcome space II Item Design RQ1 RQ2 & RQ3 Phase 1 Constructing the SETSIS Phase 2 Developing the SETSIS
III Outcome space II Item design
Internal-structure validity for content relevance and representativeness of the scale and establising items mapping Rasch Model Developing measurement model and relate
back to the theoretical construct of construct map for construct validity of
SETSPS.
Rasch
Model RQ4 I Construct Mapping Content
Analysis Fuzzy Delphi Method ii)Teaching Practice Assessment SETSIS
3.3.1.2 Building block II - Item design
Item design is a construction of tasks and contexts that assess the construct. The tasks and contexts were constructed in terms of statements that represent
observation, which can stimulate responses about the construct that are generically called items. The items were designed according to three criteria references: high item, mean item and easy item, which provide the interpretation of the item level within the construct. Each level describes characteristics of items that form groups along the continuum of the construct. Further details are described in section 4.4.1. In this study, item design was included in Phase 1 and Phase 2 of the study. These include a series of decisions to decide items that adequately represented the
construct. It started with the item pool, in which the specific items were sampled for the SETSIS and sent for translation process (see the explanation below).
All the items, in both languages, were reviewed by the experts for the purpose of assessing the content validity of the items. The Fuzzy Delphi Method (FDM) was used for analysing the experts’ judgement and obtained consensus for the reviewed items. The consensus judgement was then cross-referenced with the relevance and the association of the construct framework.
Then, the resulting items were administrated in the pre-test survey. Using a paper- and-pencil method, this survey aims to get pre-responses from the PSTs in the instrument. Feedback was sought in a small group interview, which consisted of six participants from the pre-test, in relation to the clarity of the items and the overall accessibility of the instruments.
The back-translation technique (Brislin 1970) was used to translate all the sample items of the SETSIS from the source language (English) into the target language (Malay). First, a certified translator with a science education background was chosen to translate the items in forward translation. The two versions were compared and some iterations were made in the target language based on suggestions received from the content experts panel (see 3.6.1) and pre-test survey (see 3.6.2). Then, it was given to another translator for back translation (i.e. back into the source
language). The two source language versions were compared and evaluated. There were some discrepancies in the words used, but they had semantic equivalence. The information gathered in Phase 1 was used to sort the items and design the instrument for the main research survey in Phase 2.
3.3.1.3 Building block III - Outcome space
The outcome space was used to study scores and category responses for the measure. Scores from the gathered items’ responses were categorized and the scores were the indicators of the construct, which lead to the outcome space (Wilson, 2005). The outcome space corresponds to the development of the scoring scale regarding the observed responses, where the individuals are on the construct map based on the context of the measure. The lower level of the construct should correspond with lower item scores and vice versa. The categories that define the outcome space are qualitatively distinct, and this can be validated with the
quantitative evidence.
In this study, the outcome space was initially used during constructing response options in Phase 1. The outcome space was defined in five, qualitatively distinct categories related to confidence levels of self-efficacy in teaching using science inquiry skills (see explanation in section 4.4.2). The categories defined should be represented in the quantitative evidence. Analysis of Rasch model on pre-test data examined unpredictable responses that did not fit the model and at the same time provided pre-information on the quantitative value (i.e. reliability and separation) of responses and the level of response categories (i.e. scoring of item-response categories). This will be further elaborated in section 4.6.2.
Outcome space was also used during Phase 2. In this phase, the process related the mean score from the data observed to the response categories created in Phase 1. The quantitative evidence related the score to the categories proposed and whether the interpretation of the responses could be generalised into the population. Finally, factor analysis (Field, 2013) was used to examine the representation of the items in the construct through the responses’ variances. The factor analysis was able to identify key features of the responses in the items. The result of grouping items was related to the construct framework to confirm the representative of the measure. 3.3.1.4 Building block IV - Measurement model
The final phase used the process of measurement model in order to validate and evaluate the SETSIS. Measurement model served as a process to relate scored outcomes and to compare the outcome space to the original construct (Wilson, 2005). The measurement model helps to model individual responses which are then related back into the construct map. Further, it was able to evaluate the score which interpreted the construct and guided the use of the score in practical application. For this study, in order to validate the SETSIS as a measurement model, the method used should first be able to fit response data statistically and secondly able to
validate the characteristics of the model according to the measurement framework (Wilson, 2003). The Rasch model was chosen for the formal measurement model used to validate the model of the SETSIS because it is able to explain response data and provide a probability model able to interpret the continuum between the items and the outcomes of the construct. A scaling approach based on the Rasch model is sensitive to the issue of a broader conception of construct validity (i.e.
unidimensionality). The data should be able to validate the model through the construct representative with relevant variance (Messick, 1994). If the data meets the requirements of the model and the model’s implications, it contributes to evidence for the construct validity of the measure.
In the final phase of the study, the responses from the main research were used within the contributed items from the factor analysis result. The rating scale model of Rasch was used to validate the data and further characterise the items and
responses within the model structure. Next, the responses option was also validated, and the result was used to interpret the probability scored into the estimated true score for practicality purpose. Lastly, the correlation and regression analyses were used to evaluate the scores from the item response in conjunction with the associate knowledge test and practice assessment.