presenting its purpose.
However, due to the conceptual nature of the framework, further external verification from the real world was needed to ensure its validity. This is the final step of Phase I, in preparation for the questionnaire and interviews to collect data from professionals in the field. Overall, the outcome of Phase I is a conceptual framework that is subject to subsequent revision or expansion following data collection and analysis.
3.6.2 Phase II – Development of Survey Instrument
Phase II, illustrated in Figure 3.8, begins with the experts’ pre-test of the initial draft of the survey instrument, inorder to gather initial feedback on the list of vulnerability and capability factors identified in Phase I, and to identify areas in the survey that need further development or refinement. It is also important to pre-test the survey to ensure that researchers and respondents interpret the survey in the same way and that it will function as a valid and reliable assessment tool (Converse and Presser, 1986).
The results from the pre-test of the survey instrument and the pilot study to assess reliability as shown in Figure 3.8 are further discussed below.
Pre-test Data Analysis
The following three steps were undertaken during the pre-test:
First, the initial questionnaire draft, which includes the identified variables and research constructs from the literature review, were presented to the same five experts as in Phase I; they were requested to make initial recommendations on the layout of the questionnaire, clarity of the contents and instructions (presented as the second step in Figure 3.8). This procedure allows the researcher to test the face validity and content validity of the questionnaire (as discussed in Section 3.5.3). The necessary changes were made following the experts’ feedback, as indicated in Table 3.6 below. Some irrelevant factors were deleted, and statements reworded to improve clarity.
Table 3.6: Feedback on questions and changes addressed through the pre-test Focus Description Feedback and recommended changes
C
on
te
n
t Is the content of the questions appropriate to the research?
Are the questions relevant?
• All experts agreed that the content of the questions were appropriate and relevant for the research area.
In str u cti on s an d c ove r p
age Are the instructions of the questionnaire clear? • The cover page which includes the research problem and the research aim and objectives was helpful and easy to understand.
• A suggestion was made to highlight the purpose of the questionnaire in the cover page as well.
• Some of the words in the instructions of the sections need to be underlined and bold to attract respondents’ attention.
• Instructions in Section 5 were not self-explanatory and need to be further clarified by the researcher. The instructions were therefore reworded accordingly to avoid confusion.
Q u es ti on s
Are all the wordings used in the questions and vulnerability and capability statements clear or ambiguous?
• Generally, all the wordings in the questions are clear. Some experts suggested using simple terms that all managers and operatives can understand. Some terms were therefore reworded accordingly.
• It was suggested to allow the respondents to tick more than one option for question number 4 in Section 1 as they might be involved in more than one project phase in construction.
• The term suppliers in questions in Sections 3 and 4 needs to be defined as ‘a separate firm that provides either products or services to the respondent’s firm’. A footnote was therefore added to define what suppliers meant in this context.
• To include both products/services in the vulnerability and capability statements as some respondents might be offering services instead of products in the supply chain.
Layou
t How appropriate is the layout or order of the questionnaire?
• All experts agreed that the layout and order of the sections are very good and easy to read.
Le
n
gth
How acceptable is the length of the survey to the respondents?
• Once duplicated questions are deleted, and factors are refined to shorten the questionnaire, the experts find that the duration of approximately 20 to 30 minutes was viewed as a reasonable length to complete the questionnaire.
Another key recommendation was on the classification of the vulnerability factors. Although the experts found that the pre-established scales and constructs of the capability factors in Pettit et al.’s (2013) study are useful and can be directly adopted in the survey to assess the resilience of the public sector supply chain, for the vulnerability factors, the experts argued that it would be valuable to the public sector supply chain to assess the vulnerability based on where the vulnerability arises (i.e. from within the organisation, from the supply chain, or from external factors beyond the control of the firm and its supply chain). Hence, in line with the theoretical concept and the experts’ feedback, the vulnerability factors were regrouped and classified as discussed below.
Secondly, as it was suggested that the vulnerability factors be regrouped; the content validity of the vulnerability factors was assessed by the five experts to ensure the definition of the vulnerability constructs was clear. To calculate the content validity ratio (CVR) discussed in
or irrelevant in assessing the public sector supply chain resilience. This help the researcher to further refine the vulnerability factors based in their importance.The experts were also asked to rate on a four-point Likert scale (1=not relevant, 2=somewhat relevant, 3=quite relevant and 4=very relevant) each item based on its relevance and clarity in measuring the construct it was supposed to measure, for the purpose of computing the item-content validity index (I-CVI). Using Equations 1 and 2 (Section 3.5.3), the CVR and I-CVI for the vulnerability factors were calculated accordingly, as shown in Table 3.7.
Looking at the CVR values in Table 3.7, 11 out of the 41 items fell below the 0.99 threshold, as highlighted in red. This includes items such as V3.3-operating in extreme or hazardous conditions, V3.4-loss of key personnel, V4.1-large number of members in supply chain, V4.7- limited distribution capacity, and V5.2-suppliers have limited capacity in dealing with demand changes. Despite obtaining a CVR value of 0.60, it is worth noting here that four out of the five experts selected these items as essential or important, but not essential, as seen through the breakdown of the results in Table E1 of Appendix E. Although there is 80% agreement on the essentiality of these items, Lawshe’s (1975) formula and stringent minimum CVR value of 0.99 for a small number of experts seems to require all experts to agree that those items are essential or important, but not essential be retained in the study. Hence, in this case, the researcher took into account the I-CVI value to determine whether to retain or discard the items with a CVR value of 0.60. It can be seen from Table 3.7 that these items obtained a high I-CVI value of 0.80, which is above the minimum of 0.78. Thus it was decided that these items be retained in the study.
The lowest CVR value of 0.20 was identified in items under the technology disruptions and environmental factors constructs, whereby two of the five experts perceived these items (V6.2- unforeseen technology failures and V8.2-health pandemic/spread of disease affecting employees) to be irrelevant to the study. The I-CVI values of both of these items were also below the 0.78 threshold. However, looking at the total average of the scale content validity index (S-CVI), the environmental factors construct is still within the acceptable S-CVI value of 0.80; thus the researcher decided to retain the item in the construct for further analysis. However, the technology disruptions construct fell below the S-CVI’s 0.80 threshold, with the S-CVI value of 0.70. This construct was therefore considered to have low content validity and was eliminated. Overall, considering that more than 70% of the 41 items obtained a CVR and I-CVI value of 1.00, it can be considered that the rest of the constructs have a good level of
Thirdly, Q-sorting was conducted with three researchers from Malaysia and three key industry players to ensure that the relevant vulnerability factors fall in the right constructs (as discussed in Section 3.5.3). The correct classification percentage was calculated by identifying the frequency of experts that selected the correct construct for each item. Based on the computed Q-sorting results in Table 3.8, seven items obtained 100% correct classification (value shown as 1.00 in the table), six items were correctly classified at a rate of 83%, 14 items received a correct classification rate of 67%, and 11 items were correctly classified by half (50%) of the respondents. The high percentage of correct classification shows that these 38 items exhibit consistent meaning across the panel of experts, thus confirming their adequacy in capturing the pre-specified vulnerability constructs.
Table 3.8: Results of Q-sorting analysis
Vulnerability Factors Percent
Strategic Vulnerability
V1.1 Degree of outsourcing to different suppliers 0.67 V1.2 Reliance upon specialty sources in delivering products/services 0.50
V1.3 Threat by competitive innovations 0.50
V1.4 Concentration of suppliers/operation facilities at the same area 0.67 V1.5 Complexity of services/production operations 0.50
Management Vulnerability
V2.1 Inadequate management oversight 1.00
V2.2 Late information and decision making 1.00
V2.3 Visibility of errors or deficiencies to stakeholders 0.33
V2.4 Reliance upon information flow in operations 0.50
V2.5 Budget overruns/Unplanned expenses 0.67
Personnel Vulnerability
V3.1 Shortage of skilled workers 0.67
V3.2 Labor disputes or strikes 0.67
V3.3 Operating in extreme or hazardous conditions 0.17
V3.4 Loss of key personnel 0.67
Process Vulnerability
V4.1 Large number of members in supply chain 0.33
V4.2 Unpredictability of demand by client 0.67
V4.3 Scarce or limited raw material availability 0.50 V4.4 Poor availability of utilities (electrical power, water, sewer) for
production
0.50 V4.5 The use of failure-prone equipment/product 0.67
V4.6 Limited production capacity 0.83
V4.7 Limited distribution capacity 0.67
V4.8 Product quality problem 0.67
Supplier or Customer Disruptions
V5.1 Suppliers face frequent disruptions 0.83
V5.2 Suppliers have limited capacity in dealing with demand changes 0.67
V5.3 Loss of key supplier 0.67
V5.4 Customer face frequent disruptions 0.83
Technology Disruptions
V6.1 Technology changes in the industry 1.00
V6.2 Unforeseen technology failures 1.00
Political or Legal Pressures
V7.1 Exposure to political disruptions 0.83
V7.2 Political/Regulatory changes affecting operation 0.83
Environmental Factors
V8.1 Exposure to natural disasters 1.00
V8.2 Health pandemic/spread of disease affecting employees 0.50
V8.3 Pressure from public opinion/reputation 0.50
Physical Damage Disruptions
V9.1 Products regularly stolen or vandalised 0.50
V9.2 Accidents during operation (i.e. fire, workers accident) 0.50
V9.3 Terrorism & sabotage 0.50
Market Pressures
V10.1 Fluctuations in prices 1.00
V10.2 Price pressures from competition 1.00
Liquidity or Credit Vulnerability
V11.1 Finance policies & procedures affecting management of money & assets 0.67
V11.2 Lack of financial resources 0.83
However, three items in Table 3.8 (highlighted in red) obtained below the previously selected minimum 50% correct classification rate: V2.3-visibility of errors or deficiencies to stakeholders and V4.1-large number of members in supply chain, for which only two out of the six experts (0.33 percent) classified the items to the pre-specified construct; for V3.3- operating in extreme or hazardous conditions only one expert classified it correctly (0.17 percent). It is worth noting here that these items had higher percentage values, but for a construct other than the one posited by the researcher. Hence, the decision was made to reclassify them according to the construct proposed by the majority of experts: visibility of errors or deficiencies to stakeholders was a strategic vulnerability rather than a management vulnerability; large number of members in supply chain was considered as supplier or customer disruptions instead of process vulnerability; and operating in extreme or hazardous conditions was perceived to be more suitable under the environmental factors construct that are beyond
Pilot Study to Assess Internal Consistency of Survey Instrument
After the pre-testing, the survey was ready for the reliability test, as depicted in Figure 3.8. The pilot study was conducted with 20 respondents (10 respondents from the public organisations and 10 respondents from the private organisations representing the public organisations’ supply chain members). This is a useful process to test the survey from a methodological standpoint, allowing the researcher to assess the validity and reliability of the survey instrument and to predict any difficulties that may arise during the data analysis of the complete sample (N=105), which might otherwise have gone unnoticed (Litwin, 1995).
Cronbach’s alpha and corrected item-total correlations were computed in SPSS, based on the data collected (N=20) to test the internal consistency and uni-dimensionality of the vulnerability (V1 to V11) and capability (C1 to C12) constructs. Table 3.9 presents the number of items and the Cronbach’s alpha value for each construct. The table highlights three of the 23 constructs that fell below the Cronbach’s alpha limit of 0.60 for the pilot study. The construct V6-technology disruptions obtained the lowest alpha value of 0.333. The validity of this construct was also an issue based on the low S-CVI value computed previously (Table 3.7). Plus, the item-total correlation of the items in the construct was also below the 0.30 threshold, with a value of 0.20 (see Table F1 in Appendix F). The poor correlation between these items suggests that the items are too heterogeneous to form a construct and are therefore not reliable to measure the construct technology disruptions. The researcher therefore decided to remove this construct in the main study, to ensure that the survey instrument remain valid and reliable.
The construct V9-physical damage disruptions also fell slightly below the 0.60 threshold, with an alpha value of 0.586. The item-total correlation of the items in the construct, however, was encouraging, ranging from 0.32 to 0.51 (Table F1 in Appendix F). This indicates that each item has a good correlation with the domain. The low alpha value of this construct might therefore be due to the low number of questions in the construct or the small number of respondents assessed in the pilot study. Hence it was decided that the construct remain in the study for further analysis as it could possibly bring significant managerial insights to the study.
Table 3.9: Internal reliability of vulnerability and capability factors Vulnerability Factors V1 V2 V3 V4 V5 V6 Number of Items 6 4 3 8 5 2 Cronbach's Alpha (Pilot Sample N=20) 0.723 0.615 0.814 0.916 0.852 0.333 Cronbach's Alpha (Main Sample N=105) 0.520a 0.649 0.586a 0.872 0.819 0.734 Vulnerability Factors V7 V8 V9 V10 V11 Number of Items 2 4 3 2 2 Cronbach's Alpha (Pilot Sample N=20) 0.695 0.842 0.586 0.750 0.726 Cronbach's Alpha (Main Sample N=105) 0.683 0.645 0.785 0.737 0.699 Capability Factors C1 C2 C3 C4 C5 C6 Number of Items 5 3 4 3 4 5 Cronbach's Alpha (Pilot Sample N=20) 0.929 0.537 0.917 0.788 0.796 0.840 Cronbach's Alpha (Main Sample N=105) 0.829 0.715 0.882 0.816 0.825 0.829 Capability Factors C7 C8 C9 C10 C11 C12 Number of Items 3 4 3 4 3 3 Cronbach's Alpha (Pilot Sample N=20) 0.884 0.735 0.793 0.839 0.737 0.824 Cronbach's Alpha (Main Sample N=105) 0.861 0.747 0.633 0.770 0.777 0.804
a The mean inter-item correlation for V1 is 0.2 and V3 is 0.3 indicating that each item has good correlation
with the domain (Briggs and Cheek, 1986)
Another factor that remains below the 0.60 threshold is the capability factor, C2-capacity, with an alpha value of 0.537. This construct was also an issue in Pettit’s (2013) study, whereby a lower Cronbach’s alpha value of 0.515 was obtained for the same construct. However, despite the lack of uni-dimensionality, Pettit (2008) had argued that the classification maintains a logical structure that allows for the computation of an overall capacity capability score. He believes that the construct represents multiple independent measures of capacity at the production locations, including internal assets such as inventory, equipment, labour,and utilities (Pettit, 2008). Furthermore, it can be observed through Table F2 in Appendix F that the construct’s low Cronbach’s alpha is due to the poor item-total correlation of the sub-factor, reserve capacity (materials, assets, labour, inventory). This item may not correlate well with
capacity. Hence it was suggested by Pettit (2008) to further explore this factor in detail in the subsequent phase of the study to obtain a comprehensive result. The construct therefore remains in this study for further analysis. Overall, the remaining factors’ reliability estimates ranged from 0.615 to 0.929, proving that the survey instrument was reliable and consistent for the total sample (N=105).
3.6.3 Phase III – Development of Resilience Response Framework
Once the validity and reliability of the survey instrument have been determined, data analysis of the final survey in Phase III of the study begins (see Figure 3.9). This includes an assessment of the critical areas of vulnerability and the current level of capability of the public sector supply chain.
Figure 3.9: Research process Phase III – Development of Resilience Response Framework
The critical vulnerabilities and capabilities identified from the survey were used to develop the questions for the subsequent semi-structured interviews with 12 targeted respondents (Section 3.5.2.1). By highlighting the identified critical areas in the interviews, the researcher was able