The post-meeting questionnaires were pilot-tested and iteratively refined over the course of three months, with the same 60 part-time MBA students participating in these pilot-tests as for the pilot-tests of the experimental task. The questionnaires were first reviewed by three faculty members who had experience in designing questionnaires. The questionnaires were then presented to the group of 39 pilot-test participants. The purpose of the questionnaire was explained and the participants were asked to complete the questionnaire according to the instructions and to comment on the content and the presentation of the questionnaires. The key objective of the pilot test was to ensure the content validity and the presentation of the questionnaires. As a result of this pilot-test, several sentences in the questions were re-worded to improve their clarity and the wording of the instructions was modified. The layouts of the questionnaires were also modified. Two questions in Questionnaire B were deleted because more than half of the participants thought that the questions were not relevant. On average, the participants were able to complete the questionnaire in 10 minutes. The modified questionnaires were then presented to the second group of 21 MBA students. Like the first group, the purpose of the questionnaires was explained and they were also asked to complete the questionnaires according to the instructions on the questionnaires. They were then asked to provide comments on the content and presentation of the questionnaires. Overall, members of this pilot group expressed
satisfaction with the content validity and the presentation of the post-meeting questionnaires.
4.9 Chapter Summary
This chapter described the experimental method used for testing the research hypotheses developed in Chapter 3. The research design was an experiment with a 2x3 factorial between-subjects design with two independent variables: computer support (GDSS support and non-GDSS support) and decision task structure (additive, disjunctive and conjunctive). The dependent variables were objective decision performance, group communication interaction and perceived decision satisfaction. The chapter described the subjects, the decision task, the manipulations, the procedures and the dependent variables of the experiment. The experimental manipulations and procedure were carefully pilot tested with part-time MBA students. Materials used in the experiment were also pilot tested to ensure clarity. The results of the experiment are described in Chapter 5.
Chapter 5
Analysis of Experimental Results
5.1 Introduction
This chapter presents the results of the experiment described in Chapter 4. The main focus of this chapter is on the statistical tests of the hypotheses developed in Chapter 3. The discussion, implications and conclusions drawn from these results are presented in Chapter 6. Prior to conducting the tests, the chapter deals with testing for the reliability and the validity of the research instruments, and ensuring that the data meet the requirements of the statistical tests and that the manipulation of treatment is successful.
Twenty one variables were tested. Of these, two variables involved the quality of decision and the time taken to reach a decision. Eight variables involved the coding of content in the patterns of group communication. They included the numbers of decision proposal made; the numbers of supporting comments; the numbers of clarification comments made during decision making; the numbers of critical arguments; the numbers of queries; the numbers of procedural related comments; the numbers of other unrelated comments and the number of total comments made during the whole decision process. One variable also measured the number of comments related to system support in GDSS supported groups. Eight variables involved data collected from post-experimental questionnaires on groups’ perceptions of satisfaction. They included the perceived satisfaction with the decision outcome; the perceived satisfaction with the decision making process; the perceived participation in the group decision process; the perceived conflict behaviour in the group decision process; the perceived confidence in the group decision; the perceived depth of evaluation in the decision process; the perceived willingness to remain in the group and the perceived overall satisfaction with the decision process. Two variables evaluated the usefulness of GDSS to the task and compared the effectiveness of GDSS with face-to-face meeting.
Note should be taken of the fact that eighteen of these variables applied to all six treatments: non-GDSS with additive task; non-GDSS with disjunctive task; non- GDSS with conjunctive task; GDSS with additive task; GDSS with disjunctive task and GDSS with conjunctive task. Three variables, system related comments, GDSS usefulness for task and GDSS effectiveness against face-to-face meeting, were relevant only to the three GDSS supported treatment groups.
The first step in the data analysis involved testing questionnaires, used to gather perceptual data, for their construct validity by assessing the fit between the planned constructs and the questions intended to measure these constructs (Cook and Campbell, 1979). In addition, the reliability of each construct was measured to make sure that each had a sufficient reliability level to register the true changes due to treatment differences. Factor analysis was used to assess the construct validity of the questionnaires and the reliability of the construct was assessed using the standard Cronbach coefficient alpha.
The second step in the process of data analysis was to ensure that the requirements of the statistical tests were satisfied. The statistical test used in testing the hypotheses was ANOVA. Before subjecting the data to the test, the assumptions of this statistical test were verified. Hartley’s test (Neter, et al., 1985) was used to test for homogeneity of variance of the dependent variables and the Lillefors’ test (Conover, 1971) was used to test the normality of the data.
The third step in the data analysis dealt with the manipulation check of the experimental treatments. The task structure and the GDSS support manipulations were assessed. The single sample chi-square was used to demonstrate the good fit between the experimental conditions and the subjects’ perception of the experimental environments.
The final step led to the statistical tests of the hypotheses using ANOVA. Two-way ANOVA was applied to each of the eighteen variables across six experimental treatments. If the main effects of the two-way ANOVA were significant, Newman-
Keuls post-hoc multiple comparison analysis (Kirk, 1982) was used to investigate the difference in means of individual groups. If the interaction effect was significant in the two-way ANOVA, simple effect analysis was carried out to assess the effect on each experimental treatment. Three variables were only relevant to the three GDSS support treatment groups. A one-way ANOVA was applied to each of these variables and Newman-Keuls post-hoc multiple comparison analysis (Kirk, 1982) was used to assess the difference in means of individual groups if the result of the one-way ANOVA was significant.
The remainder of the chapter is organised into seven principal parts. Section 5.2 reports the results of construct validity and Section 5.3 reports the reliability of the questionnaires used to gather the perceptual satisfaction variables. Section 5.4 reports on analysing the data to ensure that the requirements of the statistical tests were met. This section also describes how data was transformed in those cases where requirements were violated. Section 5.5 covers the manipulation check of the experimental treatments. Section 5.6 provides the results of conducting the statistical tests involved in testing the hypotheses and Section 5.7 provides a summary of the chapter.
5.2 Construct Validity
All of the post-experimental instruments were self-reported questionnaires and were subjected to both validity and reliability tests to ensure that the questions were good indicators of the construct intended, and had high internal consistency. Three post- experimental instruments were used in this study. The type and nature of these instruments are shown in Table 5.1.
Two other dependent variables were extracted directly from the experiment. They were decision quality and decision time. Another eight dependent variables were
Table 5.1 Post-experimental Instruments and the Corresponding Constructs Measures
Instruments Constructs
1. Questionnaire A: self-reported questionnaire; a modified version of Gouran, Brown and Henry (1978) and Geen and Taber (1980).
2. Questionnaire B: self-reported questionnaire; a modified version of Green and Taber (1980).
2. Questionnaire C: self-reported questionnaire for GDSS supported groups.
1. Perceived satisfaction with the decision outcome.
2. Perceived satisfaction with the group decision making process.
1. Perceived participation in the group decision process.
2. Perceived conflict behaviour in the group decision process.
3. Perceived confidence in the group decision.
4. Perceived depth of evaluation in the group decision.
5. Perceived willingness to remain in the group.
6. Perceived overall satisfaction with the decision making exercise.
1. Perceived usefulness of GDSS for the task.
2. Perceived effectiveness of GDSS compared to face-to-face meeting.
from the content coding of the comments made in the groups. These variables were objectively measured in the experiment and were not subjected to the construct validity and reliability tests.
Factor analysis was performed separately on the subjects' responses to questions in Questionnaire A, Questionnaire B and Questionnaire C, to test the construct validity of the dependent variables from these self-reported questionnaires, Principal component analysis employing a varimax rotation was carried out to determine the number of factors responsible for systematic variation in the data. All factors with eigenvalues greater than one were selected (Johnson and Wichem, 1982). As the post- experimental questionnaires were taken from the instrument developed by Gouran, Brown and Henry (1978) and Green and Taber (1980), the factors obtained in this analysis were compared with those obtained by the original developers of the instruments.