Results
The purpose of this study was to examine effects of the Tearoom Service Procedure DVD on students’ learning and their perceptions of the instruction. A quasi-experimental study design was conducted to achieve a study’s goal. Pretests and posttests scores and students’ performance grades were computed and analyzed. Results of this study are presented in the following parts: description of respondents’ demographic characteristics, analysis of tests scores, analysis of students’ performance grades, analysis of perception responses, and analysis of responses about DVD.
Demographic characteristics
One hundred eleven subjects participated in this study during the spring and fall semesters of 2007. Fifty-nine of the participants were from the control group and the remaining 52 subjects constituted the experimental group. Table 1 shows the characteristics of the subjects of the two groups. The majority of the subjects in both groups were female: 43 (73%) in the control, and 41 (79%) in the experimental group. Most of the subjects in both groups, 54 (92%) in control and 46 (89%) in experimental groups, were under 25 years old. Subjects majored in either Hotel, Restaurant, and Institution Management or Dietetics: 23 (39%) or 36 (61%) for the control group, respectively, and 38 (73%) or 14 (27%) for the experimental group, respectively. In the control group, a majority of subjects were juniors (58%) and the remainder were seniors (42%); in the experimental group, a plurality of the students were seniors (50%), followed by juniors (46%), with 2 (4%) unknown. Forty-four percent of students in the control group (26) reported a GPA higher than 3.0, whereas 58% of
Table 1. Demographic profile of the students (n=111) Control group
(n=59)
Experimental group (n=52)
Students who watched DVD (n=44) n % n % n % Gender Female 43 72.9 41 78.8 33 75.0 Male 16 27.1 11 21.2 11 25.0 Age ≤20 11 18.6 16 30.8 14 31.8 21-25 43 72.9 30 57.7 25 56.8 26-30 3 5.1 3 5.8 2 4.5 ≥31 2 3.4 3 5.8 3 6.8 Year Junior 34 57.6 24 46.2 21 47.7 Senior 25 42.4 26 50.0 21 47.7 Other 0 0.0 2 3.8 2 4.5 Major HRIM 23 39.0 38 73.1 33 75.0 Dietetics 36 61.0 14 26.9 11 25.0 GPA 3.51-4.00 12 20.3 17 32.7 16 36.4 3.01-3.50 14 23.7 13 25.0 10 22.7 2.51-3.00 19 32.2 11 21.2 10 22.7 2.01-2.50 12 20.3 10 19.2 7 15.9 1.51-2.00 1 1.7 0 0.0 0 0.0 NA 1 1.7 1 1.9 1 2.3
Work Experience (as table server)
Yes 27 45.8 19 36.5 14 31.8
No 32 54.2 33 63.5 30 68.2
students in the experimental group (30) reported being in this range. Twenty-seven (46%) of the subjects in the control group and 19 (37%) in the experimental group reported they had work experience as table servers. students in the experimental group (30) reported being in this range. Twenty-seven (46%) of the subjects in the control group and 19 (37%) in the experimental group reported they had work experience as table servers.
Among the 52 subjects of the experimental group, 44 (85%) reported they had watched the Tearoom Service Procedure DVD. Table 1 shows the demographic profile of these 44 students.
Data analysis
Test scores
To evaluate students’ knowledge of table setting, taking orders, table service, and beverage preparation and service, 15 multiple-choice items in both pretest and posttest were included. Numbers and percents of correct answers for both control and experimental groups are shown in Table 2. For the control group, the number of students who answered correctly in the posttest increased in only 4 out of the 15 test items, compared to the pretest. The control group had a lower number of correct answers in posttest than pretest in 9 test items. This may suggest that students’ hands-on experiences throughout the semester did not help them to improve their knowledge about Tearoom procedures and, instead of reinforcing information covered in orientation sessions, they made students even more confused about procedures. For the experimental group, Table 2 shows that in 12 out of the 15 test items asked in the questionnaire the number of students who answered correctly in the posttest increased, compared to the pretest. These results are descriptive only, and not the results of t- tests.
Table 2. Frequency of students responding correctly to pretest and posttest knowledge questions
Correct % Correct % Correct % Correct %
1b 39 66.1 37 62.7 46 88.5 48 92.3 2 54 91.5 50 84.7 49 94.2 44 84.6 3 55 93.2 51 86.4 48 92.3 48 92.3 4 30 50.8 23 39.0 30 57.7 28 53.8 5a,b 56 94.9 59 100.0 45 86.5 47 90.4 6b 47 79.7 44 74.6 35 67.3 45 86.5 7b 51 86.4 51 86.4 41 78.8 43 82.7 8b 52 88.1 46 78.0 36 69.2 38 73.1 9b 59 100.0 59 100.0 51 98.1 52 100.0 10a,b 51 86.4 54 91.5 44 84.6 50 96.2 11b 55 93.2 48 81.4 25 48.1 31 59.6 12a,b 38 64.4 56 94.9 36 69.2 48 92.3 13b 43 72.9 36 61.0 26 50.0 28 53.8 14a,b 47 79.7 53 89.8 41 78.8 48 92.3 15b 58 98.3 57 96.6 46 88.5 50 96.2 Question a Control (n = 59) Experimental (n = 52)
Pretest Posttest Pretest Posttest
a
Questions with higher frequency of students in control group answering correctly in posttest compared to pretest.
b
Questions with higher frequency of students in experimental group answering correctly in posttest compared to pretest
To compare scores between the groups first, total pretest and posttest scores for each subject were calculated. Fifteen points were the total possible score for each pretest and posttest. Then, gain scores (difference between posttest and pretest scores) were computed for each subject by comparing individual pretest and posttest scores. An average gain score for each treatment group was then estimated. Table 3 shows mean gain scores and standard deviations for both control and experimental groups. The experimental group obtained a positive mean gain score (M=1.14, SD=2.12), while, on average, students from the control group scored lower on the posttest than the pretest (M=-.19, SD=1.86). To identify effects of
Table 3. Results of t-test for gain scores
Group Mean SD t df p
Control (n = 59) -.19 1.86 -3.36 101 .000
Experimental (n = 44) 1.14 2.12
Gain scores
the DVD on students’ learning only 44 students who reported had watched the DVD were included in the experimental group for further analysis.
The first hypothesis developed for this study stated:
H1: Students who use new multimedia material (DVD) (experimental group) will attain higher gain scores between pretest and posttest than students who do not have access to the new multimedia material (control group).
To test this hypothesis, an independent samples t-test was computed for gain scores of control and experimental groups as shown in Table 3. There is a significant difference in gain scores between the experimental group and the control group. The experimental group (fall 2007) obtained significantly higher gain scores than the control group (spring 2007)
(p<.001). This result indicates that students who reported watching the Tearoom Service Procedure DVD improved their knowledge about service procedures, compared to students who did not (control group).
Performance grades
Students’ grades from HRI 380L were computed and analyzed to determine students’ performance as table or beverage servers. Performance grades were scored from 1 to 10 and evaluated students’ performance based on an established criterion rubric.
For the purpose of this study only grades that evaluated performance of students in the table and beverage server positions were analyzed. Since each student performed the table server position seven or eight times during the semester, an individual average grade per student of both control and experimental groups was computed. A total average grade for the control and experimental groups was then computed. For beverage servers’ performance grades, a total average grade per each group was computed. Table 4 shows the means and standard deviations of final average grades for table and beverage servers of control and experimental groups. For the experimental group, grades from only the 44 students who reported watching the DVD were considered.
Table 4. Results of t-test for students’ performance grades in HRI 380L
Performance grades Position Mean* SD t df p Table server Control (n = 59) 9.71 .29 .144 101 .443 Experimental (n = 44) 9.71 .24 Beverage server Control (n = 59) 9.55 .81 -.108 101 .457 Experimental (n = 44) 9.56 .40
The second hypothesis developed for this study stated:
H2: Students in the experimental group will attain higher table and beverage servers’ performance grades in the Quantity Food Production course than students in the control group.
To test this hypothesis, two independent samples t-tests (Table 4) were computed to compare:
1. Table servers’ performance grades of students of control and experimental groups. There was no significant difference between the two groups (t=.144, p=.443). Table servers’ performance grades of students in the experimental group were not higher than those in the control group.
2. Beverage servers’ performance grades of students of control and experimental groups. There was no significant difference between the two groups (t=-.108,
p=.457). Beverage servers’ performance grades of students in the experimental group were not higher than those in the control group.
Therefore, H2, indicating that students in the experimental group would attain higher performance grades in HRI 380L, was rejected.
Perceptions of the instruction
All students in both control and experimental groups completed the pretest
questionnaire about perceptions of the instruction at the beginning of the semester after they had the Tearoom service orientation sessions (second week). During week 14, after all students had completed the table and beverage service assignments and students in the experimental group had the opportunity to watch the DVD, posttest questionnaires were
administered. Students’ perceptions of the instruction were assessed by their level of agreement on 10 statements (see Table 5). For the experimental group, perceptions’ responses from only the 44 students who reported watching the DVD were considered.
The last hypothesis of this study stated:
H3: Students who use new multimedia material (DVD) (experimental group) will have more positive perceptions of the instruction than students who do not have access to the new multimedia material (control group).
An independent samples t-test was computed to evaluate this hypothesis. For each perception statement asked in the posttest questionnaire, a t-value was computed to compare control and experimental groups. Table 5 shows the test results for each variable. In most cases there were no significant differences in students’ perceptions of instruction between the control and experimental groups.
Only three perception statements appeared to be significantly different between the two groups. First, students in the experimental group had more positive perceptions of the amount of information covered during orientation sessions (statement 4) than the control group (p<.01). Second, students’ responses for item 7 indicated that subjects in the control group agreed more than the experimental group to the statement of making changes to instruction (p<.01)1. And, finally, subjects in the experimental group had a higher overall satisfaction (statement 10) than subjects in the control group of the Tearoom service instruction (p<.05) (see Table 5).
1
For statements 4 and 7 a higher mean score indicated a higher students’ agreement to a negative perception statement.
31 Table 5. Results of t-test for students’ perceptions of the instruction in posttest
Experimental (n =44)
Mean SD Mean SD
1) Instructional sessions' content was thorough. 4.02 .81 4.21 .77 -1.205 99 .116
2) In my opinion, the ideas presented in the instructional sessions were:
a) clear 3.93 .85 4.00 .84 -.404 101 .344
b) concise 3.88 .91 4.11 .72 -1.395 101 .083
c) easy to understand 3.86 .94 3.95 .83 -.506 101 .307
3) There was an opportunity to ask questions during the instructional sessions. 4.31 .88 4.36 .84 -.342 101 .367
4) Instructors covered too much information in the instructional sessions. 3.88 1.25 3.25 .99 2.768 101 .003
5) I have a basic understanding of service procedures. 4.47 .63 4.36 .92 .729 101 .234
6) I feel capable of the following procedures: 101
a) Set a table 4.51 .68 4.57 .76 -.420 101 .338
b) To serve customers 4.49 .70 4.57 .76 -.529 101 .299
c) To prepare beverages 4.53 .70 4.43 .90 .593 101 .277
7) I think some changes could be made in the instructional sessions to facilitate
the learning process. 3.49 1.02 2.98 .95 2.598 101 .005
8) Extent to which you understand each category of information:
a) Setting a table 4.34 .66 4.45 .59 -.920 101 .180
b) Taking orders 4.58 .59 4.57 .62 .067 101 .473
c) Serving beverages 4.47 .70 4.50 .73 -.178 101 .429
d) Serving food items 4.53 .77 4.45 .63 .498 101 .310
e) Clearing tables 4.53 .68 4.41 .69 .853 101 .198
f) Refilling beverages 4.49 .65 4.48 .73 .104 101 .459
g) Preparing beverages 4.53 .63 4.36 .81 1.144 101 .128
9) Instructional sessions were successful 4.10 .85 4.30 .59 -1.300 101 .098
10) Overall satisfaction 3.92 .89 4.20 .63 -1.835 101 .035
Note. None of the statistical tests would pass the Bonferroni significance level (p <.0025) adjusted for multiple comparisons.
Note. All items were rated using a 5-point scale: items 1 to 7 with 1=strongly disagree and 5=strongly agree, item 8 with 1=not at all and 5= a great deal, item 9 with 1=very unsuccessful and 5=very successful, and item 10 with 1=very unsatisfied and 5= very satisfied.
Students' perceptions in posttest Control (n =59)
t df p
Responses about the new instructional material DVD
Five questions included in the posttest were distributed to students in the
experimental group (see Appendix B). Of the 52 students who received the new material (experimental group), 44 (85%) reported they had watched the DVD. The majority of them watched each DVD section one or two times: 35 (80%) students for the beverage service section and 27 (65%) for the table service section of the DVD. Twenty-five (57%) students reported watching the DVD prior to their table/beverage server assignments, and 14 (32%) students did it after distribution of DVD. The time students watched the DVD might depend, however, on the rotation schedule of assignments because some students could be assigned to a table or beverage server position at the beginning of the semester (right after distribution) or be beverage server at the end of the semester.
Of the 44 students who reported watching the DVD, 40 (91%) found the DVD informative (4 or 5 rating on a 5-point range from 1=not informative to 5=very informative), 39 (89%) thought it was useful (4 or 5 rating on a 5-point range from 1=useless to 5=very useful), 17 (39%) found the information provided in the DVD interesting (4 or 5 rating on a 5-point range from 1=not interesting to 5=very interesting), 30 (68%) considered the DVD was a quality product (4 or 5 rating on a 5-point range from 1=not a quality product and 5=a high quality product), and 35 (80%) thought it was applicable in the completion of
Discussion
The analysis of gain scores indicated that subjects improved their cognitive knowledge of table and beverage service procedures after viewing the Tearoom Service Procedure DVD. However, students had difficulty answering some items related to table setting; in two of these items the number of subjects having the correct answer decreased on posttests compared to pretests. Items showing the greatest improvement in posttest answers for the experimental group were related to serving food items and especially all the questions related to preparing beverages. These procedures may have been demonstrated well in the DVD, and therefore reinforced beverage service instruction effectively (Graber, 1990).
Results supported hypothesis one: students who watched the Tearoom Service Procedure DVD (experimental group) obtained higher gain scores, comparing pretest and posttest scores, than students in the control group who did not have access to this new instructional material. Similar findings were reported by Crosby and Stelovsky (1995) and Issa et al. (1999).
Analysis of students’ performance grades in HRI 380L showed that experimental and control groups had similar table and beverage server performance grades. As shown in Table 4, mean performance grades for the experimental group were not significantly higher than those for the control group for both table and beverage server positions. These findings indicated that the Tearoom Service Procedure DVD had no effect on the actual students’ class performance. Similar results were reported by Barlett and Strough (2003), and Susskind (2005). One possible factor that may have influenced the results of hypothesis two is that students’ performance was evaluated based on teaching assistants’ observation; since
students in the control and experimental groups were evaluated by different teaching assistants, performance evaluation varied from one semester to the other.
From the analysis of responses about students’ perceptions it can be stated that, in general, no significant differences were found between control and experimental groups. Similar findings were reported by Buzzell et al. (2002). Results showed that three out of the 17 questions about students’ perceptions of instruction, students in the experimental group had more positive perceptions of the instruction than those in the control group after receiving the Tearoom Service Procedure DVD. This finding indicates that the new
instructional material had little effect on students’ perceptions. This constitutes an interesting finding. One of the major concerns of students after they were exposed to orientation
sessions was the large amount of information they had received in a short period of time and the lack of material to support that information. Thus, this DVD could be additional
instructional material for students to comprehend information presented during the orientation.
Results from the additional questions about the DVD addressed to students in the experimental group showed that the majority of the students found the new instructional material as interesting, useful, applicable in the completion of assignments, and a quality product. The DVD used in this study has been specifically designed to provide students with an additional resource to support the information they received during the instructor-led instructional sessions.
Eight (15%) of the 52 students who received a copy of the DVD, reported not having watched it. Possible causes of this may be: students previous work experience and their self- confidence in their performance as servers, length of DVD, students’ perceptions about the
usefulness of the information covered in the DVD material, and the rotation schedule of assignments. The last cause refers to students who were assigned to a table or beverage server position at the beginning of the semester, right after orientation sessions, and therefore, they did not need to review Tearoom procedures.