TESTIMONIO DE LOS “PADRES DE LA IGLESIA”
TESTIMONIO DE LOS RABINOS TALMÚDICOS
An obvious way in which the verbal protocol method differs from the written method is the demands to verbalize in the verbal protocol condition. Employing Type 1 verbalizations which require participants to concurrently think aloud whilst carrying out the task could potentially add additional demands not present in the written condition.
The possibility of verbalization interfering with the primary task is known as “reactivity effects” (Russo, Johnson & Stephens, 1989) and whether the act of producing a protocol is reactive is typically investigated
by incorporating a silent condition in the experiment. In one condition participants provide a protocol whilst completing the task and in another condition complete the same task silently. Output measures are recorded (for e.g., response time, number of correct responses) and compared to performance in the think aloud condition (Ericsson and Simon, 1993).
Therefore, in order to test whether it was providing a protocol that resulted in a detriment in performance in the think aloud condition a silent condition was included as an experiment. In this condition participants were required to interpret the graphs but at first they spent time reading the graphs silently. Once they felt they had understood the graphs as much as possible they then verbalized their interpretation to the experimenter.
The aim of this experiment is to determine whether remaining silent whilst engaging in the task resulted in any discernible benefits.
Method Participants
Fifteen undergraduate Psychology students (11 female, 4 male) from the University of Huddersfield volunteered to take part in the experiment, for which they were paid £5 in vouchers. The age of participants ranged from 18.1 to 23.2 years with a mean of 20.5 years (SD = 1.5). All participants were in the first year of a three year Psychology degree.
Materials, Design and Procedure
The experiment was carried out using the same equipment and the same procedure as Experiment 5 (verbal protocol condition). Participants were instructed that the experiment consisted of two stages – in the first “quiet” stage they could take as long as they wanted to understand the graph they were viewing as much as possible. In the second “talking” stage they were required to tell the experimenter what they had understood about the graph.
Results
Results were similar to the written condition – only a small proportion of the sample was classified pre- elementary. The high rate of pre-elementary performance found in the think aloud condition in Experiment 5 was not replicated with only 7% of the sample classified as such in this condition. Similar to the colour match graph and written interpretations, the majority of participants were classified as elementary and a smaller proportion pre-elementary and intermediate. Figure 22 displays the number of users in each
comprehension category, alongside the results of the think aloud condition in Experiment 5 for comparison.
Figure 22: Percentage of line graph users in the three performance categories, think aloud and silent condition
The improvement that emerged from the silent condition was very similar to the improvements observed in the written condition and the colour match graph. 40% of participants interpreted all six trials correctly in this condition. This compares favourably to 13% of line graph readers in the think aloud condition. However, this association was not significant (Fisher’s Exact test p = .22).
A chi-square test revealed that the number of participants classified as pre-elementary in the silent and think aloud line graph condition was significant (chi-square = 10.4; df = 2; p < .01). A comparison of the number of correct trials between the conditions also revealed that the silent condition resulted in a significant increase in the number of correctly interpreted trials (think aloud mean ranks = 11.7, silent mean ranks = 19.93). This difference was significant: U = 46, p = <.01
Discussion
The results of this experiment suggest that the high rate of pre-elementary performance consistently found in the line graph condition in the think aloud conditions was a result of the demands of verbalizations. When participants were allowed to remain silent whilst formulating their interpretation pre-elementary performance was equal to the bar graph condition and level of comprehension was similar to the written condition.
Therefore, despite the impressive research evidence reviewed by Ericsson and Simon (1993) supporting their assertion that protocols are not reactive it would appear that the conclusion drawn by Russo, Johnson & Stephens (1989) receives further support from the results of this experiment – the effect of providing a protocol is dependent on the interaction between task demands and the demands of verbalization.
There are only a limited number of studies in the literature which have found reactivity effects in tasks employing the think aloud method. In fact, Russo, Johnson & Stephens (1989) concluded that because of the lack of studies finding reactivity effects, there is an agreement in the literature that employing the think aloud method slows down processing but does not alter it.
To determine whether providing a protocol can be reactive Russo, Johnson & Stephens (1989) tested four different tasks in an attempt to test Ericsson and Simon’s (1993) claim that verbal protocols are accurate reflections of underlying cognitive processes. They found that for two of the four tasks - Raven’s Matrices and anagrams - providing a protocol resulted in no differences between the silent and think aloud condition.
However, for the other two tasks the effect of thinking aloud differed depending on the nature of the task. Thinking aloud whilst making a choice between two gambles significantly improved task performance compared to the silent condition. However, when participants were required to add three digit numbers
performance deteriorated. This led them to conclude that the reason why reactivity effects did not often occur is because the demands of verbalization interact with the demands of the task to affect output.
Russo, Johnson & Stephens (1989) proposed a processing resources explanation to account for their results. They suggest that when the demands of verbalization compete with task demands then participants will have to choose where to allocate processing resources. Russo, Johnson & Stephens (1989), following on from Kahneman (1973) suggest that participants draw on slack resources which are not being used up by the task to verbalize whilst completing the task. If the demands to think aloud are minimal and resources are
available then there will be no change in task performance.
However, when the availability of slack resources is minimal and demands to verbalize requires more resources than are available then participants face a choice: they can either continue to verbalize which will result in a drop in the resources available for the task or stop thinking aloud which violates the instructions of the task. If participants choose the former option then reactivity effects can result.
Therefore, when participants are required to solve a difficult problem then there will be little or no
processing resources available for the demands posed by verbalization, whereas if the problem is simple then there are slack resources available to draw upon, thus resulting in little or no difference in problem solving despite the demands to verbalize. Kahneman (1973) suggests the more difficult the problem the higher the likelihood problem solving will be impaired if there is a competition for resources.
This explanation could account for the pattern of results found for the experiments conducted for this research. Although it could be problematic defining line graphs as more difficult to interpret than bar graphs, Kahneman (1973) suggests problem difficulty could be measured by number of errors made whilst solving problems. Since the pattern of results from these experiments consistently show participants make more errors in the line than the bar graph condition, the line graph format can be considered to be more difficult to interpret than bar graphs.
Russo, Johnson & Stephens’ (1989) finding that task demands interact with the think aloud method to affect output is consistent with the findings of this research. Performance in the bar graph condition appears to be
unaffected by the demands to think aloud whereas the requirement to think aloud in the line graph condition results in a marked detriment in performance when compared to a silent condition.
Therefore, one potential explanation for these findings would be that when interpreting line graphs - which are difficult to understand without training - there is a high degree of competition for processing resources. When additional demands posed by verbalization are also present then resources required to interpret the information presented in line graphs are diverted to demands posed by the instructions to verbalize. This competition for resources could potentially result in the pre-elementary performance observed in the line graph condition in numerous experiments which does not emerge when the demands to verbalize are not present – in the silent and written condition.
However, this proposed explanation by Russo, Johnson & Stephens (1989) has been challenged by Schooler, Ohlsson and Brooks (1993) who proposed a very different explanation for why reactivity effects occur with certain types of problems. Schooler, Ohlsson and Brooks (1993) tested for reactivity effects by comparing performance on insight and non-insight problems whilst participants completed the problem silently or whilst thinking aloud. They found that the demands of thinking aloud resulted in significantly fewer insight
problems being solved compared to non-insight problems.
As these two types of problems were matched for difficulty, Schooler, Ohlsson and Brooks (1993) argued that the proposed explanation by Russo, Johnson & Stephens (1989) could not explain their results. If a processing resources explanation was valid then a detriment in performance should occur in the non-insight problems where participants were also required to think aloud throughout. However, they found no detriment in performance for these types of problems. Therefore, they concluded that verbalization was not reducing the availability of resources to solve a problem.
Instead, they explain their results by drawing on the verbal overshadowing paradigm. They argue that the demands to verbalize directs the way resources are allocated, rather than consuming resources and making them unavailable for use in the primary task. Specifically, attention is directed to aspects of the problem that are easy to verbalize which draws attention away from processes which are difficult to verbalize.
This explanation could also perhaps explain the results of this research. An analysis of the Gestalt principles present in interaction bar and line graphs revealed that bar graph displays have a perceptual feature allowing readers to relate the pattern to both independent variables. However, the line graph display has a perceptual feature allowing readers to relate the pattern to the legend but not to variables plotted on the x axis.
Therefore, the verbal overshadowing explanation proposed by Schooler, Ohlsson and Brooks (1993) could be applied to these types of diagrams. Although not insight problems it is possible that the demands to verbalize draws attention to information that is easy to verbalize in the line graph condition. This information would be the variables in the legend, as a simple colour matching process allows participants to match the lines at the centre of the display to the variables in the legend.
As there is no equivalent grouping process available allowing readers to relate the pattern to variables plotted on the x axis, this information would be difficult to verbalize and so attention would be drawn away from it. This could perhaps explain the pattern of errors found when participants are required to interpret line graphs whilst thinking aloud – the large majority of participants ignored the x variable and described the
relationship between the legend and y axis variable.
Although Schooler, Ohlsson and Brooks (1993) discount a processing resources explanation of reactivity effects and present convincing evidence this theory cannot account for their results, this explanation cannot be discounted completely. This is because the task Russo, Johnson & Stephens (1989) found a significant detriment in performance in does not consist of information that is difficult to verbalize. Their task required participants to add three-digit numbers, information which is consistent with a verbal code and so easy to verbalize (Ericsson and Simon, 1993).
The research literature demonstrating reactivity effects which occur from thinking aloud whilst completing a task is limited and therefore any theoretical account attempting to explain such effects is at an early stage. It is possible that the conclusions Russo, Johnson & Stephens (1989) drew about thinking aloud interacting with task demands to affect the output of a task can also be applied to the two theoretical accounts proposed to explain reactivity effects.
It is possible that when information is difficult to verbalize, attention is focussed on the components of a problem that are easily to verbalize (Schooler, Ohlsson and Brooks, 1993). This does not necessarily discount the explanation however, that processing resources are divided between task demands and the demand to verbalize which can cause a detriment in performance (Kahneman, 1973 Russo, Johnson & Stephens, 1989).
Based on the results of this experiment it would appear that the pre-elementary performance found in the think aloud line graph conditions is a result of the requirements to verbalize. However, due to the nature of the task it could be argued that the second stage of the task – interpreting the graphs out loud to the
experimenter, could be influencing the results. The communicative aspect of providing an interpretation to someone else could be resulting in an improvement. Therefore, splitting the task into two stages could potentially result in a confounding variable. Comparing the silent condition to the think aloud condition is not appropriate because of this potential confound.
Therefore, the final experiment will test the explanation that communicating understanding to someone else is resulting in an improvement in performance. Again the task will be split into two stages. In the next experiment participants were required to interpret the graphs whilst thinking aloud so the first stage of the task was identical to the think aloud condition. Once they felt they had understood the graphs as much as possible, they then provided an interpretation to the experimenter.