Petersen and his colleagues were interested in understanding how linguistic informa- tion is processed in the human brain. They started with individual words– the basic building-blocks of language. Many different types of information are relevant to the normal course of reading, writing, or conversing. There is visual information about the shape and layout of the word, as well as auditory information about how the word sounds and semantic information about what the word means. The interesting question is how these different types of information are connected together. Does silently reading a word to oneself involve processing information about how the word sounds? Does simply repeating a word involve recruiting information about what the word means?
The two leading information-processing models of single-word processing (often called lexical access) answer these two questions very differently. Within neurology the dominant model, derived primarily from observing brain-damaged patients, holds that the processing of individual words in normal subjects follows a single, largely invariant path. The information-processing channel begins in the sensory areas. Auditory infor- mation about how the word sounds is processed in a separate brain region from infor- mation about the word’s visual appearance. According to the neurological model, however, visual information about the word’s appearance needs to be phonologically recoded before it can undergo further processing. So, in order to access semantic infor- mation about what a written word means, the neurological model holds that the brain needs to work out what the word sounds like. Moreover, on this model, semantic processing is an essential preliminary to producing phonological motor output. So, for example, reading a word and then pronouncing it aloud involves recruiting information about what the word means.
Exercise 3.7 Draw a flowchart illustrating the distinct information-processing stages in single- word processing according to the neurological model.
The principal alternative to the neurological model are various different varieties of cognitive model (derived primarily from experiments on normal subjects, rather than from studies of brain-damaged patients). The neurological model is serial. It holds that information travels through a fixed series of information-processing“stations” in a fixed order. In contrast, the cognitive model holds that lexical information processing is parallel. The brain can carry out different types of lexical information processing at once, with several channels that can feed into semantic processing. Likewise, there is no single route into phonological output processing.
Petersen and his colleagues designed a complex experiment with a series of conditions to determine which model reflects more accurately the channels of lexical information processing in the brain. The basic idea was to organize the conditions hierarchically, so that each condition could tap into a more advanced level of information processing than
its predecessor. The hierarchy of conditions mapped onto a hierarchy of information- processing tasks. Each level involved a new type of information-processing task. Success- fully carrying out the new task required successfully carrying out the other tasks lower in the hierarchy. What this means is that by looking at which new brain areas are activated in each task we can identify the brain areas that are specifically involved in performing that task– and we can also see which brain areas are not involved.
The base-line condition was simply asking subjects to focus on a fixation point (a small cross-hair) in the middle of a television screen. The point of asking the subjects to do this was to identify what is going on in the brain when subjects are visually attending to something that is not a word. The second condition measured brain activity while subjects were passively presented with words flashed on the screen at a rate of forty words per minute. The subjects were not asked to make any response to the words. In a separate condition the same words were spoken to the subjects. Combining the results from these two different conditions allowed Petersen and his colleagues to work out which brain areas are involved in visual and auditory word perception. The key to doing this is to subtract the image gained from the first condition from the image derived from the second condition. The image of brain activity while fixating on the cross-hair acts as a control state. In principle (and we will look much more closely at some of the methodological difficulties in functional neuroimaging inChapter 11), this allows us to filter out all the brain activation that is responsible for sensory processing in general, rather than word perception in particular.
The third and fourth levels of the experimental hierarchy measured brain activation during more complex tasks. The aim here was to trace the connections between initial sensory processing and the semantic and output processing that takes place further “downstream.” In the third condition subjects were asked to say out loud the word appearing on the screen. Subtracting the resulting image from the word perception image allowed Petersen and his colleagues to calculate which brain areas are involved in speech production. Finally, the highest level of the experimental hierarchy involved a task that clearly requires semantic processing. Here the subjects were presented with nouns on the television monitor and asked to utter an associated verb. So, for example, a subject might say “turn” when presented with the word “handlebars.” As before, Peterson and his colleagues argued that subtracting the image of brain activation during this semantic association task from the image obtained from the speech production task would iden- tify the brain areas involved in semantic processing.
Exercise 3.8 Make a table to show the different levels in the hierarchy and the aspects of single- word processing that they are intended to track.
Statistical comparison of the brain images in the different stages of the experiment produced a number of striking results. As we see in Figure 3.9, each of the tasks activated very different sets of brain areas. (The areas with the maximum blood flow are colored white, followed in decreasing order by shades of red, yellow, green, blue, and purple.)
Moreover, the patterns of activation seemed to provide clear evidence against the neurological model. In particular, when subjects were asked to repeat visually presented words, there was no activation of the regions associated with auditory processing. This suggested to Petersen and his colleagues that there is a direct information pathway from the areas in the visual cortex associated with visual word processing to the distributed network of areas responsible for articulatory coding and motor programming, coupled with a parallel and equally direct pathway from the areas associated with auditory word processing. Moreover, the areas associated with semantic processing (those identified in the condition at the top of the hierarchy) were not involved in any of the other tasks, suggesting that those direct pathways did not proceed via the semantic areas.
The situation can most easily be appreciated in an information-processing diagram.
Figure 3.10is drawn from a paper by Petersen and collaborators published in the journal Naturein 1988. Unlike many information-processing flowcharts, this one is distinctive in that it identifies the particular brain areas that are thought to carry out each distinct stage. This is not an accident. It reflects how the information-processing model was reached – on the basis of direct study of the brain through PET scan technology. This model, and the methodology that it represents, is a powerful illustration of how the bottom-up study of the brain can be used in developing higher-order models of cognition.
Figure 3.9 Images showing the different areas of activation (as measured by blood flow) during the four different stages in Petersen et al.’s lexical access studies. (From Posner and Raichle1994)
OUTPUT TASK
Motor output Motor (rolandic) cortex
Articulatory coding. Motor programming SMA. Inferior premotor sylvian areas.
L. Premotor ASSOCIATION TASK Generate uses covert monitoring Semantic association Area 47 SENSORY TASK Passive words Auditory presentation Auditory (phonological) word form Temporoparietal
cortex
Early auditory processing Primary auditory cortex
Passive words Visual presentation
Visual word form Extrastriate cortex
Early visual processing Striate visual cortex
SENSORY TASK
Figure 3.10 A flowchart relating some of the areas of activation in Petersen et al.’s study to the different levels of lexical processing. The dashed boxes outline the different subtraction. The solid boxes outline possible levels of coding and associated anatomical areas of activation. (From Petersen et al.1988)
Summary
This chapter has explored the “turn to the brain” that took place in cognitive science during the 1980s. This involved the development of experimental paradigms for studying the information pathways in the brain from the bottom up. These experimental paradigms included lesion studies on monkeys, as well as neuroimaging of human brains. We looked at two examples of how these different techniques allowed cognitive scientists to develop models of cognitive capacities that were much less abstract and functional than those we looked at inChapter 2. One example came from the two visual systems hypothesis developed primarily on the basis of monkey experiments, and another from a model of single-word processing developed from neuroimaging studies. Another important factor in the turn to the brain was the development of computational modeling techniques based on an idealized model of how neurons work.
Checklist
Ungerleider and Mishkin’s two visual systems hypothesis
(1) The cross-lesion disconnection paradigm, coupled with various other anatomical and neurological methods, was used to identify two different information-processing pathways for visual information. (2) Both pathways start from the primary visual cortex.
(3) Information relevant to object identification and recognition travels along the ventral pathway, from the primary visual cortex to the inferior temporal cortex via the prestriate cortex. (4) Information relevant to locating objects flows from the primary visual cortex to the posterior
parietal lobe.
Information processing in artificial neural networks
(1) These networks are designed to reflect certain high-level features of how the brain processes information, such as its parallel and distributed nature.
(2) The neuron-like units in artificial neural networks are organized into layers, with no connections between units in a single layer.
(3) The overall behavior of the network is determined by the weights attached to the connections between pairs of units in adjacent layers.
(4) Networks “learn” by adjusting the weights in order to reduce error. (5) Artificial neural networks are particularly suited to pattern recognition tasks.
Functional neuroimaging: The example of single-word processing
(1) Allows brain activity to be studied non-invasively by measuring blood flow in the brain while subjects are performing particular cognitive tasks.
(2) The paired-subtraction paradigm aims to focus on the brain activity specific to the task by subtracting out the activity generated by carefully chosen control tasks.
(3) In studies of how single words are processed experimenters constructed a four-level hierarchy of tasks of increasing complexity.
(4) The patterns of activation they identified across the different tasks supported a parallel rather than a serial model of single-word processing.
Further reading
Ungerleider and Mishkin’s paper “Two cortical visual systems” is reprinted in Cummins and Cummins2000. Mishkin, Ungerleider, and Macko 1983/2001 is a little more accessible. David Milner and Melvyn Goodale have developed a different version of the two visual systems hypothesis, placing much more emphasis on studies of brain-damaged patients. See, for example, their book The Visual Brain in Action (2006). A more recent summary can be found in Milner and Goodale2008(including discussion of Ungerleider and Mishkin). A different development in terms of vision for action versus vision for higher mental processes has been proposed by the cognitive neuroscientist Marc Jeannerod, as presented in Ways of Seeing, co-authored with the philosopher Pierre Jacob (Jacob and Jeannerod2003). A recent critique of the two system account (with commentary from Milner, Goodale, and others) can be found in Schenk and McIntosh2010
The Handbook of Brain Theory and Neural Networks (Arbib2003) is the most comprehensive single-volume source for different types of computational neuroscience and neural computing, together with entries on neuroanatomy and many other “neural topics.” It contains useful introductory material and “road maps.” Dayan and Abbott2005and Trappenberg2010are other commonly used introductory textbooks.Scholarpedia.orgis also a good source for introductory articles specifically on topics in computational neuroscience. McLeod, Plunkett, and Rolls1998is a good introduction to connectionism that comes with software allowing readers to get hands-on experience in connectionist modeling. Bechtel and Abrahamsen (2002) is also to be recommended. Useful article-length presentations are Rumelhart1989(in Posner1989, reprinted in Haugeland
1997) and Churchland1990b(in Cummins and Cummins2000). A more recent discussion of connectionism can be found in McClelland et al.2010, with commentary and target articles from others in the same issue. The mine/rock network described in the text was first presented in Gorman and Sejnowski1988and is discussed in Churchland1990a
A very readable book introducing PET and functional neuroimaging in general is Posner and Raichle (1994), written by two senior scientists participating in the lexical access experiments discussed in the text. These experiments are discussed in the article by Petersen et al. cited in the text and also (more accessibly) in Petersen and Fiez2001. Rowe and Frackowiak2003is an article-length introduction to the basic principles of functional neuroimaging. Another good introduction to neuroimaging, including discussion of many of the experiments mentioned in this chapter (and with a lot of colorful illustrations), is Baars and Gage2010
T H E I N T E G R A T I O N
C H A L L E N G E
I N T R O D U C T I O N
The chapters inPart Ihighlighted some of the key landmarks in the development of cognitive science. We saw how the foundations for cognitive science were laid in psychology, linguistics, and mathematical logic. We looked at three key studies that helped to establish cognitive science as a field of study in the 1970s. These studies provided different perspectives on the idea that the mind could be modeled as a form of digital computer. In their different ways, they each reflected a single basic assumption. This is the assumption that, just as we can study computer software without studying the hardware that runs it, so too can we study the mind without directly studying the brain. As we saw inChapter 3, however, cognitive science has moved away from this confidence that the brain is irrelevant. Cognitive scientists are increasingly coming to the view that cognitive science has to be bottom-up as well as top-down. Our theories of what the mind does have to co-evolve with our theories of how the brain works.
Two themes were particularly prominent inPart I. The first was the interdisciplinary nature of cognitive science. Cognitive science draws upon a range of different academic disciplines and seeks to combine many different tools and techniques for studying the mind. This interdisciplinarity reflects the different levels of organization at which the mind and the nervous system can be studied. The second theme was the idea that cognition is a form of information processing. As we saw, this is one of the guiding ideas in the prehistory of cognitive science and it remained a guiding assumption both for theorists who modeled the mind as a digital computer and for theorists who favored the direct study of the brain and neurally-inspired models of computation. These two themes are the focus ofPart II.
Chapter 4shows how the interdisciplinary nature of cognitive science gives rise to what I call the integration challenge. Cognitive science is more than just the sum of its parts and the integration challenge is the challenge of developing a unified framework that makes explicit the relations between the different disciplines on which cognitive science draws and the different levels of organization that it studies. We will look at two examples of what I call local integrations. These are examples of fruitful “crosstalk” between different levels of organization and levels of explanation. The first example is relatively high-level. We will look at how evolutionary psychologists have proposed a particular type of explanation of experimental results in the psychology of reasoning. The second is much lower-level. It concerns the relation between systems-level cognitive activity, as measured by functional neuroimaging, and activity at the level of individual neurons, as measured by electrophysiology.
InChapter 5we look at two global models of integration in cognitive science. One model is derived from reflections on the unity of science in the philosophy of science. This model proposes to think about integration directly in terms of the relation between levels of explanation, by reducing cognitive science to a single, fundamental theory of the brain. A second model, very popular among cognitive scientists, is derived from Marr’s study of the visual system (discussed in
section 2.3). We see that neither model is really appropriate for solving the integration challenge. Insection 5.3I propose a more modest approach. The mental architecture approach proposes tackling the integration challenge by developing an account (1) of how the mind is organized into different cognitive systems, and (2) of how information is processed in individual cognitive systems.