Visual salience
Entities are said to be visually salient when they stand out from
surrounding stimuli. Evolutionarily speaking, the detection of threat and reward (predators, food, etc.) may be served by these entities being visually salient. Indeed, many plants propagate via seed dissemination mechanisms that rely on their berries or fruit being eaten and as such have evolved brightly coloured fruit or berries to signal their presence to potential feeders. Conversely, certain species such as the poison dart frog (Summers & Clough, 2000) and cinnabar moth (Zoelen & Meijden, 1991) have evolved brightly coloured markings to signal to predators that they are poisonous, a phenomenon known as aposematism.
These efficient signalling mechanisms rely on the intended observer having a corresponding mechanism that allows bright colours to stand out in the visual field (i.e. be salient). Salience, therefore, is not strictly speaking a property of a physical stimulus, but is a phenomenon that arises when an observer and stimulus come together in certain specific situations (see below).13
Visual salience may be driven by both bottom up and top down factors. Bottom up factors relate to the physical properties of the stimulus. A classic example of this is the feature search task. In this task an observer is required to find among distractors a stimulus that has a particular feature or combination of features. In many instances of the visual search task the target stimulus will ‘pop
13
Technically, it is more accurate to say that the interaction of the stimulus with the perception systems of the observer is what results in salience; individual stimuli are in fact only salient with regards to any given observer and what may be salient to one observer may not be salient to another. As an example, a green stimulus among red stimuli may be visually salient when observed by an observer with colour vision that allows discrimination between these two colours but may not be visually salient to a colour-blind observer. Nonetheless, referring to stimuli as being salient or not is a useful shorthand and will be used herein under the assumption that it is generally understood that this in fact incorporates the notion of stimulus/observer interaction.
out’ and be easily and quickly identified. However, manipulation of the type of distractor stimuli can impact the visual salience of the search target (see Figure 3.1).
Figure 3.1. Examples of visual search task. In instances where distractors share only a single feature with the target they can be processed in parallel and search is efficient. When multiple distractor types are present, each sharing different features with the target (conjunction search; Treisman & Gelade, 1980), search is inefficient and proceeds in a serial fashion. Graphics reproduced from
http://www.scholarpedia.org/article/Visual_salience.
Saliency mechanisms are important to facilitate day-to-day commerce with the world; but how is visual saliency realised by the human mind? One answer is through saliency maps. Itti et al. (Itti, Koch & Neiber, 1998) provide an elegant description of this mechanism. Assume a stimulus as in Figure 3.2 below. Each rectangle is equally (or very nearly equally) intense in terms of its difference in luminance from the background. The high peaks in the intensity map shown in Figure 3.2 correspond to the large degree to which each bar stands out from the
background (note that the intensity map coordinates correspond to the two dimensional x and y spatial coordinates of the stimulus). However, because all the peaks are high, the intensity map is relatively uninformative when trying to distinguish differences between the rectangles. Therefore, a normalisation factor
N(.) is applied to the intensity map, down-weighting the peaks. A similar process
is applied with regards to orientation, except that here the map is informative since one of the bars has different orientation to the others, so the contribution of the orientation dimension to salience is upweighted. In this way, an overall saliency map corresponding to spatial locations is generated (essentially this is a
compound of the normalised intensity and orientation maps shown in Figure 3.2).
Figure 3.2. Bottom up saliency maps for a simple display of white rectangle stimuli. Reproduced from Itti et al. (1998).
Visual salience is also driven by top down factors. The most obvious of these is conscious search; if one is searching for a particular entity (say, one’s own car in a car park) then this entity may pop out from the crowd. Additionally, it is possible for less complex features to take on salience when they are
situationally relevant. If I own a red car and need to find it in a car park, then the colour red may serve as an initial guiding factor to allow me to limit processing only to a consideration set of red cars. The way in which top-down factors guide visual search in this manner has been extensively studied and termed ‘guided search’ (Cave & Wolfe, 1990; Wolfe, 1994; Wolfe, 2007; Wolfe, Cave, & Franzel, 1989; Wolfe & Gancarz, 1996).
Each of these top down factors can be described as being motivational in nature inasmuch as each relates to a particular goal an individual has at a
particular time. However, not all motivational states need be consciously held, and factors such as hunger, thirst, tiredness etc. may each play a role in
determining which stimuli or stimulus features may be salient at any given time.
Motivational salience
The visual search literature is replete with examples of the effect of both bottom up and top down factors on salience and the subsequent deployment of attention.14 Indeed, the deployment of attention to a stimulus (e.g. as indexed by saccades; Pomplun, 2006) is often used to determine salience in visual search tasks. Thus, salience often denotes that a stimulus is worthy of selective attention.
14
There is an ongoing debate as to the extent to which bottom-up and top-down factors separately influence the immediate saliency of stimuli and the associated initial feed-forward sweep of visual processing and deployment of visual attention. The two poles of this argument are represented by a review from Theeuwes (2010) in which he concludes that ‘top–down knowledge regarding non-spatial features of the objects cannot alter the initial selection priority’, and the contingent-capture hypothesis (Folk, Remington, & Johnston, 1992), which argues that bottom-up signals must match to a top-down goal state if they are to capture visual attention.
Selective attention, however, is not a processing resource in its own right; rather it is a guiding mechanism that serves to allocate neural processing resources
towards particular entities. More generally then, we can say that salience may denote a stimulus as worthy of more processing. This is an important distinction since it also allows for the level of processing at post- selective/encoding levels (i.e. once an entity has been encoded in memory) to possibly be biased by salience.
In the visual search example depicted in Figure 3.1 above, the colour red is salient due to its uniqueness in the display. In the natural world, bright red fruit and berries are salient both because they tend to stand out from distractors (e.g. green leaves with which they share few features), and because they have intrinsic value (they are food). It is also possible that features with no intrinsic value may carry learned value codes and so also result in saliency. For example, it is possible to imbue a particular colour with a positive or negative value by
repeatedly pairing it with either reward or punishment (see below). Such a colour then carries associated motivational value, which may bias salience even in situations where the colour is no longer task relevant. In Chapter 7 I imbue value into face stimuli and test whether the resulting motivational salience impacts WM for such faces.
Value is a thus a higher order dimension that may bias salience through a variety of top down processes. Once stimulus processing has occurred, and the fact that a particular stimulus has value has been established, then value may serve to alter the saliency map such that a greater saliency is realised at the location of the value laden stimulus. This could be achieved through the
feeds in to the overall saliency map. Alternatively, value may serve to bias the weights assigned to particular features. For example, a red stimulus and blue stimulus presented together with a crowd of green stimuli will both stand out from the crowd, but if the red colour has previously been paired with (say) monetary reward then the reward association history may serve to decrease the threshold at which neurons representing the colour red (in early visual areas) fire. It is not the purpose of this thesis to distinguish between these two mechanisms, and indeed both may play a role.
In the same way that visual salience is affected by both bottom-up and top-down factors, motivational salience is also influenced by top-down goals (see below) and by bottom-up factors such as the automatic motivational saliency accorded to items of evolutionary threat (snakes, spiders; Öhman, Flykt, & Esteves, 2001; Öhman & Mineka, 2001) and items of modern threat (knives, syringes; Brown, El-Deredy & Blanchette, 2010). Stimuli such as snakes, spiders, guns and syringes can be thought of as carrying threat information within the context of an omnipresent goal; survival. As such, they may be expected to always carry such importance. However, the overall goal to survive is subserved by a number of sub-goals; sustenance, shelter, reproduction, rest and
exploration15 (see Table 3.1).
15
Surprisingly, exploration is rarely thought of a being a low-level goal, more traditionally being discussed as a higher level goal (see for example Maslow’s hierarchy of needs; Maslow, 1943). This is probably because strictly speaking it does not directly and immediately subserve survival. If an organism lives within a group in which food, shelter and reproductive partners are readily available then there may be no need for this individual to explore beyond the boundaries of the group in order to successfully pass on genetic material. However, if we consider low level ‘second order’ goals as being driven by an evolutionary pressure that works over generations rather than merely during the lifetime of a single individual, then since the long term survival of a group depends upon its eventual expansion to acquire new resources then the absence of exploration as a hard-wired goal would have negative consequences for survival. Note that the above scenario of territorial expansion is only one extreme form of exploration. More generally we can
Survival/reproduction related sub-goal Motivating factors Satiating stimuli Demotivating factors Sustenance Hunger Thirst Food Water Satiation Shelter Cold Heat Presence of predators Shelter Comfort
Reproduction Potential sexual
partner
Sexual partner
Presence of sexual competitor
Rest Tiredness Sleep Absence of safe resting
place
Exploration Absence of
reward in current situation
‘Discovery’ Reward already obtainable in current situation
Perceived risk / cost of exploration
Table 3.1. Low-level goals, motivating factors, satiating stimuli, and demotivating factors.
There is, of course, an opportunity cost of pursuing any particular goal since only a limited number of goals may be pursued at one time (we cannot seek out food while asleep). Opportunity costs act in a similar way to directly
demotivating factors in controlling an organism’s propensity to pursue any given goal. It is evident from Table 3.1 that low-level goals may be broadly divided into two categories; threat negating and reward seeking (with shelter perhaps being a compound goal comprising elements of both). This distinction is important since different neural substrates underlie the acquisition of positive and negative value codes for motivational stimuli (see below).
define exploration as expenditure of energy to create or move into a new situation in which potential rewards may be obtainable (see Table 3.1).
So far I have discussed low-level goals common to higher organisms. Humans, however, are complex social animals and in addition to homeostatic requirements are driven to pursue a number of ‘higher’ goals to fulfil complex social motivations (e.g. Maslow, 1943). More specifically, it is of note that each ‘need’ can be satisfied by the setting and fulfilment of certain goals, as illustrated in Table 3.1 above. In this way it is possible to conceptualise almost all human behaviour as resulting from a process of goal setting and fulfilment. This is not to say that goals are necessarily usually consciously set. Indeed, conscious
development of a goal-plan need only occur when a series of ordered steps are required: If I am hungry and I have on my person a bar of chocolate then I can consume it without deliberation, but if I do not possess food then I must formulate multi-step plan to acquire some (e.g. find money, find a shop, select food,
purchase).
It is apparent from the discussion of setting goals to fulfil low level needs above that motivational salience is also affected by top-down factors. Thus, motivational salience may be regarded as a property of a stimulus (or more formally a stimulus and observer interaction) that derives from the value that that stimulus has to the observer at any particular time. Value is a complex construct, representing as it does a compound of various stimulus attributes such as
valence, predictiveness of learned outcome related to stimulus, delay to outcome predicted, goal congruence, intensity of outcome predicted, action trade-off status (energy that must be expended to utilise stimulus), and rarity of stimulus
(probability of encountering later if not approached/utilised on this occasion), the net result of which is modulated by current goals to derive final stimulus value or utility to the observer.
Given that WM is a crucial part of the response planning apparatus, it would be beneficial if motivationally salient items had preferential access to, were efficiently manipulated in, and were easily recovered from WM. In Chapter 2 I noted that emotional and in particular angry faces have been observed to
experience a boost in WM relative to non-emotional (neutral) faces. It is possible that such a boost derives from their status as motivationally salient items rather than their threat status per se. To investigate the possibility that motivational salience might effect a boost in WM processing, in Chapter 7 I imbue value into expressively neutral face stimuli and test whether this affects WM processing for them. Imbuing of stimuli with value can be achieved using a value-learning
paradigm. In such a procedure, different stimuli are consistently paired with either a negative, neutral, or positive outcome, usually in the form of a game in which points or small amounts of money are won and lost. In-procedure response bias analysis can be used to confirm that learning of stimulus values and
reward/punishment prediction has taken place, and post-procedure stimulus evaluation tests can be used to ascertain whether this learning translates to an emotional bias across stimuli (whether, for example, stimuli paired with highly positive outcomes are then rated as more trustworthy than those paired with negative outcomes). The mechanisms underlying the acquisition of positive and negative value codes seem to be based on different neurological structures, with the amygdala being associated with acquisition of negative value codes (Kahn, Yeshurun, Rotshtein, Fried, Ben-Bashat & Hendler, 2002), and the ventral striatum and right OFC more with positive value codes (Yacubian, Gläscher, Schroeder, Sommer, Braus, & Büchel, 2006), although it is possible that these systems may in fact share some neural resources, with the amygdala playing a role in
assignation of both positive and negatively valenced codes to objects (Paton, Belova, Morrison & Salzman, 2006).
In Experiment 8 I use an operant conditioning paradigm that has been reported before (O’Brien & Raymond, 2009) to imbue value associations directly to specific face stimuli. In Experiments 6 and 7 I use a novel value learning
paradigm to imbue value associations into particular classes of stimuli as defined by different colours. The notion of using colour to define value association has been successfully utilised elsewhere.16 For example, Anderson et al. (Anderson, Laurent & Yantis 2011a; 2011b) successfully imbued high and low positive value into colour categories (using an associative learning task in which colour was orthogonal to the task)17 and then, in a separate test phase, used stimuli of different value-associated colours as either distractors (2011a) or salient
distractors (2011b), finding attentional capture by value associated items in both instances.
Motivationally salient stimuli in visual WM
There is an inherent advantage in having lower level visual detection and representation mechanisms that are able to quickly prioritise certain types of information for efficient evaluation and appropriate response generation. In daily life, we encounter numerous stimuli that require near instantaneous evaluation if we are to generate the optimal response quickly enough to take advantage of, or avoid a situation. One mechanism that facilitates such differential stimulus processing is attention. Attention mechanisms are differentially deployed under
16 Indeed, the ability to imbue learnt status into colour is prevalent across species. For example, pigeons
can be trained to associate pecking a coloured key with reward (e.g. Lea, 1979).
17 See Chapter 7 for discussion of their method. See also Hickey et al. (Hickey, Chelazzi & Theeuwes, 2010)
conditions of high and low arousal (e.g. Shapiro & Lim, 1989), such that in
conditions of high arousal attention deployment is biased toward auditory attention over visual attention and also biased within visual attention to the periphery of the visual field. Such differential deployment of attention may serve an evolutionary purpose, increasing awareness of surroundings in situations of high arousal (danger). The degree to which particular stimuli generate positive or negative affect depends in part upon the way in which they fit current and future goals. When hungry, food may generate positive affect, but when hunger is satisfied food may become an emotionally neutral stimulus (Brendl, Marknan, & Messner, 2003). Additionally, stimuli perceived as neither instrumental nor disinstrumental to
achieving a current focal goal may be subject to an unconscious ‘devaluation’ effect such that they are evaluated more negatively, with this devaluation thought to be due to a mechanism whereby evaluative responses are blocked rather than inhibition of attention to stimuli (Brendl et al, 2003).
Additional to the role that mood may play in directing attention, emotional states associated with specific stimuli may also serve to direct attention. In particular, negative stimuli better capture attention than positive stimuli (Pratto & John, 1991), make disengaging attention more difficult (Fox, Russo, & Dutton, 2002), and generate greater interference on detection tasks (Pessoa, Mckenna, Gutierrez, & Ungerleider, 2002).
The relationship between attention and motivational salience is bi-
directional, such that attentional state can directly influence emotional evaluation of a stimulus. In particular, under conditions where a stimulus must be ignored, and an inhibitory ‘tag’ attached to the stimulus to facilitate task demands (non- response to the stimulus), the inhibited stimulus may be subject to a devaluation
effect so that it is rated less emotionally favourably (Raymond, Fenske, and Tavassoli, 2003). Such ‘inhibitory tags’ are recoverable at a later date (one possible mechanism of reinstating do-not-respond locations in interrupted visual search; Tipper, Grison, and Kessler, 2003). Devaluation effects can also be reinstated in this way and the original formation of such devaluation associated tags requires working memory (Goolsby, Shapiro & Raymond, 2009).
Recent research (Raymond & O’Brien, 2009) using a value-learning task followed by attentional blink task has demonstrated survival of memory traces for positively valenced but not negatively valenced stimuli under conditions of low attention availability at recall. This finding, along with the fact that emotional