The current thesis explored and illustrated spatial simulations towards more precise simulation models within the framework of grounded-embodied theories of memory and language.
Simulation of word locations in the absence of words during memory retrieval was addressed in Chapter 3 (Experiment 1) and Chapter 4 (Experiment 2). Simulation of spatial locations triggered directly and automatically by word meanings was focused in Chapter 5 (Experiment 3) and Chapter 6 (Experiment 4).
Experiment 1 results showed that participants encoded locations of words upon their presentation in an unintentional and effortless manner: First, word locations were encoded along with the words themselves although participants did not have to remember locations intentionally. Second, words locations were encoded without any objects (e.g., faces) to explicitly associate with as in previous studies (thus, effortless) (D. C. Richardson & Spivey, 2000; Scholz et al., 2018, 2011, 2016). Spatial indices tagging word locations induced simulation of the location when the information previously occurred there was re-accessed with memory retrieval. In return, simulation of word locations guided eye movements to the relevant locations (i.e., looking at nothing). In other words, participants relied on the simulation of word locations. In this manner, memory work was offloaded onto the body and the environment through eye movements and spatial simulations, respectively.
Experiment 1 results also showed that spatial indices can be modulated with additional visual information. A visual cue appearing in the same location as to the previous location of the probe word emphasised the locational information and strengthened its simulation. As a result, looks to the relevant, blank locations were more frequent following a spatially congruent visual cue compared to trials where no cue was shown. On the other hand, a visual cue appearing in a diagonal location to the previous location of the probe word formed a new spatial
index. Competition between the spatial indices standing for the probe word and the incongruent visual cue interfered with “looking at nothing” behaviour but it did not delay memory retrieval.
The key findings of Experiment 1 are the correlations between visuospatial memory span measured with Corsi-block tapping task and looking behaviour: First, there was a positive correlation between visuospatial memory and percentage of fixations in the central interest area (across all cue conditions). Second, there was a negative correlation between visuospatial memory and percentage of fixations in the relevant, blank quadrant (in incongruent cue condition) and in the irrelevant, blank quadrant (in no cue condition). We conclude that participants with worse visuospatial memory relied more on the environment and thus, simulation of word locations than participants with better visuospatial memory.
Almost all patterns of the looking behaviour observed in Experiment 1 were replicated in Experiment 2: First, participants looked at the relevant, blank locations (the quadrant where the probe word was presented) more than the irrelevant, blank locations (the other three blank quadrants) when remembering words. Second, visual cues shown between encoding and retrieval stages affected looks in the same way as found in Experiment 1. Participants looked at “nothing” when no cue was presented or following a spatially congruent cue. However, spatially incongruent cues disrupted the looks to blank locations. Third, the link between visuospatial memory and looking behaviour was replicated. Lastly, looking at nothing did not improve memory performance in either of the studies.
What is the novel contribution of Experiment 2? The key finding is that participants looked more at “nothing” when remembering words that are more difficult to maintain and retrieve from memory compared to easier words. Mixed-effects models that were fit to investigate the contribution of lexico-semantic variables revealed that word imageability and concreteness modulated looks to blank locations. There were more looks in the relevant, blank locations during the retrieval of less imageable and abstract words as compared to more imageable and concrete words. Notably, word length modulated memory performance (hit rate and hit latency)
but not looks to relevant, blank locations suggesting a special role for imageability and by extension, mental imagery. In support of this, participants with more control on their mental images as measured with Gordon Test of Visual Image Control (Gordon, 1949) relied on spatial simulations more frequently (i.e., looked more at nothing) than participants with worse mental imagery control when there was no visual cue between encoding and retrieval stages. The contribution of semantic variables (i.e., word imageability and concreteness) but not perceptual variables (i.e., word length) indicates that participants did not treat words as physical images.
Rather, they accessed word meanings and a semantic component modulated looking behaviour.
Such a behaviour highlights the tight links between spatial and conceptual representations (Boot & Pecher, 2011; Martarelli et al., 2017).
We attempted to specify the mechanism of location simulation in memory for language in Experiment 1 and 2. In particular, our evidence illuminated the conditions under which individuals tend to rely on location simulations and look at relevant, blank locations during memory retrieval. We conclude that there is a balanced trade-off between internal and external sources driven by cognitive differences between individuals and cognitive demands coming from the words to make the most of environmental opportunities and cognitive capacity.
In Experiment 3, we asked participants to read 1439 concrete and abstract nouns selected from a wide range of domains (e.g., temporal words such as “yesterday”, directional words such as “north”, emotional words such as “bewilderment” etc.). Then, participants were asked to associate words with locations on a precise, two-dimensional coordinate system. Thereby, we collected semantic coordinates for each word on the horizontal and vertical axes just as geographical coordinates in topographic environments in the real-world. For example, the word
“moon” was located on the centre of the horizontal scale as of 17 pixels and on the upward side of the vertical scale as of 419.83 pixels. As such, “agony” was located on the leftward and downward position at (-243, -413.83). Location ratings showed that most of the words were positioned on a diagonal line from bottom left to top right. Words that denote objects were
associated with locations where the objects typically occur in the real world. “Good” words with positive valence were associated with upward and rightward locations; whereas “bad”
words with negative valence were associated with downward and leftward locations. Ratings revealed that vertical space was more dominant than horizontal space. That is, participants tended use the whole range of the vertical scale but rated the words more conservatively on the horizontal scale. Importantly, there was a high degree of agreement between participants in spatial ratings. We conclude that mappings between words and locations are even more consistent and fine-grained than previously documented (Dunn et al., 2014; Estes et al., 2015;
Lachmair et al., 2011; Louwerse, 2008; Marmolejo-Ramos et al., 2013; Meteyard & Vigliocco, 2009). This is particularly interesting considering that language itself does not provide any rules or conventions as to spatial associations. In this respect, Experiment 3 results support a perceptual basis of language within grounded-embodied cognition (Barsalou, 1999;
Pulvermüller, 1999).
Experiment 4 demonstrated that words, which are associated with locations in space either perceptually or metaphorically, give rise to simulations of these locations upon their presentation. We evidenced that language-based reactivations of space have consequences for memory in relation with physical locations and their simulations. Words that were presented in locations incongruent to the locations they imply (e.g., “moon” in a downward location or
“agony” in an upward location) were remembered faster than words that were presented in congruent locations (e.g., “moon” in an upward location or “agony” in a downward location).
A visual cue shown at retrieval guided participants’ attention to the previous locations of the probe word and/or semantically implied location. Performance was better when the cue location did not overlap with the location of the language-based spatial simulation. Consider the word “moon”. “Moon” guides attention to the upward side of the screen because it is associated with an upward location in space as Experiment 3 indicated. Experiment 4 showed that participants’ recognition memory was more accurate when they saw a cue in the bottom
right quadrant, but it was slower when they saw a cue in the top left corner after encoding
“moon”.
Crucially, visual cues occurring in simulated locations deteriorated memory performance only with the modulation of imageability for words encoded in congruent positions. In other words, simulations invoked by high imageable words (but not low imageable words) interfered with visual cues if words were encoded in the congruent positions. Whereas, visual cues interfered with spatial simulations without any lexico-semantic modulation if the words were encoded in incongruent locations. Put differently, abstract words with spatial associations triggered strong spatial simulations and interfered with visual cues only when they were encoded in incongruent locations. We conclude that incongruency between physical and semantic space at the encoding stage produces an unexpectedness effect (Schmidt, 1991).
Unexpectedness due to spatial incongruency has two impacts on memory for language: (1) It deepens memory traces which results in better retrieval performance. (2) It strengthens language-based spatial simulations which elicits a more robust interference effect.
Spatial simulations examined in this thesis have connections to language; yet, they are of two different types: (1) Simulation of locations (in which words are perceived) in the absence of words (offline spatial simulation). (2) Simulation of space suggested by word meanings (online spatial simulation). We evidenced that online and offline simulations of space have consequences on memory for language even if space does not seem to be a part of the retrieval process. We conclude that spatial perception and spatial cognition underlie memory and language operations in robust and systematic ways. In this respect, experiments provide compelling evidence for a memory and a language conceptualisation that is grounded in the sensorimotor system and extended into the body and the environment.
7.2 Implications
The overarching theme of the present thesis is space and spatial simulation. Experiments connect to each other in that space is involved in cognitive processing in different ways. Thus, the overall implication of the current work is showing that simulations and particularly, simulations of space play key roles in the human cognition. Along with that, experiments reported in this thesis extend into different domains of cognition; in particular, memory, language, grounded-embodied and extended cognition. Hence, results have considerable implications for the architecture of the abovementioned domains and potentially, on more practical fields.
7.2.1 Implications for Grounded-Embodied and Extended Cognition
Our results can be thought as further steps towards more precise models of grounded-embodied and extended cognition in a broader sense. In four experiments, we specified the mechanisms of simulation (Experiment 1, 2, 3 & 4) and cognitive offloading (Experiment 1 & 2). We observed a flexible and intelligent coordination between internal and external sources across all experiments. In Experiment 1 and 2, participants moved from internal to external bits of information wherever the “cost of an operation” (Kirsh, 2010) is lower according to their own sources and task conditions. In Experiment 4, the combination of external information (i.e., physical location) and internal information (i.e., semantic location) affected memory performance. In this respect, our results demonstrate not only the existence but also the importance of mental representations within grounded operations. Results support a relatively