Over the millennia, the human brain has evolved to excel in collecting (percep- tion) and processing information (cognition) in an incredibly efficient manner. Essentially, the way our brain has evolved and has been structured is what makes us so different from all other animal species, many of which have significantly larger brains with even higher number of synapses[207]. For the human brain to continue evolving, certain organs may have to become bigger, including the hu- man brain per se. For example, for processing more information, wider synapses are needed, resulting in a demand for greater in-brain blood flow and in turn a larger heart. However, even if the human brain and its supporting organs and networks will eventually grow in size, further evolution will certainly face lim- its imposed by laws of Physics, as well as diminishing efficiency after a certain threshold of brain size increase[112]. Nevertheless, these changes cannot occur naturally and in a timely manner for satisfying the frenetically increasing de- mands in information processing of the today’s era. In fact, lifelogging, could be considered a plausible culprit for aggravating information overload in a future uptake, simply due to the utter volume of data it entails[64].
Evidently, the way people collect and process information has always been in- fluenced by technology. For example, the introduction of stone-headed javelins during hunting would have greatly altered decision making and strategic think- ing of prehistoric humans. Similarly, the introduction of smartphones has greatly influenced the way we seek and consume information, highly disrupting the en- tire spectrum of our cognitive processes (i.e., attention, memory, learning, de- cision making, problem solving, etc.). Albeit technology so far had a beneficial role on how we perceive and process the world around us, in recent years it has undertaken a rather disruptive and double-edged role. The era of ubiqui- tous technologies and the Internet of Things (IoT) finds our brains unprepared for handling the sheer volume of information produced daily by a multitude of
48 2.2 Lifelogging
sources. Attention deficit disorders, the multi-tasking illusion, learning difficul- ties, sleep deprivation, weak memory, chronic stress, and others, are just a few examples of the negative side effects that modern technologies impose on ev- eryday life. As modern technologies become increasingly pervasive and even addictive [174], while blurring the line between personal and professional life, such negative side effects are expected to further exacerbate. In response, some companies limit access to technology after a certain hour (e.g., the "right to dis-
connect"), while individuals decide to abstain from using smart devices or social media for a period of time or even completely. But why should one have to resort to such practices after all? Is technology per se not meant to help us improve our quality of life, and eventually realize our full potential as human beings?
Several endeavours promise to bring closer together the human mind with technology, in what has been named "Human-Machine Confluence", essential the vision in which the human brain converges with the machine[81]. An EU project under this title has attempted to showcase that the concept may be viable in the future, identifying a set of research challenges 10 years ago in which very few advancements have happened since then. In the US, the BRAIN initiative11 re-
ceived initial funding of approximately $110 million from the Defence Advanced Research Projects Agency (DARPA), the National Institutes of Health (NIH), and the National Science Foundation (NSF). The EU Human Brain project12, involv-
ing researchers from over 100 institutions, received funding over one billion Euros, together with criticism from Europe’s leading neuroscientists. More re- cently, SpaceX and Tesla CEO Elon Musk has joined the BCI venture with a newly founded company called Neuralink13. This company is centred on creating im-
plantable interfaces in the human brain, with the eventual purpose of helping human beings merge with software for a true human-machine symbiosis. Face- book unveiled a project on a BCI that could also be used by patients with severe paralysis. This will be a system that allows one to type even faster than with one’s physical hands, at upwards of 100 words per minute.
We believe that modern technologies hold the potential to greatly amplify hu- man cognition in the entirety of its spectrum, seizing in a way the role of natural evolution[205]. In this work, we argue that the negative effects observed by in- creased technological use are simply side effects of our inability to keep up with the pace in which technology advances. We attribute this phenomenon to the fact that despite the overall technological proliferation, the devices and systems (i.e., machine side) we use daily, still remain oblivious of our cognitive and affec-
11https://www.braininitiative.nih.gov 12https://www.humanbrainproject.eu 13https://neuralink.com
49 2.2 Lifelogging
tive states, assuming always the maximum of our cognitive capacities. We name this discrepancy "cognitive gap", and we theorize that an intermediate software architecture (see Chapter 10), between human and machine could bridge this gap, essentially paving the way towards human-machine convergence[170].
Chapter 3
Augmenting and Measuring Memory
Recall
In this chapter, we present the methodology we follow for achieving the afore- mentioned thesis goals. In principle, we design and develop memory interven- tions in the form of mobile and desktop applications that are later deployed on participants’ devices. Participants are usually volunteers recruited via Univer- sity newsletters (and advertisements) and agree to go through the entire study design for testing a memory intervention. Each intervention, aims at testing a different aspect of augmenting one’s memory recall. Often, participants are also invited to our lab for additional tests. So far, we have conducted (or participated in conducting) a set of 6 deployments with a total of 199 participants, in field studies that lasted from one hour up to 5 weeks. In the following sections, we first present our contextual information sources for generating memory cues, fol- lowed by a section on cued recall, the primary psychological theory behind most of the trials for augmenting one’s memory with technology. Next, we describe the apparatus (i.e., equipment) we have used for data collection, intervention delivery, and feedback acquisition. Then, we outline our evaluation methodol- ogy, where we present the methods we have used for evaluating the effectiveness of our interventions throughout this work. Finally, we present a brief overview of the statistical analyses we typically perform for assessing the effectiveness of the interventions tested in each deployment.
3.1
Memory Cues
Given the great effectiveness of pictures (and video) in aiding memory recall [48], we focus on visual information as the central means for tailoring "primary
52 3.1 Memory Cues
Method: Cued Recall
Human Memory Memory Cues Actual Experience 7Figure 3.1. Contextual information sources that produce cues for supporting one’s ability to recall a past experience or prior knowledge.
memory cues", with video and images comprising the "primary contextual informa-
tion sources". Nevertheless, additional contextual information sources may pro- duce "secondary memory cues" that can either enhance visual (i.e., primary) cues or independently be used as memory cues (see Figure 3.1). Typical secondary memory cues are location, time and social context (i.e., co-presence), though each comes with its own strengths and weaknesses. For example, location cues have been found to support recall by eliciting patterns of behaviour (e.g., "Here I was
on my way to work and hence I was feeling stressed")[125]. However, location cues need to vary significantly (e.g., work and home) for supporting recall without the aid of additional cue information [243]. Time, on the other hand, is central to episodic memory, as episodic memories are clustered together in episodes and placed in a temporal order when being registered in memory [48].
Visual information as captured via pictures and videos often include people’s co-presence and interactions in the form of social context. Social context cues, in the form of people we encounter, are often assumed effective memory cues for triggering autobiographical memories. In fact, an analysis of which elements within SenseCam pictures provide the best triggers for memory recall showed that people in pictures were often associated with vivid recollections [144]. In