In order to create a perfect lifelog, a hypothetical solution might simply take a mobile device and add all necessary (even more hypothetical) sensors for a perfect, around-the- clock tracking. However, most of lacking aspects of ideal listening histories are far too complex to be captured with a sensor (e.g., the social surroundings and their current implications), or not available for sensing (e.g., the listener’s emotional situation). Yet, even without such a perfect, unattainable sensing device, at least some of the missing context can be restored through other sources: One common dimension of all lifelogging data is time and this attribute also allows merging data from various sources. This section contains possible data sources for enriching listening histories from three areas:
Episodic memory
One powerful data source that sometimes allows for explaining otherwise cryptic listen- ing decisions is the listener’s memory. Lifelogs can only serve as extensions to biological memory as they lack data and contain mostly low-level information (e.g., blurry photos, non-descript audio recordings, single songs). In order to make sense of this data and tell the stories that happened at that time, the owner of the lifelog is needed (for this section cf. [13]).
It is interesting to compare the attributes of lifelogs and biological memory: While the former contains perfect representations of data with little significance, the latter stores vague renditions of events condensed to their main aspects (if needed, details can al- ways be re-imagined on-the-fly as so-calledfalse memories[141]). The former also allows direct access to any data based on its time stamp, while the latter struggles with exact dates ("What did you have for dinner last Tuesday?"[13]) but provides rich, content- based associations between items.
One main attribute of human memory is that its store of autobiographical events is or- ganized episodically [162]. Such episodes can be accessed using memory triggers that can be of very different nature and based on all sensory impressions: from visual cues, via melodies, voices, to smells or tastes (Proust’s narrator inÁ la recherche du temps perdu
starts his seven-volume narration based on the memories triggered by the taste of a madeleine). Human memory, however, is also highly efficient storage-wise and only keeps information that it deems relevant at the time. Therefore, even providing "right" memory triggers might not cause a recall, simply because the person’s mind at the time decided to ignore this fact. A way to solve that is to store as much information as possi- ble electronically to be able to provide a wealth of possible triggers.
A second aspect that is important regarding episodic memory is the so-calledgeneration effect[82]: People are better at remembering items created by themselves (written text, photos), than ones they only consumed (read text, seen photos) and this even works (partially) with automatically captured ones (Sellen et al. showed this in a study about this aspect with Sensecam pictures [148]). Therefore, episodic memory triggers that al- low understanding a lifelog should be as varied as possible and created by the owner of the lifelog, if possible.
Local data
Various types of data are created regularly by people using computers (electronic com- munication, written pieces) or stored on them (photos, images) and can be made avail- able for enriching lifelogs. And there is also data that is actually used for organizational purposes but also contains context, most prominently calendars.
Lifelog applications can simply rely on this data as memory triggers. Photosare one of the most powerful sources for not only associations but also context. Oftentimes, peo- ple only have to see a glimpse of a photo to remember the surrounding event and their feelings at the time. Also, the function to take pictures is by now included in almost any portable device from cell phones to media players, so that the problem of creating
photos has long been displaced by the problem of finding and organizing them.
Another important source are calendar entries. While they are mainly used as prospec- tive devices for remembering and planning, in hindsight they also become parts of and enrichments for lifelogs. Depending on the details in an event description they can con- tain information about current tasks, problems and also social surroundings.
Similarly, (locally stored) electronic communication such as emails or instant messaging can also work as a backdrop for episodic memory. Highlighting relevant topics of a certain time [166] or the most important contacts can make other decisions easier to un- derstand.
In the context of listening histories, actual representations of themusic heard are valu- able to interpret the logs. Even complex textual descriptions with extensive meta-data cannot capture the richness of music, so simply playing back a certain piece from a his- tory is often the easiest and most expressive way to help the owner remember.
Online services
Not every aspect of one’s life is kept on the personal harddrive and more and more tasks and tools are transferred to online counterparts. Photos are moved to flickr and Face- book, and calendar entries and electronic communication to Google Calendar or Mail, or Facebook. The trend towards cloud-based computing promises to rather sooner than later move every type of personal data to online databases coupled with easy synchro- nization and access. The downsides regarding network failures and privacy only show little effect on the growth rates of online services.
In addition to transferring existing data to the cloud, online services also capture other, formerly non-existent data: Status updatesin social networks such as Facebook, Twitter, or Google+ make the idea of a static "About me" representation dynamic. Facebook’s ("What’s on your mind?") and Twitter’s ("What’s happening?") titles for their status boxes both aim at learning about the current context in an informal way. In connection with a lifelog owner’s memories such status updates can be very helpful in understand- ing.
Another automatically or semi-automatically collected source of expressive data arelo- cation histories. Services for smartphones such as foursquare32 or Gowalla33 capture (manual) "check-ins" to different venues. Automatic versions such as Google’s Lati- tude34take location snapshots in regular intervals. Location data is helpful in retrieving specific information ("Where have I gone for lunch yesterday") but also allows deriv- ing context: Specific venues stand for specific activities, and such information can be as helpful as manually-created calendar events.
Finally, music listening histories can be enriched with all the metadata available online. The fact that songs in such a history are uniquely identifiable allows adding informa-
32http://foursquare.com 33http://gowalla.com
tion about its content (using an analysis service such as The Echo Nest35), its current trendiness (provided by last.fm and also Echo Nest), or other information such as the biographical background of the artists (via Musicbrainz36 or Wikipedia).
3.6
Summary
In this chapter, I discussed the concept of lifelogging and music listening histories as one concrete instance of this idea. Lifelogging projects are currently focussed on the problems of capturing and storing the data, while accessing it is not yet central. After- wards, I presented the attributes of an ideal listening history and put it in contrast to its real-world implementations that suffer from lower complexity and noise and gaps. I then gave details on a study that we performed and where we analyzed 310 long-term listening histories in detail. The results showed that seasons and personal taste were very important for listening decisions and that a surprising level of descriptive detail can be reached by calculating aspects and enriching the data with additional meta-data. In the last section of the chapter, I discussed what real-world listening histories lack and how to make up for it using additional data sources.
35http://the.echonest.com 36http://musicbrainz.org
Chapter
4
Visualizing Listening Histories
This is one for the good days
And I have it all here
In red, blue, green
Red, blue, green.
– Radiohead -Videotape–
Music listening histories are a large and complex type of data. As we have seen in the previous chapter, an average history contains thousands of unique tracks and can span several years. Providing listeners with simple lists of songs and time stamps as last.fm does might not be enough to allow understanding underlying patterns.
To provide this understanding, having asummarizingapproach to presentation (cf. last chapter 3.1.2) for the listening history can unearth relevant patterns that were not obvi- ous before. Simply calculating charts or trends from the data shows the most favorite artists, genres and songs and allows matching the taste between friends (last.fm presents this comparison in taste as a simple, one-dimensional musical compatibility bar, but at least explains which artists overlap in both profiles). Yet, summarizing approaches al- ways suffer from the inherent abstraction in their presentation. Digging deeper and trying to understand how a certain result came about (e.g., why a musical compatibility value is so low) is usually not possible - the source and explanation for all such summa- rizing is the black box of statistics.
Yet, there is no way around summarizing, as having direct access to all items equally works only up to a point. With thousands of songs and artists accessing each one man- ually is not feasible and does not contribute to seeing a bigger picture.
One methodology for solving this problem of understanding and exploring is interac- tive information visualization. Information visualization techniques rely on the massive
parallelism of the human optical system to allow for a more effective transmission of in- formation. While text-based displays of information force attention to hop from item to item, visualization displays all relevant information at once and allows a self-chosen path of exploring the data. Combined with interaction capabilities to filter for relevant items or adjust the display based on some chosen parameter (e.g., sort by an attribute), visualizations are powerful tools for exploring and understanding.
With all the strengths of visualizations comes also a certain complexity. Being able to use a visualization tool in detail and knowing about all its aspects can require dedicated training. Additionally, depending on the data being visualized, understanding all traits of the data might even need a solid background in statistics. So, while visualization is a valuable tool for making data graspable, it can be overwhelming for non-visualization- experts. The imposing rows of buttons and interface widgets of standard tools like Tableau1 can already signal to the uninitiated that this software was off limits, just as the resulting graphics and textual output. Yet, the owners and producers of lifelogs are usually also the most skilled experts on the data, so their analysis can produce the most insightful results.
A solution to this dilemma between complex software and complex data is casual infor- mation visualization [133]. The complexity of a visualization interface can be reduced by focusing on important or interesting questions instead of trying to support every query imaginable (and additionally allowing for discovery of patterns in the data that one was unaware of before). Pousman et al. explain the difference between traditional infovis and casual infovis: "In core infovis, a system should have a tool-like ability to do work to display data, uncover trends and outliers, and generate hypotheses. Casual In- fovis systems are useful artifacts that are helpful for providing representations of data, but without a clear task focus" [ibid]. While Pousman et al. also included more abstract cases such as artistic visualizations (that often completely ignore the exploration and are very explicit in their statements) into their casual terminology, I argue that even ca- sual visualizations can produce serious insight. An aesthetically pleasing appearance and a presentation that emphasizes interesting bits about the owner’s data can entice even non-infovis-experts to interact with a visualization tool. Once this initial stum- bling block is cleared, people are ready to dive deeper into the analysis and play with their data, given the tool remains friendly and understandable. Finally, relying on ex- isting and known interface metaphors and keeping the interface itself as simple and non-threatening as possible can allow even non-experts to analyze their complex lifelog information.
In this chapter, I will first present the previous work on the visualization of listening histories. These fall into two categories:summarizingvariants that present an abstracted overview andsingle-purpose versions that succinctly describe one interesting aspect of the data without allowing much exploration. The main part of the chapter is taken up with the discussion of the design space of visualizations for music listening histories and its main design dimensions and additional aspects that are useful to keep in mind.
4.1
Related Work - Visualizations for Listening Histo-
ries
The convenient availability of last.fm listening histories through their API made their data a popular target for simple visualizations and statistical tools. Some of the profile pages are embroidered with various widgets and applications that stand as proof for the enthusiasm of their owners and their adorement of their musical idols. They are often used for putting an explicit emphasis on a certain fact about one’s listening behavior (e.g., ’Top-10 listener of Justin Bieber’).
Accordingly, most visualizations that are produced are single-purpose or an a very ab- stract level. They are good enough for a short aha moment, but are not used beyond that. Usually, these small visualizations do not allow exploration and are not even inter- active. The approach is that not all tasks are supported in a single tool but spread across the landscape of available last.fm visualizations.
Almost all of the visualizations available for listening histories have been written by fans or companies. Only rarely is research involved with this type of data, even though visualization of lifelogging data can be expected to be relevant in the near future. Right now, all this data is collected without suitable ways for accessing it.
Visualizations for listening histories can be separated into summarizing and single- purpose approaches.
Figure 4.1Three examples for summarizing listening history visualizations: (a) bar charts for top artists, (b) streamgraph, (c) histogram for scrobbles (Source: last.fm).