This study has several limitations. First, most of the findings did not reach significance relative to the standard alpha level of .05, and thus should be interpreted with caution.
Replicating this study with a larger sample size may improve confidence in the interpretations based on these results.
Use of a relatively homogenous sample also limits the generalizability of these findings.
This study relied on a convenience sample of college students who are younger and generally more educated than the broader population. Undergraduates may be more accustomed to processing subtextual messages, have greater cognitive capacity, and they could also be more prone to engaging in multitasking during viewing. Any one of these factors could have had unique effects on the narrative processing strategies they chose to use. As such, these findings may not be applicable to other groups.
The possibility of self-selection bias also interferes with the ability of these findings to be generalized. Students were offered course credit in exchange for participation, and only students
who completed the first survey on time and showed up for the lab session were included in the final sample. Accordingly,
Furthermore, because the student participants in this study were not regular viewers of Numb3rs, or the target audience for the show, their reactions may not reflect the responses of more typical viewers. One reason the “Harvest” episode of Numb3rs was selected for the current study was because Morgan et al. (2009) found evidence that exposure to the episode had an effect on a number of persuasive outcomes. However, notably, the average age of the sample for their study was slightly older (29), and survey data were collected online, in fan websites. Hence, compared to participants in the current study, respondents to Morgan et al.’s survey were most likely regular, intended viewers of Numb3rs, who enjoyed the television show enough to visit the fan website. For this reason, it stands to reason that, compared to participants in the current study, they were probably more familiar with and favorably disposed to the characters, and perhaps even more amenable to the episode’s persuasive message. These notable differences in sample characteristics could explain why “Harvest” had persuasive effects in one study but not the other. They also highlight the importance of studying differences in how fans and new audiences process entertainment content in the future.
This research is also limited in its ability to shed light on narrative persuasion because, by and large, the audience appeared to already be supportive of organ donation. Over half of the sample for this study had already registered as organ donors. This may explain why, with the exception of belief in an organ black market in the U.S., exposure to the episode did not have any effect on pro-donation persuasive outcomes. A large proportion of the sample appears to have already been convinced of the importance of organ donation. As Dal Cin et al. (2004) point out, stories’ primary persuasive asset is their ability to camouflage extreme arguments so that
audiences do not recognize their extremity. Research that investigates narrative persuasion effects in response to more polarizing subtexts may yield more robust results.
The study’s external validity was also compromised by conducting the experiment in a tightly controlled, computer lab classroom setting. Participants watched the episode on a computer using headphones, and they were unable to engage in other activities, adjust viewing settings on their computer, or interact with others. These measures helped to maintain
experimental control, but also created a relatively artificial viewing situation for participants.
Therefore the findings from this study may not generalize to more natural viewing situations.
Outside of the lab setting, for instance, people have the opportunity to make themselves more comfortable while viewing entertainment. Also, it is not uncommon for viewers to engage in many different tasks during viewing, such as talking to friends, surfing the Internet, or cooking dinner. Any number of activities that happen when people consume entertainment media in more natural settings could affect how they attend to and receive persuasive messages embedded in narratives. For instance, dialogue between audience members could potentially help viewers members tap into different subtextual messages, or it could distract viewers from the intended message. For this reason, research on narrative persuasion processing would benefit greatly from naturalistic studies that account for different message-relevant and irrelevant distractions that occur when people consume stories.
The results of this study also may not generalize well to other types of television shows or films. One strength of this study is that it examined audience processing of a real, professionally designed, entertainment-education program. However, as discussed previously, different shows and different genres may prompt different types of processing strategies. More research is needed to understand how well the current findings apply to other entertainment education
programs.
One strength of this study is that it employed a thought-listing technique to gauge participants’ narrative processing strategies. This measurement permitted an analysis of authentic, cognitively accessible thoughts viewers had while watching a persuasive television episode. However, to some extent, the thought-listing measurements of narrative and subtext processing also limit this study’s ability to draw concrete conclusions about psychological processing in narrative persuasion. Participants’ ability to report thoughts and feelings they had while viewing a television program that lasted over 40 minutes may be limited. As such, the reported thoughts may provide a better indicator of participants’ most recent thoughts, rather than their most frequent thoughts. The use of both thought-listing tasks as well as close-ended scales that ask participants to assess how frequently they found themselves thinking about different aspects of a story may paint a clearer picture of how prominent different processing strategies were during viewing. But care should be taken to ensure that these indexes are precise enough to detect the nuances in the actual processing strategies audiences use while they are engaged with a story. This is particularly true of measurements of subtext processing that could potentially be confused with measures of narrative processing (Moyer-Gusé & Nabi, 2010).
Finally, the focus on trait motivation, as opposed to a state measure of processing strategy limits this study’s ability to advance claims related to how motivation affects narrative
processing. The weak or nonsignificant associations among eudaimonic motivation, processing measures, and persuasive outcomes suggest that dispositional entertainment use motivations may not be the best indicators of how people process entertainment media. Participants’ overall eudaimonic disposition may affect media selection, but does not necessarily determine whether they adopt eudaimonic processing strategies (e.g., scrutinizing the subtext; relating the
persuasive message to real-life scenarios) while they are actually engaging with entertainment content. The development and use of a reliable eudaimonic processing scale would permit a more precise investigation into how entertainment use motivation at the time of viewing affects narrative processing outcomes.