Overall, the Panel found it challenging to characterize many health product risk communication tools because of a lack of readily available information summarizing and describing tools in use. In many cases, the information presented in Tables 3.1, 3.2, and 3.3 was deduced through reading several examples of the tool in question.8 Furthermore, in some cases, regulator documents reference an internal evaluation, but the Panel could not find any
7 The recall information released publicly by the MHRA (or any national regulator in Europe) is often supplied by the EMA.
8 Given that this exercise was not meant to be exhaustive, there are health product risk communication tools that exist in the different jurisdictions examined, which the Panel chose not to include due to space constraints or because information was difficult to locate. It is also possible that the information described under tool characteristics may need updating or revision as new information becomes available.
concrete information on the nature of the study, timeframe, methods, or results. As this report is based on publicly available information, these evaluations could not be included in the summary tables.
Although the Panel identified a range of established risk communication tools, several shared similar features. Many tools, for example, were primarily text- based with few visuals and sparse colour. While some tools used images, these were generally illustrations or pictures rather than graphic risk presentations. The most common method of dissemination was through posting online (with the notable exception of leaflets), although some of the tools aimed at healthcare professionals were also disseminated by other methods such as mail. Occasionally, tools consisted of an addition or change to another established tool, such as a black box on a leaflet. Most of the tools that targeted the public often did not quantify risk, instead using terms such as “increased risk,” “rare,” or “chance of.” Detailed information about risk was generally available in some of the comprehensive ongoing communication documents (e.g., summary basis of decision documents), but these documents were also longer and written in more technical language. Many of these observations, however, do not align with the evidence-informed communication practices outlined in Chapter 2 and Table 2.6.
For evaluation to be undertaken successfully, the goals of a risk communication need to be defined. Although the desired populations that a communication is intended to reach might vary, the goal of sending the communication in a timely and appropriate manner was constant in the identified evaluations. Generally, evaluations used individual interviews, surveys, standardized tests, or expert analysis to mostly measure the readability or clarity of a given tool. Importantly, these evaluations only provided evidence on the content of a communication tool and did not evaluate if it achieved broader goals such as whether it was understood and incorporated into the behaviours and decision-making of the targeted receivers of information.
In most cases of the evaluations reviewed by the Panel, the desired goals of the communication tools could be characterized simply as either informing select groups, changing their behaviour (and possibly seeking out more information), or both. For instance, a public recall has the goal of changing behaviour (i.e., stop using a product) while a product insert has the goal of informing patients about the risks and benefits associated with a medication. Some evaluations also explored broader goals such as enhancing the health of Canadians. The outcome evaluations that examined use and impact most often used medical or pharmacy data (e.g., prescribing rates, new users) as
indicators. There was no apparent evaluation of larger overarching goals related to relationship-building such as establishing trust with the public, empowerment, or becoming the go-to source for health information.
Overall, the Panel found little publicly available and publicly conducted evaluation results on the effectiveness of health product risk communication tools or template documents. Of the evaluations identified, the majority related to tools used by the FDA. For example, Dusetzina et al. (2012) identified 49 outcome evaluations of FDA dear doctor letters, public advisories, or black box warnings over a period of 20 years. However, most of these evaluations were done by independent researchers rather than the FDA and it is not clear how the regulator is applying the evaluation results. It seems that regulators have either not evaluated the effectiveness of their health product risk communication tools or used the results of external evaluations, and in any case have not made these results public or easily accessible. This gap could have implications for the quality of risk communication.
Apart from the evaluation of specific health product risk communication tools, Health Canada has released some broader evaluations of programs with a range of communication activities, which demonstrate some learning (see Box 3.2). It has also discussed plans for specific evaluation and public engagement activities related to risk communication, but the results of these activities are either not yet completed or have not been released. For example, public consultations and public surveys have been identified as methods to engage receivers of information and other stakeholders and to “collect important data to inform the Department of the effectiveness of current methods used to communicate drug safety information and to identify areas where the Department needs to make improvements” (Health Canada, 2011a). The specific results of these activities, however, do not appear to be public and the Panel could only find limited evidence of similar surveys conducted between 2003 and 2007 that indicated a number of opportunities for improved awareness among the public and health professionals (Evaluation Directorate, 2014c).
Box 3.2.
Broader Health Canada Program Evaluations
Although there are few targeted evaluations of specific Health Canada communication tools, the regulator has included communication elements in the evaluations of broader health product programs including those relating to biologics, medical devices, and human drugs. These recent evaluations include assessment of all program activities (i.e., all ongoing, incident(s), and defect related tools were included in addition to all other Health Canada activities) and a range of outcome variables related to “relevance and performance (effectiveness, efficiency, and economy).” The studies were carried out by an external evaluation firm using literature, document (government) and administrative data reviews, case studies, surveys of industry and other stakeholders (patients, healthcare professionals), focus groups with manufacturers, and key informant interviews. These studies illustrate a comprehensive approach to reviewing regulatory activities.
With respect to communication-related findings, the program evaluations note that in the past decade Health Canada has implemented a number of new initiatives to improve communication, openness, and engagement. These initiatives include the publication of summary basis of decision documents, public consultations on proposed policies, and improvements to the MedEffect™ website (e.g., adding an advanced search option). While the evaluation states that Health Canada’s activities are expected to “lead stakeholders…to adopt safe behaviours” and “produce increased awareness and understanding…of risks and benefits,” it did not determine whether communication tools were actually achieving these outcomes or other communication goals related to development, reach, use, or impact. The evaluation also found no evidence of other evaluations that draw these types of conclusions, apart from a small number of limited public opinion surveys.
In all of these more comprehensive program evaluation activities, it was noted that the “vast scope and complexity of the subject matter” was a challenge for document review as was locating all relevant internal documents, and that there were problems with very low survey response rates and difficulties in organizing interviews. These limitations highlight the opportunity for Health Canada to carry out more targeted evaluations of specific health product tools, which would better scope the subject matter, better explore communication specific goals, and better engage more specialized evaluation participants.
Health Canada has also stated that it is working with experts in knowledge translation to help evaluate the effectiveness of its public advisory template in terms of “reducing the health literacy burden on Canadians,” carrying out readability tests on its public advisories template (see Section 3.2.2), and collecting data to establish whether prescribing patterns are influenced by dear healthcare professional letters (Health Canada, 2011a). The results of the evaluation on doctor letters have not yet been released. Finally, the Panel is aware that Health Canada is currently planning an evaluation of the effectiveness of risk communications. However, no results of this evaluation have been made public and no description of the types of activities that are/ would be included were available at the time of report publication (Evaluation Directorate, 2014a, 2014b, 2014c).
Ultimately, a health agency has a responsibility to the citizens it protects. The Panel identified clear instances where Health Canada has demonstrated its recognition of the importance of openness and evaluation (e.g., plain language initiative, public communication documents to coincide with communication documents for healthcare professionals, release of safety reviews, readability evaluations). These exemplars signify that Health Canada could position itself as a leader in the evaluation of health product risk communication. However, these evaluations must be publicly available if any agency wants to establish a long-term commitment to building and establishing trust with the people it intends to reach. The benefits, process, and challenges associated with evaluation are discussed further in Chapter 4.
3.3 EMERGING TOOLS FOR HEALTH PRODUCT