CAPÍTULO III 3 VARIABLES E HIPOTESIS
4.2. Diseño de la investigación
Quantitative and qualitative analysis revealed insights into the structure and functioning of Montana’s Chronic Disease Prevention and Health Promotion Bureau. It is tempting to look for causal relationships between the elements and themes identified in
the analysis. However, it is important to remember that this case study reflects a point in time and is designed to be descriptive in nature. These results suggest that relationships do exist between the elements of the conceptual model, but we cannot infer the exact causal nature of those relationships.
The results are presented in three sections and then synthesized. First, the documents that were included are described and analyzed. The next section reports the quantitative results from the interviews. The third section highlights the qualitative findings from the interviews. Finally, the last section of this chapter relates the comprehensive findings to the original research questions.
Document Review
Three documents were reviewed for this case study. They include the “Bureau of Chronic Disease Prevention and Health Promotion Organizational Chart,” the “Montana Chronic Disease Plan,” and the “Public Health Framework Assessment Tool (PHFAST)” results collected and reported by the Bureau.
Bureau of Chronic Disease Prevention and Control Organizational Chart
The organizational chart for the Bureau of Chronic Disease Prevention and Health Promotion (Appendix 7) is divided into three levels: Bureau Chief, Sections, and Programs (Figure 3). Section Chiefs report to the Bureau Chief and oversee three to four program areas. Programs generally include a program manager and an epidemiologist. They
sometimes also include administrative specialists, data specialists, prevention specialists, or communications specialists.
Figure 3: Basic Structure of the Bureau of Chronic Disease Prevention and Health Promotion
Leadership of the Bureau includes 1 bureau chief, 4 section supervisors, and 14 program managers. While administrative specialists often sit in a particular program, they generally function section-‐wide and sometimes function bureau-‐wide. Epidemiologists are also assigned to a particular program, but they identify also as part of a bureau-‐wide team. There are 57 total positions within the Bureau. There are 6 epidemiologists working in 8 positions (2 epidemiologists support 2 programs each) and there are 7 data/quality improvement/quality assurance specialists.
Program
Secron
Bureau
Bureau ChiefSecron Supervisor Program Manager
Montana Chronic Disease Plan
As a deliverable for a grant from CDC, Montana’s Chronic Disease Prevention and Health Promotion Bureau developed a document to guide statewide activities related to chronic disease prevention for the next five years called the “Montana Chronic Disease Plan.” The components of this plan include roles for state agencies and partners in all sectors. The foundation of this plan is the assertion that “program coordination will increase efficiency, reduce duplication of work, [and] expand and maximize the impact of program activities.”
The plan documents the rigor of a performance management system currently in development in Montana. The Division of Public Health Safety is using this system to
prepare for voluntary accreditation from the Public Health Accreditation Board. The Bureau of Chronic Disease Prevention and Health Promotion is the first organizational unit within the Division to go through this process and begin using the tools.
The Bureau anticipates that these two directives, performance management and chronic disease coordination, will result in improved organizational effectiveness. Per “The Montana Chronic Disease Plan,” these processes together:
“[provide] opportunities to work together, [promote] collective thinking and problem solving, and [support] working together in new ways so that impact of all chronic disease programs is improved.”
Specific elements of these activities are identified as:
“building the capacity of staff and stakeholders to effectively implement chronic disease activities; increasing chronic disease leadership in cross-‐cutting skill areas and leveraging shared services; enhancing collaborative processes that establish shared ownership and responsibility; development of a chronic disease
communication plan and the Montana Chronic Disease Plan.”
Public Health Framework Assessment Tool (PHFAST)
In May 2012, the Montana Chronic Disease Prevention and Control Branch used the Public Health Framework ASsessment Tool (PHFAST) to examine its organizational capacity and inform the work towards chronic disease coordination in response to the recent grant from CDC to all states enabling cross-‐category action for chronic disease prevention. This grant enables states to re-‐envision a system that has historically required that they manage several siloed categorical programs and move towards a system that allows for efficiencies and synergies that were previously impossible.
Although PHFAST was originally designed as a tool to generate discussion, the Bureau of Chronic Disease Prevention and Health Promotion made the decision to pilot it as a survey tool in order to gather as much input from as large a proportion of their staff as possible. This was done using SurveyMonkey, a web-‐based survey tool. The epidemiologist for the Coordinated Chronic Disease Program in the Chronic Disease Prevention and Health Promotion Branch analyzed the responses and reported them by staff category of the respondents. She used two staff categories: management team and
program/administrative staff. Indicators in eight framework domains were rated as: “not present,” “present – weak,” “present – adequate,” “present – strong,” or “don’t know.” These categorical ratings were assigned a number, one through four, that served to weight the responses. “Don’t know” responses were treated as missing. A mean response was calculated for each respondent for each element. The Coordinated Chronic Disease
Epidemiology and the Coordinated Chronic Disease Program Manager shared the summary data tables with me, including the qualitative comments regarding opportunities and
follow-‐up as collected by the survey tool. Qualitative comments were not identified by staff category in the document I received. I calculated the mean overall score. The mean score by staffing category had already been calculated by the epidemiologist.
The mean overall scores in each PHFAST element suggest that managers and staff are in agreement regarding the weakest and strongest elements, although the staff consistently rate each element higher than do the managers (Figure 4).
Figure 4: PHFAST Survey Results
In its traditional use, the PHFAST tool invites users to identify opportunities related to each indicator in each domain and prompts users to note issues or items for follow-‐up. Items for follow-‐up often include highlighting information to use in further assessment, identifying resources to assist capacity development in a particular area, researching an
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Program Coordinaron Program Management Evaluaron Intervenrons State Plans Partnerships Epidemiology & Surveillance Leadership
Overall Managers Staff
administrative policy or process, or assigning responsibility for a particular task. In the survey use, responses in the follow-‐up field seem to lean more towards challenges, which may be a function of the preceding question regarding opportunities. Interesting themes emerged in these responses (Table 8).
Table 8: PHFAST Opportunities and Follow-‐up Themes
Indicators Opportunities Follow-‐up
Leadership Improved communication will
yield improved leadership.
Program to program
communication, program to manager communication, Bureau to policy maker communication all need improvement.
Missed opportunities using new technologies and social media platforms to share information and increase effectiveness of
communications efforts.
Lack a unified voice. Epidemiology and
surveillance
Developing Bureau-‐wide journal articles and reports would raise visibility.
Excellent epidemiology and surveillance capacity.
Improvements in
dissemination and translation of data will benefit multiple audiences.
Invite communities and stakeholders to be partners in epidemiology.
Translation for wide audiences.
Partnerships Leverage initiatives in other parts of the agency to nurture partnerships.
Identify redundancy in requests to partners.
Key partners are not included.
Successful partner
relationships require more staffing than currently available.
More internal communication needed.
State plans Review of plans across categories would be useful.
Too many separate state plans – should be more coordinated.
Clear measures should be standard in all plans.
Plans should be shared with partners.
Indicators Opportunities Follow-‐up
Interventions Inconsistent levels of funding and staffing for intervention delivery.
Disparities not addressed in intervention planning.
Communication regarding the evidence supporting decisions should be shared more widely.
Translation of traditional public health approach into other sectors needs more support.
Evaluation Relevance of evaluation work day to day is unclear.
Intra-‐bureau dissemination and communication is very important.
Regular communication about evaluation findings is needed.
Internal and external audiences are important.
Program
management and administration
Uptake of new technologies is lacking.
New employee orientation specific to the Bureau would be beneficial.
Agency commitment to workforce development is unclear.
Coordination and integration training is necessary.
Program coordination
Leadership commitment is crucial.
Communication is crucial.
Balancing program specific duties with understanding Bureau wide activities is challenging.
Interviews -‐ Quantitative Findings
As described in the previous chapter, open-‐ended interview questions were paired with Likert scaled response categories. Not only did this serve to focus the discussion and aid in prompting more detail, it also allowed for quantitative analysis of each of the model elements.
Collaboration
Collaboration was assessed through five questions about the extent of collaboration and proficiency of collaboration. Respondents reported a much higher frequency of
whole (27% frequently or always). The vast majority of respondents (82%) reported collaboration with external partners as frequent or always. While most respondents considered their program’s collaborative ability to be at least somewhat strong (85%) and more than half (53%) rated collaborative ability of their program as very strong, perceptions of the collaborative ability of the Bureau varied (Figure 5).
Figure 5: Perception of the Bureau’s Collaborative Ability
Evidence-‐Based Decision-‐Making
Overall, Bureau staff clearly have a great deal of confidence in decision-‐making (Figure 6).
Figure 6: Confidence in Decision-‐Making
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Very strong Somewhat strong Somewhat weak Weak 0% 10% 20% 30% 40% 50% 60%
Not confident or Somewhat unconfident Somewhat confident Very confident
Goal Ambiguity
Respondents reported the greatest clarity in proximal, program goals. Nearly the same proportion of respondents rated both Bureau and Divisional goals as somewhat clear, although Bureau goals received slightly higher responses of both very clear and somewhat unclear/not clear (Figure 7).
Figure 7: Goal Ambiguity by Organizational Level
Political Support
Support within the Department of Public Health and Human Services for the work of the Bureau of Chronic Disease Prevention and Health Promotion as well as support for the Bureau from outside the Department are both considered part of political support. Internal support was characterized as stronger than external support (Figure 8).
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Program Goals Bureau Goals Division Goals
Somewhat unclear/Not clear Somewhat clear
Figure 8: Internal and External Political Support
Responsiveness
The Bureau’s ability to address emerging needs was rated somewhat strong by 59% of respondents and very strong by 21%. The Bureau’s ability to respond strategically was rated very strong or somewhat strong by 90% of respondents.
Workforce Competency
The Division of Public Health and Safety is planning an assessment of the entire workforce in the coming months. To prevent assessment fatigue, we substituted the planned comprehensive assessment with five questions regarding public health education, confidence in self-‐competency and bureau-‐wide competency, and access to professional
0% 10% 20% 30% 40% 50% 60%
Very weak Somewhat weak Somewhat strong Very Strong
Internal Support External Support
development. Particularly striking were the results regarding public health education experience within the Bureau (Figure 9).
Figure 9: Educational Background within the Bureau
Of the six Bureau employees who have an MPH, four are epidemiologists. An additional four Bureau employees have public health education other than an MPH. These four are all program managers. Other public health education experience reported included: health education, health promotion, community health management, and public health certificates. Even given the low frequency of public health education within the Bureau, most
respondents were very confident that the skills and knowledge needed to accomplish the work of the Bureau are indeed present (Figure 10).
Figure 10: Confidence in Bureau-‐wide Workforce Competency
0 5 10 15 20 25 30 Other public health MPH Not public health
Educaronal Background within Bureau
0 5 10 15 20 25 30 35 Somewhat unconfident Somewhat confident Very confident
Performance
Effectiveness ratings were high overall and highest at the program level, again increasing with proximity (Figure 11).
Figure 11: Effectiveness by Organizational Level
Correlation
Association of the quantitative variables was examined by chi-‐square analysis (Table 9).
0% 10% 20% 30% 40% 50% 60% 70% 80%
Somewhat effecrve Very Effecrve
Program effecrveness Bureau effecrveness Division Effecrveness
Table 9: Chi-‐Square Analysis of Conceptual Model Elements
Evidence-‐ based Decision-‐
making Ambiguity Goal
Political Support Responsiveness Workforce Competency Effectiveness Collaboration (df 16) 22.49 22.97* (df 12) 28.97* (df 16) 33.82* (df 16) (df 8) 8.77 (df 4) 1.6 Evidence-‐based Decision-‐making 36.10* (df 12) 16.44 (df 16) 32.38* (df 16) 9.91 (df 8) 6.86 (df 8) Goal Ambiguity (df 12) 13.64 (df 12) 15.97 (df 6) 7.48 (df 6) 9.29 Political Support (df 16) 8.48 (df 8) 7.17 (df 8) 5.41 Responsiveness 5.84 (df 8) 6.22 (df 8) Workforce Competency 2.54 (df 4) *Statistically significant ≥ 95% confidence
Workforce competency and effectiveness did not show a statistically significant
association with any of the model elements. Associations between collaboration and goal ambiguity, collaboration and political support, collaboration and responsiveness, evidence-‐ based decision-‐making and goal ambiguity, and evidence-‐based decision-‐making and responsiveness each were statistically significant with at least 95% confidence. The nature of these relationships is unknown. For each pair, we can expect a change in one element would be associated with a change in the other. However, whether it is a direct relationship or an indirect relationship and whether one change causes the other is not elucidated in this case study.
Interviews -‐ Qualitative Findings
These qualitative findings are a product of the open-‐ended interview questions. Study participants were asked a series of questions aimed at documenting their opinions about
the performance of the Bureau of Chronic Disease Prevention and Health Promotion in each element of the conceptual model.
Collaboration
Collaboration was examined through a series of five questions (Appendix 4). Aspects of the Bureau that were reported as facilitating collaboration included culture and a
willingness to pitch in to help each other, open and regular communication, physical
proximity to each other, and leadership at the Section, Bureau, and Division levels. External respondents noted specifically that the Bureau’s practice of approaching collaboration purposefully and strategically helped these partners understand how they could contribute, and trust that their time would not be wasted. The Division’s new integrated performance management system was also identified as supporting collaboration. This system is
organized around the work being done and not around organizational structure. Collaboration is influenced both negatively and positively by funder involvement, especially in the case of CDC. CDC’s National Center for Chronic Disease Prevention and Health Promotion is working to encourage coordinated chronic disease prevention and health promotion within state health departments. Montana is leveraging CDC’s grant for coordinated chronic disease prevention into support for collaborative approaches within the Bureau. However, guidance from CDC’s categorical programs is sometimes at odds with broader collaboration and rather than encourage a coordinated approach, it reinforces existing position bias related to the categorical silo.
Collaboration was valued as a method to improve effectiveness. Respondents noted that collaboration had a positive impact on workforce development and mentoring. The collaborative culture of the Bureau was credited with encouraging individuals to seek out advisers and peer-‐to-‐peer learning opportunities. This was true for learning new skills or quick information sharing as well as longer term mentoring for new managers. This has resulted in efficiency in data and information sharing that minimized the need for collaborators to “reinvent the wheel.”
Identifying an appropriate balance between specialization and shared tasks and expertise emerged as an important component of successful collaboration. Some specialist groups meet across program areas. This allows each individual to develop specialized knowledge in a program area and share skill-‐based knowledge across programs. Program-‐ based finance analysts meet regularly as a Bureau-‐wide group. Epidemiologists meet regularly in the Bureau and regularly but less often across the Division. Some respondents expressed a desire to engage other specialist groups across programs such as health promotion specialists or communications specialists.
Evidence-‐Based Decision-‐Making
Evidence-‐based decision-‐making was explored through three open ended questions (Appendix 4). Nearly all respondents reported that within the Bureau, there is a clear expectation of evidence-‐based decision-‐making. Per Bureau culture and practice, the typical decision-‐making framework includes documenting needs and pairing them with programs, processes, and interventions that have the strongest likelihood of success.
Sources of evidence cited include peer-‐reviewed literature, Cochran Reviews, the Guide to Community Preventive Services, the US Preventive Services Task Force recommendations, and guidance or direction from federal agencies including CDC, NIH, and SAMSHA. Other sources of input include coalition and stakeholder recommendations, surveillance data, evaluation data, and performance forecasting. Logic models were identified as a tool to assist in evidence-‐based decision-‐making.
Translation was identified as a sometimes difficult component of evidence-‐based public health practice. This seems to be true when interventions exist but have been created for demographics that vary substantially from Montana, or when interventions do not yet exist. While traditional public health literature is well employed, literature and expertise from fields such as sociology and communications are not often used.
Positive influences on evidence-‐based decision-‐making included a culture of
accountability, clear expectations, open and empowered leadership, weighing alternatives against program and Bureau goals, and frequent communication. Senior leadership of the Bureau and the Division began their careers in the programs within the Bureau. This seems to have imbued the leadership with a certain level of implicit expertise, resulting in
confidence from the staff that decisions are based in solid evidence.
Goal Ambiguity
Goal ambiguity was investigated through three open-‐ended questions (Appendix 4). Most respondents rated proximal goals as having more clarity than distal goals. While they
stated that their immediate program goals were very clear, the objectives became less obvious at the Bureau level, and even less apparent at the Divisional level (Figure 12).
Figure 12: Goal Ambiguity by Structural Level
Characteristics contributing to goal clarity included funder instructions defining goals, training offered by the Coordinated Chronic Disease Program, the Public Health
Accreditation Board accreditation preparation process, program maturity, coordination, and communication. Several respondents noted that for CDC-‐funded programs, the CDC-‐
mandated work plan was more pertinent to their daily work than organizational goals at any level. Many respondents were not sure that organizational goal clarity at the program, Bureau or Division level was important; they felt they were able to successfully complete their job tasks without such clarification.
“Sometimes I’m not filled in until a decision is made and don’t really know why something is happening the way it is. But do I need to know why? Probably not.”