5. PROPUESTA
5.05 Metodología
5.07.01 ELABORACION DEL MANUAL DE FUNCIONES
Evaluation plays several within the program, organization, and community. Within the bounds of the program, the outcomes will be measured in order to gauge the effectiveness of the
program intervention and highlight areas for improving the program. Identification of areas in need of improvement will strengthen communication between the program staff, clinicians, and administrators in order to build accountability and promote efficient implementation of the program. Furthermore, the goals of continuing program improvement and program dissemination will motivate program staff to conduct an evaluation for the purpose of strengthening implementation and program outcomes.
The Piedmont Health Collaborative Care program community and organizational
contexts reinforce the need for a program evaluation. First, the program goals and interventions are consistent with Piedmont Health organizational goals for reduction of the number of
individuals with a hemoglobin A1c greater than 9 and the improvement of the mental health care provided within their clinics. Second, the Latino community possesses an interest in the
evaluation because it will shed light about the effectiveness of the program intervention within a community health center that cares for underserved Latinos.
Evaluation will strengthen understanding of the sustainability needs of the program. For example, evaluation will highlight the need for additional funding and support to improve the sustainability of the program during a time of scarce resources that is magnified in a community health center. Moreover, the diverse administrative, participant, and provider concerns will be more clearly articulated in the evaluation process, promote stakeholder buy-in, and therefore promote sustainability.
I will serve as an evaluator and facilitator of the evaluative efforts of those involved in the program. This role will involve bringing evaluation questions to the team for ongoing feedback, facilitating communication about perceived problems, and anticipating sustainability needs.
Limited funding prevents employment of an external program evaluator. As a program staff member and part of the planning team, I will bring historical knowledge about the formation of the program and the numerous strengths and weakness to the current structure of the team. Internal and external evaluations provide unique strengths to the development and improvement of the program. For example, the involvement of organizations such as the UNC Center for Excellence in Community Mental Health and Community Care of North Carolina will bring an external collective evaluation knowledge and experience to the evaluation. In addition, an external evaluator without institutional or organizational ties may bring a less biased
perspective about the program dynamics than an internal evaluator. On the other hand, the historical knowledge and understanding of program development, planning, and
implementation, possessed by the internal evaluator will require time for an external evaluator to develop. Both types of evaluators will need an understanding of the program rationale, literature base, and unique clinic structure, such as Federally Qualified Health Center regulations. Finally, an evaluator needs to possess sensitivity to the competing and overlapping interdisciplinary and institutional interests in order to promote a shared constructive dialogue among stakeholders. Ideally, the clinic providers, care managers, office managers, consulting psychiatrist, corporate managers, policy makers, and participants will be involved in the evaluation. The administrators and providers will most likely want to know how the program improved outcomes and whether or not the magnitude of improvements require a manageable amount of time, funding, and effort. Office managers will likely prioritize whether the program coincides with the existing flow of the clinic or if the program creates disruption for staff or participants. Policy- makers will likely value whether the program health outcomes, program costs, and potential savings are merit dissemination. Participants will most likely value whether or not the program improves their health at a manageable financial and time cost. As the program grows, the local mental health providers could also become involved in providing feedback about referral from the program to the public mental health system.
Finally, Several challenges for useful program evaluation exist. For example, the numerous competing priorities faced by stakeholders may result in insufficient time for a meaningful evaluation product.
Evaluation Study Design
The evaluation design is quasi-experimental and performs prospective and longitudinal comparisons between two groups. The participant group will be compared to the eligible participants who choose not to enroll. Depression severity, hemoglobin A1c, quality of life scores, and total health care costs are unbounded outcomes and will be gathered before and after the intervention.
There are several factors that limit using a quasi-experimental design for the evaluation. First, because the search for eligible patients in the EMR will only occur prior to the program, patients who establish care with the clinic after the program starts will be excluded. This will limit comparison of the participant’s hemoglobin A1c values measured after the program to the values of all of the patients with a hemoglobin A1c greater than 9 that are followed by the pilot provider. Second, participant variables, such as improved compliance with medication
schedules, improved diet, and improved medical illness, will be potential confounders to consider when interpreting changes in hemoglobin A1c levels that will be measured after the program. Third, a lack of access to Medicare claims will limit reporting of health costs and saving for individuals with Medicare. Finally, selection bias will also be a concern since the eligible unenrolled group actually receives depression screening and comes to the attention of the provider. An alternative comparison group could be individuals with a hemoglobin A1c greater than 9 and seen by another provider within the clinic who will not implement systematic depression screening.
Implementation Evaluation. Fidelity to the program model will face challenges such as competing priorities within the clinic and the need for efficient delivery of care under financial and time constraints. For example, the program calls for weekly clinical supervision between the care manager, consulting psychiatrist, and primary care provider (PCP). Program staff will prioritize this collaboration as a regularly scheduled activity.
The program evaluation topics for discussion during weekly meetings will be ongoing logistical, staff, and funding issues. In addition, conformity to the treatment algorithm must be monitored to increase the likelihood of achieving targeted outcome measures.
Chart reviews will be conducted to monitor the team’s compliance with the treatment algorithm and the goals for a timely response to changes needed in the treatment plan. In addition, implementation evaluation will incorporate estimations of care manager and consulting psychiatrist time commitments to the program to promote development of accurate sustainability estimates. Finally, because the program staffing brings together individuals from diverse
organizations, implementation evaluation potentially gauges provider, administrator, and participant buy-in.
Outcome Evaluation. Improvement or resolution of depressive symptoms represents the primary program outcome. Additional outcome measures will include hemoglobin A1c level, quality of life, and health care costs. Program staff will track depression severity, hemoglobin A1c, and medical costs using existing health information technology within the Piedmont Health Services and CCNC data systems. The current Piedmont Health electronic medical record allows for inclusion of the PHQ-9 to measure depression severity and quality of life measure into an automated system that can be queried in the future for evaluation purposes.
Evaluation Methods
The Piedmont Health Collaborative Care program evaluation will use both qualitative and quantitative methods.
Qualitative methods will include observation of interactions between the care manager and the participant by the consulting psychiatrist. Individual interviews of program staff will be conducted on an ongoing basis to solicit feedback about how the program implementation can be improved. The care manager and the consulting psychiatrist will track the time spent in the planning process and in weekly reviews of the clinical record. Open-ended interviews will take place on an ongoing basis between the care manager, consulting psychiatrist, and PCP to address participant care and program logistics issues. The care manager and the consulting psychiatrist will discuss challenges facing the program on a weekly basis. The PCP and care manager will meet multiple times a week to discuss clinical care and program logistics. The program team will meet monthly.
Quantitative methods will be used in the evaluation process as well. An implementation fidelity tool will be used to monitor fulfillment of his or her implementation responsibilities and barriers to executing these responsibilities. Clinical data systems from North Carolina Medicaid and the Piedmont Health system will be used to document the intervention, monitor health care service use, and health care costs. Documentation of the primary intervention outcomes and the dates of participant contact with the care manager or the consulting psychiatrist will occur in a Microsoft Excel spreadsheet. Spreadsheets for active and eligible participants will be kept separate to monitor the effectiveness of screening effort.
Additional methods may be utilized as well. For example, an online anonymous survey represents an available tool for confidential feedback about program progress from the program
staff. Web-based video and telephone conferencing are technologies that could improve the efficiency of program staff communication and ensure that ongoing evaluation actually occurs.
Finally, the inclusion of stakeholder and administrative dialogue may prove challenging and secondary to time constraints, but will be key to expanding the program and ensuring sustainability.
Logic Model
Resources Activities Outputs Short &
Long-term Outcomes Impact -Care manager -Psychiatrist -Time available -Space for patient visits -Data tracking -Screening tool -Access to cost data -Printing capabilities -Relationship with local public mental health system -Cultural competency input -Consumer input -Literature source - Reimbursement method (fee- for-service)
-Intervention training for care manager
-Care manager and psychiatrist meet to plan program
-Care manager and psychiatrist develop complete needs
assessment
-Care manager and psychiatrist generate questions for
providers and administrators -Care manager and psychiatrist meet with IT for medical record changes for intervention
-Care manager and psychiatrist meet with local mental health providers
-Use the PHQ-9 depression screening tool
-Use the QOL scale
-Use educational materials for patients
- Care Estimate potential number of participants by searching the EMR.
-Care manager records data about program participant outcome tracking.
-Care manager and consulting psychiatrist complete Medicaid provider portal training to track cost data for participants with Medicaid. -Staff completes intervention training. -Weekly care manager/psychiatrist meeting to review participant progress -Weekly care manager/psychiatrist meeting to plan program -Monthly program planning team meeting (clinic leaders, Piedmont, UNC, CCNC) -Initial PHQ-9 to qualified participants YEARLY -Face-to-face or telephone contact with participants WEEKLY and within 2 days of positive screen -Change treatment plan if improvement ≤ 50% reduction in PHQ-9 score -Care manager and consulting
psychiatrist review cost data every month. Short term: -Reduced depression (lower PHQ-9 score) -Improved Diabetes care (lower Hemoglobin A1C) -Decreased medical costs Long term: -Improved care manager comfort with addressing depression -Improved PCP comfort with treating depression -Expand collaborative care model to other Piedmont Health Sites -Reduced depression -Improved Diabetes care -Improved quality of life -Improved integration of mental health and primary care Healthier participants .
Evaluation Planning Tables Short Term objectives:
Short Term Objective 1: By December 2011, 50% of participants will experience a 50% or greater reduction in the severity of depressive symptoms as measured by the PHQ-9. Evaluation Question Participant Evaluation Method Did 50% of participants
experience a 50% or greater reduction in the severity of depression symptoms?
Project coordinator Review of patient tracking information; pre and post PHQ-9.
Did primary care providers (PCP) experience greater support for depression care from the care manager and consulting psychiatrist?
Primary care providers (PCP)
Project coordinator
Electronic survey Open-ended interview
Did care managers experience greater confidence in educating patients and providing support for depression care?
Care managers Project coordinator
Electronic survey Open-ended interview
Did participants feel the program involved affordable and accessible
interventions?
Participants
Project coordinator
Paper survey
Did the care manager and consulting psychiatrist follow the depression treatment algorithm?
Project coordinator Care manager
Consulting psychiatrist
Chart review
In what ways can depression care be improved? PCP Care managers Consulting Psychiatrist Open-ended interview
Did the care manager feel training and support from the consulting psychiatrist was effective?
Care managers
Consulting psychiatrist
Short Term Objective 2: By May 2012, 70% of participants will experience a one-point reduction in their hemoglobin A1c.
Evaluation Question Participant Evaluation Method Did 70% of participants
experience a one-point reduction in their
hemoglobin A1c (hgb A1C) by May 2012?
Project coordinator Review of patient tracking information; pre and post hgb A1c
Did participants receive referral for diabetes- supported care such as nutrition counseling,
medication adherence, and foot care?
Project coordinator Care manager PCP
Chart review
Did participants receive education about the relationship between depression and diabetes?
Project coordinator Care manager PCP
Chart review
Did care managers feel confident in educating patients about diabetes and depression in addition to navigating referral for diabetes supports?
Care manager Project coordinator
Open-ended interview
How can the diabetes management for participants be improved? Care manager Project coordinator PCP Participant Open-ended interview
Short Term Objective 3: By May 2012, 70% of participants will experience a 0.5% decrease in total healthcare costs
Evaluation Question Participant Evaluation Method Did 70% of participants
experience a 0.5%
decrease in total healthcare costs?
Project coordinator Review of cost data
What kinds of health costs were decreased?
Project coordinator Review of cost data What kinds of costs were
included or excluded for this estimate?
Project coordinator Review of cost data
Did participants feel the program was costly?
Participants
Project coordinator
Open-ended interview How can we improve the
cost effectiveness of the program?
Project coordinator Piedmont corporate management
Site business offices participants
Open-ended interview
Long Term Objectives:
Long Term Objective 1: By December 2012, 80% of eligible participants within the entire clinic are screened for depression.
Evaluation Question Participant Evaluation Method Did 80% of eligible
participants within the clinic undergo screening for depression?
Project coordinator Review of patient tracking information
Was sufficient time
available to care managers for screening? Project coordinator Care manager Open-ended interview Did PCPs receive communication regarding screening results and participant progress in the program? Care manager Project coordinator PCP Open-ended interview Did participants in
maintenance phase receive a yearly screening?
Care manager Review of patient tracking information
How can the fidelity of screening and communication of results be improved? Care manager Project coordinator Open-ended interview
Long Term Objective 2: By May 2013, the care manager is able to sustain 50% time on program activities.
Evaluation Question Participant Evaluation Method Is the care manager able to
sustain 50% time on program activities? Care manager Piedmont corporate leadership Open-ended interview
What other priorities compete for the care manager’s time? Care manager Piedmont corporate leadership Project coordinator Open-ended interview
What are the sustainability needs of the care manager and consulting psychiatrist?
Care manager Piedmont corporate leadership
Project coordinator
Open-ended interview
How can the time
management and funding of the care manager position be improved? Care manager Piedmont Corporate leadership Project coordinator Open-ended interview
Long Term Objective 3: By December 2013, the consulting psychiatrist is able to sustain 40% time on program activities.
Evaluation Question Participant Evaluation Method By December 2013, did the
consulting psychiatrist sustain 40% time on program activities? What other priorities compete for the consulting psychiatrist's time? Consulting psychiatrist Piedmont corporate leadership Center of Excellence leadership Project coordinator Open-ended interview
How secure are the current sources of position funding? Consulting psychiatrist Center of Excellence leadership Piedmont corporate leadership Project coordinator Open-ended interview
How can the time and funding of the consulting psychiatrist position be improved? Consulting psychiatrist Piedmont Corporate leadership Center of Excellence leadership Project coordinator Open-ended interview
Dissemination Plan
Dissemination of the Piedmont Health Collaborative Care program evaluation will play several roles. First, the evaluation will inform stakeholders about the program. To effectively disseminate the evaluation, evaluators must carefully consider translation of evaluation methods and results into a meaningful language for stakeholders. The care manager and the consulting psychiatrist will serve as the project staff primarily responsible for presenting this information for site administration and for directing supervisors. Together, this group will develop the
evaluation presentation format for various stakeholders. In addition, evaluation of pilot data could be incorporated into a grant proposal for project expansion.
The information conveyed in the disseminated evaluation will include several key items. First, progress toward the primary program outcomes; improvement in depression severity, reduction in hemoglobin A1c, improvement in quality of life, and reduction of medical costs,; primary outcomes for program effectiveness. The audience for the dissemination will be the Piedmont Health Services leadership, the UNC Center for Excellence in Community Mental Health, and CCNC. These stakeholders will participate in ongoing program meetings and give direct feedback on implementation and opportunities for improvement.
Outcomes, logistical, staffing, and funding issues are the primary discussion points at these meetings. These issues will be presented in a Microsoft Word document format. Meeting minutes will be sent to stakeholders in order to clarify the major points, identify potential
changes to the program, and outline issues for further discussion. We set a goal of monthly meetings in which interim reports occur. A yearly report will occur as well. These reports will allow the program staff to identify areas for improvement such as staff training, scheduling, and patient education and adherence.
Participants, local mental health providers, and Latino health advocate groups possess interest in the program evaluation. Participants will be given updates in document format on
their progress within the program.
Program staff will meet with local Latino mental health providers on a quarterly basis to solicit feedback. Furthermore, the program context builds upon existing collaborative depression care management in primary care literature and may be of interest to those interested in the intervention or those serving minority or immigrant populations. Additional opportunities for dissemination include presentation of materials at professional meetings, CCNC network meetings, and publication in peer-reviewed journals.
Discussion
The Piedmont Health Collaborative Care program aims to improve the depression care for individuals living with poorly controlled diabetes. The program tries to reduce the specific problems with disease severity, everyday life, and high health care costs faced by individuals with depression and diabetes (1).
The presence of mental health conditions worsens the effects of other illnesses (2,3,4). For individuals with diabetes, the addition of a mental illness leads much higher health care costs (2,3). Furthermore, individuals with diabetes and major depression, compared to those