TIPOS DE DATOS

Simulation modeling is often a participatory and iterative process. Subject matter experts are involved throughout the model development process, from conceptualization of the system to validating that the model represents the actual system. As initial, simple models expand to include more detail in order to explore more outcomes or to assess a new set of policy questions, the modeling process is repeated.

This iterative process is outlined in Figure 3-1, below.

Figure 3-1: The model development process is iterative, and a single model may require many cycles through the process.

The model used to assess the research questions in this thesis is the fourth iteration of a modeling process focused on addressing the question of buprenorphine capacity. Early model iterations included simple simulated geographies and were presented at SAMSHA’s 2014 Buprenorphine Summit (iteration 1: SAMHSA 2014c), at the American Association for the Treatment of Opioid Dependence (AATOD) annual conference in 2015 (iteration 2: Alexandra Nielsen, 2015), and submitted to an

academic journal (iteration 3). In the third iteration, model geography was based on a real map, and the scope of the model expanded to include: 1) a greater number of relevant outcomes, 2) more detail on patients seeking treatment, 3) people not seeking treatment as agents, 4) methadone treatment offered at Opioid Treatment Programs (OTPs), 5) children and pregnant women, and 6) more policy options. Model revision followed each dissemination activity in response to reviewer comments and critique. In

the fourth modeling iteration, the model was streamlined and several outcomes and populations were removed. For example, because child buprenorphine poisonings are a significant public health concern, children and this outcome were included in iteration 3, to increase applicability, but removed in iteration 4, to address more targeted

dissertation research questions.

Each step of the modeling process detailed in Figure 3-1 is detailed below:

3.1.1 Conceptualize the system

I sought understanding of the target system to be modeled through extensive literature search and through interviews with experts in OB buprenorphine treatment.

Prior to initial modeling, I interviewed two addiction medicine specialists who had experience developing the eight-hour training course for DATA 2000 waiver

certification, Dr. Margaret Kotz, and Dr. Stephen Wyatt (Kraus et al., 2011). In order to involve a broader base of stakeholders, I empaneled a diverse group of experts to guide model conceptualization and specification. Expert panel members consulted during model development for iterations 2 through 4 are listed below. I interviewed each panel member at least twice using an unstructured format, and asked specific questions via email as they arose in the modeling process. Dr. Alane O’Connor and Dr. Andrew Saxon were also consulted to assess iteration 4 face validity.

3.1.1.1 Expert Panel Members

• Dr. Todd Korthuis—Oregon Health and Science University, General internal medicine, HIV research program director—Primary Care Provider in Oregon

• Timothy Lepak—president National Alliance of Advocates for Buprenorphine Treatment—Patient Advocate

• Melvania Briggs, PA-C—Academic coordinator for the Duke University Physician Assistant Program. Co-Principal Investigator of a NIDA funded Buprenorphine Clinical Trial for a community based mental health organization and Duke University Medical Center—Physician Assistant in North Carolina

• Dr. Kelly Clark—Chief Medical Officer of CleanSlate Addiction Treatment Centers, president of the American Society of Addiction Medicine (ASAM)—

Addiction Medicine specialty provider in Kentucky and Pennsylvania

• Dr. Andrew Saxon—Center of Excellence in Substance Abuse Treatment and Education, VA Puget Sound Health Care System, Director Addiction Psychiatry Residency Program, University of Washington, associated with the American Association of Addiction Psychiatrists (AAAP)—Addiction Psychiatry provider in Washington

• Dr. Alane B. O’Connor, DNP, FNP—Maine Dartmouth family Medicine Residency Faculty Adjunct Instructor, author of NP buprenorphine health

policy article, investigator of clinical trials of buprenorphine use in pregnancy—Family Practice Nurse Practitioner in Maine

3.1.2 Build a model

The operational model was developed in NetLogo 6.0.2, an ABS tool. I used the pattern oriented modeling technique (Grimm & Railsback, 2012), in which a model is developed to fit several quantitative and qualitative patterns. The Overview, Design concepts, and Details (ODD) protocol (Grimm et al., 2010) was employed to convert the understanding of the system into an operational model. The model is documented in Section 3.2 using the ODD protocol, which is the gold standard for model reporting and replicability.

3.1.3 Add data to the model

Modeling best practice requires that all aspects of the system that are

considered important in model conceptualization be included in an operational model whether or not high quality data are available to support them (Caro, Briggs, Siebert, &

Kuntz, 2012). When possible, I obtained empirical support for model parameters from peer-reviewed published literature through both comprehensive and targeted literature searches. I obtained empirical support for model parameters for which high quality data were not available from smaller studies that are not necessarily generalizable, “grey”

literature, expert opinion, and through the process of model calibration.

3.1.4 Test the Model

For models to serve as credible proxies for the target system, models should be calibrated, verified, and validated to the degree possible. The model should also be tested for sensitivity to specific parameter values and critical assumptions about model structure. I calibrated the iteration 4 model manually to reproduce historical trends in opioid overdose deaths and the number of unique buprenorphine recipients for a given year. I conducted model verification—testing that the model performs as expected—

throughout the iterative modeling process, as model code was developed and as new data added to the model. The model was assessed for face validity by two panel experts, Dr. O’Connor and Dr. Saxon, and externally validated by comparing model-generated annual opioid overdose deaths and number of unique buprenorphine recipients against data that was not used in model calibration. I conducted one-way sensitivity testing by varying each model parameter by 30%. I tested the model’s sensitivity to the geography selected by conducting baseline model runs with 10 different maps. I tested sensitivity to assumptions about provider preferences for patient loads by running the model under several alternative assumptions about providers’ patient load preferences.

In general, model testing built confidence that the model is a sufficiently faithful representation of the world to be able to consider the results of policy experiments to be meaningful. Calibration can show that a model is capable of generating known behavior, verification shows that the operational model matches the conceptual model developed with subject matter experts, and validation can show that the model is

capable of generating real world behavior in a way that conforms to experts’

understanding of the target system. Sensitivity testing builds confidence by showing the degree to which model performance is dependent on assumptions and on parameter values which may or may not have strong theoretical or empirical support. While none of these tests can prove that a model is “right,” they do build confidence that a model may be useful for its intended purpose.

Model testing specifics and results are reported in Section 3.4.

3.1.5 Run Experiments

I conducted policy experiments by systematically changing the values of policy variables and observing model outcome values after a year of modeled time. Because the model contains several random variables, each experiment was repeated 35 times to establish the possible range of outcomes. One-way ANOVA power analysis indicated that with 35 replications, one could detect medium to large effects (f = 0.30), with an alpha of 0.05, beta of 0.1, and 5 variable levels. Fewer replications would risk low power, or inability to detect differences in outcome variables that are present in simulation data. Policy experiments and results are detailed further in Section 3.7 and Section 5.

I also conducted spatial potential access metric exploration experiments using alternative formulations of the aggregated spatial disparity metric to assess the relative

sensitivity and usefulness of each formulation. The process is further detailed in Section 3.6, results detailed in Section 4.