EVALUATION OF THE ILLINOIS ADOPTION PRESERVATION AND LINKAGES PROGRAM (APAL) USING A REGRESSION DISCONTINUITY DESIGN
Abstract
This study evaluated the impact of the Illinois Adoption Preservation and Linkages (APAL) program on post-adoption and guardianship families using a regression discontinuity design. APAL is a needs assessment and service referral program designed to prevent adjustment difficulties and foster care reentry, or post-permanency discontinuity, for adolescents residing in legally permanent adoptive or guardianship homes. The purpose of this study was to examine whether APAL participation was associated with two outcomes considered proximal to discontinuity. Specifically, it was hypothesized that APAL would be related to fewer child behavior problems and higher caregiver commitment to permanency. Results showed that APAL participation was associated with fewer child behavior problems, but findings related to caregiver commitment were inconclusive. Results suggest implications for intervention design, practice, and future research with post-adoption or guardianship families.
Evaluation of the Illinois Adoption Preservation and Linkages (APAL) Program Using a Regression Discontinuity Design
The number of children in foster care in the United States has decreased from a historical high of over 500,000 children in the mid-1990’s to about 407,000 in 2011 (Testa, 2004; USDHHS, 2011). This decrease in the number of children in care can be at least partially attributed to changes in child welfare policy and practice over the past several decades that have led to increases in both adoptions and legal guardianships of foster youth (Hartinger-Saunders, Trouteaud, & Matos-Johnson, 2014; Nalavany, Ryan, Howard, & Smith, 2008; Simmel, Barth, & Brooks, 2007; Testa, 2004). For example, the number of children adopted from public child welfare agencies rose from about 36,000 in 1998 to approximately 52,000 in 2012 (Annie E. Casey Foundation, 2014; USDHHS, 1998), and from 2003 to 2012 the percentage of exits from foster care due to adoption increased from 18% to 21% (USDHHS, 2013). Similarly, from 2003 to 2012, the percentage of exits from foster care due to guardianship increased from 4% to 7% (USDHHS, 2013).
Federal child welfare policy has increasingly provided directives and incentives for child welfare agencies to expedite permanency for foster youth through adoption and guardianship when reunification is not possible (Allen & Bissell, 2004; Testa, 2004). For instance, the Adoption and Safe Families Act of 1997 (ASFA) specified timelines for terminating parental rights, provided exceptions to the requirement that child welfare
agencies show “reasonable efforts” to reunify foster youth with their biological parents prior to pursuing adoption, and legitimized guardianship as a valid permanency goal for foster youth (Allen & Bissell, 2004; Golden & Macomber, 2009). More recently, the Fostering Connections to Success and Increasing Adoptions Act of 2008 provided incentives for states
to find adoptive homes for children with special needs (e.g., older or disabled youth), created more opportunities for adoption assistance for children with special needs, and expanded the availability of subsidized guardianship payments for relatives (Children’s Defense Fund, 2008).
The increase in foster youth adoptions and guardianships over the past two decades is generally a positive development for child-welfare involved youth. However, even after adoptions and guardianships are legally finalized, some former foster youth still experience placement instability. Estimates for rates of foster care reentry after adoption or guardianship, or discontinuity (Testa et al., 2014), range from about 2% to 15%, with higher risks for certain at-risk groups, such as adolescent youth or youth with mental health or behavior problems (Barth, Berry, Yoshikami, Goodfield, & Carson, 2001; Barth & Miller, 2000; Berry, Propp, & Martens, 2007; Festinger, 2002; Hartinger-Saunders et al., 2014; Henry, 1999; Koh & Testa, 2011; McDonald, Propp, & Murphy, 2001; Selwyn, Wijedasa, & Meakings, 2014; Testa, 2004). Although the risk for discontinuity is much lower than child welfare scholars feared after the passage of ASFA, it is also much higher than the risk of foster care entry for the general population, which is about .34% (USDHHS, 2011).
Post-permanency discontinuity is generally considered to be a negative child welfare outcome, because adoptive and guardianship families are screened and carefully vetted by child welfare agencies and courts prior to legal finalization. In addition, placement instability has been associated with numerous negative outcomes for foster youth, including behavior problems, mental health issues, and poor educational achievement (Bruskas, 2008; Bruskas, 2010; D’Andrade, 2005; Newton, Litrownik, & Landsverk, 2000; Rubin, O’Reilly, Luan, &
foster youth, who have already experienced traumatic experiences related to child abuse or neglect, and research shows that adverse childhood experiences (ACE’s) are associated with poor adult outcomes (Anda et al., 2006; Brown et al., 2009). For instance, people who report four or more ACE’s are between four and 12 times more likely to experience alcoholism, drug abuse, depression and suicide in adulthood as compared to adults who report no ACE’s (Felitti et al., 1998).
Therefore, post-permanency interventions are needed to support families after legal finalization to prevent poor family adjustment and discontinuity. A limited number of peer- reviewed studies have examined the impact of post-adoption interventions on child or family outcomes and generally found positive results. For example, Berry and colleagues (2007) showed that post-adoption families who participated in Intensive In-Home Services (IIS) to address child behavior problems were more likely to be intact at 12 months follow-up, and that the number of days that families received IIS services was positively related to family intactness at 12 months. IIS services were provided to families with children at-risk for out- of-home placement within 72 hours, and services included intensive case management, family assessment and engagement, parenting training, and assistance to meet concrete material needs.
Similarly, in a qualitative study that assessed the impact of intensive adoption preservation services, Zosky and colleagues (2005) showed that post-adoption intervention helped parents better understand their children’s behaviors and obtain services to help decrease children’s behavior problems. Also, Belanger, Cheung, and Cordova (2012)
employed mixed methods to examine the impact of flexible caseworker services on outcomes of rural African-American adoptive families and concluded that caseworker services were
essential for stable adoptions. Finally, Liao and Testa (2014) examined the impact of APAL on child and family permanency and well-being outcomes using the same data set this study examines, but with an instrumental variables design. The authors found that APAL was associated with less child behavior problems, higher caregiver commitment, and lower odds of placement discontinuity. Therefore, previous adoption studies generally suggest that flexible, family-centered post-permanency services provided by child welfare agencies after legal finalization have positive effects on child and family outcomes.
Method Intervention Description
APAL is a post-permanency needs assessment and service referral program
developed by the Illinois Department of Child and Family Services (IDCFS) and the Child and Family Research Center at the University of Illinois. The intervention was designed to prevent adjustment difficulties and discontinuity for adolescent children placed in legally permanent adoptive or guardianship homes (Koh & Rolock, 2010). APAL services were delivered via phone contact or home visits, and consisted of two components: (1) a brief caseworker assessment of child and family needs and (2) caseworker referrals to post- adoption services.
Liao (2014) provides a detailed description of the APAL program, but in general, IDCFS contracted with three private agencies in Illinois to provide APAL, and each APAL worker carried a caseload of between 25 to 40 families. Families were first contacted by letter to attempt to schedule a home visit to complete the APAL instrument. If families did not make contact with APAL agencies in response to the letter, efforts were then made to
the APAL assessment with caregivers during home visits, but the assessment could also be completed by phone if necessary. APAL started on October 1, 2007 and services were provided for about a year, until program funding was discontinued by IDCFS (Liao, 2014). APAL is not a manualized intervention, but provides a stark contrast to post-permanency services as usual (SAU), in which there is typically no personal contact at all between child welfare caseworkers and families after legal finalization of an adoption or guardianship.
Research Question and Hypotheses
The research question of interest in this study is whether participation in APAL has a significant impact on children’s behavior problems or caregivers’ commitment to
permanency. Therefore, it was hypothesized that, compared to routine post-permanency services-as-usual (i.e., SAU), APAL would be associated with:
(1) Less behavior problems of adopted or guardianship youth;
(2) Increased caregiver commitment to youth in adoptive or guardianship placements.
Study Design
Participants. The sample for this study was comprised of 437 former foster youth ages 12 to 17 years old who resided in adoptive homes in Illinois. The youths’ caregivers were surveyed by the Illinois Department of Child and Family Services in 2008 as part of the second round of a post-permanency survey (Round 2) undertaken to assess family outcomes after adoption or guardianship. The population from which the Round 2 survey was drawn consisted of primary caretakers providing care for 4,155 foster children who (1) were taken into adoption or guardianship between July 1997 and June 2004 and resided in the Chicago area, (2) had an active subsidy case between October 2007 and September 2008, and (3) had ever been assigned to the Illinois title IV-E Subsidized Guardianship Waiver Demonstration.
Six months after the APAL intervention was implemented, a stratified random sample of 670 households from the population was drawn for the Round 2 survey. Specifically, 335 households were randomly chosen as the intervention group from those families assigned to the APAL intervention, and 335 households were randomly selected as the comparison group from those families who were not assigned to the APAL program. In cases where a family had more than one target child, the child with the earliest case opening date was selected as the focal child for both the APAL intervention and the Round 2 interview. Just 439 of the 670 randomly selected cases for the Round 2 survey consented to link their survey responses to administrative data, and two cases had to be dropped because survey data did not match foster care records, leaving a total sample of 437 households (a response rate of
approximately 65%).
Questions in the post-permanency survey included items regarding caregiver and child characteristics, family relationships and social support, and caregiver thoughts about the permanent placement. As noted above, caregivers were interviewed by phone or in person to complete the surveys. Administrative data regarding child characteristics and placement history were then obtained from the IDCFS Integrated Database and linked to the survey data.
Sampling weights were included in the post-permanency dataset to account for sampling differences across six strata. For this sample, cases fell into one of six sampling strata according to whether they were assigned to APAL intervention or SAU, whether they were assigned to the subsidized guardianship experimental or comparison condition, and whether they were adoption or guardianship placements. Sampling weights were included in
descriptive and outcome analyses shown below to approximate results for the full Round 2 post-permanency survey population.
Regression discontinuity. This study used a regression discontinuity (RD) design to estimate the effects of the APAL intervention on two proximal outcomes related to post- permanency discontinuity, child behavior problems and caregiver commitment. Although widely used in economics, RD has received less attention and application in social work and other social sciences (Cook, 2008). However, the design has potential to be used in many social work applications because, under particular conditions, the design allows the estimation of treatment effects that are comparable to those obtained using randomized experiments, with weaker assumptions than those required in typical observational studies (Shadish, 2011).
The RD design may be applied in any situation where participants are assigned to treatment conditions on the basis of an assignment score or scores that reflect constructs such as merit, need, or age (Thomas, Lemieux, Rhodes, & Vlosky, 2011). In RD, assignment to treatment conditions (i.e., treatment versus control) is completely or partially determined by whether the value of a predictor variable is smaller than, or equal to or larger than, a fixed “cutoff” value. The assignment variable may or may not be correlated with the outcome, but if a correlation exists, the assignment variable must change smoothly with respect to the outcome so that any discontinuity at the cutoff may be interpreted as a treatment effect (Imbens & Lemieux, 2007).
Causal inference in RD. According to the Neyman-Rubin framework, participants in a study are selected into treatment or comparison groups, but they also have potential
they been selected into the alternative treatment condition (Guo & Fraser, 2010; Neyman, 1923; Rubin, 1974; Rubin 1986). The fundamental problem of causal inference (Holland, 1986) is that only one state for each group is observable. It is impossible to observe individual-level causal effects. But in a randomized experiment, estimation of an unbiased group-level or average treatment effect (ATE) is theoretically trivial because the probability of assignment to treatment is equal for all participants, and thus, randomization creates groups that are statistically equivalent, on average, in regard to baseline characteristics that may cause differences in outcomes (Shadish, Cook, & Campbell, 2002). In contrast, estimation of an unbiased ATE is problematic in observational studies because the probability of treatment assignment is unknown, and thus, characteristics other than treatment may differentially affect outcomes for treatment and control groups.
The RD design is unique among observational studies in that, treatment assignment is based on a cutoff score for an assignment variable, and thus, the probability of receiving or at least being offered treatment is known (Shadish, et al., 2002). Because participants in the neighborhood of a cutoff on either side are assumed to be similar on all characteristics other than treatment assignment, the RD design can be seen as creating local randomization around the cutoff (Imbens & Lemieux, 2007). Under this assumption, treatment participants just above the cutoff provide the counterfactual for those below the cutoff and vice versa. However, one drawback of RD is that the treatment effect estimated applies locally, to the neighborhood of the cutoff, rather than globally. This average treatment effect at the cutoff (ATEC) is limited in that extrapolation beyond the neighborhood of the cutoff requires stronger assumptions, such as constant treatment effect (DeGiorgi, 2005; Shadish, 2011).
Fuzzy or sharp discontinuity. When the probability of assignment to treatment jumps from 0 to 1 or from 1 to 0 at a cutoff value, the treatment assignment mechanism is completely known, and the design is said to be sharp regression discontinuity (SRD; Bloom, 2009). However, an estimation of the ATEC is also valid in fuzzy regression discontinuity (FRD) designs, where the probability of receiving treatment jumps by less than 1 (Lee & Lemieux, 2010). This allows the application of the RD design to situations where there is a stochastic component to treatment assignment near the cutoff. However, an important
assumption for FRD is that participants have no more than imprecise control of the receipt of treatment. If participants have complete control, and can thus, self-select into conditions, the RD design is not valid (Lee & Lemieux, 2010).
The ATEC estimated in a FRD can only be said to apply to “compliers” in the study (Imbens & Lemieux, 2007), or those who would receive treatment if assigned to treatment and would not receive treatment if assigned to control (Angrist, Imbens, & Rubin, 1996). The ATEC for compliers may be estimated as the ratio of two discontinuities at the cutoff: the discontinuity in the outcome variable over the discontinuity in the probability of treatment (Lee & Lemieux, 2010). Alternatively, a two-stage least squares (2SLS) procedure may be implemented, in which treatment receipt is first modeled as a function of treatment
eligibility, and then the outcome is regressed on the probability of treatment receipt and the assignment variable (Imbens & Lemieux, 2007). In this study, FRD regression models were estimated for both outcomes using 2SLS because, for each of the cutoff values examined, the probability of treatment changed by less than 1.
Assignment variable and cutoff scores. For youth in the study sample, the APAL intervention was allocated based on age, because program administrators were uncomfortable
with random assignment of children to treatment conditions. Specifically, youth that were either 13 or 16 years old on October 19, 2007 were assigned to treatment, and children ages 12, 14, 15, or 17 were assigned to the comparison condition of child welfare post-
permanency SAU. Thus, the assignment variable for this study was child’s age in years, and there were four discontinuities in the assignment variable that allowed for an estimation of the ATEC at four cutoff scores (i.e., ages 13, 14, 16, and 17). The age variable (ch_agey) was continuous, with a range of 12 to 17.89 (M = 15.00; SD = 1.72).
Treatment variables. Because this study required estimation of the ATEC for fuzzy discontinuities, two treatment variables were used in analyses. First, the dichotomous
variable apal indicated whether participants were at or above the cutoff age in the assignment variable, with apal = 1 indicating that, based on age, a participant was eligible for the
intervention and apal = 0 indicating that, based on age, a participant was not eligible for the intervention. Second, the variable treatment was a dichotomous caregiver-report variable that indicated whether the participant actually received APAL (i.e., contact from a caseworker to assess family needs), with treatment = 1 indicating that a participant did receive APAL and treatment = 0 indicating that a participant did not.
Outcome variables. Two outcome variables were of interest in this study. First, child behavior problems was measured using the variable bpiscore, a continuous variable with values that ranged from 0 to 27 (M = 10.14; SD = 7.54). This variable was derived from responses to the Behavior Problems Index (BPI), with higher numbers indicating more child behavior problems as reported by the caregiver. The BPI is a 28-item rating scale of
study with an adolescent population to have a Cronbach’s alpha coefficient of .92 (Brand & Brinich, 1999).
Second, the variable for caregiver commitment, commit, was a scale derived from summing caregiver responses to 7 questions in “Section H” of the Round 2 post-permanency survey (see Figure 3.1 below). The first 5 items (i.e., H4-H11) were Likert-type questions with five possible response options that ranged from “strongly agree” to “strongly disagree” (corresponding to 1 and 5, respectively) and included a “neutral” option (corresponding to 3). The last two items, H12 and H14, were also Likert-type variables with 5 and 4 response options, respectively. Five variables in the commitment scale (i.e., items H4, H7, H8, H12, and H14) were reverse scored so that higher numbers indicated higher caregiver commitment to the adoption or guardianship. In a previous study (White, 2015), exploratory and
confirmatory factor analysis was used to provide evidence that a latent variable for caregiver commitment caused responses to the 7 items shown in Figure 3.1. Also, Cronbach’s alpha for the caregiver commitment measure in the Round 2 sample was .82 (White, 2015), suggesting that the scale was acceptable for research purposes (DeVellis, 2003). The caregiver
commitment variable in this study was continuous, with a range of 12 to 34 (M = 30.09; SD = 3.89).
“Thoughts About Your Adoption/Guardianship” Variable Name
H4. I am able to manage [NAME’S] behavior. manage
H5. I always feel angry with [NAME]. angry
H7. I feel close to [NAME]. close
H8. I feel pleasure in parenting [NAME]. pleasure
H11. If I could, I would end this adoption/guardianship. endthis H12. Overall, would you say the impact of [NAME’S] adoption/guardianship on your
family has been… famimp
H14. How often do you think of ending the adoption/guardianship? thinkend