Numerous studies have examined the neural correlates of habitual and goal-directed control systems in the past decades, mostly working with animal models and the classical experimental paradigms of outcome devaluation and contingency degradation. Findings in this regard showed that the behavioral effect of outcome devaluation and contingency degradation in extinction was abolished after lesions of the dorsomedial striatum, which is the rodent equivalent of the (posterior) caudate, whereas lesions of the dorsolateral striatum, corresponding to human (posterior) putamen, prevented instrumental behavior to become habitual after extended training (Balleine & Dickinson, 1998; Goodman & Packard, 2016; Graybiel, 2008; Voon, Reiter, Sebold, & Groman, 2017; Yin, Knowlton, & Balleine, 2006, 2004; Yin, Ostlund, Knowlton, & Balleine, 2005; Yin & Knowlton, 2006; Zapata, Minney, & Shippenberg, 2010). In addition, lesions of NAcc shell did not result in changes of the sensitivity towards outcome values or contingencies and, hence, did not affect the performance of goal-directed or habitual responses, while lesions of NAcc core left sensitivity to contingency degradation intact but interfered with rats’ sensitivity to outcome devaluation (Corbit, Muir, & Balleine, 2001; Yin et al., 2008). These dissociable effects of lesions to various striatal regions suggest that NAcc may be involved in the acquisition but not performance of goal-directed and habitual behavior, while there seems to be a focus of control of goal-directed behavior in caudate and of habitual behavior in putamen (Graybiel, 2008; Yin, Ostlund, & Balleine, 2008).
Furthermore, there is evidence for a shift of control from dorsomedial to dorsolateral striatum in rats and primates accompanying the behavioral shift from goal-directed to habitual control under certain circumstances, for example extended training (Haruno & Kawato, 2006; Tricomi et al., 2009; Yin et al., 2008). In addition to these findings regarding different areas of the striatum, there is empirical evidence for a role of prefrontal regions in goal-directed and habitual control. Findings here are less conclusive, but suggest a role of prelimbic cortex in goal-directed and infralimbic regions in habitual control (Balleine & Dickinson, 1998; Graybiel, 2008; Voon et al., 2017). Unfortunately, it is still unclear, what the human analogue regions to those rodent prefrontal areas are. It has been suggested that, based on the pattern of thalamic input to these regions, prelimbic rodent cortex corresponds to the pregenual ACC (Brodmann area 32) and infralimbic cortex to human subgenual ACC (Brodmann area 25; Gass & Chandler, 2013).
Translational neuroimaging research in humans corroborated findings from animal studies using equivalent paradigms. Valentin, Dickinson, and O’Doherty (2007) found
30
activation in OFC to encode option values during an outcome devaluation task using specific satiation that corresponded to the valuation scheme of a goal-directed controller. Tricomi, Balleine, and O’Doherty (2009) showed increasing task-related activation of posterior putamen with extended but not limited training in a similar devaluation task suggesting this area to be related to habitual control. Two more fMRI studies using the slips-of-action task (de Wit et al., 2007) have shown goal-directed behavioral control to be associated with increased activation in vmPFC (de Wit et al., 2009) and increased structural connectivity between caudate and vmPFC and decreased connectivity between posterior putamen and premotor cortex measured with diffusion tensor imaging (S. de Wit et al., 2012). Furthermore, Gläscher and colleagues could show that the model-based system relies on state representations being stored and updated in the intraparietal sulcus and dlPFC (Gläscher, Daw, Dayan, & O’Doherty, 2010). The cognitive map, which is based on these state representations and is assumed to underlie goal-directed control and model-based RL, seems to be represented in a network comprising the hippocampus, OFC, dlPFC, and posterior parietal cortex (O’Doherty, Cockburn, & Pauli, 2017). Moreover, Daw et al. (2011) showed blood-oxygen level-dependent (BOLD) responses in ventral striatum and vmPFC to be associated with model-free RPEs but to also comprise signatures of the model-based valuation system using the Two-Step task to investigate neural correlates of model-free and model- based reinforcement learning. In addition, model-based control in the Two-Step task has been shown to be modulated by various manipulations: application of L-DOPA to healthy volunteers, which enhances dopamine levels systemically, enhanced model-based control (Wunderlich, Smittenaar, & Dolan, 2012); disrupting right dlPFC activation via transcranial magnetic stimulation reduced model-based control in favor of model-free behavior (Smittenaar, FitzGerald, Romei, Wright, & Dolan, 2013), but anodal transcranial direct current stimulation of the same prefrontal area did not change model-free and model-based control (Smittenaar, Prichard, FitzGerald, Diedrichsen, & Dolan, 2014).
In summary, goal-directed behavioral control is assumed to be dependent on vmPFC and OFC in interaction with dorsomedial striatum (i.e. caudate; Everitt & Robbins, 2016), which fits well with OFC’s role in representing various facets or features of options and vmPFC’s role in the online integration of these to calculate subjective values (see section 1.3.1). Caudate also integrates information from hippocampus, ACC, intraparietal sulcus, and dlPFC, which could give rise to the representation of the mental map of the state space goal- directed control is based upon. In contrast, habitual behavior seems to be based on dorsolateral striatum (i.e. putamen) and possibly (pre-)motor cortical areas (Balleine &
31 O’Doherty, 2010; Everitt & Robbins, 2016; Yin et al., 2004). These findings map well onto the reward-processing circuits, where information from vmPFC, OFC, and ACC is combined in ventral striatum and conveyed to dorsal striatum via loops comprising midbrain nuclei. While goal-directed valuation depends on the integration of a vast amount of information, habitual valuation might circumvent this process and directly activate response representations being cached in dorsal striatum-motor cortex connections.
Crucially, the numerous neural adaptations to chronic alcohol misuse affect among other things the dopaminergic innervation of striatal medium spiny neurons (Volkow & Baler, 2014; Volkow, Wang, Tomasi, & Baler, 2013). These effects are assumed to drive the proposed shift from goal-directed to habitual alcohol use in AUD, because they “hijack” the normal dopaminergic signal cascade (Keramati & Gutkin, 2013).