Marco Legal

In order to understand how conceptual features are modulated in combined concepts, we created a set of predictive models that made different predictions about how concepts combine. Each model generates predictions reflecting the conceptual brightness of the adjective-noun combinations (BCOMBO) based on the conceptual brightness of the adjective (BADJ) and noun (BNOUN), and thus also generates a prediction of brightness change (BCHANGE), which corresponds to the ground-truth adjective effects described above. We constructed and tested an adjective model, a noun model, a weighted additive model, and a Bayesian model.

The adjective- and noun-models are non-combinatorial and are useful because their outputs can be considered baseline predictions that the combinatorial models should outperform. A similar approach is found in distributional or vector- based semantics (e.g., Mitchell & Lapata, 2008; 2010; Chang et al., 2009). In the adjective model, the predicted brightness of the combined concept (e.g., “dark diamond”) is identical to the brightness of the adjective (e.g., “dark”), which is formalized as:

BCOMBO = BADJ

Where BADJ corresponds to the extreme values of the scale (dark=50; light=0). Within a single adjective (e.g., “dark”), the adjective model does not predict differences in BCOMBO across the 45 nouns but it does predict differences in BCHANGE based on differences in BNOUN for “dark” and “light” separately. That is, it does make predictions about how observed feature activation will be influenced by each combination. However, when adjective effects are averaged across adjectives — as we did to define our ground-truth behavioral adjective effects — the adjective model does not predict differences in BCHANGE. This is an important point: the non-combinatorial adjective model is not able to capture variability in the extent to which brightness can be modulated across our noun concepts. In the non-combinatorial noun model, the predicted brightness of the combined concept is identical to the brightness of the unmodified noun:

BCOMBO = BNOUN

The noun model predicts differences in BCOMBO across items. It does not predict any variance in BCHANGE in the “dark” and “light” combinations separately, nor when averaged together. Thus, like the adjective model, the noun model is also unable to capture variability in the extent to which brightness can be modulated across the 45 noun concepts.

84

We constructed a combinatorial additive model in which the predicted brightness of the combined concept is a weighted sum of BADJ and BNOUN. This model has been proposed as a candidate combinatorial mechanism in both cognitive (e.g., Smith et al., 1988) and computational models of distributional semantics (e.g., Mitchell & Lapata, 2010). The simple form is:

BCOMBO = W∙BADJ + BNOUN

In our case, BADJ represents the extreme brightness values (dark=50, light=0). Our implementation of the additive model makes predictions for “dark” and “light” combinations that can be represented as:

BCOMBO-DARK = W∙50 + BNOUN BCOMBO-LIGHT = BNOUN - W∙50

We optimized W between 0 W 1 (intervals of 0.01), separately for “dark” and “light” combinations. This resulted in a value of WDARK that minimized the mean squared error (MSE) of BCOMBO-DARK predictions relative to the ground-truth dark- combo brightness values, and the value of WLIGHT that minimized the mean squared error (MSE) of BCOMBO-LIGHT predictions (Fig. 15D).

We also constructed a combinatorial Bayesian model of adjective-noun

combinations (Fig. 15A-C). The motivation behind this approach is that feature uncertainty may influence how concepts combine. In addition to using entropy as a measure of feature uncertainty, feature uncertainty can be embedded in

probabilistic feature models, like the one described here. We represented

conceptual brightness for adjective and noun concepts as probability distributions over possible brightness values. In a Bayesian model, the predicted brightness of a combined concept is the maximum a posteriori (MAP) estimate of the product of the constituent concepts’ brightness distributions. If the brightness probability distributions for adjectives (PADJ) and nouns (PNOUN) are each captured by a gaussian defined by a mean (µ) and standard deviation (𝜎), then the Bayesian predictions of the brightness of combined concepts is the maximum a posteriori (MAP) estimate of the product of these distributions:

BCOMBO = arg max f {PADJ (µ,𝜎)∙PNOUN (µ,𝜎)}

We derived the 45 PNOUN distributions (e.g., for DIAMOND, PAINT, CHARCOAL) by fitting gaussian distributions to histograms reflecting the frequency of responses in the explicit brightness judgment task (Fig. 15A). We do not have data from which the PADJ distributions can be calculated; for simplicity, we assumed that PDARKµ=50 and PLIGHTµ=0. We optimized separately for PDARK𝜎 and PLIGHT𝜎 (0≤ 𝜎 ≤50; intervals of 0.01); two example PDARK distributions with PDARK𝜎 =8 and PDARK𝜎 =15 are shown in Fig.2B-C. This procedure resulted in values for PDARK𝜎

85

and PLIGHT𝜎 that minimized the MSE of BCOMBO predictions relative to the ground- truth BCOMBO values (Fig. 15E).

The two combinatorial models (i.e., additive and Bayesian) each had one free parameter. For each model we generated predictions by optimizing the parameter (i.e., W or PADJ𝜎 ) across the 45 concepts and then used that

parameter value to make BCOMBO and BCHANGE predictions for each concept. We Figure 15: Predictive combinatorial models. (A) In our Bayesian model, conceptual brightness was represented as a probability distribution over brightness values for each noun concept. Greater values on the scale indicate increased conceptual darkness. Each distribution is defined by a mean and sigma, derived from our behavioral task (see Fig. 14). We defined the means of the “dark” and “light” distributions as the extreme ends of the brightness scale, and optimized for sigma. (B-C) Different “dark” 𝜎 values result in different

BCOMBO predictions for “dark diamond”, and our goal was to find the sigma that generated the

most accurate predictions of BCOMBO across the 45 noun concepts. (D) In the additive model,

we optimized the adjective-weight for “dark” (W=0.35) and “light” (W=0.33) separately. (E) In the Bayesian model, we optimized the standard deviation of the “dark” (𝜎= 8.42) and “light” (𝜎 = 10.27) distributions separately. We averaged the “dark” and “light” parameters within each model to analyze fMRI data.

86

compared the accuracy of these models in their ability to predict the behavioral data.

In our analysis of fMRI data, we simplified the combinatorial models by averaging the “dark” and “light” parameter estimates within each model, resulting in a single value for W and a single value for PADJ𝜎 . One of the goals of the fMRI analysis was to determine which of any neural regions exhibit responses to combined concepts that reflect a combinatorial feature-based mechanism. The additive model and Bayesian model are both combinatorial, and success of either one in predicting neural response would satisfy our goal. We thus created a “composite combinatorial model” by averaging the BCHANGE predictions across the additive and Bayesian models. This resulted in BCHANGE predictions for the 45 dark- combos and the 45 light-combos.