DEFENSORÍA DEL PUEBLO

To compare the two groups, we see how the overall acquisition rate improved over time. We expect the static group’s aggregate acquisition rate to remain flat, on av- erage. By contrast, we expect the rate for the adaptive (RPM-GLMM) group to increase on average over time. Figure 2.2 confirms those expectations, showing the results of the experiment involving 700 million impressions over the span of two months. We com- pare the cumulative conversion rates, aggregated across all ads and websites, computed as PJ j=1 PK k=1yjkt/ PJ j=1 PK

k=1mjkt, whereyjkt and mjkt are already defined to be the

cumulative conversions and impressions for ad k on website j through periods 1, . . . , t. Note, throughout this empirical portion of this chapter, all conversion rates reported are rescaled versions of the actual data from ING Direct, at the request of the firm to mask the exact customer acquisition data. We performed this scaling by a factor, so it has no effect on the relative performance of the policies. The scaling factor is small enough so that al- most all values of interest are within the same order of magnitude as their actual observed counterparts.

Compared to the static balanced design, the RPM-GLMM policy improves overall acquisition rate by 8%. For instance, due to this policy, we achieved approximately 240

200 300 400 500 600 700 −10 −5 0 5 10 ● ● ● ● ● ● ● ● ● ●

cumulative impressions through time (millions)

change in cum ulativ e con v ersion r ate (% of con v ersion r

ate in initial per

iod)

Figure 2.2: The actual field experiment results show the RPM-GLMM (adaptive group, solid line) achieves a higher cumulative improvement than the balanced design (static group, dashed line), relative to the cumulative conversion rate after the initial period. The cumulative conversion rate is the cumulative conversions per cumulative impressions. The impressions were delivered continuously over time (two months). For the adaptive policy, the circles indicate when reallocations occurred (every five to seven days).

extra new customers out of approximately 3000 new customers acquired.

From a substantive perspective, we note that these extra conversions come at no additional cost because the total media spend does not increase. They are the direct result of adaptively reallocating already-purchased impressions across ads within each website. Therefore, the cost per acquisition decreases (CPA = total media spend / by total number of acquisitions). In essence, we increased the denominator of this key performance metric by 8%. The new CPA has consequences beyond the gains during the experiments; it provides guidance for future budget decisions (e.g., how much the firm is willing to spend for each expected acquisition). We return to this in the general discussion, when we discuss potential linkage to post-acquisition activities like customer lifetime value.

We note that we have not changed the actual conversion rate of any ad. Instead, we assume each ad on a website has a constant conversion rate, but the aggregate conversion rate of ads, which is a weighted average, does increase due to our adaptive allocation. This is because we have allocated more impressions to better performing ads on each website by controllingwjkt. The expected aggregate conversion rate for adkacross allJ websites in periodtisPJ j=1 PK k=1wjktMjtE[µjk(θ)]/ PJ j=1Mjt.

One may ask a range of questions regarding stationarity. First, is it reasonable to assume that each ad within a website has a constant conversion rate? Second, assuming they are truly stationary, why does the aggregate conversion rate for the static group us- ing a balanced design appear non-stationary (i.e., not a perfectly flat line)? The aggregate conversion rate varies slightly over time, but any sources of its variation seem to be uncor-

related with our effects of interest.

While we observe the adaptive RPM group improve by 8% over a baseline, are those results really meaningful in a statistical sense? While the above results are aggregate, they do not reflect any uncertainty in performance. That is because we only observe one realization of a stochastic process. However, we can compute the implied distribution of performance through Monte Carlo simulation. The static group’s balanced design sets the number of impressions for each website and ad within each website. Keeping that constant across simulated “worlds,” we generate simulated conversions. The data-generating pro- cess is binomial with the constant probability set to equal the long-run probability actually observed using all data from the experiment. We compute 100 worlds in parallel applying the same balanced design in each world. We further detail this process for all other policies in Section 2.6.

Figure 2.3 shows observed results for RPM-GLMM and the observed results for the balanced design, compared to a predictive distribution of results for a balanced design. The interval of performance over time (upper and lower bounds and mean) for the balanced design remains lower than the RPM-GLMM policy, beginning after about 350 million im- pressions were delivered. That is, the RPM-GLMM policy achieves levels of improvement that are outlying with respect to a null distribution, but it takes time for the policy to learn and reach that higher level of performance.

200 300 400 500 600 700 −10 −5 0 5 10 ● ● ● ● ● ● ● ● ● ●

cumulative impressions through time (millions)

change in cum ulativ e con v ersion r ate (% of con v ersion r

ate in initial per

iod)

Figure 2.3: The actual cumulative performance of RPM-GLMM (adaptive group is dark solid line) is even better than the simulation-based predictive distribution’s 95% interval for balanced design performance, at the end of the experiment. This variability around the actual performance of the balanced design is summarized as the predictive distribu- tion’s mean, 2.5% quantile, and 97.5% quantile (middle, low, and high, light dashed lines, respectively). By the end of the experiment, the predictive performance distribution for the balanced design is centered near the actual performance of the balanced design (static group is dark dashed line). For the RPM-GLMM policy, reallocations occurred every five to seven days (circles).