MIES VAN DER ROHE CLARIDAD ESTRUCTURAL “Nosotros no hubiéramos hecho lo que Sullivan Nosotros vemos con

Two basic strategies can be applied to a benchmark for relative measures. One is to apply a completely naïve benchmark that holds no pretence to fitting the theory or the empirical evidence on the form of the phenomena. This sort of benchmark would not compete with any model under assessment; rather it would provide a somewhat distant and impartial reference, the random walk is the most popular (C. R. Nelson & Plosser, 1982), and despite its pretensions to be naïve is very well performing in many situations primarily because of its typical application within a one step ahead protocol. (Kilian & Taylor, 2003). Another approach not mentioned in the literature is a benchmark that is deliberately chosen for competitiveness. Such a benchmark is closer to the everyday understanding of a benchmark as a standard, which sets a level of performance that is currently the best and which any other model needs to beat to be given the benchmark title. This study demonstrates the use of relative measures with both types of benchmark.

In forecasting decline, as defined in this study, a random walk with drift might seem the best approach (Haldrup & Hylleberg, 1995), however, because of the restrictions placed on the study to use fixed origin forecasts see section 5.2.1, two alternatives were considered. A straight line fitted to the last three estimation points, and a stationary point benchmark representing the last point in the estimation data.

The single stationary point is a traditional naïve standard, given the use of a single origin for all forecasts, this benchmark is the equivalent of the random walk naïve model. The forecasts from this model are simply the last known value of the variable of interest, thus it does not represent the S-curve decline very well, because it does not follow the curve, as the classic naïve model, the random walk, would do with step ahead forecasting tests.

A simple investigation of how the stationary point benchmark performed was undertaken by plotting MAPE, the results are presented in Figure 8. Clearly, the stationary benchmark is not a competitive forecast method, so is a conceptually poor benchmark, its MAPE error rate growth mirroring the rate of decline in the data series it measures. It does however, remove the benchmark duties from the straight-line model, allowing it to be assessed as a model alongside the classic marketing science diffusion models, because all models are then measured relative to this stationary point benchmark.

Figure 8. Mean forecast MAPE of the stationary point benchmark model.

The straight-line model fitted to and then extrapolated from, the last three estimation data points is demonstrated fitted to a series with a non-ideal curve in Figure 9.

Figure 9. Simulated fit of the straight-line to an example decline data series.

Even when applied to a series with a local non S-curve slope, this model is still a reasonable model, in its expression of the slope of the last three data points and of the central part of the curve in general. It is clear that on an empirical basis, and in the restricted context of this study, the straight-line model makes a better benchmark for S-curve decline in the context of this study, than does the naïve stationary point benchmark. Figure 10 demonstrates that MAPE for the straight-line model is predictably lower than the stationary point model, see Figure 8, but also confirms that the benchmark’s MAPE performance deteriorates in some

S-curve fashion reminding us that one might expect the S-curve model’s MAPE error performance to deteriorate in a much more linear fashion.

Figure 10. Mean forecast MAPE of the straight-line model.

All the above indicators, point to the straight-line model being the better benchmark. However, when the straight-line was utilised as a benchmark in early investigations in CUMRAE, the models under investigation performed similarly to the straight-line model. This caused a problem, as sometimes the performance was worse sometimes better. Consequently, as work progressed these observations became increasingly difficult to explain in a simple understandable manner, because across the horizons any given model might be better or worse than the straight-line at any given horizon. Whereas when the models were compared with the stationary point (a less competitive benchmark), it was easy to explain the situation as in most estimates the models beat the naïve stationary point benchmark, greatly enhancing understanding. As a result, it was decided to use the less competitive benchmark but to include the straight-line model into the pool of models investigated. The challenges in multi horizon forecasting of using a competitive benchmark versus a weaker benchmark is not mentioned in the forecasting literature, although the limited usefulness of linear models is mentioned by Meade and Islam (2001). They warn against the blind adoption of the simpler models forecast better than more complex models

premise, because linear models will not model saturation, and will only be competitive short term as a result.