METODO AASHTO

Figure 29: Example of the effect of updating at different lead times

4.2Validation results of the processing strategies

In section 3.5 3 set-ups of QM in pre-processing and 4 strategies of combining pre- and post- processing are introduced. In this paragraph these set-ups (section 4.2.1) and strategies (section 4.2.2) are evaluated and the most appropriate set-up and strategy are chosen.

The effectiveness of the different QM set-ups and strategies will be evaluated with the CRPS. In section 3.7.3 it has been explained that this is a general evaluation score that combines different properties of forecast quality. This score is also used by other studies to evaluate the effectiveness of pre-processing and post-processing techniques, as single score (Kang et al., 2010) or as one of the evaluation scores (Madadgar et al., 2014; Pagano et al., 2013; Verkade et al., 2013; Zalachori et al., 2012). For an adequate evaluation of the performance of a forecasting system a set of evaluation scores is needed (also argued for in section 3.7), but in this study one of the processing strategies needs to be selected and it is considered that it is not needed to thoroughly analyse the performance on all properties of forecast quality. The set-up with the lowest CRPS over lead times from 0 days to 9 days will be used as pre-processing set-up and the strategy with the lowest CRPS will be used as processing strategy. In addition the RMAE (section 3.7.4.1) and rank histogram flatness coefficient (section 3.7.4.2) are used to show the effect of processing on the relative bias of the ensemble mean and on dispersion of the ensembles. The validation period is from 1-11-2007 to 31-10-2011.

4.2.1Best pre-processing set-up

In Figure 30, Figure 31 and Figure 32 the evaluation results of the QM set-ups are presented. Regarding precipitation the three QM set-ups all result in an almost equal (little) improvement of the CRPS. The RMAE and especially the flatness coefficient improve considerably by QM. These results indicate that the correction has improved the precipitation forecasts. The CRPS and RMAE are almost equal for all QM set-ups and the flatness coefficient of QM separately for each lead time (set-up 2) is on average slightly better, so it has been chosen to apply set-up 2 to pre-process the precipitation forecasts. However the differences are very small, especially when it is remembered that there is a random element in the rank histograms of precipitation forecasts (explained in section 3.7.4.2).

4.2 Validation results of the processing strategies

Regarding temperature the CRPS of QM with separate lead times and two seasons (set-up 3) is slightly better than the other QM set-ups for most lead times and this is on average approved by the RMAE and flatness coefficient. Therefore it is chosen to use set-up 3 to correct the temperature forecasts, which was also expected based on Figure 18 (explained in section 3.5.3.1). A larger improvement is achieved for the temperature forecasts than for the precipitation forecasts. The reason for this is that in the temperature forecasts a more consistent bias is present, so the temperature forecasts are easier to pre-process based on a training period. This bias might be introduced by calculating the daily average temperature (see section 2.3). Over a large part of the cumulative probability domain the CDF of the pre-processed precipitation forecasts during the validation period is actually farther away from the CDF of the observations during the validation period than before the correction. In the temperature forecasts the bias is more consistent, so after pre-processing the CDF of the forecasts is closer to the CDF of the observations than before.

In appendix 5 the rank histograms of precipitation and temperature forecasts are presented for each lead time, before and after pre-processing. The rank histograms of uncorrected precipitation and temperature forecasts are substantial non-uniform. These U-shaped histograms indicate under- dispersion or conditional biases (Hamill, 2001), possibly as a result of under-dispersion and/or bias in the initial states of the meteorological model and as a result of the meteorological model itself. At larger lead times the flatness improves, which is a well-known feature of operational meteorological ensemble prediction systems (Candille & Talagrand, 2005). At larger lead times the influence of initial conditions is less and the spreading of the forecasts becomes larger. In general QM improves the flatness of the rank histograms considerably for both precipitation and temperature forecasts. Regarding precipitation, for small lead times (up to a lead time of 3 days) the histograms still show a non-uniform distribution after pre-processing. Regarding temperature, after pre-processing the rank histograms are non-uniform at all lead times and more non-uniform than the precipitation forecasts.

4.2 Validation results of the processing strategies

Figure 30: CRPS of the QM set-ups and uncorrected forecasts of precipitation (a) and temperature (b), over the validation period 2008-2011. Lines of the different set-ups are almost on top of each other.

Figure 31: RMAE of the QM set-ups and uncorrected forecasts of precipitation (a) and temperature (b), over the validation period 2008-2011. Lines of the different set-ups are almost on top of each other.

4.2 Validation results of the processing strategies

Figure 32: Rank histogram flatness coefficients of different QM set-ups and uncorrected forecasts of precipitation (a) and temperature (b) forecasts, over the validation period 2008-2011

4.2.2Best processing strategy

The best QM set-up for precipitation and temperature forecasts from section 4.2.1 is applied in strategy 1 and 2 (see Table 11). In Figure 33, Figure 34 and Figure 35 the validation results of the different pre- and post-processing strategies are presented. It is very remarkable that strategy 0 (no correction) results in the best CRPS values. Since the flatness coefficient has also worsened by post- processing, the spread of the flow forecasts is worse after post-processing. There is not a large effect on the RMAE, so on average the ensemble mean flow forecasts have not clearly been improved or got worse. The best performing processing strategy over the validation period will be applied to the flow forecasts, so no correction at all (strategy 0). Since no processing will be applied, the training period does not need to be excluded from the evaluation period of the ensemble flow forecasts. So the ensemble flow forecasts can be evaluated over the period 2008-2013.

In section 3.5.3 it has already been mentioned that the effectiveness of QM depends on whether during the validation period the same bias is present between the CDF of observations and the CDF of forecasts as during the training period. In Figure 36 it is visible that this bias is not the same over the whole cumulative probability domain for all years. Especially during the hydrological year 2010 (green line) the bias is different, but also for other years on some parts of the cumulative probability domain the correction is in the other direction than during the validation period. Over the cumulative probability domains 0 - 0.2 and 0.85 - 1 the bias during the validation period is in the other direction than during the training period. In addition between a cumulative probability of about 0.4 and 0.7 the bias during the training period is much larger than during the validation period, so in this range the correction is in the good direction but the correction is too large. As a result of these points many corrected forecasts deviate further away from the observations than the uncorrected forecasts. QM usually functions effectively in cases with distant CDFs (consistent bias over whole cumulative

4.2 Validation results of the processing strategies

probability domain) (Madadgar et al., 2014), which is clearly not the case here. Madadgar et al. (2014) describe that when two distributions are relatively close, which will be the case for a well- calibrated model, this deficiency of QM becomes more significant. This is further elaborated in the discussion in section 5.2.

In appendix 6 the effects of the different processing strategies on the CRPS, RMAE and flatness coefficient during the training period are shown. These figures show the potential of processing with QM when a consistent bias is present. Over the training period strategy 3 with seasonal distinction gives the best performance.

Figure 33: CRPS of the post-processing strategies, over the validation period 2008-2011