Estudios científicos en Pacomarca

Table 5.5: Summary of verification metrics. Event type BS BSS AUC

June BN 0.16 0.27 0.79 NN 0.20 0.07 0.76 AN 0.17 0.25 0.79 July BN 0.17 0.20 0.75 NN 0.21 0.04 0.70 AN 0.19 0.16 0.73 August BN 0.14 0.29 0.78 NN 0.18 0.14 0.73 AN 0.16 0.27 0.77

5.6

Forecast verification

5.6.1

Deterministic forecast

In this work we use a deterministic forecast for multiple categories. It is obtained from the set of linear equations by computing the mean over all ensemble members in a single SPI-1 value. This value characterises dry or wet conditions in the SPI-1 gradations for each given FR.

Table 5.2 summarizes verification metrics for characterizing the skills of the direct SPI-1 hindcasts of the SL-AV model together with those of our forecasting model 1 based on one-month lead-time forecasts of H500 and MSLP from SL-AV. Here, "true" and "false" indicate whether or not the deterministic forecast made by our algorithm falls into the same of the seven SPI classes defined by the WMO as the corresponding observation.

In our case, the reliability characteristic ρ (see Section 3.5.1) takes values of around 0.5, with the weakest results for July (0.48), slightly better values in August (0.5) and the highest in June (0.54). The local correlation coefficients vary between 0.27 and 0.73 (Fig. 5.17 a), with the best results again being obtained for June, whereas in July and August, the correlation is generally weaker. Figure 5.17 (a,b) highlights areas with relatively good, as well as such with relatively poor forecast accuracy. Specifically, a good forecast can be expected in June in the northwestern, southern European and far-eastern parts of Russia. In July, reasonable results can be obtained, especially in the Far East and some parts of southern Russia. Finally, in August, we expect acceptable forecasts in the westernmost EPR as well as in the western to central parts of Siberia. The model generally tends to exhibit lower forecast skill for the central

parts of Russia, with maximum values of about 0.5 for both ρ and local correlation, which meets the expectations raised by the spatially distinct influences of different large-scale atmospheric circulation patterns (see Section 5.4).

As additional verification metrics, we also calculated the Mean Absolute Error (MAE) and root mean square error (RMSE) of the SPI-1 forecasts for each FR. Both character- istics indicate good accuracy. In accordance with the metrics discussed above, June is found to be the month with the lowest forecast error (values of 0.11 and 0.35 for MAE and RMSE, respectively). In August, both error measures have slightly higher values (0.15 and 0.38, respectively), whereas July shows the highest values (0.20 and 0.45, respectively). Figure 5.17 (c) shows the corresponding spatial patterns of the RMSE. As expected, in general the lowest errors are found in areas with the highest local cor- relation and forecast reliability ρ. In turn, large errors together with low correlation and reliability indicate regions with poor forecast accuracy.

In June, small errors can be observed on the Taymyr peninsula, in northern Siberia, some smaller areas in the centre of the EPR, close to the Ural Mountains, and in the Far East of Russia close to the Sea of Okhotsk. In July, larger errors are found in several smaller areas across the entire study region, with the highest values in the southern parts of Russia. In August, much of the EPR, Eastern Siberia on the south of the Russian Far-East exhibit the largest forecast errors. However, there are some regions with relatively small forecast error during all months, like the westernmost part of Russia and some areas in the Far-East.

An additional verification of our deterministic forecasts is provided by the scatter plots between observed and predicted SPI-1 values shown in Figure 5.18. The curves demonstrate the different capabilities of model 1 in catching dry, normal and wet episodes and their intensity in the boreal summer. Based on these results, it is con- cluded that months classified as having normal weather conditions are not necessarily well-represented in the SPI forecasts. The actual forecast scores depend strongly on the specific region, reflecting different dominant atmospheric circulation patterns in different regions. In general, the deterministic forecasts showed the best accuracy in June and somewhat lower ones in August and July.

5.6.2

Probabilistic forecast

A big advantage of probabilistic forecast is the explicit consideration of forecast un- certainty. Specifically, the probabilistic forecast takes all ensemble members explicitly into account and produces the prediction along with a certain probability of the occur- rence of the event of interest. To check the quality of the obtained forecasts, we use two metrics for probabilistic forecast verification: ROC curves and reliability diagrams (as discussed in Section 3.5.2). The corresponding results for our pressure-based fore- casts for each class of conditions and each boreal summer month are presented in Figure 5.19 and Table 5.5. The best accuracy is found in June, with a maximum area under the ROC curve (AUC) value of 0.79 for below normal (BN) and above normal (AN) conditions, while a generally lower prediction accuracy is observed for normal

5.6. Forecast verification 85

Figure 5.17: Deterministic forecast accuracy: a) local (point-wise) correlation, b) forecast reliability characteristic ρ and c) RMSE between deterministic forecast and CAMS-derived SPI-1 values.

Figure 5.18: Scatter plot between observed and predicted SPI-1 values for the deter- ministic forecasts for the months June, July and August (from top to bottom).

conditions (0.76). The same ranking between the different types of events is recovered for the two other summer months (with AUC generally slightly higher for dry than for wet conditions). However, the corresponding AUC values are consistently lower for August than for June and indicate the lowest forecast accuracy for July.

Figure 5.20 shows the resulting reliability diagrams. The corresponding values of Brier Score (BS), Brier score baseline (BSB) and Brier Skill Score (BSS) are provided in Ta- ble 5.5. The best skills are identified (highest BSS) for all three types of conditions during August, slightly less skilful forecasts for June, and the lowest skills for forecasts in July, which resembles the outcome of the ROC analysis. In general, the reliability diagrams indicate the highest reliability (despite not possessing the highest BSS) for the prediction of dry conditions in June. According to Wilks (2011) and Weisheimer et al. (2014) and their classification of characteristic forms in reliability diagrams, in our case, dry and wet conditions generally exhibited good forecast calibration. For wet conditions, the forecast line often lay slightly above the diagonal during June and July, indicating a minor tendency towards underforecasting, except for very high fore- cast probabilities. In contrast, for dry conditions during August, the reliability curves fall below the main diagonal, revealing a loss of predictive skills at low-to-moderate forecast probabilities. Unlike for wet and dry conditions, the reliability diagrams for normal conditions indicate generally poorer skills of our probabilistic forecasts (yet slightly higher skills than pure climatological forecasts) and reveal a general and sta- ble tendency towards overforecasting at all but the very low forecast probabilities for all boreal summer months.

As for the deterministic forecasts, it should be underlined that the accuracy presented here exhibits large differences among the different regions of Russia due to the differ- ential impact of large-scale atmospheric circulation patterns (not shown). Specifically,