Figure 4.2 shows the performance of the probabilistic radar tracker Rad-TRAM as time series of the Brier score (top), the CSRR (middle), and the area under the ROC curve (bottom) over the entire period from 8 to 16 August 2007.
The Brier score (Fig. 4.2, top) gives information about the magnitude of the forecast error. During the entire investigated period, the Brier score has a large variability. In comparison to the uncertainty component (Fig. 4.1), a similar shape of the curves can be seen. The deviation of short lead times from uncertainty is an indication for their skill in reliability and resolution. Hence, the performance of the forecasts in terms of the Brier score is highly dependent on the observed relative frequency of the event. A clear ranking following lead times can be identified on all days. The number of distinguishable lead times varies and seems to be dependent on the meteorological regime. If more than the first forecast hour is distinguishable the loss in skill is largest in this first hour.
The use of the CSRR (Fig. 4.2, middle) has the advantage that the exact values of various days can be compared more reliably. The score is independent of the observed frequency as it is weighted with the size of the rain domain (sec. 2.4). Therefore, the variability within the days is smaller. The quality of the forecasts and the rate of loss decreases with increasing lead time. Also in the CSRR, the number of distinguishable forecasts might be taken as indication of the predictability of the situation. The more lead times are distinguishable, the larger the amount of precipitation and the better the description of the development of the precipitation field by advection. Large outliers in the CSRR are a sign that there was almost no precipitation in the domain and should not necessarily be taken as a very large error of the forecast (i.e. 14 August 2007).
Figure 4.2: Development of Brier Score, CSRR, and area under ROC curve for Rad-TRAM based probabilistic forecasts from 8 to 16 August 2007. Colour-coded as explained in Fig. 3.7.
The area under the ROC curve (Fig. 4.2, bottom) varies for the different lead times and days over the entire possible range of values. The skill of the forecasts concerning discrimination is in the first forecast hour very high (between 0.9 and 1.0), except some outliers (i.e. 12 and 14 August). The ROC area has a clear ranking in quality of the forecasts following lead time. But longer lead times are not necessarily well ordered (8 August), but might show a larger variability (14 August). On eight of the nine investigated days, values of longer lead times are below the 0.5 threshold. Then, the forecasts do not have any skill in discrimination if the event occurs or not, but are even anticorrelated.
Comparing all three scores, there are some similarities. All scores show a ranking following lead time with the forecasts based on the latest observations having more skill than the older. All scores show in agreement that the loss of skill is largest in the first forecast hour. After the first hour, it depends on the meteorological situation if the differences between the lead times can be distinguished or if they are well ordered. The number of distinguishable lead times can be seen as an indication for the predictability. In this overview, the results the scores provide are consistent. That means, a forecast with high skill in the Brier score is also good in CSRR and the area under the ROC curve. Nevertheless, there are some exceptions to this finding. For example, on 12 August 2007 around noon, the Brier score and the CSRR show a rapid increase in forecast skill, whereas the area under the ROC curve shows a decrease (sec. 3.2).
Reliability diagrams display the full distribution of forecasts and observations in terms of the refinement calibration distribution (sec. 2.4). The reliability diagram consists of the so-called calibration function and the refinement distribution (small diagram in the top left of each plot). The refinement distribution is shown on a logarithmic scale (Fig. 4.3). The 15 minutes nowcasts have very high skill in reliability as the calibration function is very close to the diagonal (Fig. 4.3, top left). The aspect of resolution is well represented by the forecasts as well as the distance to the horizontal no resolution line is large. The histogram shows that the forecasts are relatively sharp, because the extreme bins are two mostly populated bins. As reaching 19 dBZ is a relatively rare event, the 0 % bin is mostly populated.
The forecast with 2.25 hours lead time also has very high skill (Fig. 4.3, top right). Relia- bility and resolution are still large. But sharpness has already decreased as can be seen by the lower population of the bins near 1.0. This is also visible in the calibration function that already is lower than the prefect reliability line for forecasted probabilities of 0.9 and 1.0. Nevertheless, forecasts of all categories are skillfull as they are far above the no skill line. The reliability diagram of the 4.25 hours forecast (Fig. 4.3, middle left) shows that for this lead time, not all bins are populated. This means, the 1.0 forecast is never issued at this lead time. The other forecast categories have a high reliability and resolution except the 0.0 bin. It can be seen that it is above the no skill line, indicating that if 0.0 was forecasted there was an observed frequency of nearly 0.05.
In the reliability diagram of forecasts with a lead time of 6.25 hours (Fig. 4.3, middle right) a decrease in skill can be seen. Although the forecasts are still above the no skill line, their distance to the perfect reliability line is for bins larger 0.3 larger than to the no skill line. The high bins are rarely populated resulting in a further loss of sharpness.
The forecasts of a lead time of almost 7.75 hours (Fig. 4.3, bottom) show further decreased skill. The forecasts up to 0.7 are very close to the no skill line. This indicates that they hardly have skill concerning reliability and resolution. The high bins are if populated only sparsely populated but their calibration function is above the no skill line.
Figure 4.3: Reliability diagram for Rad-TRAM based probabilistic forecasts from 8 to 16 August 2007, for forecasts with lead times of 15 minutes (n=1), 135 minutes (n=9), 255 min- utes (n=17), 375 minutes (n=25), and 465 minutes (n=31) with the perfect reliability (solid), the no resolution (dashed) and the no skill line (dotted).
grams. Even the long forecasts are skillful and the short lead times have very high skill in sharpness, reliability, and resolution.