In Section 7.2 the uncertainty introduced by the rainfall-runoff model structure choice was isolated in an attempt to evaluate the impact of the uncertainty introduced by signature regionalisation in the identification of the optimal parameter set. In contrast, in this section the knowledge gained in Section 7.2 about the importance of considering the inter-signature error dependencies will be applied to real data in order to evaluate the quality of the resultant streamflow predictions. The use of real data and the lack of knowledge on model structural and observational error discourage the use of the Bayes factor for performance assessment, mainly because the non-account for those sources of uncertainty results in residuals that are often autocorrelated in time. Nash-Sutcliffe for probabilistic predictions, despite also presenting limitations when there is structure in the residuals series, is used to evaluate the quality of the streamflow probabilistic predictions in terms of its accuracy and precision.
Figure 7.37 shows for same two sub-catchments of Figure 7.35 (the Nezinscot River at Turner Center sub-catchment and the Susquehanna River at Conklin sub-catchment) how NSprob changes when response signatures are added to constrain simulations of streamflow.
Figure 7.37 – NSprob for the Nezinscot River sub-catchment and Susquehanna River sub-catchment showing variations in performance between catchments for different combinations of signatures and for real data.
For the Nezinscot River sub-catchment, NSprob tends to improve when more signatures are added up to a maximum of four signatures. Adding a fifth signature leads to negligible additional improvement.
For the Susquehanna River sub-catchment, NSprob tends to improve when more signatures are added up to a maximum of three signatures. Adding more signatures may lead to a decrease in performance depending on the order that the signatures are added to the conditioning process. This
0
166 pattern is general over all catchments, i.e. after adding more than three or four signatures the performance usually decreases.
It is worth noting that any performance measure is a statistical summary of the predicted time series, and as such necessarily focuses on particular aspects of the hydrograph. The choice of the performance measure should therefore consider the specific requirements of the given application (e.g. if it is more important to adequately capture peak flows than low flows). However, ultimately the choice of performance measure remains a largely subjective decision. The performance measure used here judges the quality of the predictions in terms of precision and accuracy. Figures 7.35 and 7.36 show for the same catchments the results in terms of precision and accuracy separately. The y-axis is formulated as 1-Precision and 1-Accuracy, respectively, in order to ensure that in both cases a value of 1 corresponds to perfect performance. A decrease in NSprob performance measure as more signatures are added could reflect a decrease in the accuracy of predictions despite an improvement in precision (i.e. the prediction bounds become smaller). Other explanations can be found for a decrease in performance. Most importantly, predictions have not been designed to match NSprob in particular, but different response signatures.
Figure 7.38 – [1-Precision] for the Nezinscot River sub-catchment and Susquehanna River sub-catchment for different combinations of signatures. [ ] ∑ [ ̂] ∑⁄ [ ] .
0 0.2 0.4 0.6 0.8 1
0 1 2 3 4 5
1-Precision
# Signatures 0
0.2 0.4 0.6 0.8 1
0 1 2 3 4 5
1-Precision
# Signatures
167
Figure 7.39 – [1-Accuracy] for the Nezinscot River sub-catchment and Susquehanna River sub-catchment for different combinations of signatures. [ ] ∑ [ ̂] ⁄∑ [ ] .
Figure 7.40 summarises for all 84 catchments the variability in NSprob for different combinations of signatures, with each boxplot referring to a unique combination of signatures used to constrain the model simulations. Similarly to Figure 7.36, boxplots are colour coded by the total number of signatures combined. Additionally, boxplots corresponding to the results obtained when observed signatures are used to calibrate the hydrological model are also shown (grey dashed lines) and can be used as benchmarks. In order to calculate NSprob for this case it is necessary to quantify the uncertainty affecting the observed signatures. For that purpose, all five signatures are calculated for five different decadal periods. For each signature and catchment, the variance is calculated based on the relevant five calculated values (one per decadal period). The signature variance, averaged by catchment, is calculated and assumed to represent the variance of that signature. Assuming that the observed signatures are corrupted by an error that follows a Gaussian distribution, centred on the observed signature and variance determined, it is possible to calculate the value of NSprob for calibration and subsequently use these to plot the dashed grey boxplots. It is acknowledged that strong assumptions are inherent in these calculations of NSprob for calibration. Specifically, the way the variance for observed streamflow is calculated reflects other uncertainties that do not compare with the variance of the regionalised signatures. For example, the calculated variance for the observed signatures may include uncertainty related to extrapolation to a new time period. On the other hand, this source of error is not accounted (at least explicitly) when estimating uncertainty for the regionalised signatures. For the aforementioned reasons, the comparison of the coloured with the grey dashed boxplots should be done with caution. Nevertheless, this enables a basic comparison of how the results obtained using the methodology suggested in this thesis for ungauged catchments
0 0.2 0.4 0.6
0 1 2 3 4 5
1-Accuracy
# Signatures
0 0.2 0.4 0.6
0 1 2 3 4 5
1-Accuracy
# Signatures
168 compare with the results obtained using traditional calibration techniques (assuming observations would be available for the latter case).
Figure 7.40 – Boxplots representing the distribution of NSprobvalues for each combination of response signatures.
Figure 7.40 shows that any combination of four signatures, where HPC is present (i.e. [RR, BFI, SE, HPC], [RR, BFI, SFDC, HPC], [RR, SE, SFDC, HPC], [BFI, SE, SFDC, HPC]), gives worse (or equal) performance than when all five signatures are used at the same time. However, the combination of four signatures that does not include HPC (i.e. [RR, BFI, SE, SFDC]) gives better results than when all
-1.5 -1 -0.5 0 0.5 1
169 five signatures are used at the same time. This seems to indicate that HPC has a negative impact on model performance in terms of NSprob. This negative impact of HPC is not been observed when synthetic data is used (Figure 7.36). It can therefore be speculated that this decrease in performance may be due to model structural deficiencies. Nevertheless, it should be noted that the predictions were not designed to match NSprob, but response signatures instead. Comparing the results obtained from constraining the hydrological model on regionalised response signatures with those obtained based on signatures calculated from actual streamflow measurements, indicates that the regionalisation method suggested here gives almost as good results as when observed signatures are available.