RIVPACS - Model validation: Assessing the accuracy of predictions

In order to validate the model we must assess the accuracy of our predictions of reference condition. One obvious way to determine the accuracy of the prediction is to calculate the agreement between observed and predicted values of indices derived from the lists of observed and predicted taxa. Figure 1 shows the relationship between observed and predicted values of BMWP-ASPT. Ideally model validation should be on an independent dataset of undisturbed sites but these independent sites still need to be of the same range of high qualities as the reference sites themselves. In developing RIVPACS, a wide range of different methods of classifying the reference sites and predicting the expected fauna were tested. The methods were compared in various ways, but mostly by assessing the ability to predict the expected values of the BMWP indices for a completely independent set of high quality sites.

Figure 1. Relationship between observed and predicted values of BMWP-ASPT

An alternative way and perhaps more direct way to assess the accuracy of predictions is to compare the observed data with the predictions. This is not easy when the observed data are highly multivariate in the sense that they incorporate the presence-absence and abundances of each of a large number of taxa. One method is to calculate the statistical likelihood or probability of getting the observed taxa given the model predictions of their expected probability of occurrence and abundance.

Figure 2 shows the probability of the observed data for each of the RIVPACS reference sites plot against observed/expected number of taxa. No reference sites had an observed fauna which was statistically unlikely (namely less than 1% probability) based on its predicted expected fauna. In contrast the right-hand plot shows the same relationship amongst the sites for the national survey for General Quality Assessment of UK rivers (across a range of water qualities) which shows that the statistical likelihood has the power to detect lack of fit. In fact in the absence of other ecological knowledge, this conditional likelihood probability could be used an index of ecological quality and classified into ecological status classes, as an alternative to using O/E ratios. However, its use is not recommended as it does not incorporate any ecological knowledge on individual taxa tolerance to environmental stresses.

Figure 2. Probability of observed data for each of the RIVPACS reference sites plotted against observed/expected number of taxa

Cross-validation is a third method that could be used to assess the accuracy of the model at an early stage. This process checks whether sites have been allocated to the correct group by the MDA model. The process of constructing the predictive model is followed as detailed above but one of the reference sites is omitted from the data set and used as a test site. A different test site is omitted each time and the model reconstructed. The test site is considered to be correctly allocated if it is placed in the classification group with the highest probability of membership.

There is also a very important iterative step whereby the reference site dataset that was used to develop the predictions is passed through the model to identify any reference sites that are of dubious biological quality ie that have less than 75% of the taxa that the model predicts they should have based on the site’s environmental characteristics. Such sites are then removed from the dataset and the model reconstructed.