RIVPACS - Quantifying Uncertainty in Ecological Quality Indices

It is important to realise that all methodological decisions have implications for the reliability, precision and robustness of any resulting indices for assessing the ecological quality and status of individual river stretches. Whatever EQI and classification system is chosen, it is vital to have some idea of the effects of sampling variation and other errors on the value of the index for any particular site. An index of ecological quality is of little value without some knowledge of its levels of "uncertainty".

The WFD stipulates that government environment agencies estimate the confidence limits and precision of their monitoring results for individual water bodies, be it rivers or lakes. It is therefore necessary to be able to estimate the confidence of the assignment of a site to a particular Ecological Status class. In other words, what are the risks of misclassifying sites to the wrong grade?

Also it is important to be able to assess whether there is a statistically significant or ecologically real difference in the quality estimates obtained for two samples, whether from the same site at two points in time or from two different sites, e.g. is an EQI value of 0.75 significantly less than 0.95?

The decision on how many grades or classes to have and where to place the class limits in terms of the EQI values should take account of statistical aspects, such as acceptable misclassification rates. Having more classes gives finer apparent discrimination of quality but greater actual misclassification rates. The WFD requires quality indices to be divided into five ecological status classes, not the six grades currently used in UK.

The actual uncertainty and reliability of estimates of the biological condition for a site based on particular O/E ratios must depend on sampling variation and errors in estimating the observed fauna and hence the observed index value as well as errors in predicting the site-specific Reference Condition and expected index value for a site. The potential sources of errors in estimating the Reference Condition have been discussed earlier. The two sources of errors in estimating the observed fauna at a test site are due to sampling variation and sampling processing (including taxonomic identification) errors. Spatial heterogeneity in stream microhabitat structure at a site results in variation in macroinvertebrate taxonomic richness and community composition between samples taken during the same period. A UK study found that one standardised three minute RIVPACS sample typically contained 50% of the species and 60% of the families found amongst six replicate samples at a site. Therefore sampling variation and sampling processing errors need to be quantified for a wide range of types and qualities of site.

There has been considerable thought and discussion on how to assess the reliability of O/E estimates. It is clear that the choice of reference sites, environmental variables and prediction method must all be treated as an integral part of the definition of the quality index. The only errors in estimating the expected fauna and values for new sites are assumed to arise from errors in measuring their values for the environmental predictor variables.

The current version of RIVPACS in the UK has a powerful function whereby it can generate statistical information on the reliability of classification or risk of misclassification. It achieves this using an uncertainty simulation model developed by CEH scientists which integrates errors from a variety of sources to generate simulated values of O/E (Figure 1).

Figure 1. Component sources of error in the RIVPACS uncertainty simulation model

All of these different sources of variation and error have been incorporated into the uncertainty simulation module within the RIVPACS software system to estimate the uncertainty distribution of O/E values for one or more samples. Five hundred or more simulated O/E values are generated to estimate the distribution and range within which the "true" O/E value for the site probably lies.

The example given in Figure 2 shows the simulated potential values of O/E for two samples; 1 and 2. The simulated O/E values have also been classified into grades a, b, and c, shown in blue, green and red, which can be thought of as "high", "good" and "moderate" status classes. Sample 1 has an O/E of 0.94, but the simulated distribution gives it 95% confidence limits from 0.78 to 1.10. The probability of belonging to grade "a" or "high" is 85%, and there is a 15% chance it belongs to grade "b" or "good". Therefore we can say that it is definitely of "good" or better’ status.

In this example, sample 2 is of poorer quality than sample 1. Although classified as of "good" ecological status, the site has an estimated 30% probability of being of "moderate" ecological quality. The simulation distribution, on the right, showing the possible difference between the two sample O/E values enables us to assess the likelihood and extent of real change in ecological quality between the two sites or two surveys at the same site.