3.2: Statistics – Assessing data

In deciding how the performance of a water supply system should be assessed, it is necessary to consider:

the statistical implications of the assessment mechanism;
possible health implications of using different statistical measures;
community perceptions of what constitutes good quality water.

Three commonly used procedures measure performance respectively against a maximum value, a mean, or a percentile (Ellis 1989).

Assessing data against a maximum value

Using this approach, performance is measured by quoting the percentage of scheduled samples tested that are below the guideline value. Although the approach is used often and is superficially easy to understand, it has some serious deficiencies:

While measurements will show how a system is performing at the time of sampling, there is no way of determining what the water quality is like between sampling events. Statistical procedures cannot be used to indicate whether or not the measurements are representative of the quality at other times. (Other methods of assessing performance, however, can provide this information.)
There is no way of reliably estimating what the true maximum value is, as this may well occur between samples. Any sampling program can only provide a biased estimate of the true maximum value, which it will invariably underestimate. There is always the possibility that the next sample analysed may have a higher value.

Assessing data against a mean

Performance is assessed by comparing the mean value of measurements with the guideline value over a period (usually 12 months). Such an approach has a number of attractions:

For characteristics not related to health, the guideline values are generally set at values that have the potential to generate a change that is noticeable by the customer. In many cases, it is sudden large increases in a value that can bring an increased number of consumer complaints. Therefore, when looking at trends over time, it can be argued that it is the mean or average value that is the significant value in relation to system performance.
Simple and well recognised statistical procedures can be used to provide statistically unbiased estimates of the mean with a known degree of confidence. The degree of confidence (expressed as the confidence interval) will indicate how much the values are likely to vary from the mean between sampling events.

The disadvantage of this approach is that a few high values can be offset by a number of low values.

Assessing data against a percentile

Using this approach, performance is satisfactory if a large percentage of results (although not necessarily all) are less than the guideline value. Like the use of a mean, this approach has a number of attractions:

For health-based characteristics, performance could not be regarded as satisfactory if the guideline values were exceeded more than rarely. This is consistent with using a high percentile such as a 95th percentile.
It is possible, using statistical procedures, to estimate with a known degree of confidence how well the results of sampling represent the quality of water at other times.
Using a percentile to assess performance against the guideline is consistent with the requirement that the upper control limit of the control chart be equal to or less than the guideline value. For example, for normally distributed data, if the 95th percentile is used, control limits can be placed at 1.64 times the standard deviation on either side of the mean. These control limits will then encompass about 90% of the data, and of the remaining 10%, about 5% will be above the upper control limit and 5% below the lower. This means that if the upper control limit is the same as, or less than, the guideline value, then 95% or more of the data should be below the guideline value.
More samples need to be analysed to assess performance against a percentile than are needed for a mean. This is reasonable for health-based characteristics, as exceeding the guideline may, in some cases, have significant health effects. More sampling provides a greater degree of protection.

The main disadvantage of this approach is that estimates of percentiles are inherently more uncertain than estimates of means.

If more detailed or involved data analysis techniques are to be considered the advice of a statistician should be sought.

Assessing microbial health-based target microbial band allocation data

A microbial band allocation is used to provide a measure of the overall level of faecal contamination of the source water using E. coli as the microbial indicator. It is an important step in establishing the category of the source water and subsequently the required level of treatment to meet the microbial health-based targets. Section 5.3 describes this process in more detail.

The suggested monitoring period to characterise microbial risk in the source water is two years, which would provide at least 100 data points with weekly sampling. The maximum E. coli results should be used for the allocation of the microbial band (Walker et al. 2015) unless the dataset is robust enough to use the 95th percentile. Adoption of a percentile should first be discussed with the relevant health authority or drinking water regulator to determine if this is an appropriate option.

References

Ellis JC (1989). Handbook on the Design and Interpretation of Monitoring Programmes. Water Research Centre, Medmenham, United Kingdom, Technical Report NS29.

Walker E, Canning A, Angles M, Ball A, Stevens M, Ryan G, Liston C, Deere D (2015). Semi Quantitative Assessment of Microbial Source Risk. Australian Experience from Pilots of implementing a Health Based Target for Microbial Water Quality, Occasional Paper, Water Research Australia.

Previous3.1: Statistics – Visualising data Next3.3: Statistics – Statistical principles

Last updated 5 months ago

Assessing data against a maximum value

While measurements will show how a system is performing at the time of sampling, there is no way of determining what the water quality is like between sampling events. Statistical procedures cannot be used to indicate whether or not the measurements are representative of the quality at other times. (Other methods of assessing performance, however, can provide this information.)

There is no way of reliably estimating what the true maximum value is, as this may well occur between samples. Any sampling program can only provide a biased estimate of the true maximum value, which it will invariably underestimate. There is always the possibility that the next sample analysed may have a higher value.

Assessing data against a mean

Performance is assessed by comparing the mean value of measurements with the guideline value over a period (usually 12 months). Such an approach has a number of attractions:

For characteristics not related to health, the guideline values are generally set at values that have the potential to generate a change that is noticeable by the customer. In many cases, it is sudden large increases in a value that can bring an increased number of consumer complaints. Therefore, when looking at trends over time, it can be argued that it is the mean or average value that is the significant value in relation to system performance.

Simple and well recognised statistical procedures can be used to provide statistically unbiased estimates of the mean with a known degree of confidence. The degree of confidence (expressed as the confidence interval) will indicate how much the values are likely to vary from the mean between sampling events.

The disadvantage of this approach is that a few high values can be offset by a number of low values.

Assessing data against a percentile

For health-based characteristics, performance could not be regarded as satisfactory if the guideline values were exceeded more than rarely. This is consistent with using a high percentile such as a 95th percentile.

It is possible, using statistical procedures, to estimate with a known degree of confidence how well the results of sampling represent the quality of water at other times.

Using a percentile to assess performance against the guideline is consistent with the requirement that the upper control limit of the control chart be equal to or less than the guideline value. For example, for normally distributed data, if the 95th percentile is used, control limits can be placed at 1.64 times the standard deviation on either side of the mean. These control limits will then encompass about 90% of the data, and of the remaining 10%, about 5% will be above the upper control limit and 5% below the lower. This means that if the upper control limit is the same as, or less than, the guideline value, then 95% or more of the data should be below the guideline value.

More samples need to be analysed to assess performance against a percentile than are needed for a mean. This is reasonable for health-based characteristics, as exceeding the guideline may, in some cases, have significant health effects. More sampling provides a greater degree of protection.

The main disadvantage of this approach is that estimates of percentiles are inherently more uncertain than estimates of means.

If more detailed or involved data analysis techniques are to be considered the advice of a statistician should be sought.

Assessing microbial health-based target microbial band allocation data

References

Ellis JC (1989). Handbook on the Design and Interpretation of Monitoring Programmes. Water Research Centre, Medmenham, United Kingdom, Technical Report NS29.