Quality assurance program
Introduction
To ensure that the data collected and presented by Beachwatch are accurate and reliable, a quality assurance (QA) program is undertaken by Beachwatch. It includes QA of:
Sample collection
Background
Water samples for the Beachwatch Programs are collected by a number of different agencies. These sampling activities are audited during the year.
Sydney - Beachwatch Programs staff collect water samples from 28 Sydney northern and city ocean beaches, one lagoon site and 59 swimming locations in Pittwater, Sydney Harbour, Botany Bay, Lower Georges River and Port Hacking.
Sutherland Shire Council lifeguards collect water samples from the eight southern beaches in Bate Bay.
Hunter - Hunter Water collect water samples from 17 ocean beaches in the Port Stephens, Newcastle and Lake Macquarie shires.
Illawarra - Sydney Water collects water samples from 18 ocean beaches in the Wollongong, Shellharbour and Kiama shires.
Method
The collection of water samples by Sutherland Shire Council, Hunter Water and Sydney Water is audited by Beachwatch Programs Field Officers several times throughout the year. Audits on Beachwatch Programs field officers are conducted during the year by the Beachwatch Senior Project Officer.
Audits include an assessment of the field officer's performance according to established Beachwatch Programs sampling protocols, including aseptic sampling technique, sample collection, and storage and documentation of field observations. These protocols are based on internationally recognised methods for the collection of water samples in recreational bathing areas (APHA 1998).
Percentage compliance is calculated for each of the three components of the audit: sampling technique, sample collection, and storage and field observations. A total compliance is then calculated as the average compliance from each of these three components.
After each audit, a report is prepared that describes the results of the audit and any issues of concern. A copy of the report is provided to the audited organisation for its information and for action, if necessary.
Results and discussion
The results of the quality assurance audits conducted between May 2008 and April 2009 are summarised in this report.
Overall compliance was calculated as the average total compliance achieved from the audits.
Sydney results -Beachwatch field officers achieved 100% compliance with the established sampling protocols for the beach sampling run. Sutherland Shire Council lifeguards also achieved an overall compliance of 100% with the established sampling protocols.
Hunter results -Sample collection in the Hunter region complied well with established Beachwatch sampling protocols. Hunter Water achieved an overall compliance of 100% from the quarterly audits.
Illawarra results -Sample collection by Sydney Water in the Illawarra region complied well with established Beachwatch sampling protocols, with a compliance of 100%.
Conclusion
Auditing of sample collection in the Sydney, Hunter and Illawarra regions has shown a high level of compliance with established Beachwatch sampling protocols.
Sampling officers displayed a good understanding of water sampling, collection and storage techniques and a sound knowledge of marine water quality issues and potential beach pollution sources.
These results indicate that water quality samples collected by, and provided to, Beachwatch are of an acceptable and high standard.
Laboratory analysis
Background
Since February 1993, Beachwatch has regularly sent sets of water samples to a number of accredited microbiological laboratories in order to determine the reliability of the routine data provided by the contracted laboratory relative to those from other laboratories. The quality assurance program started with three laboratories but has since been expanded to six.
Preparation of samples
At the start of every month, three duplicate environmental samples from the lower Parramatta River were submitted to six laboratories participating in the quality assurance program. Each laboratory analysed its three subsamples for faecal coliforms and enterococci bacteria. The exception was during January 2009, when only five labs provided results for faecal coliforms.
Data analysis
The geometric mean of each of the samples was calculated from the results of all laboratories; this is termed the consensus mean. This consensus mean was used to represent the best estimate of the true density of the particular bacterial group in the sample. The individual results for each laboratory were then compared with the consensus mean to calculate the relative deviation.
Over a large number of samples, a trend develops for each laboratory of being over, under or similar to the consensus mean.
Results and discussion
Figures showing the relative accuracy of quality assurance data for each laboratory and the deviation of results of the contracted laboratories from the consensus mean are contained in Appendix 2, together with details of the statistical analysis used to assess the performance of the contracted laboratory.
A total of 81% of the contracted laboratory's faecal coliform results were within 0.3 log units of the consensus mean. Values greater than +0.3 log units or less than -0.3 log units reflect samples that were double or half the consensus mean, respectively. In most cases, faecal coliform outside this range were under-reported by the contracted laboratory. As can be seen from Figure 32 and Figure 33, four values were under-reported and outside the 0.3 log unit. The laboratory was notified of this incident and since then all results have been reported within the acceptable range.
For enterococci, 78% of the contracted laboratory's enterococci results were within 0.3 log units of the consensus mean (equivalent to a range of half to twice the consensus mean). Most enterococci results outside this range were under-reported, only slightly outside the 0.3 log unit range and at low bacterial densities, as shown in Figure 33.
Statistical analysis of the data indicated that the faecal coliform results were not significantly different among laboratories. The results for faecal coliforms would therefore not be significantly different if another contracted laboratory had been used.
Similar to the faecal coliform results, no significant difference was found among laboratories for enterococci results. The results from the contracted laboratory were consistant with those from most other laboratories.
Conclusions
In summary, although there were some differences among the performance of all laboratories, these were relatively small, and similar accuracy and precision were achieved for each laboratory for both faecal coliform and enterococci results.
The contracted laboratory did not differ significantly from the majority of laboratories in its estimation of faecal coliform or enterococci densities, hence confirming that its analyses of bacterial counts are within the acceptable range. Therefore, there is confidence in the accuracy of the results reported for water quality in the Sydney region.
This quality assurance program highlights the observation that a single result from any one laboratory can be substantially different from those reported by the other laboratories. Thus, when monitoring and reporting on water quality at the ocean and harbour beaches, Beachwatch should not emphasise any single result. It is preferable, and more reliable, to report the results of population and trend analyses to minimise the effect of the occasional aberrant result.
Reporting to the community
Background
Providing the community with current beach water quality information is a core function of Beachwatch, so consequently reporting has been incorporated into the quality assurance program. The quality assurance program on reporting enables Beachwatch to measure the accuracy, consistency of content and punctuality of all reports released. When necessary, this information is used to improve the reporting process.
Introduction: Sydney daily bulletins
Beachwatch and Harbourwatch bulletins are generated daily to report on the likelihood of bacterial contamination at Sydney ocean and harbour swimming areas. This information can be accessed by the public through the Beachwatch website (www.environment.nsw.gov.au/beach) and the Beachwatch and Harbourwatch telephone information line (1800 036 677). The information is also sent by fax and email to a range of stakeholders and media on the morning each daily bulletin is created.
Beachwatch and Harbourwatch bulletins are based on telemetered rainfall data and any reported pollution incidents that could affect beach water quality. The bulletins include a prediction of the likelihood of pollution at ocean beaches and harbour swimming areas, as well as the daily weather, tides and coastal conditions, based on the Australian Bureau of Meteorology's Metropolitan Forecast and Coastal Waters Forecast.
Method
Daily bulletins are audited weekly to determine the accuracy of reported information and the punctuality of distribution. There are eight parameters measured in total.
For Beachwatch daily bulletins:
For Harbourwatch daily bulletins:
Weather information common to both:
In addition, the accuracy and punctuality of any updates to the daily bulletins are measured.
The results from the quality assurance audits are stored in an electronic database, with a weekly summary of any detected errors distributed to the field officers for their attention and action, if necessary.
Results and discussion
The results of the quality assurance audits for the daily bulletins are presented in Figure 31, with results for both the summer and winter seasons shown. A high level of compliance was achieved for all of the measured variables.
For the Beachwatch daily bulletin, the pollution scenario was accurately reported 100% of the time in summer and in winter. The reports were faxed out within ten minutes of the issue time on 96% of occasions in winter and 95% of the time in summer. Where bulletins were late, this was usually only by a matter of minutes and the result of technical problems.
For the Harbourwatch daily bulletin, the pollution scenarios for swimming areas in Pittwater, Sydney Harbour and Botany Bay (including the Lower Georges River) were accurately reported at least 99% of the time in both summer and winter. Pittwater pollution scenarios were accurate for 100% of the time for winter and summer, and the Sydney Harbour pollution scenario was accurate for 100% of the time during winter and 99% of the time during summer. The Botany Bay pollution scenario was accurate 99% of the time during both winter and summer. The Harbourwatch bulletin was timely on 95% of occasions in both winter and summer. Again, where the fax-out of bulletins was delayed, this was usually only by a matter of minutes and resulted from technical problems.
The reported rainfall was correct 100% of the time in winter and 96% of the time in summer. The tidal information was correct 98% of the time in both winter and summer. In all cases, the errors were minor and were most often related to formatting.
Only six Harbourwatch bulletin updates were required during 2008-2009. All were accurate and only one was not faxed out within ten minutes of the issue time. The Beachwatch bulletin updates were accurate 100% of the time in both winter and summer. The bulletin updates were faxed out within ten minutes of the issue time on 100% of occasions in winter and on 96% of occasions in summer.
Conclusion
The daily bulletins generated by Beachwatch were accurate 96% to 100% of the time and were distributed on time 95% to 100% of the time. Where errors were detected, these were minor and generally related to formatting. Where delays in bulletin distribution were detected, these were only in the order of minutes and were the result of technical problems.
Weekly star ratings
Introduction
Each week during the summer season, Hunter Water and Sydney Water publish water quality data in local newspapers as 'star ratings'. The star ratings are based on NHMRC (1990) guidelines, with one star indicating poor water quality, through to a four-star rating indicating excellent water quality.
Results for the Hunter region are published in the Newcastle Herald and the Port Stephens Examiner newspapers. Results for the Illawarra are published in the Illawarra Mercury newspaper.
Method
The star ratings published in the Hunter region are calculated by Beachwatch, confirmed by Hunter Water and then submitted to the newspaper for preparation of the graphic. A copy of the graphic is then supplied to Beachwatch for checking and approval prior to publication.
The same procedure is followed in the Illawarra region, with Beachwatch calculating the star ratings and Sydney Water confirming them before submission to the newspaper. The graphic produced by the newspaper is checked and approved by Beachwatch prior to publication.
Results and conclusion
As the star ratings for the Hunter and Illawarra beaches are both quality assured by two different agencies, any potential errors in calculations or in the presentation of data were detected prior to publication. No erroneous reports were published in 2008-2009.
Monthly reports
The Beachwatch and Harbourwatch monthly reports present the level of compliance with swimming guidelines for the previous month. These reports are available on the Beachwatch website under 'Reporting'. Before distribution, all computer-generated compliance calculations are checked manually to ensure accuracy. As a result, all errors are detected before distribution of the reports.
Figure 31: Results of the quality assurance program for Beachwatch and Harbourwatch daily bulletins during winter 2008 and summer 2008-2009.
Page last updated: 26 February 2011
