Verification development: current status

 
Ben Wichers Schreur, KNMI
 

The report on the expert meeting on verification (HIRLAM newsletter no. 33) and the invitation to comment and criticise on its outcome disappointingly evoked no response from the project staff, despite the expressed urgency for action with regard to the development of the HIRLAM verification package. This is taken to be a sign both of support for the strategy outlined by the expert meeting and of the conceptual difficulties in mesoscale and precipitation verification. 

The general struggle in the NWP community to define meaningful mesoscale verification methodologies was also highlighted by discussions at the recent ECMWF seminar on 'Diagnosis of Models and Data Assimilation Systems'. A particular point that was brought up, which has its implications for the HIRLAM verification system, is that different countries use different definitions of what constitutes an extreme event that a mesoscale NWP systems is required to predict. 

From this discussion it was concluded that the HIRLAM mesoscale verification package should use verification methodologies that are flexible and, to a degree, user definable to accommodate the different requirements of HIRLAM member states. This prompted the development of a masking tool that may serve as the basic building block for the selection of data in response to complicated verification questions. Furthermore in line with the strategy outlined by the expert meeting first versions were developed of a field verification package and of a precipitation package. At FMI a bug in the observation verification package was detected. The use of a very coarse resolution in the GRIB recoding of BUFR observations of height resulted in poor scores for HIRLAM height fields and basically made model improvements undetectable.

Current work on verification is focussed on correcting errors in the observation verification package and on implementing the masking tool, field verification package and precipitation verification package in the reference system.

Observation verification package

The HIRLAM observation verification package is based on two principles: it uses the same BUFR observation files that are input to the analysis system and BUFR observations are recoded to a GRIB field. The first principle allows the verification system to be run without modification both in the reference system and in local implementations of the HIRLAM system in member states. The second principle allows the verification package to use the field processing routines of the HIRLAM system, such as interpolation and also the new masking tool, to process observations. In fact it allows the application of the same verification methods and scores to both observation verification and field verification. These synergies are a powerful reason to maintain the current observation package. At the same time it is recognized that the system is in need of an update. It still uses rather rudimentary data quality control. It is worth considering using the results of the analysis data quality control in the verification. The bug report from FMI suggests further that some preventive maintenance may be in order. 

Masking tool

A masking tool has been developed by extending the functionality of the field manipulation routine MANPFD in the library util. It allows data from one field to be extracted on the basis of a masking criterion applied to a second field, e.g. temperature when wind speed < 2 m/s. The routine is controlled by a namelist, that may be called iteratively, allowing the construction of masks of arbitrary complexity. 

Using this masking tool in the verification package can provide answers to such questions as:

  • what is the rainfall error for a river catchment area;
  • what is the error in the heat flux over sea when the wind speed drops below 1 m/s;
  • what are the temperature errors over high ground in Ireland.
Thus the verification system has the required flexibility to respond to application oriented questions. At the same time it can act as an external diagnostics tool. As both observation verification and field verification use GRIB fields, the masking tool applies to both and verification issues may be answered both by observation verification and field verification.

Masks may be based on model fields or on more traditional masking fields as are defined in the climate files, e.g. land-sea mask, height of terrain, land use classification. For application to precipitation verification KNMI is in the process of acquiring rights to data on catchment areas of all major European rivers and their subsidiaries and plans to make this available to the HIRLAM community.

Field verification package

A general field verification package has been developed that allows the verification of model forecast fields against model analysis or any other, user defined, truth. The package calculates the major moments of the error distribution and writes these out to file. At the same time these error moments are accumulated as fields over a user definable period, to allow the assessment of the geographic distribution of errors. In addition to the error moments the field verification package calculates the correlation between forecast field and verification field relative to a, again user definable, reference field. In this way anomaly correlations, where the reference is climatology, or tendency correlations, where the reference is persistence, may be calculated. 

Precipitation verification package

A precipitation verification package has been developed as an extension to the current observation verification package. It uses the precipitation observations from synoptic stations contained in the BUFR observation file. Thus, in line with reporting practice, it verifies 12 hour sums at 06 and 18 UTC and 6 hour sums at 00 and 12 UTC. It calculates the major moments of the error distribution and accumulates observation/forecast pairs in a contingency table. The category limits in this table are user definable through a namelist.

Work plan

Priority is given to the implementation of current developments in the reference system. This would fulfil the short term aims of the precipitation and field verification projects as defined by the expert meeting. Completion is foreseen for the first quarter of the year 2000. Subsequently a mesoscale verification project will be started. This project aims to deliver methodologies for the subjective and objective scoring of mesoscale forecasts and severe/rare weather events. A report of the definition phase of this project will be presented by June 2000.

SRNWP Lead Centre for Verification

Within the framework of the EUMETNET Short Range Numerical Weather Prediction Programme a new lead centre has been created for verification. This lead centre will be under the responsibility of the HIRLAM project, which has delegated all the operative aspects to KNMI. Aim of the lead centre for verification is to deal with the scientific issues related to the verification of numerical models, in particular those related to the steadily increasing resolution of present day models. Other issues identified are the verification of turbulent and radiative fluxes, involving the relation between point measurement flux values and grid averged fluxes, verification of the vertical structure of the atmosphere and the use of new observational data such as AMDAR in verification. Your input on other verification topics to be included in a SRNWP/HIRLAM workshop on verification is very welcome.

Your comments please

Again I urge all HIRLAM staff to help the verification development effort by directing comments, suggestions and criticisms to me (ben@knmi.nl).