Benchmarking and Assessment

This page exists to document progress of the Benchmarking and Assessment Working Group led by Kate Willett

Members (as of 12/23/10):

Kate Willett (UKMO Hadley Centre, UK) (Chair)

Claude Williams (NCDC, USA)

Ian Jolliffe (Exeter Climate Systems, University of Exeter, UK)

Robert Lund (Department of Mathematical Sciences, Clemson University, USA)

Lisa Alexander (Climate Change Research Centre, University of New South Wales, Australia)

Olivier Mestre (Meteo France, France)

Stefan Bronniman (University of Bern, Switzerland)

Lucie A. Vincent (Climate Research Division, Environment Canada, Canada)

Aiguo Dai (Climate and Global Dynamics Division, NCAR, USA)

Steve Easterbrook (Department of Computer Science, University of Toronto, Canada)
Victor Venema (Meteorologisches Institut, University of Bonn, Germany)
David Berry (National Oceanography Centre, Southamton, UK)
Ingvild Antonsen (Justervesenet - The Norwegian Metrology Service, Norway)

Purpose:

To facilitate use of a robust, independent and useful common benchmarking and assessment system for temperature data-product creation methodologies to aid product intercomparison and uncertainty quantification.

Blogsite for discussion of ideas/thoughts/work in progress:

http://surftempbenchmarking.blogspot.com
This blogsite is open to all and contructive comments are welcome.

Minutes:

Jan 25th 2011 14:00 GMT
Mar 30th 2011 14:00 GMT
Jun 15th 2011 14:00 GMT
Aug 11th 2011 14:00 GMT

Conferences and Workshops:

December 2011 - Ian Jolliffe's 5th International Verification Methods Workshop, Melbourne, Australia:Benchmarking and Assessment (Verification) of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI), report.

October 2011 - Steve Easterbrook's WCRP Open Science Conference, Denver, CO, USA:Benchmarking and Assessment of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI). See Steve Easterbrook's blog.
                        - Kate Willett's COST HOME 7th Seminar for Homogenisation and Quality Control of Climate Databases, Budapest, Hungary: Creating a Global Benchmark Cycle for the International Surface Temperature Initiative.. Meeting Report (see blog too).

May 2011        - Kate Willett's presentation for MARCDATIII, Frascati, Italy, 2011: Is it good enough? Benchmarking homogenisation algorithms and cross-cutting with efforts for land observations

April 2011        - Kate WIllett's Poster for EGU 2011: Robust Benchmarking of Homogenisation Algorithms for the Surface Temperature Initiative

February 2011 - Kate Willett's informal presentation at the National Climate Data Center (NC, USA): Devising a Benchmarking System for Homogenisation Methods of Climate Data-Products

Working Group Documents:

White Paper 9 formed the basis for breakout group discussion at the Exeter meeting. Discussion outcomes are summarised in the final session.
Outline for the planned Homogenisation Review Paper to be written by the working group members
Terms of Reference agreed by the Benchmarking and Assessment Working Group (15/6/11)
Working draft of Benchmarking and Assessment Paper describing the methodological background to benchmarking and assessment
October 2011 Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee 10/11/2011

Objectives and Timelines:

Activity	Details	Owner	Due date
Advocacy of the benchmarks and support for users	All group members should be encouraging use of the benchmarks and providing support where necessary	Benchmarking and Assessment working group, Steering Committee	Ongoing
Up to date reference list of work on inhomogeneities in surface temperatures on the website (www.surfacetemperatures.org/benchmarking-and-assessment-working-group)	Ongoing throughout but will have formed the basis for defining error model spread.	Benchmarking and Assessment working group led by Kate Willett	Ongoing
Benchmarking and Assessment working group Terms of Reference	These will fit in with the Implementation Plan and Steering Committee Terms of Reference	Benchmarking and Assessment working group	June 2011 (Completed)
Benchmarking Position paper submitted for peer review	A descriptive paper presenting background concepts and methods for creation of the benchmark programme co-authored by the working group	Benchmarking and Assessment working group led by Kate Willett	April 2012
Creation and release of pilot benchmark analog-known-worlds and analog-error-worlds (monthly data)	Use frozen version 1 of the consolidated master database, Legacy code created (well documented) with easily tweakable parameters.	Benchmarking and Assessment working group - assigned error model creation group to choose parameters and run code.	November 2012 (8 month lag to release of databank version 1)
Creation of official cycle 1 benchmark analog-known-worlds and analog-error -worlds (monthly data) for official release and release of the analog-error-worlds	Use frozen version 1 of the consolidated master database, and legacy code created previously with tweaked parameters.	Benchmarking and Assessment working group - assigned error model creation group to choose parameters and run code.	November 2012 (8 month lag to release of databank version 1)
Release of official analog-known-worlds	The analog-known-worlds should be released prior to the workshop so that results/necessary improvements can be brought together during the workshop.	Benchmarking and Assessment working group	November 2014
Workshop to discuss results of benchmarking	To include Benchmarking and Assessment working group and all analysts who submitted	Benchmarking and Assessment working group	April 2015
Summary paper submitted to peer reviewed journal	To include assessment of cycle 1 and recommendations for cycle 2	Benchmarking and Assessment working group	1^st draft August 2015 submitted November 2015
Begin cycle 2 – creation of benchmarks and release – monthly and daily		Benchmarking and Assessment working group	November 2015

Reference Literature:

Kate Willett's work on 'pseudo-worlds' - a set of benchmarks for homogenisation of daily Tmax and Tmin - please leave comments on the blogsite thread
Example plots to be uploaded shortly

Holly Titchner et al's work on radiosonde error models for validating the homogenisation

Claude William et al's work on homogenising USHCN with benchmarking of the methods:
Williams, C. N., Jr., M. J. Menne, and P. Thorne, in press: Benchmarking the performance of pairwise homogenization of surface temperatures in the United States. J. Geophys. Res., doi:10.1029/2011JD016761,
BLOGPOST

Victor Venema et al's work on benchmarking the COST HOME homogenisation algorithms:
Venema, V., O. Mestre, E. Aguilar, I. Auer, J.A. Guijarro, P. Domonkos, G. Vertacnik, T. Szentimrey, P. Stepanek, P. Zahradnicek, J. Viarre, G. Müller-Westermeier, M. Lakatos, C.N. Williams, M. Menne, R. Lindau, D. Rasol, E. Rustemeier, K. Kolokythas, T. Marinova, L. Andresen, F. Acquaotta, S. Fratianni, S. Cheval, M. Klancar, M. Brunetti, Ch. Gruber, M. Prohom Duran, T. Likso, P. Esteban, Th. Brandsma., 2012: Benchmarking homogenization algorithms for monthly data, Climate of the Past, 8, pp. 89-115, 2012.
BLOGPOST

Links to Related Projects:

www.homogenisation.org - website for the COST HOME action on homogenisation

Last modified by Kate Willett: Jul 19th 2011

Subpages (1): What is Benchmarking?

Surface Temperatures Home	Search this site

Surface Temperatures Home