Benchmarking and Assessment

This page exists to document progress of the Benchmarking and Assessment Working Group led by Kate Willett

Members (as of 12/23/10):

Kate Willett (UKMO Hadley Centre, UK) (Chair)

Claude Williams (NCDC, USA)

Ian Jolliffe (Exeter Climate Systems, University of Exeter, UK)

Robert Lund (Department of Mathematical Sciences, Clemson University, USA)

Lisa Alexander (Climate Change Research Centre, University of New South Wales, Australia)

Olivier Mestre (Meteo France, France)

Stefan Bronniman (University of Bern, Switzerland)

Lucie A. Vincent (Climate Research Division, Environment Canada, Canada)

Aiguo Dai (Climate and Global Dynamics Division, NCAR, USA)

Steve Easterbrook (Department of Computer Science, University of Toronto, Canada)
Victor Venema (Meteorologisches Institut, University of Bonn, Germany)
David Berry (National Oceanography Centre, Southampton, UK)
Mike Finney (Department of Mathematical Sciences, Clemson University, USA)
Rachel Warren (College of Engineering, Mathematics and Physical Sciences, University of Exeter, UK)
Giuseppina Lopardo (Istituto Nazionale di Ricerca Metrologica (INRiM), Italy)

Ex-officio:
Peter Thorne (CICS-NC, USA)

Purpose:

To facilitate use of a robust, independent and useful common benchmarking and assessment system for temperature data-product creation methodologies to aid product intercomparison and uncertainty quantification.

Blogsite for discussion of ideas/thoughts/work in progress:

http://surftempbenchmarking.blogspot.com
This blogsite is open to all and contructive comments are welcome.

Minutes:

Jan 25th 2011 14:00 GMT
Mar 30th 2011 14:00 GMT
Jun 15th 2011 14:00 GMT
Aug 11th 2011 14:00 GMT
Jan 31st 2013 15:00 GMT
May 3rd 2013 16:00 GMT+1

Conferences and Workshops:

November 2012 - Kate Willett's presentation at the 5th ACRE Meeting, Toulouse, France: presentation

June 2012 - Peter Thorne's presentation at the Earth Temperature Network workshop, Edinburgh, UK: presentation

May 2012    - Kate Willett's NCDC Visit with Claude Williams and Robert Lund - presentation
                  - Kate Willett's Clemson University/Robert Lund visit - presentation

December 2011 - Ian Jolliffe's 5th International Verification Methods Workshop, Melbourne, Australia:Benchmarking and Assessment (Verification) of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI), report.

October 2011 - Steve Easterbrook's WCRP Open Science Conference, Denver, CO, USA:Benchmarking and Assessment of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI). See Steve Easterbrook's blog.
                        - Kate Willett's COST HOME 7th Seminar for Homogenisation and Quality Control of Climate Databases, Budapest, Hungary: Creating a Global Benchmark Cycle for the International Surface Temperature Initiative.. Meeting Report (see blog too).

May 2011        - Kate Willett's presentation for MARCDATIII, Frascati, Italy, 2011: Is it good enough? Benchmarking homogenisation algorithms and cross-cutting with efforts for land observations

April 2011        - Kate WIllett's Poster for EGU 2011: Robust Benchmarking of Homogenisation Algorithms for the Surface Temperature Initiative

February 2011 - Kate Willett's informal presentation at the National Climate Data Center (NC, USA): Devising a Benchmarking System for Homogenisation Methods of Climate Data-Products

Working Group Documents:

White Paper 9 formed the basis for breakout group discussion at the Exeter meeting. Discussion outcomes are summarised in the final session.
Outline for the planned Homogenisation Review Paper to be written by the working group members
Terms of Reference agreed by the Benchmarking and Assessment Working Group (15/6/11)
Working draft of Benchmarking and Assessment Paper describing the methodological background to benchmarking and assessment
October 2011 Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee 10/11/2011
October 2012 (submitted Feb 2013) Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee xx/xx/2013

Objectives and Timelines:

Activity	Details	Owner	Due date
Advocacy of the benchmarks and support for users	All group members should be encouraging use of the benchmarks and providing support where necessary	Benchmarking and Assessment working group, Steering Committee	Ongoing
Up to date reference list of work on inhomogeneities in surface temperatures on the website (www.surfacetemperatures.org/benchmarking-and-assessment-working-group)	Ongoing throughout but will have formed the basis for defining error model spread.	Benchmarking and Assessment working group led by Kate Willett	Ongoing
Benchmarking and Assessment working group Terms of Reference	These will fit in with the Implementation Plan and Steering Committee Terms of Reference	Benchmarking and Assessment working group	June 2011 (Completed)
Benchmarking Position paper submitted for peer review	A descriptive paper presenting background concepts and methods for creation of the benchmark programme co-authored by the working group	Benchmarking and Assessment working group led by Kate Willett	June 2013 (in progress)
Analog-known-worlds proof-of-concept	Create software to produce analog-known-worlds and a proof-of-concept scale and submit methods paper	Team Creation lead by Robert Lund and Kate Willett	May 2013 (in progress)
Analog-known-worlds global scale production	Produce analog-known-worlds for as many ISTI land meteorological databank stage 3 stations as possible	Team Creation - code probably run and data hosted by Kate Willett	May 2013
Analog-error-worlds concepts finalised	Decide up number and type of error models to create (including how to ensure that these are blind tests for each cycle for some period of time)	Team Corruption - lead by Claude Williams	May 2013
Validation/Assessment concepts finalised	Decide on number and type of tests with which to perform validation	Team Validation - lead by Ian Jolliffe	August 2013
Analog-error-worlds proof-of-concept	Create software to produce analog-known-worlds and a proof-of-concept scale and submit methods paper (if desired)	Team Corruption - lead by Claude Williams	October 2013
Working Group meet up/code sprint	Attempt to get as many together as possible, possibly a networked code spring with a USA and Europe (Australia?) hub	All - kickstarted organisation by Kate Willett in April 2013	September/October 2013
Validation/Assessment proof-of-concept	Create software and score system/intercomparison table to run the validation on a proof-of-concept scale and submit methods paper (if desired)	Team Validation - lead by Ian Jolliffe	October 2013
Analog-error-worlds global scale production	Produce analog-error-worlds from the analog-known-worlds ready for distribution	Team Corruption - lead by Claude Williams	November 2013
Benchmark Cycle Official release of analog-error-worlds	Release first official benchmarks, publicise widely.	All - lead by Kate Willett	November/December 2013
Validation/Assessment global scale production	Produce software and framework ready for running on the global scale - automated or manual?	Team Validation - lead by Ian Jolliffe	End 2013
Benchmark platform design	Create webpage showing step-by-step 'How to benchmark' with appropriate links to data, validation and intercomparison tables with registration so that feedback can be provided and contact maintained	All - lead by Kate Willett	December 2014 - ideally earlier but more important to get benchmarks created first
Benchmark cycle release of analog-known-worlds answers	Publish the analog-known-worlds underlying the analog-error-world benchmarks	All - lead by Kate Willett	June 2016
Workshop to discuss results of benchmarking	To include Benchmarking and Assessment working group and all analysts who submitted	Benchmarking and Assessment working group	June 2016 ready for late 2016?
Summary paper submitted to peer reviewed journal	To include assessment of cycle 1 and recommendations for cycle 2	Benchmarking and Assessment working group	2017
Begin cycle 2 – creation of benchmarks and release – monthly and daily		Benchmarking and Assessment working group	2017

Reference Literature:

Peter Thorne et al.'s overview of ISTI including the need for benchmarking:
Thorne, P., Willett, K. M., et al., 2011: Guiding the creation of a comprehensive surface temperature resource for 21st century climate science. BAMS, 92 (11), ES40-ES47, doi: 10.1175/2011BAMS3124.1

Kate Willett's work on 'pseudo-worlds' - a set of benchmarks for homogenisation of daily Tmax and Tmin - please leave comments on the blogsite thread
Example plots to be uploaded shortly

Holly Titchner et al's work on radiosonde error models for validating the homogenisation

Claude William et al's work on homogenising USHCN with benchmarking of the methods:
Williams, C. N., Jr., M. J. Menne, and P. Thorne, in press: Benchmarking the performance of pairwise homogenization of surface temperatures in the United States. J. Geophys. Res., doi:10.1029/2011JD016761,
BLOGPOST

Victor Venema et al's work on benchmarking the COST HOME homogenisation algorithms:
Venema, V., O. Mestre, E. Aguilar, I. Auer, J.A. Guijarro, P. Domonkos, G. Vertacnik, T. Szentimrey, P. Stepanek, P. Zahradnicek, J. Viarre, G. Müller-Westermeier, M. Lakatos, C.N. Williams, M. Menne, R. Lindau, D. Rasol, E. Rustemeier, K. Kolokythas, T. Marinova, L. Andresen, F. Acquaotta, S. Fratianni, S. Cheval, M. Klancar, M. Brunetti, Ch. Gruber, M. Prohom Duran, T. Likso, P. Esteban, Th. Brandsma., 2012: Benchmarking homogenization algorithms for monthly data, Climate of the Past, 8, pp. 89-115, 2012.
BLOGPOST

Links to Related Projects:

www.homogenisation.org - website for the COST HOME action on homogenisation

Last modified by Kate Willett: Jul 19th 2011

Subpages (1): What is Benchmarking?

International Surface Temperature Initiative