 This page exists to document progress of the Benchmarking and Assessment Working Group led by Kate Willett Contents:
Purpose Blogsite Minutes Conferences and Workshops Working Group Documents Objectives and Timeline Reference Literature Links to related projects Members (as of 31/5/13):Kate Willett (Met Office Hadley Centre, UK) (Chair)
Claude Williams (NCDC, USA)
Ian Jolliffe (Exeter Climate Systems, University of Exeter, UK)
Robert Lund (Department of Mathematical Sciences, Clemson
University, USA)
Lisa Alexander (Climate Change Research Centre, University of
New South Wales, Australia)
Stefan Brönniman (University of Bern, Switzerland)
Lucie A. Vincent (Climate Research Division, Environment Canada, Canada)
Steve Easterbrook (Department of Computer Science, University
of Toronto, Canada)
Victor Venema (Meteorologisches Institut, University of Bonn, Germany)
David Berry (National Oceanography Centre, Southampton, UK) Rachel Warren (College of Engineering, Mathematics and Physical Sciences, University of Exeter, UK) Giuseppina Lopardo (Istituto Nazionale di Ricerca Metrologica (INRiM), Italy) Renate Auchmann (Oeschger Center for Climate Change Research & Institute of Geography, University of Bern, Switzerland) Enric Aguilar (Centre for Climate Change, Universitat Rovira i Virgili, Spain)
Matt Menne (NCDC, USA) Colin Gallagher (Department of Mathematical Sciences, Clemson University, USA)
Zeke Hausfather (Berkeley Earth,USA)
Thordis Thorarinsdottir (Statistical Analysis, Pattern Recognition, and Image Analysis (SAMBA), Nowegian Computing Centre, Norway)
Robert Dunn (Met Office Hadley Centre, UK)
Ex-officio: Peter Thorne (NERSC, Norway) Purpose:To facilitate use of a robust, independent and useful common benchmarking and assessment system for temperature data-product creation methodologies to aid product intercomparison and uncertainty quantification.Blogsite for discussion of ideas/thoughts/work in progress:http://surftempbenchmarking.blogspot.comThis blogsite is open to all and constructive comments are welcome. 2013/05 Call for regional inhomogeneity info
2012/01 Mailing list on homogenisation of climate data 2012/01 New article: Benchmarking homogenization algorithm...benchmarking, climate variability, HOME, Homogenization, open-access publishing 2012/01 Benchmarking of USHCN 2011/12 Metadata 2011/11 Team Validation - thoughts from the Homogenisation... 2011/11 Team Corruption - Thoughts from the Homogenisation... 2011/11 Team Creation - thoughts from the Homgenisation Me... 2011/11 2011 Progress Report Now Published 2011/18 Team Validation 2011/07 Benchmark for real-world problems 2011/07 Another radiosonde benchmarking paper 2011/07 Generating inhomogeneous worlds 2011/07 Benchmarking temperature networks 2011/06 Homogenization seminar 2011/06 If I had but one analog I could create ... 2011/06 Big questions with which to test homogenisation al... 2011/03 Creating the Benchmark 'Truths' 2011/02 Assessing the Benchmarks 2011/02 Review paper references 2011/02 My first time using blog... 2011/01 Homogenization aspects that scare me 2011/01 Kate's Pseudo-worlds work 2011/01 Benchmarking and Assessment Open Comment - January...
RELATED BLOG POSTS:
2012/01 What distinguishes a benchmark?
#22 Aug 10th 2015 13:00 GMT #21 Aug 12th 2014 14:00 GMT #20 Feb 17th 2014 13:00 GMT #19 Jan 28th 2014 15:00 GMT #18 Dec 12th 2013 13:00 GMT #17 Nov 27th 2013 18:00 GMT #16 Nov 15th 2013 14:00 GMT #15 Oct 28th 2013 13:00 GMT Benchmarking workshop (NCDC, July 2013) Workshop agenda Overall report #14 Jul 3rd 2013 14:00 GMT+1 #13 Jul 2nd 2013 14:00 GMT+1 #12 Jul 1st 2013 16:00 GMT+1 #11 Jun 18th 2013 16:00 GMT+1 #10 Jun 14th 2013 16:00 GMT+1 #9 Jun 7th 2013 16:00 GMT+1 #8 May 29th 2013 16:00 GMT+1 #7 May 10th 2013 16:00 GMT+1 #6 May 3rd 2013 16:00 GMT+1 #5 Jan 31st 2013 15:00 GMT #4 Aug 11th 2011 14:00 GMT #3 Jun 15th 2011 14:00 GMT #2 Mar 30th 2011 14:00 GMT #1 Jan 25th 2011 14:00 GMT Conferences and Workshops:July 2015 - Kate Willett's poster presentation at the Copernicus Climate Change workshop on Climate Data, ECMWF, Reading, UKApril 2015 - Victor Venema's poster presentation at EGU, Vienna, Austria: Benchmarking and Assessment of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI) February 2015 - Kate Willett's presentation at the WCRP Grand Challenge on data for extremes workshop, University of New South Wales, Sydney, Australia: The ISTI: Land surface air temperature datasets for the 21st Century (also CSIRO, Hobart and BOM, Melbourne) December 2014 - Kate Willett's presentation at the University of Bern, Switzerland: The International Surface Temperature Initiative and Benchmarking for Homogenisation Algorithms October 2014 - Kate Willett's presentation at Edinburgh University: The International Surface Temperature Initiative and Benchmarking for Homogenisation Algorithms July 2014 - Kate Willett's presentation at the SAMSI/IMAGe workshop, NCAR, Boulder, CO, USA: The ISTI Benchmarks - their construction and characteristics April 2014 - Enric Aguilar's presentation at EGU, Vienna, Austria: January 2014 - Kate Willett's presentation at the Met Office, Exeter: Creating synthetic global surface temperature data for benchmark testing homogenisation algorithms and understanding uncertainty. November 2013 - Kate Willett's presentation at the University of Reading Meteorology Department lunchtime seminars: The ISTI: Dragging the land surface temperature data kicking and screaming into the 21st Century
July 2013 - Kate Willett's presentation at the Benchmarking workshop, NCDC, USA: presentation
June 2013 - Kate Willett's presentation at the 12th International Meeting on Statistical Climatology (IMSC), Jeju, South Korea: An overview of benchmarking data homogenisation procedures for the ISTI November 2012 - Kate Willett's presentation at the 5th ACRE Meeting, Toulouse, France: presentation June 2012 - Peter Thorne's presentation at the Earth Temperature Network workshop, Edinburgh, UK: presentation May 2012 - Kate Willett's NCDC Visit with Claude Williams and Robert Lund - presentation - Kate Willett's Clemson University/Robert Lund visit - All things CLIMATE DATA and our maths and statistics headaches December 2011 - Ian Jolliffe's 5th International Verification Methods Workshop, Melbourne, Australia:Benchmarking and Assessment (Verification) of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI), report. October 2011 - Steve Easterbrook's WCRP Open Science Conference, Denver, CO, USA:Benchmarking and Assessment of Homogenisation Algorithms for the International Surface Temperature Initiative (ISTI). See Steve Easterbrook's blog. - Kate Willett's COST HOME 7th Seminar for Homogenisation and Quality Control of Climate Databases, Budapest, Hungary: Creating a Global Benchmark Cycle for the International Surface Temperature Initiative.. Meeting Report (see blog too). May 2011 - Kate Willett's presentation for MARCDATIII, Frascati, Italy, 2011: Is it good enough? Benchmarking homogenisation algorithms and cross-cutting with efforts for land observations April 2011 - Kate Willett's Poster for EGU 2011: Robust Benchmarking of Homogenisation Algorithms for the Surface Temperature Initiative February 2011 - Kate Willett's informal presentation at the National Climate Data Center (NC, USA): Devising a Benchmarking System for Homogenisation Methods of Climate Data-Products Working Group Documents:BETA BENCHMARK CLEAN WORLDS: August 2015 BNCHCAAA_beta1 32522 stations:STATION LISTS: Station List in ISTI format: 5Mb Station list in GHCN format: 2.5Mb Station list with pseudo-countries (filled in using nearest neighbour with country assignment) and land use classification based on ESA CCI-land cover (http://www.esa-landcover-cci.org/): 6.6Mb Note 'nearest neighbour' is preferentially that with the same first two letters of the station ID (FIPS country code) if its within 1000km or else it is the nearest neighbour within 500km.
In some cases (32) countries are listed twice with slightly different names - these have been reconciled resulting in 219 unique countries. THESE PSEUDO-COUNTRY LISTINGS ARE NOT OFFICIAL AND SHOULD ONLY BE USED FOR ERROR WORLD NETWORK CLASSIFICATION. CLEAN WORLD DATA MASKED WITH REAL WORLD MISSING DATA: A gzipped single ascii file version (stations in rows, months in columns Jan 1860 to Dec 2018) can be downloaded here: 38Mb Tarballs of data in ISTI and GHCN format, split by first letter of station ID, are available from this directory (tar -xzf filename) ISTI_BCNHCAAA_beta1_A GHCN_BCNHCAAA_beta1_A 2035 stations 23Mb, 9.1Mb ISTI_BCNHCAAA_beta1_B GHCN_BCNHCAAA_beta1_B 671 stations 7.2Mb, 2.9Mb ISTI_BCNHCAAA_beta1_C GHCN_BCNHCAAA_beta1_C 5314 stations 60Mb, 25Mb ISTI_BCNHCAAA_beta1_D GHCN_BCNHCAAA_beta1_D 66 stations 722Kb, 292Kb ISTI_BCNHCAAA_beta1_E GHCN_BCNHCAAA_beta1_E 219 stations 2.4Mb, 981Kb ISTI_BCNHCAAA_beta1_F GHCN_BCNHCAAA_beta1_F 571 stations 6.4Mb, 2.7Mb ISTI_BCNHCAAA_beta1_G GHCN_BCNHCAAA_beta1_G 1040 stations 12Mb, 4.8Mb ISTI_BCNHCAAA_beta1_H GHCN_BCNHCAAA_beta1_H 70 stations 787Kb, 327Kb ISTI_BCNHCAAA_beta1_I GHCN_BCNHCAAA_beta1_I 672 stations 7.2Mb, 3.0Mb ISTI_BCNHCAAA_beta1_J GHCN_BCNHCAAA_beta1_J 330 stations 3.6Mb, 1.6Mb ISTI_BCNHCAAA_beta1_K GHCN_BCNHCAAA_beta1_K 363 stations 4.0Mb, 1.7Mb ISTI_BCNHCAAA_beta1_L GHCN_BCNHCAAA_beta1_L 128 stations 1.5Mb, 599Kb ISTI_BCNHCAAA_beta1_M GHCN_BCNHCAAA_beta1_M 812 stations 8.8Mb, 3.6Mb ISTI_BCNHCAAA_beta1_N GHCN_BCNHCAAA_beta1_N 701 stations 7.8Mb, 3.2Mb ISTI_BCNHCAAA_beta1_P GHCN_BCNHCAAA_beta1_P 363 stations 4.0Mb, 1.6Mb ISTI_BCNHCAAA_beta1_Q GHCN_BCNHCAAA_beta1_Q 1 stations 11Kb, 5.2Kb ISTI_BCNHCAAA_beta1_R GHCN_BCNHCAAA_beta1_R 1301 stations 15Mb, 6.2Mb ISTI_BCNHCAAA_beta1_S GHCN_BCNHCAAA_beta1_S 1446 stations 16Mb, 6.6Mb ISTI_BCNHCAAA_beta1_T GHCN_BCNHCAAA_beta1_T 533 stations 5.8Mb, 2.4Mb ISTI_BCNHCAAA_beta1_U GHCN_BCNHCAAA_beta1_U 15694 stations 175Mb, 74Mb ISTI_BCNHCAAA_beta1_V GHCN_BCNHCAAA_beta1_V 113 stations 1.2Mb, 481Kb ISTI_BCNHCAAA_beta1_W GHCN_BCNHCAAA_beta1_W 26 stations 283Kb, 113Kb ISTI_BCNHCAAA_beta1_Y GHCN_BCNHCAAA_beta1_Y 9 stations 98Kb, 41Kb ISTI_BCNHCAAA_beta1_Z GHCN_BCNHCAAA_beta1_Z 44 stations 483Kb, 196Kb ISTI/Benchmarking Glossary - open for editing White Paper 9 formed the basis for breakout group discussion at the Exeter meeting. Discussion outcomes are summarised in the final session. Outline for the planned Homogenisation Review Paper to be written by the working group members Terms of Reference agreed by the Benchmarking and Assessment Working Group (12/12/13) Terms of Reference agreed by the Benchmarking and Assessment Working Group (15/6/11) Working draft of Benchmarking and Assessment Paper describing the methodological background to benchmarking and assessment October 2011 Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee 10/11/2011 October 2012 (submitted Feb 2013) Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee xx/xx/2013 October 2013 (submitted Nov 2013) Progress Report of the Benchmarking and Assessment working group submitted to and accepted by the Steering Committee xx/xx/2013
Objectives and Timelines:
Reference Literature:Papers on benchmarking:The Benchmarking working groups paper on benchmarking concepts:
Willett, K. M., C. N. Williams, I. Jolliffe, R. Lund, L. Alexander, S. Brönniman, L. A. Vincent, S. Easterbrook, V. Venema, D. Berry, R. E. Warren, G. Lopardo, R. Auchmann, E. Aguilar, M. Menne, C. Gallagher, Z. Hausfather, T. Thorarinsdottir, P. W. Thorne, 2014: A framework for benchmarking of homogenisation algorithm performance on the global scale, Geoscientific Instrumentation, Methods and Data Systems, 3, 187-200, doi:10.5194/gi-3-187-2014. Peter Thorne et al.'s overview of ISTI including the need for benchmarking: Thorne, P., Willett, K. M., et al., 2011: Guiding the creation of a comprehensive surface temperature resource for 21st century climate science. BAMS, 92 (11), ES40-ES47, doi: 10.1175/2011BAMS3124.1 Kate Willett's work on 'pseudo-worlds' - a set of benchmarks for homogenisation of daily Tmax and Tmin - please leave comments on the blogsite thread Example plots to be uploaded shortly Holly Titchner et al's work on radiosonde error models for validating the homogenisation Titchner, H. A., Thorne, P. W., McCarthy, M. P. et al. 2009: Critically Reassessing Tropospheric Temperature Trends from Radiosondes Using Realistic Validation Experiments. Journal Of Climate. 22, 465-485. Claude William et al's work on homogenising USHCN with benchmarking of the methods: Williams, C. N., Jr., M. J. Menne, and P. Thorne, 2012: Benchmarking the performance of pairwise homogenization of surface temperatures in the United States. J. Geophys. Res., 117, D05116 doi:10.1029/2011JD016761, BLOGPOST Victor Venema et al's work on benchmarking the COST HOME homogenisation algorithms: Venema, V., O. Mestre, E. Aguilar, I. Auer, J.A. Guijarro, P. Domonkos, G. Vertacnik, T. Szentimrey, P. Stepanek, P. Zahradnicek, J. Viarre, G. Müller-Westermeier, M. Lakatos, C.N. Williams, M. Menne, R. Lindau, D. Rasol, E. Rustemeier, K. Kolokythas, T. Marinova, L. Andresen, F. Acquaotta, S. Fratianni, S. Cheval, M. Klancar, M. Brunetti, Ch. Gruber, M. Prohom Duran, T. Likso, P. Esteban, Th. Brandsma., 2012: Benchmarking homogenization algorithms for monthly data, Climate of the Past, 8, pp. 89-115, 2012. BLOGPOST ____________________________________ Papers on known sources of inhomogeneity:
---------------------------------------------------------------
Papers on homogenisation:
Links to Related Projects:www.homogenisation.org - website for the COST HOME action on homogenisationLast modified by Kate Willett: Jul 19th 2011 |
