This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640276.
G3.02 Missing standards for, and evaluation of, co-location criteria
The impact of a particular choice of co-location criterion is only rarely studied in the scientific literature reporting on satellite validation results. However, without some quantification of the impact of the co-location criterion that was adopted, it is virtually impossible to assess the contribution of natural variability to the total error budget of the data comparisons. As such, this gap impacts significantly the potential interpretation of the data comparison result in terms of data quality. Some in-depth studies do exist, but testing multiple criteria, or using criteria based on the latest results of such exploratory work, is far from common (indeed, often arbitrary) practice(s). This gap thus concerns the need for more awareness among validation teams, for more detailed studies comparing the (dis-)advantages of various co-location criteria, and for community-agreed standards on co-location criteria that are broadly adopted in the context of operational services.
Part I Gap description
- Uncertainty in relation to comparator
- Governance (missing documentation, cooperation etc.)
- Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
- Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
- International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
- Climate research (research groups working on development, validation and improvement of ECV Climate Data Records)
- Independent of instrument technique
G3.04. To be addressed before G3.02
Argument: Ideally, co-location criteria take into account the smoothing and sampling properties of the measurements. Consequently, studies on co-location criteria can benefit from a proper characterization of these smoothing and sampling properties.
G3.06. To be addressed before G3.02
Argument: The merit of certain co-location criteria can best be assessed when the uncertainty budget of the comparisons is decomposed in measurement and co-location mismatch uncertainties.
G6.03. To be addressed after G3.02
Argument: Deciding on the best time and location for targeted reference observations should be informed by information on the optimal co-location criteria
Co-location criteria should represent an optimal compromise between the obtained number of co-located measurements (as large as possible to have robust statistical results) and the impact of natural variability on the comparisons (as low as possible to allow a meaningful comparison between measured differences and reported measurement uncertainties). Hitherto, only a few ground-based satellite validation studies explored the impact of the adopted co-location criteria on the comparison results (e.g. Wunch et al., 2011, and Dils et al., 2014, for CO2 , Verhoelst et al., 2015, for O3, Pappalardo et al., 2010, for aerosols, Lambert et al. 2012, for water vapour, Van Malderen et al. 2014, for integrated water vapour). Still, atmospheric variability is often assumed –or even known- to impact the comparisons, but without detailed testing of several co-location criteria (or by extensive model-based simulations), this impact is hard to quantify. Besides the need for dedicated studies, this gap also concerns the “community practices” regarding co-location approaches, which are neither consistent across different studies, nor optimal as they often rely on historical co-location criteria, which are not necessarily fit-for-purpose for the accuracy and spatiotemporal sampling properties of current measurement systems. Consequently, to ensure reliable and traceable validation results, as required in an operational context, community-agreed standards for co-location criteria should be developed, published, and adopted.
- Independent of specific space mission or space instruments
- Radiance (Level 1 product)
- Geophysical product (Level 2 product)
- Gridded product (Level 3)
- Assimilated product (Level 4)
- Time series and trends
- Representativity (spatial, temporal)
- Calibration (relative, absolute)
- Spectroscopy
- Auxiliary parameters (clouds, lightpath, surface albedo, emissivity)
- GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future
Two activities within GAIA-CLIM targeted this gap to some extent:
Within GAIA-CLIM, a dedicated task (T3.2 in WP3) dealt with data intercomparison studies, focusing on a closure of the comparison uncertainty budget and including an exploration of different co-location criteria, see for instance the results on total ozone columns published by Verhoelst et al. (2015, their Fig. 11).
The Virtual Observatory developed within GAIA-CLIM offers the user the possibility to adjust co-location criteria and to visualize the resulting impact on the comparison results.
However, no attempt has been made within GAIA-CLIM to produce an authoritative analysis and resulting documentation on this matter.
Part II Benefits to resolution and risks to non-resolution
Identified benefit | User category/Application area benefitted | Probability of benefit being realised | Impacts |
---|---|---|---|
Greater awareness of the impact of natural variability on the comparison results; |
|
| More reliable feedback on data quality, in particular on the reported uncertainties. This in turn increases confidence in the data for the end user and allows more meaningful use in a variety of applications. |
Improved definition of appropriate co-location criteria for validation work, minimizing errors due to co-location mismatch. |
|
| Lower uncertainty due to co-location mismatch will result in tighter constraints on the products from validation work, supporting further instrument and algorithm development. |
Facilitates intercomparison of different validation studies |
|
| More reliable comparisons between different products (each having its own validation report) to better assess their fitness-for-purpose for a specific user application. |
Identified risk | User category/Application area at risk | Probability of risk being realised | Impacts |
---|---|---|---|
Poor feedback on data quality (in particular on the reported uncertainties) from validation studies due to unknown/unquantified influence of atmospheric variability. |
|
| Poor confidence in data and services; potential over-interpretation; difficult/unreliable assimilation; Potential of both EO and ground segments not fully realized. |
Difficulty to compare validation results on similar products performed by different teams |
|
| Sub-optimal choice of data product for a given application. |
Part III Gap remedies
Remedy 1: Systematic quantification of the impacts of different co-location criteria
Dedicated studies are required which explore in detail the advantages and disadvantages of several co-location methods and criteria. Dedicated working groups or activities could/should be set up within the framework of the ground-based observing networks, as already initiated for meteorological variables at a GRUAN-GSICS-GNSSRO WIGOS workshop on Upper-Air Observing System Integration and Application, hosted by WMO in Geneva in May 2014. Dissemination among, and acceptance by, the key stakeholders may be challenging and can probably best be achieved in the context of overarching frameworks such as the CEOS Working Group on Calibration & Validation (WGCV). The financial cost should be very low. Also, the space agencies and service providers could/should insist on sufficient attention for (and analysis of) the adopted co-location criteria in the validation protocols followed by their validation teams.
These studies and the proposed associated governance support target this gap directly. They will provide stakeholders with a traceable, authoritative reference on which to base their validation requirements and protocols regarding co-location criteria. It will also facilitate meta-analysis of different validation studies without the need to take into account differences in results due to differences in the impact of co-location mismatch on the results.
Peer-reviewed publications or widely distributed technical notes on the subject, from an authorative body; Explicit inclusion of requirements on the co-location methodology and criteria in validation protocols.
- High
- Single institution
- Consortium
- Less than 3 years
- Low cost (< 1 million)
- No
- EU H2020 funding
- Copernicus funding
- National funding agencies
- National Meteorological Services
- WMO
- ESA, EUMETSAT or other space agency
- Academia, individual research institutes
- Dils et al., “The Greenhouse Gas Climate Change Initiative (GHG-CCI): comparative validation of GHG-CCI SCIAMACHY/ENVISAT and TANSO-FTS/GOSAT CO2 and CH4 retrieval algorithm products with measurements from the TCCON”, AMT v7, 2014
- Lambert, J.-C., et al., “Comparing and merging water vapour observations: A multi-dimensional perspective on smoothing and sampling issues”, in “Monitoring Atmospheric Water Vapour: Ground-Based Remote Sensing and In-situ Methods”, N. Kämpfer (Ed.), ISSI Scientific Report Series, Vol. 10, Edition 1, 326 pp., ISBN: 978-1-4614-3908-0, DOI 10.1007/978-1-4614-3909-7_2, © Springer New York 2012
- Pappalardo et al., “EARLINET correlative measurements for CALIPSO: First intercomparison results”, J.G.R.: Atmospheres v115, 2010
- Van Malderen, R. et al., “A multi-site intercomparison of integrated water vapour observations for climate change analysis”, AMT v7, 2014
- Verhoelst et al., “Metrology of ground-based satellite validation: Co-location mismatch and smoothing issues of total ozone comparisons”, AMT v8, 2015
- Wunch et al., “A method for evaluating bias in global measurements of CO2 total columns from space”, ACP v11, 2011