⇒ IQuOD:International Quality-controlled Ocean Database Projecthttps://repository.oceanbestpractices.org/handle/11329/15902024-03-29T05:04:18Z2024-03-29T05:04:18ZA Framework to Quality Control Oceanographic Data.Castelao, Guilherme P.https://repository.oceanbestpractices.org/handle/11329/24102024-01-15T12:53:39Z2020-01-01T00:00:00ZA Framework to Quality Control Oceanographic Data.
Castelao, Guilherme P.
The ocean is an intrinsically challenging environment to collect data, which makes spurious
measurements inevitable. Thus, the quality of oceanographic datasets is highly dependent on
the ability to identify and remove bad samples. Quality control (QC) of oceanographic data
has mostly relied on manual QC by experts, which, despite resulting in the best data quality,
is not scalable and becomes impractical to handle large datasets or real-time data streams. To
address this issue, automatic QC procedures have been proposed and widely used for decades
(e.g., IOC/IODE, 1993; DATA–MEQ working group, 2010; GTSPP Real–Time Quality Control
Manual, 2010; Morello et al., 2014; QARTOD group, 2016; Wong, Keeley, Carval, &
Argo Data Management Team, 2015); however, these procedures are seldom organized and
distributed as packages, so it is still common for new users to have to implement them from
scratch. Additionally, different applications of the same dataset may require different QC
procedures. For example, a particular user faced with the QC of a small dataset might be
willing to apply a less conservative QC in order to preserve a larger number of data points,
paying the price of having some false positives. CoTeDe is an Open Source Python package
that provides a flexible way to automatic QC oceanographic data by combining multiple QC
standards while allowing the users to fully control and tune the parameters according to their
own needs.
2020-01-01T00:00:00ZAustralian XBT Quality Control Cookbook Version 2.1.Cowley, RebeccaKrummel, Lisahttps://repository.oceanbestpractices.org/handle/11329/127.32024-01-15T14:06:08Z2023-01-01T00:00:00ZAustralian XBT Quality Control Cookbook Version 2.1.
Cowley, Rebecca; Krummel, Lisa
Expendable Bathythermographs (XBTs) have been used for many years by oceanographers to
measure the temperature of the upper ocean. These instruments are simple devices which are
designed to be deployed from moving vessels, enabling the use of ships of opportunity to collect data
in repeated transects. The XBT has accordingly played an important role in several large international
research programs, and the global data archives reflect this. Quality Control (QC) procedures are
described for data recorded by XBTs. Examples are shown and described for commonly observed
oceanographic features and instrument malfunctions. A QC system is described, which aids in the
process of future validation and documentation of real features, and in the elimination of erroneous
temperature profiles. There are some modes of malfunction of the XBT which appear very similar to
real oceanographic features. This manual enables the user to better distinguish between the two. A
knowledge of the different types of real and erroneous features, when combined with a local
knowledge of water mass structure, statistics of data anomalies, the depth and gradient of the
thermocline, and cross validation with climatological data in a statistical sense, ensures a data set of
the best possible quality.
This document is an update to the original ‘Quality Control Cookbook for XBT Data’ Version 1.1
(Bailey et al, 1994). Over time, the QC routines used by the Australian team have developed and
many codes have become redundant due to improvements in the recording systems and our
understanding of failure modes. The older, redundant codes are now summarised in the Section 4.8
and the Appendices A to C. We urge the reader to refer to Bailey et al (1994) for more detail on these
historical codes.
2023-01-01T00:00:00ZBenchmarking of automatic quality control checks for ocean temperature profiles and recommendations for optimal sets.Good, S.A.Mills, BillCastelao, GuilhermeCowley, RebeccaGoni, GustavoGouretski, ViktorDomingues, Catia M.Boyer, TimBringas, Francishttps://repository.oceanbestpractices.org/handle/11329/21462023-03-01T22:03:07Z2023-01-01T00:00:00ZBenchmarking of automatic quality control checks for ocean temperature profiles and recommendations for optimal sets.
Good, S.A.; Mills, Bill; Castelao, Guilherme; Cowley, Rebecca; Goni, Gustavo; Gouretski, Viktor; Domingues, Catia M.; Boyer, Tim; Bringas, Francis
Millions of in situ ocean temperature profiles have been collected historically using various instrument types with varying sensor accuracy and then assembled into global databases. These are essential to our current understanding of the changing state of the oceans, sea level, Earth’s climate, marine ecosystems and fisheries, and for constraining model projections of future change that underpin mitigation and adaptation solutions. Profiles distributed shortly after collection are also widely used in operational applications such as real-time monitoring and forecasting of the ocean state and weather prediction. Before use in scientific or societal service applications, quality control (QC) procedures need to be applied to flag and ultimately remove erroneous data. Automatic QC (AQC) checks are vital to the timeliness of operational applications and for reducing the volume of dubious data which later require QC processing by a human for delayed mode applications. Despite the large suite of evolving AQC checks developed by institutions worldwide, the most effective set of AQC checks was not known. We have developed a framework to assess the performance of AQC checks, under the auspices of the International Quality Controlled Ocean Database (IQuOD) project. The IQuOD-AQC framework is an open-source collaborative software infrastructure built in Python (available from https://github.com/IQuOD). Sixty AQC checks have been implemented in this framework. Their performance was benchmarked against three reference datasets which contained a spectrum of instrument types and error modes flagged in their profiles. One of these (a subset of the Quality-controlled Ocean Temperature Archive (QuOTA) dataset that had been manually inspected for quality issues by its creators) was also used to identify optimal sets of AQC checks. Results suggest that the AQC checks are effective for most historical data, but less so in the case of data from Mechanical Bathythermographs (MBTs), and much less effective for Argo data. The optimal AQC sets will be applied to generate quality flags for the next release of the IQuOD dataset. This will further elevate the quality and historical value of millions of temperature profile data which have already been improved by IQuOD intelligent metadata and observational uncertainty information (https://doi.org/10.7289/v51r6nsf).
2023-01-01T00:00:00ZInternational Quality-Controlled Ocean Database (IQuOD) v0.1: The Temperature Uncertainty Specification.Cowley, RebeccaKillick, Rachel E.Boyer, TimReseghetti, FrancoKizu, ShoichiPalmer, Matthew D.Cheng, LijingStorto, AndreaLe Menn, MarcSimoncelli, SimonaMacdonald, Alison M.Domingues, Catia M.https://repository.oceanbestpractices.org/handle/11329/15922021-06-29T07:00:52Z2021-01-01T00:00:00ZInternational Quality-Controlled Ocean Database (IQuOD) v0.1: The Temperature Uncertainty Specification.
Cowley, Rebecca; Killick, Rachel E.; Boyer, Tim; Reseghetti, Franco; Kizu, Shoichi; Palmer, Matthew D.; Cheng, Lijing; Storto, Andrea; Le Menn, Marc; Simoncelli, Simona; Macdonald, Alison M.; Domingues, Catia M.
Ocean temperature observations are crucial for a host of climate research and forecasting activities, such as climate monitoring, ocean reanalysis and state estimation, seasonal-to-decadal forecasts, and ocean forecasting. For all of these applications, it is crucial to understand the uncertainty attached to each of the observations, accounting for changes in instrument technology and observing practices over time. Here, we describe the rationale behind the uncertainty specification provided for all in situ ocean temperature observations in the International Quality-controlled Ocean Database (IQuOD) v0.1, a value-added data product served alongside the World Ocean Database (WOD). We collected information from manufacturer specifications and other publications, providing the end user with uncertainty estimates based mainly on instrument type, along with extant auxiliary information such as calibration and collection method. The provision of a consistent set of observation uncertainties will provide a more complete understanding of historical ocean observations used to examine the changing environment. Moving forward, IQuOD will continue to work with the ocean observation, data assimilation and ocean climate communities to further refine uncertainty quantification. We encourage submissions of metadata and information about historical practices to the IQuOD project and WOD.
2021-01-01T00:00:00ZSalaciaML: A Deep Learning Approach for Supporting Ocean Data Quality Control.Mieruch, SebastianDemirel, SerdarSimoncelli, SimonaSchlitzer, ReinerSeitz, Steffenhttps://repository.oceanbestpractices.org/handle/11329/15602023-12-22T22:43:38Z2021-01-01T00:00:00ZSalaciaML: A Deep Learning Approach for Supporting Ocean Data Quality Control.
Mieruch, Sebastian; Demirel, Serdar; Simoncelli, Simona; Schlitzer, Reiner; Seitz, Steffen
We present a skillful deep learning algorithm for supporting quality control of ocean
temperature measurements, which we name SalaciaML according to Salacia the roman
goddess of sea waters. Classical attempts to algorithmically support and partly automate
the quality control of ocean data profiles are especially helpful for the gross errors in the
data. Range filters, spike detection, and data distribution checks remove reliably the
outliers and errors in the data, still wrong classifications occur. Various automated quality
control procedures have been successfully implemented within the main international
and EU marine data infrastructures (WOD, CMEMS, IQuOD, SDN) but their resulting
data products are still containing data anomalies, bad data flagged as good and
vice-versa. They also include visual inspection of suspicious measurements, which is
a time consuming activity, especially if the number of suspicious data detected is large.
A deep learning approach could highly improve our capabilities to quality assess big
data collections and contemporary reducing the human effort. Our algorithm SalaciaML
is meant to complement classical automated quality control procedures in supporting
the time consuming visually inspection of data anomalies by quality control experts. As a
first approach we applied the algorithm to a large dataset from the Mediterranean Sea.
SalaciaML has been able to detect correctly more than 90% of all good and/or bad data
in 11 out of 16 Mediterranean regions.
2021-01-01T00:00:00ZAn Algorithm for Classifying Unknown Expendable Bathythermograph (XBT) Instruments Based on Existing Metadata.Palmer, Matthew D.Boyer, TimCowley, RebeccaKizu, ShoichiReseghetti, FrancoSuzuki, ToruThresher, Annhttps://repository.oceanbestpractices.org/handle/11329/6272021-06-24T02:12:11Z2018-01-01T00:00:00ZAn Algorithm for Classifying Unknown Expendable Bathythermograph (XBT) Instruments Based on Existing Metadata.
Palmer, Matthew D.; Boyer, Tim; Cowley, Rebecca; Kizu, Shoichi; Reseghetti, Franco; Suzuki, Toru; Thresher, Ann
Time-varying biases in expendable bathythermograph (XBT) instruments have emerged as a key un-
certainty in estimates of historical ocean heat content variability and change. One of the challenges in the
development of XBT bias corrections is the lack of metadata in ocean profile databases. Approximately 50%
of XBT profiles in the World Ocean database (WOD) have no information about manufacturer or probe type.
Building on previous research efforts, this paper presents a deterministic algorithm for assigning missing XBT
manufacturer and probe type for individual temperature profiles based on 1) the reporting country, 2) the
maximum reported depth, and 3) the record date. The criteria used are based on bulk analysis of known XBT
profiles in the WOD for the period 1966–2015. A basic skill assessment demonstrates a 77% success rate at
correctly assigning manufacturer and probe type for profiles where this information is available. The skill rate
is lowest during the early 1990s, which is also a period when metadata information is particularly poor. The
results suggest that substantive improvements could be made through further data analysis and that future
algorithms may benefit from including a larger number of predictor variable.
2018-01-01T00:00:00Z