Data Processing – IMOS toolbox
Spirit of Tasmania data – work to add quality control (QC) to the data
Four (4) sets of transects, approximately 233 files, collected between 06 October 2013 and 22 April 2014, have been recently processed by the toolbox.
Recent improvements to the IMOS toolbox permitted a set of 1Hz data files, sampling real time temperature, salinity, fluorescence and turbidity, collected by the Spirit of Tasmania between Devonport and Melbourne, to be read, pre-processed, quality controlled and converted to NetCDF.
Pre-processing routines performed include -
- Conversion of turbidity and fluorescence in count into turbidity nephelometric turbidity unit (NTU) and Chlorophyl-a proxy in umol/l,
- Correction from linear/quadratic drift of the instrument when necessary (in particular for turbidity and sometimes salinity), and
- Linear correction from observed differences between sensor measurements performed just before and just after a cleaning service (in particular for turbidity, fluorescence and salinity).
QC tests performed include some of the ones found in the toolbox for moored time-series -
- Impossible date,
- Impossible location,
- Global range,
- Regional range,
- Stationarity, and
- Salinity from P and T.
In addition -
- A new QC algorithm, to remove data spikes, has been used to automatically flag spikes in turbidity,
- Manual QC was extensively performed on all datasets, flagging any dubious data,
- A set of diagnostic and visual validation plots have been generated, and
- FV02 product files have been produced from the original quality controlled datasets averaged over a time period of 10 seconds and where only data with a flag 'Good' or 'Probably good' have been considered.
Newly QC data files are now accessible through the AODN portal (https://portal.aodn.org.au/search?uuid=02640f4e-08d0-4f3a-956b-7f9b58966ccc) and the IMOS THREDDS catalogue (http://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/transect/catalog.html)