Since early 2017, a small team has been working on improving the software infrastructure that handles all the data files received by the Australian Ocean Data Network (AODN) for publication. Commonly called "pipelines", these programs are triggered when any file is uploaded. They then automatically check the file for compliance with required conventions, save the file to the public cloud storage (provided by Amazon), and harvest its metadata and some data into a database. Compliant files are thus made available on the AODN Portal within minutes.
The aim of this project was to make the pipeline system more reliable, efficient and transparent, giving better feedback to the data provider (and us) about what happens to every uploaded file. Until recently this has only been "behind the scenes" development, but as of February 1, we have started rolling out he new software on our production servers. Currently files from Argo, NSW moorings, Autonomous Underwater Vehicles, Wireless Sensor Networks, Ships of Opportunity; Air Sea Flux, Sea Surface Temperature, Tropical Research Vessels, Expendable Bathythermograph and National Reference Station Darwin and Yongola Real-Time are being handled by the new pipelines. In the coming months we'll be gradually switching over all sub-facilities and data streams one by one. All affected data providers will be notified in advance.