V.DIG.2201 - Data Platform Resilience
MLA has a data platform supporting the Australian red meat industry to adopt a data-driven decision culture.
| Project start date: | 04 April 2022 |
| Project end date: | 31 October 2025 |
| Publication date: | 30 January 2024 |
| Project status: | In progress |
| Livestock species: | Grain-fed Cattle, Grass-fed Cattle, Sheep, Goat, Lamb |
| Relevant regions: | National |
Summary
The ISC Data Team has adopted a cloud-first architecture in its implementation of modern data platform to support the Australian meat and livestock value chain in shifting to adopt data-driven decision culture. Following Agile development principles, the development of the Data Platform is completed in short iterations and incremental improvements.
Objectives
The objective of this project is to make the Data Platform more resilient, secure, stable and able to accommodate new types of work loads by processing tasks in batches.
Key findings
The Data Platform supports data to be processed via batches. For the next phase of the project, the team will implement near real time data streaming to allow the end users to have access to more frequent data.
Benefits to industry
The data platform is an enabler to the MLA/ISC sub programs.
Future research
The next phase will include several improvements for the administrators of the platform.
This will include improving:
- logging
- emails on data pipeline execution
- metrics additions to the new relic dashboard
- service level agreements to be put in place to help the support team.
More information
| Project manager: | Inn Inn Er |
| Contact email: | reports@mla.com.au |
