V.ISC.1930 - CAPEX: Data Platform Foundation
| Project start date: | 21 July 2019 |
| Project end date: | 29 November 2022 |
| Publication date: | 21 December 2023 |
| Project status: | Completed |
| Livestock species: | Grain-fed Cattle, Grass-fed Cattle, Sheep, Goat, Lamb |
| Relevant regions: | National |
Summary
The purpose of this project is to deliver the key infrastructure of the data platform and consolidate disparate datasets within ISC into a cohesive and comprehensive repository.
Objectives
1. Efficient and reliable data ingestion pipeline with participating ISC data sets for both historical and ongoing data.
2. Data storage utilising AWS technology with high durability, scalability and optimised performance built in. Solution should handle both intensive workload, fast streaming data and infrequent access data cost efficiently.
3. Robust and efficient pipeline to validate data quality and ensure the source data conformed to the predefined data type.
Key findings
There are many opportunities in the data platform.
Benefits to industry
The data platform foundation project will deliver the key infrastructure of a data lake andintegrate with ISC datasets including NLIS, LPA, LDL, eNVD and myMLA. Once populated, ISC will begin evaluating technologies designed for big data analytics to assist with better understanding this data, its value and the opportunities provided by the platform and deliver this analysis to core parts of the business. This is an important enabler to underpin the implementation of the Integrity System 2025 Strategy (IS2025). It is the cornerstone of one of the IS2025 Strategy pillars:
- Leveraging integrity data to add value through the chain.
MLA action
To monitor the state of the data platform and continue improvements as required.
Future research
The data platform foundation project is complete but future work into the data platform is encouraged for continual improvement and innovation.
|
For more information Contact Project Manager: Jun Chen |
