Developing a data lakehouse for a South African government-sector training authority : implementing quality control for incremental extract-load-transform pipelines in the ingestion layer
No Thumbnail Available
Date
2024-12
Authors
Govender, Priyanka
Naicker, Nalindren
Patel, Sulaiman Saleem
Joseph, Seena
Moonsamy, Devraj
Akinola, Ayotuyi Tosin
Madamshetty, Lavanya
Govender, Thamotharan Prinavin
Journal Title
Journal ISSN
Volume Title
Publisher
IGI Global
Abstract
The Durban University of Technology is undertaking a project to develop a data lakehouse system for a South African government-sector training authority. This system is considered critical to enhance the monitoring and evaluation capabilities of the training authority and ensure service delivery. Ensuring the quality of data ingested into the lakehouse is critical, as poor data quality deteriorates the efficiency of the lakehouse solution. This chapter studies quality control for ingestion-layer pipelines to propose a data quality framework. Metrics considered for data quality were completeness, accuracy, integrity, correctness, and timeliness. The framework was evaluated by practically applying it to a sample semi-structured dataset to gauge its effectiveness. Recommendations for future work include expanded integration, such as incorporating data from more varied sources and implementing incremental data ingestion triggers.
Description
Keywords
Citation
Govender, P. et al. 2024. Developing a data lakehouse for a South African government-sector training authority: implementing quality control for incremental extract-load-transform pipelines in the ingestion layer. In: Ogunleye, Olalekan Samuel. Machine learning and data science techniques for effective government service delivery. Hershey, Pa.: IGI Global, 157-184. doi:10.4018/978-1-6684-9716-6.ch006
DOI
10.4018/978-1-6684-9716-6.ch006