---
abstract: 'Consider the situation where a data analyst wishes to carry out an analysis on a given dataset. It is widely recognized that most of the analyst’s time will be taken up with *data engineering* tasks such as acquiring, understanding, cleaning and preparing the data. In this paper we provide a description and classification of such tasks into high-levels groups, namely data organization, data quality and feature engineering. We also make available four datasets and example analyses that exhibit a wide variety of these problems, to help encourage the development of tools and techniques to help reduce this burden and push forward research towards the automation or semi-automation of the data engineering process.'
author:
- Alfredo Nazabal
- 'Christopher K.I. Williams'
- Giovanni Colavizza
- Camila Rangel Smith
- Angus Williams
bibliography:
- 'autods\_ckiw.bib'
title: |
    Data Engineering for Data Analytics:\
    A Classification of the Issues, and Case Studies
---

Introduction {#sec:introduction}
============

Overview of Case Studies \[sec:case-studies\]
=============================================

Classification of Wrangling Challenges \[sec:class\]
====================================================

Related Work \[sec:rel-work\]
=============================

Conclusions \[sec:conc\]
========================

In this paper we have identified three high-level groups of data wrangling problems, those related with obtaining a proper representation of the data (data organization), those related to assessing [and improving]{} the quality of the data (data quality), and feature engineering issues, which heavily depend on the task at hand and the model employed to solve it. Furthermore, we have presented the full analysis of four use cases, where we have provided a systematic pipeline for each of the datasets to clean them while identifying and classifying the main problems the data scientists faced during the wrangling steps. We hope that this work helps to further explore and understand the field of data engineering, and to value a part of every data scientist’s work that most of the time goes unnoticed both in research and industry. Additionally, we would like to encourage practitioners to provide their raw data and the scripts necessary to clean it in order to advance the field. In future work we would like to study data engineering workflows across multiple datasets in order to identify (if possible) common structures concerning the ordering of the various wrangling operations. We note that there can be feedback loops in the process, as described e.g. in [@crisp-dm-00].

Acknowledgments {#acknowledgments .unnumbered}
===============

This work was supported in by The Alan Turing Institute under the EPSRC grant EP/N510129/1 and by funding provided by the UK Government’s Defence & Security Programme in support of the Alan Turing Institute.

We would like to thank the AIDA team including Taha Ceritli, Jiaoyan Chen, James Geddes, Zoubin Ghahramani, Ian Horrocks, Ernesto Jimenez-Ruiz, Charles Sutton, Tomas Petrick and Gerrit van den Burg for helpful discussions and comments on the paper.
