  Organizations routinely accumulate semi-structured log datasets generated as
the output of code; these datasets remain unused and uninterpreted, and occupy
wasted space - this phenomenon has been colloquially referred to as "data lake"
problem. One approach to leverage these semi-structured datasets is to convert
them into a structured relational format, following which they can be analyzed
in conjunction with other datasets. We present Datamaran, an tool that extracts
structure from semi-structured log datasets with no human supervision.
Datamaran automatically identifies field and record endpoints, separates the
structured parts from the unstructured noise or formatting, and can tease apart
multiple structures from within a dataset, in order to efficiently extract
structured relational datasets from semi-structured log datasets, at scale with
high accuracy. Compared to other unsupervised log dataset extraction tools
developed in prior work, Datamaran does not require the record boundaries to be
known beforehand, making it much more applicable to the noisy log files that
are ubiquitous in data lakes. Datamaran can successfully extract structured
information from all datasets used in prior work, and can achieve 95%
extraction accuracy on automatically collected log datasets from GitHub - a
substantial 66% increase of accuracy compared to unsupervised schemes from
prior work. Our user study further demonstrates that the extraction results of
Datamaran are closer to the desired structure than competing algorithms.
