A new month, a new release! We're excited to announce the immediate availability of Snowplow version 0.8.0. This has been our most complex release to date: we have done a full rewrite our ETL (aka enrichment) process, adding a few nice data quality enhancements along the way.
This release has been heavily informed by our January blog post, The Snowplow development roadmap for the ETL step - from ETL to enrichment. In technical terms, we have ported our existing ETL process (which was a combination of HiveQL scripts plus a custom Java deserializer) to a new Hadoop-only ETL process which does not require Hive. The new ETL process is written in Scala, using Scalding, a Scala API built on top of Cascading, the Hadoop ETL framework.
Redshift only - the new ETL process only supports writing out in Redshift format. We discuss this further in A note for Infobright/Hive users below.
On points 2 and 3: rest assured that improving the performance of the new Hadoop ETL (not least by tackling the small files problem) is a key priority for the Snowplow Analytics team going forwards.
If you are using Infobright or plain-Hive to store your Snowplow data, we understand that you'll be feeling a little left out of this release. Unfortunately, supporting Redshift, Infobright and Hive all in this version 1 simply wasn't feasible from a development-effort perspective.
This does not mean that we are giving up on Hive and Infobright: on the contrary, we have big plans for both data storage targets.
For Hive users - we are working on a new Avro-based storage format for Snowplow events. Being based on Avro, it should be less fragile than our existing flatfile approach, easily queryable from Hive using the AvroSerde, and faster to query (because data is stored more efficiently in binary format). Evolving the Avro schema to incorporate our event dictionary will also be much more straightforward. This will be the 0.9.x release series and should come later in Q2 or early Q3.
For Infobright users - we will be adding Infobright support into the new Hadoop-based ETL later this year. If you would rather not wait, we recommend switching to Redshift now or switching to PostgreSQL support when this is released in late Q2.
We should also stress: it is totally safe for Infobright/Hive users to upgrade to 0.8.0: the Hive-based ETL process continues to work as before, and we will be continuing to support the Hive ETL with bug fixes etc for the foreseeable future.
As always, you can check out upcoming features on our Product roadmap wiki page.
Nothing else is changed in this release.
:out_bad_rows: ADD HERE # Leave blank for Hive ETL.
:out_errors: ADD HERE # Leave blank for Hive ETL.
The out_bad_rows bucket will contain any raw Snowplow log lines which did not pass the ETL's validation. If you set continue_on_unexpected_error to true, then the out_errors bucket will contain any raw Snowplow log lines which caused an unexpected error.
The job_name should make it easier to identify your ETL job in the Elastic MapReduce console.
Change implementation to "hive" to use our alternative Hive-based ETL process.
That's it! Once you have made those configuration changes, you should be up-and-running with the new ETL process.