<NME> _installation.adoc
<BEF> == Installation

Installing Genie is easy. You can run Genie either as a standalone application with an embedded Tomcat or by deploying
the WAR file to an existing Tomcat or other servlet container. There are trade-offs to these two methods as will be
discussed below.

=== Standalone Jar

The
ifeval::[{isSnapshot} == true]
http://oss.jfrog.org/libs-snapshot/com/netflix/genie/genie-app/{revnumber}/[standalone jar]
endif::[]
ifeval::[{isSnapshot} == false]
https://bintray.com/netflixoss/maven/download_file?file_path=com%2Fnetflix%2Fgenie%2Fgenie-app%2F{revnumber}%2Fgenie-app-{revnumber}.jar[standalone jar]
endif::[]
is the simplest to deploy as it has no other real moving parts. Just
put the jar somewhere on a system and execute `java -jar genie-app-{revnumber}.jar`. The downside is it's a little
harder to configure or add jars to the classpath if you want them.

Configuration (application*.yml or application*.properties) files can be loaded from the current working directory or
from a `.genie/` directory stored in the users home directory (e.g. `~/.genie/application.yml`). Classpath items (jars,
.jks files, etc) can be added to `~/.genie/lib/` and they will be part of the application classpath.

Properties can be passed in on the command line two ways:

. `java -Dgenie.example.property blah -jar genie-app-{revnumber}.jar`
. `java -jar genie-app-{revnumber}.jar --genie.example.property=blah`

Property resolution goes in this order:

. Command line
. Classpath profile specific configuration files (e.g. application-prod.yml)
. Embedded profile specific configuration files
. Classpath default configuration file (e.g. application.yml)
. Embedded default configuration file

For more details see the
http://docs.spring.io/spring-boot/docs/{springBootVersion}/reference/htmlsingle/#boot-features-external-config[Spring Boot]
documentation on external configuration.

=== Servlet Container Deployment

If you want to deploy to an existing Servlet container deployment you can re-package the `genie-web` jar inside a WAR.
You can see the https://docs.spring.io/spring-boot/docs/current/reference/html/howto-traditional-deployment.html#howto-create-a-deployable-war-file[Spring Boot]
docs for an example of how to do this.

=== Configuration

Genie has a lot of available configuration options. For descriptions of specific properties you can see the
<<Properties>> section below. Additionally if you want to know how to configure more parts of the application you
should have a look at the http://docs.spring.io/spring-boot/docs/{springBootVersion}/reference/htmlsingle/[Spring Boot]
docs as they will go in depth on how to configure the various Spring components used in Genie.

==== Profiles

Spring provides a mechanism of segregating parts of application configuration and activating them in certain
conditions. This mechanism is known as
http://docs.spring.io/spring-boot/docs/{springBootVersion}/reference/htmlsingle/#boot-features-profiles[profiles]. By
default Genie will run with the `dev` profile activated. This means that all the properties in `application-dev.yml`
will be appended to, or overwrite, the properties in `application.yml` which are the defaults. Changing the active
profiles is easy you just need to change the property `spring.profiles.active` and pass in a comma separated list of
active profiles. For example `--spring.profiles.active=prod,cloud` would activate the prod and cloud profiles.

Properties for specific profiles should be stored in files named `application-{profileName}.yml`. You can make as many
as you want but Genie ships with `dev`, `s3` and `prod` profiles properties already included. Their properties can be
seen in the <<Properties>> section below.

==== Database

By default since Genie will launch with the `dev` profile active it will launch with an in memory database running as
part of its process. This means when you shut Genie down all data will be lost. It is meant for development only. Genie
ships with JDBC drivers for `MySql`, `PostgreSQL` and `H2`. If you want to use a different database you should load
the JDBC driver jar file somewhere on the Genie classpath.

For production you should probably enable the `prod` profile which creates a connection pool for the database and then
override the properties `spring.datasource.url`, `spring.datasource.username` and `spring.datasource.password` to match
your environment. The datasource url needs to be a valid JDBC connection string for your database. You can see examples
http://www.java2s.com/Tutorial/Java/0340__Database/AListofJDBCDriversconnectionstringdrivername.htm[here] or
http://alvinalexander.com/java/jdbc-connection-string-mysql-postgresql-sqlserver[here] or search for your database
and JDBC connection string on your search engine of choice.

Genie also ships with database schema scripts for MySQL and PostgreSQL. You will need to load these into your database
before you run Genie if you use one of these databases. Genie no longer creates the schema dynamically for performance
reasons. Follow the below sections to load the schemas into your table.

NOTE: Genie 3.2.0+ software is not compatible with previous database schema. Before upgrading existing Genie servers to
3.2.0, follow the steps below to perform database upgrade, or create a new database with 3.1.x schema. Database upgrades
beyond 3.2.0 are handled automatically by the Genie binary via https://flywaydb.org/[Flyway].

===== MySQL

NOTE: This assumes the MySQL client binaries are installed

WARNING: Genie requires MySQL 5.6.3+ due to certain properties not existing before that version

Ensure the following properties are set in your `my.cnf`:

[source,text]
----
[mysqld]
innodb_file_per_table=ON
innodb_large_prefix=ON
innodb_file_format=barracuda
----

NOTE: Restart MySQL if you've changed these properties

Run:

[source,bash]
----
mysql -u {username} -p{password} -h {host} -e 'create database genie;'
----

====== 3.1.x to 3.2.0 database upgrade

WARNING: Genie requires MySQL 5.6.3+ due to certain properties not existing before that version

If you have an existing Genie installation on a database version < 3.2.0 you'll need to upgrade your schema to
3.2.0 before continuing.

Download the:
ifeval::[{isSnapshot} == true]
https://github.com/Netflix/genie/blob/master/genie-ddl/src/main/sql/mysql/upgrade-3.1.x-to-3.2.0.mysql.sql[3.1.x to 3.2.0 Schema Upgrade].
endif::[]
ifeval::[{isSnapshot} == false]
https://github.com/Netflix/genie/blob/v{revnumber}/genie-ddl/src/main/sql/mysql/upgrade-3.1.x-to-3.2.0.mysql.sql[3.1.x to 3.2.0 Schema Upgrade].
endif::[]

Ensure the following properties are set in your `my.cnf`:

[source,text]
----
[mysqld]
innodb_file_per_table=ON
innodb_large_prefix=ON
innodb_file_format=barracuda
----

NOTE: Restart MySQL if you've changed these properties

Then run:

probably create a larger directory you can store the job working directories and other places in. These correspond to
the `genie.jobs.locations.*` properties described below in the <<Properties>> section.

=== Wrap Up

This section contains the basic setup instructions for Genie. There are other components that can be added to the

Then run:

[source,bash]
----
psql -U {user} -h {host} -d genie -f upgrade-3.1.x-to-3.2.0.postgresql.sql
----

==== Local Directories

Genie requires a few directories to run. By default Genie will place them under `/tmp` however in production you should
probably create a larger directory you can store the job working directories and other places in. These correspond to
the `genie.jobs.locations.*` properties described below in the <<Properties>> section.

===== S3

If your commands, applications, or jobs depend on artifacts referenced via S3 URI, you will need to configure the S3
subsystem. If you're not assuming a role there is nothing you necessarily have to do provided a default credentials
provider chain can be created. See
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html[here]
for the rules for that.

If you need to assume a order to access Amazon resources from your Genie node set the property
`genie.aws.credentials.role` to the ARN of the role you'd like to assume. This will force Genie to create a
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html[STSAssumeRoleSessionCredentialsProvider]
instead of the default one.

Example role setting:
```
genie:
  aws:
    credentials:
      role: <AWS ROLE ARN>
```

=== Wrap Up

This section contains the basic setup instructions for Genie. There are other components that can be added to the
system like Redis, Zookeeper and Security systems that are somewhat outside the scope of an initial setup. You can see
the <<Properties>> section below for the properties you'd need to configure for these systems.

<MSG> Document S3 configuration (#506) (#509)


<DFF> @@ -144,6 +144,28 @@ Genie requires a few directories to run. By default Genie will place them under
 probably create a larger directory you can store the job working directories and other places in. These correspond to
 the `genie.jobs.locations.*` properties described below in the <<Properties>> section.
 
+===== S3
+
+If your commands, applications, or jobs depend on artifacts referenced via S3 URI, you will need to configure the S3 subsystem.
+
+1. Make sure `application-s3.yml` exists in `~/.genie` (or at a custom location, referenced by `--spring.config.location=`)
+2. An example of `application-s3.yml` is available below. Set either `credentials.file` or `credentials.role`
+  * For `credentials.file`, provide a filename of a file **in classpath** that contains the AWS access key and secret. A convenient location to place this file is `~/.genie/lib`.
+  * For `credentials.role`, provide an inline role value.
+3. Enable the Spring Boot `s3` profile during Genie launch, for example via command-line: `--spring.profiles.active=prod,s3`
+4. To confirm things are properly configured, head to `/actuator/beans` and verify that the `file.system.s3` bean was created
+
+Example `application-s3.yml`
+```
+genie:
+  aws:
+    credentials:
+      # File in classpath containing aws credentials, i.e. accessKey and secretKey
+      #file: <AWS CREDENTIALS FILENAME>
+      # OR role arn string
+      #role: <AWS ROLE ARN>
+```
+
 === Wrap Up
 
 This section contains the basic setup instructions for Genie. There are other components that can be added to the

Document S3 configuration (#506) (#509)