Q: Using Spark to retrieve properties in an Oozie Workflow I'm using Spark 1.3.0 and Oozie 4.1.0
I have defined an Oozie workflow for a Spark action as follows (slightly condensed for readability).
 <action name="sparkler-kicker">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.output.dir</name>
                <value>/user/oozie/output/</value>
            </property>
        </configuration>
        <master>yarn-cluster</master>
        <mode>cluster</mode>
        <name>sparkler-kicker</name>
        <class>com.sparklet.SparkClientCount</class>
      <jar>${nameNode}/apps/${JobName}/${JobStack}/lib/${JobName}.jar</jar>
        <spark-opts>... more here ...</spark-opts>
        <arg>...args here...</arg>
    </spark>
    <ok to="mark-job-end"/>
    <error to="mark-job-fail"/>
</action>

I would like to specify a Spark driver to write output based on a path provided within <configuration>, say for example the property with name mapred.output.dir. Is it possible for my Spark driver to programmatically read those properties? I can't seem to access them through the SparkConf or JavaSparkContext.hadoopConfiguration() objects. Based on other documentation I have found, almost all Spark programs use <arg>...</arg>: I have not found any examples of reading <property> from within <configuration>.

A: I'm also in the process of learning Oozie myself, but I believe you can do it this way:
In your job.properties file you could define the output dir as a variable:
outputDir=/user/oozie/output
Then in your workflow, refer to that variable in <configuration> and also pass it in as an additional <arg> to your Spark app.
 <action name="sparkler-kicker">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.output.dir</name>
                <value>${outputDir}</value>
            </property>
        </configuration>
        <master>yarn-cluster</master>
        <mode>cluster</mode>
        <name>sparkler-kicker</name>
        <class>com.sparklet.SparkClientCount</class>
      <jar>${nameNode}/apps/${JobName}/${JobStack}/lib/${JobName}.jar</jar>
        <spark-opts>... more here ...</spark-opts>
     <arg>${outputDir}</arg>   
     <arg>...some more args...</arg>
    </spark>
    <ok to="mark-job-end"/>
    <error to="mark-job-fail"/>
</action>

Update: adding an alternative method to defining the variables in job.properties, as pointed out by @SamsonScharfrichter in his comment. Instead of defining outputDir in job.properties, you can define them instead in your workflow in a parameters element.
<parameters>
    <property>
        <name>outputDir</name>
        <value>/user/oozie/output</value>
    </property>
</parameters>
<action name="sparkler-kicker">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.output.dir</name>
                <value>${outputDir}</value>
            </property>
        </configuration>
        <master>yarn-cluster</master>
        <mode>cluster</mode>
        <name>sparkler-kicker</name>
        <class>com.sparklet.SparkClientCount</class>
      <jar>${nameNode}/apps/${JobName}/${JobStack}/lib/${JobName}.jar</jar>
        <spark-opts>... more here ...</spark-opts>
     <arg>${outputDir}</arg>   
     <arg>...some more args...</arg>
    </spark>
    <ok to="mark-job-end"/>
    <error to="mark-job-fail"/>
</action>

