Q: pyspark store rdd in an existing folder without overwrite I have tried to use the saveAsTextFile() function in PySpark 1.3.1; however, there will be an exception that Output directory already exists, if I did not delete the existing directory.
I was wondering that is there a way to store a spark RDD incrementally to an existing folder without overwrite?  

A: No, you can't do that: this is to avoid accidental overriding of files. 
What you may do is to specify a subfolder of the existing directory and then you can retrieve all the data in all the subfolders when you need that. 
Otherwise the only option you have is to put the resulting files in a temporary folder and then move them wherever you need. This can be achieved by hand of course or you can use oozie.

A: Or you could read the output directory to RDD and union two RDDs and write with overwrite to that directory.
