In Sqoop, We can import data from MySQL and write to HDFS. It basically utilizes Mapper that opens up a pipe to mysqldump and pulls data directly. If you look at the following MySQLDumpMapper, it creates CopyingAsyncSink to process the MySQLDump stream via the reader thread, CopyingStreamThread. The thread reads data from mysqldump and cleans up any extra header/characters before writing to HDFS.
On the other hand, to export data from HDFS back to MySQL, it uses MySQLExportMapper that starts a mysqlimport process and uses that to export rows from HDFS to a MySQL database at high speed. Basically, it creates FIFO, a named pipe (using mknod –mode=0) to interface with mysqlimport behind the scene. A BufferedOutputStream is used to write bytes straight to mysqlimport process via the above named pipe, which is basically can be thought as a file.
* Mapper that opens up a pipe to mysqldump and pulls data directly.
// AsyncSinks used to import data from mysqldump directly into HDFS.
* header and footer characters that are attached to each line in mysqldump.
// Actually do the read/write transfer loop here.
int preambleLen = -1; // set to this for "undefined"
// across all lines, so just figure this out once.
// flag this error so we get an error status back in the caller.
LOG.error("InterruptedException reading from mysqldump: "
* output, and re-emit the text in the user's specified output format.
// Configure the output with the user's delimiters.
LOG.warn("ParseError reading from mysqldump: "
continue; // Skip emitting this row.
int recordLen = 1; // for the delimiter.
// flag this error so the parent can handle it appropriately.
// TODO(aaron): Refactor this method to be much shorter.
* the database and upload the files directly to HDFS.
// We need to parse the connect string URI to determine the database name.
LOG.info("Performing import of table " + tableName + " from database "
args.add(MySQLUtils.MYSQL_DUMP_CMD); // requires that this is on the path.
// --defaults-file must be the first argument.
// confuse Java, and adding in surrounding quotes confuses Java as well.
// If the user supplied extra args, add them here.
// begin the import in an external process.
// Actually start the mysqldump.
// read from the stdout pipe into the HDFS writer.
LOG.debug("Output delimiters conform to mysqldump; "
// Start an async thread to read and upload the whole stream.
// Start an async thread to send stderr to log4j.
// block until the process is done.
// Remove the password file.
// block until the stream sink is done too.
// Try to wait for stderr to finish, but regard any errors as advisory.
LOG.info("Thread interrupted waiting for stderr to complete: "
throw new IOException("mysqldump terminated with status "