Q: Load 1 million records from a file and save to PSQL database I have a file of 1 million records in which I have to pass one by one record to elastic search and save the resulted data into the database.
But the issue is, it is taking very long time to do that as the records are streaming one by one to elasticsearch then it saves the data into PSQL database.
I want some suggestions that how can I improve on this or should use some other tools.
Right now I am using Nodejs with some packages:
I upload the file in nodejs application and convert it to json file using
const csv=require('csvtojson')
I use
const StreamArray = require('stream-json/streamers/StreamArray');
const {Writable} = require('stream');

For reading json and parsing it through these packages using stream as the file is too big.
I use this code
const fileStream = fs.createReadStream(this.fileName);
            const jsonStream = StreamArray.withParser();
            const incomingThis = this;
            const processingStream = new Writable({
                write({key, value}, encoding, callback) {
                    incomingThis.recordParser(value, (val, data) => { // pass the data to elasticsearch to get search data
                        incomingThis.processQueue(data); // save the data to the PSQL database
                        callback();
                    });
                },
                //Don't skip this, as we need to operate with objects, not buffers
                objectMode: true
            });
            //Pipe the streams as follows
            fileStream.pipe(jsonStream.input);
            jsonStream.pipe(processingStream);
            //So we're waiting for the 'finish' event when everything is done.
            processingStream.on('finish', async () => {
                console.log('stream end');
                const statistics = new Statistics(jobId);
                await statistics.update(); // update the job table for completion of data
            });

Please suggest How can I improve on this to parse 1 million records file in couple of hours rather than days or minimum less time.
I am open to use any other tools too like redis, spark if these will help me out.
Thanks.

A: Instead of one by one pressing from stream . use batch approach ( create multiple batches ) to get data in  elastic and save in batch .
