Q: Fast method to transfer multiple large files to a USB disk with progress update in command-line I need to transfer a number of large files of specific types from an internal disk to an USB SSD drive. The files I need to transfer are in the range of 5-200GB.
I would like to monitor the overall progress for the transfer of all files and have a fast transferspeed to the USB disk.
I started out with rsync and found the following to provide the best speed result:
rsync -W --size-only --protocol=29 --block-size=131072 --no-compress --info=progress2 /home/user/rosbags/*.bag /media/upload_disk/

This gives me a transfer speed of 150-180MB/s and I get a total progress for all files that need to be transfered. It seems to be helping to change the protocol in rsync to version 29 or earlier with 15-20% additional MB/s.
I specificly set the blocksize to the max value, to ensure this i not the limiting factor. Since I use the -W (whole file) setting it should use normal transfer and not delta compression. My assumtion is this should be similar to what one with get with a direct copy, but I could have misunderstood the settings.
When I use dd to copy the individual files:
dd if=/home/user/rosbags/2021-10-25-12-22-33.bag bs=131K status=progress of=/media/upload_disk/2021-10-25-12-22-33.bag

I get transfer speeds in the range of 500-600MB/s for the same setup.
In this case, I am using a similar block-size to rsync, to ensure they are as similar as possible.
I could make a script that uses dd to copy all the files individually and at the same time monitors the progress,
but I assume there is a smarter way?
Either using rsync or some other standard linux tool.
I have started writing a sh script that uses dd instead:
    #!/bin/sh
    total_size=$(du -sb /home/agco/ros_bags/| cut -f1)
    for i in  /home/user/ros_bags/*.bag; do
    echo $i
    #filename=$(basename $i)
    #echo $filename
    dd if=$i status=progress bs=64M of=/media/upload_disk/$(basename $i)
    done

The main challenge that I am facing is the ability to calculate the total progress for this I need to extract the current transferred number of bits.
I assume again, there is some smart way of doing this, but have been unable so far to find a solution.
