Q: Scanning a large DynamoDB Table - 1.5TB in size with 10 Billion records Background
I am in need to scan a quite large DynamoDB table and am looking for ways to optimize scan speed. I have tried running a simple scan query and the scan wasn't complete even after a day of running. The table is 1.5TB in size with 10 Billion records, with an avg record size of 0.15KB
Details
PrimaryKey: "{userId}:{barcode}"
SortKey: "{epochTimeInMillis}"

In our usecase, we have access to just the userId and we have to scan the table to find rows where the userId is in the PrimaryKey.
Question
Any ideas on what I can try with the request to expedite the ScanTime? Right now, I've set 30K Read Capacity Units but while the Scan is running I only seeing around 1500 RCUs being utilized, so I don't think that's a problem.
Best,

A: I'm not aware of any way to speed up a DDB Scan().. (Edit duh...from F_SO_K's answer use parallel scan)
Although I wouldn't expect 10 billion rows to take more than day. (EDIT yeah actually a day and a half to read 1.5TB in batchs of 1MB 85ms response time per batch)
If this is something you anticipate doing more than once, I'd set up a GSI with userId as the hash key and something else as the sort key.  Perhaps barcode:epochTimeInMillis
Honestly, I'd have considered userId as the hash key for my table...does a user really have more than one barcode?  Or is the barcode not unique to the user?
EDIT
I can understand not wanting to pay for a GSI that you don't need often... but storagewise, 1.5TB is only $375/mo or so (US East).  And you that assumes the entire record is replicated.  You'd want to have WCU match the table, but RCU could be 1 or 2, just bump it up if need be on the days you actually need to use it.

A: Yes, you can speed it up loads using parallel scan. I've done a load of testing on my own tables before and found the following rules of thumb:
A parallel scan is more efficient than a regular scan when there is more than 2MB of data to get, and seems to be optimised when the number of segments equals the number of MBs in the scan. For example a scan that evaluates 400MB of data is optimal with 400 segments.
Here is a CLI example of using a parallel scan:
aws dynamodb scan --table-name YOUR-TABLE --total-segments 1 --segment 0 --select COUNT


aws dynamodb scan --table-name YOUR-TABLE --total-segments 4 --segment 0 --select COUNT

