<p><strong>TL;DR</strong>
Try reducing the number of concurrent connections used by awscli to 1 using this command: <code>aws configure set default.s3.max_concurrent_requests 1</code></p>

<hr>

<p>You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the <code>aws s3</code> commands (not <code>aws s3api</code>). </p>

<p>In the <a href="https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html#using-s3-commands-managing-objects" rel="nofollow noreferrer">cli user guide</a> under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.</p>

<p>I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of <code>lsof</code> I could see my browser was only opening a single connection for a ~500MB file download, whereas using <code>aws cp</code> 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.</p>

<p>Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times:
<code>
s3.max_concurrent_requests=10   8m22.965s
s3.max_concurrent_requests=1    6m47.772s
</code></p>

<p>So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.</p>

<p>You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: <a href="https://docs.aws.amazon.com/cli/latest/topic/s3-config.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/cli/latest/topic/s3-config.html</a>. </p>

CC BY-SA 4.0
2018-08-22T16:17:45.433Z
It works better in the CLI, not worse.
Uberhumus
CC BY-SA 4.0
2018-08-22T16:16:06.633Z
2018-08-22T16:16:06.633Z
Luke Richardson
<p>I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.</p>

<p>Is there a known reason for this? </p>

<p>Is there something that can be done about it?</p>

CC BY-SA 4.0
2018-08-19T18:26:42.237Z
Which one is slower? How large is the file?  Are the cli and the browser both running on your local machine?
Michael - sqlbot
CC BY-SA 4.0
2018-08-19T21:27:53.107Z
It’s slower in the console. I’ve checked with files ranging from 10Kb to 150Mb. Tried with chrome and firefox. Both in my machine.
Uberhumus
CC BY-SA 4.0
2018-08-19T21:49:27.850Z
There's no easily-explainable "standard" reason for that.  Every interaction with S3 takes place using the same APIs and same endpoints, whether you are using the console via a web browser or the CLI or one of the SDKs.  S3 does support parallel (byte range splitting) uploads (files > 5MB) and downloads (any size).  A browser download will never do parallel, but a browser upload sometimes does, and the CLI... I'm unsure of its standard behavior.  I suspect that's the most likely explanation, unless you are using a web proxy behind your browser that has performance issues.
Michael - sqlbot
CC BY-SA 4.0
2018-08-19T21:51:32.870Z
If it’s of any use, this is reliably repeated in our LAN, from any computer for over a week.
Uberhumus
CC BY-SA 4.0
2018-08-19T21:56:23.033Z
I suspect your LAN may have a web proxy.  Sometimes, a telltale sign of this is to visit a guaranteed-invalid domain such as `www.invalid.invalid` or `invalid.example.com` and the proxy may reveal itself in the error message, which will not be a standard browser error.
Michael - sqlbot
CC BY-SA 4.0
2018-08-20T08:57:57.233Z
I've checked and double checked, no web proxy.
Standard browser error.
Uberhumus
<amazon-web-services><amazon-s3><aws-cli>
Different download speeds from AWS S3 CLI vs Console/Websites
Uberhumus
# Different download speeds from AWS S3 CLI vs Console/Websites

(Asked by: username_0 on 2018-08-19)

I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.

Is there a known reason for this?

Is there something that can be done about it?

**Tags:** amazon-web-services, amazon-s3, aws-cli

> **Comments:**
> **username_1**: Which one is slower? How large is the file?  Are the cli and the browser both running on your local machine?  
> **username_0**: It’s slower in the console. I’ve checked with files ranging from 10Kb to 150Mb. Tried with chrome and firefox. Both in my machine.  
> **username_1**: There's no easily-explainable "standard" reason for that.  Every interaction with S3 takes place using the same APIs and same endpoints, whether you are using the console via a web browser or the CLI or one of the SDKs.  S3 does support parallel (byte range splitting) uploads (files > 5MB) and downloads (any size).  A browser download will never do parallel, but a browser upload sometimes does, and the CLI... I'm unsure of its standard behavior.  I suspect that's the most likely explanation, unless you are using a web proxy behind your browser that has performance issues.  
> **username_0**: If it’s of any use, this is reliably repeated in our LAN, from any computer for over a week.  
> **username_1**: I suspect your LAN may have a web proxy.  Sometimes, a telltale sign of this is to visit a guaranteed-invalid domain such as `www.invalid.invalid` or `invalid.example.com` and the proxy may reveal itself in the error message, which will not be a standard browser error.  
> **username_0**: I've checked and double checked, no web proxy. Standard browser error.  

---

## 1 answers

### Answer by username_2 on 2018-08-22. Score: 1

**TL;DR**
Try reducing the number of concurrent connections used by awscli to 1 using this command: `aws configure set default.s3.max_concurrent_requests 1`

---

You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the `aws s3` commands (not `aws s3api`).

In the [cli user guide](https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html#using-s3-commands-managing-objects) under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.

I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of `lsof` I could see my browser was only opening a single connection for a ~500MB file download, whereas using `aws cp` 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.

Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times:
`s3.max_concurrent_requests=10 8m22.965s
s3.max_concurrent_requests=1 6m47.772s`

So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.

You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: <https://docs.aws.amazon.com/cli/latest/topic/s3-config.html>.

> **Comments:**
> **username_0**: It works better in the CLI, not worse.  

---

