If a server is hosting a download, will that server's speed determine the download speed on the side of the downloader or will that not matter since it's the downloader's own internet connection speed that determines how long the download will take?
It partially depends on what you mean by "server's speed". The server's network connection speed will absolutely have an impact on the transfer rate of data. It's likely the server has a much faster network connection than the clients, but enough clients could saturate the link and result in slower downloads.
It's also true that driving a network interface uses CPU resources, and so a server doing a lot of network IO will also be using some CPU. However, these days, that CPU load is probably very small and you'll run out of network bandwidth before you run out of CPU cycles to process it.
It's also true that lots of network connections consume other resources on a server, including memory, so lots and lots of long running downloads will tie up other resources and may or may not result in performance bottlenecks.
Lastly, there's an awful lot of network infrastructure usually between a client and the server, and it could all potentially impact the rate of download.
There is always a bottleneck somewhere. Determining a maximum network transfer speed is a matter of locating that bottleneck. Possible bottlenecks in their order of likelihood are as follows.
You see, even a server on a 100Mbits/sec Internet connection and a client on a 25Mbits/sec fiber-to-the-home Internet connection may not see transfer speeds of 25Mbits/sec for a single tcp session. The latency between the client and the server will likely impact the speed at which the tcp sliding window can move forward based on the rate of the receipt of acknowledgement. See this article about tcp perfmance: http://www.cisco.com/web/about/ac123/ac147/ac174/ac196/about_cisco_ipj_archive_article09186a00800c8417.html
This latency-induced factor is the main reason for things like Content Delivery Networks being used by smaller sites that don't need the additional capacity to handle their user's requests.