You are here
Evaluating Load Balancers and Network Performance in the Cloud
Abstract Cloud computing has emerged as a solution to bring the most popular online applications to large numbers of users all across the Internet. As part of this solution, load balancers perform a necessary function of distributing requests from a large number of clients among several servers. Two cloud computing providers, Amazon Elastic Cloud Compute (EC2) and GoGrid, offer proprietary load balancing solutions that require minimal configuration and potentially deliver significant performance benefits. To quantify the performance of these load balancing solutions, they were measured for maximum throughput while distributing web page requests across a small cluster of web servers. Requests were generated by a single source in one scenario and then from distributed generators in a second scenario. The networks inside the provider’s cloud are measured and analyzed looking for differentiating factors including network throughput and latency or evidence of network congestion. A significant performance advantage was observed with GoGrid’s load balancer, which uses specialized hardware from f5 , over Amazon EC2’s Elastic Load Balancer. The latency observed in the network between less expensive instances was more variable than in the more expensive instances for a particular provider. This indicates that there is an external factor that disproportionally hinders the network performance of the more affordable instances.
Cloud Computing as a Service
Whether intentional or not, the term cloud computing brings along with it a host of mysterious connotations and imprecise definitions. It is after all a paradigm shift in computing that involves abstracting the details of computing infrastructures. The key concept of cloud computing is providing Internet-accessible computing resources as a service. Furthermore, these resources are not restricted to any particular geographic location. Files and applications could be on an unknown server far away, but they are still instantly accessible somewhere “in the cloud.”
There are different products and services in the cloud computing marketplace and companies implement them in their own unique ways. Amazon EC2 and GoGrid belong to the Infrastructure as a Service (IaaS) paradigm in cloud computing, which is based on providing the raw access to a computing infrastructure. Other cloud computing offerings fall into the Software as a Service (SaaS) paradigm and offer complete online applications such as email or accounting solutions. Google’s AppEngine falls into this category. The final broad category of cloud services is called Platform as a Service (PaaS) to which Microsoft Azure belongs. PaaS provides an online development platform to create custom applications . An important benefit of cloud computing is that the buyer of the service doesn’t necessarily need to understand the details of its implementation. The type of service and acceptable conditions in which it will be delivered are established in a contract, but cloud computing providers are often free to determine how exactly they will provide the service.
In the crowded cloud computing marketplace it has become important for providers to differentiate their products. Reducing capital expenditures and the administration associated with IT were early selling points that enticed many businesses to buy into the cloud computing model. Now with more providers entering the market, it is not in a provider’s best interest to differentiate solely based on cost, but to highlight new features and performance enhancements so that they can demand a higher price. It can be to a provider’s advantage to hide the implementation details, as they are proprietary elements that distinguish their service. This is especially true within the Infrastructure as a Service (IaaS) paradigm in cloud computing, which is based on providing the raw hardware for the customers to use.
Virtualization in the Cloud
The emergence of cloud computing could never have been a possibility without virtualization. As a technology, virtualization has existed since IBM developed a method of partitioning mainframes into separate virtual machines in the 1960’s . However, it was isolated within a large machine room and accessible only to those with the privilege of physical access. Along came the Internet, connecting computers all across the world with speeds reaching into the multi-gigabit per second range today. Entire data centers running virtualization software can be connected and anyone connected to the Internet can access their aggregated resources.
Virtualization has taken away many of the restrictions that existed in previous data centers such as physical location, lead times to provision new hardware, and having to provision enough resources to accommodate peak utilization. A virtualized data center looks much like a conventional data center consisting of racks upon racks of servers. It requires technical staff to operate and maintain it and needs adequate power, cooling, and network infrastructures. However the software running on these commodity servers draw on the old concept of virtualization and allows for many virtual machines, or instances, to coexist on a single physical server. These virtual machines are made accessible over the Internet, making physical location insignificant. It is no longer necessary to purchase equipment in advance to accommodate the worst-case scenario of maximum utilization. An arbitrary number of servers can be instantly deployed under times of high application load and conversely, decommissioned under times of reduced load to save on energy costs. This translates into savings for the customer since they only have to rent additional computing capacity when it is needed.
A complication that arises is that the virtual machines are competing for common physical resources such as CPU, memory, and I/O buses. Ideally, these physical resources can be shared among virtual machines in a manner that that keeps them from sitting idle. However, fixed resources such as network and disk I/O buses become bottlenecks when multiple machines attempt to use them at the same time. In an IaaS cloud, virtual instances are a blank slate for the customer to use as they see fit. They can run an operating system of their choice, be it Microsoft Windows or Linux, and deploy customized applications. Applications that provide services such as media hosting, online backup and storage, and large computations are examples of applications that have been deployed successfully on IaaS clouds.
Performance and Load Balancing in the Cloud
The performance of load balancing solutions such as GoGrid’s hardware-based solution and Amazon’s Elastic Load Balancer (ELB) is important for any application in the cloud that is providing access to web content for many concurrent users. 3rd party load balancers, such as HAProxy, may be used by the customer configures them on a general-purpose virtual machine within the IaaS cloud. This approach has the disadvantage of restricting the load balancer to hardware and network connections that have not been specifically optimized for the task. The use of a load balancer allows web content to be served from many separate web servers instead of placing the entire load on a single machine, or in the case of cloud computing, a machine instance. The success of social networking, e-commerce, and auction applications in the cloud is contingent on being able to satisfy concurrent web requests from a large number of users.
Since the implementation details surrounding the load balancing offerings from GoGrid and Amazon EC2 are vague, this paper not only benchmarks load balancing performance, but looks for explanations of which factors are the most significant determinants of load balancing performance. By understanding the factors driving performance, it is possible to determine how the results for a specific experiment can be applied to other applications. This will ultimately help to provide a customer with the knowledge they need to choose the best cloud computing provider for their specific application. Here it is hypothesized that the network latency specific to a provider will be a significant determinant of load balancing performance. Also important to understanding load balancing performance is the variability of latency, also known as jitter, because it can be a warning sign of “sub-standard” network infrastructure  or network congestion. Examples where latency has already been identified as a limiting factor are in financial trading applications, video streaming services, and scientific High Performance Computing (HPC) applications [5- 7]. In these applications the computation is minimal compared to the time required to communicate the information across the network. This paper explores whether the latency bottleneck extends to load balancer performance.
The remainder of the paper is structured as follows. Section 2 introduces the load balancers to be measured from Amazon and GoGrid and the common network technologies we expect to be present in cloud computing. We discuss previous work in the area of cloud benchmarking in Section 3. Section 4 outlines the testing methodology used to collect the load balancing and network performance metrics and then Section 5 presents the measurements obtained. In Section 6 we draw comparisons between providers and look at load balancing performance in light of the network performance measurements we collected. Section 7 suggests some additional tests that could be done to eliminate possible sources of error and extrapolates on trends shown in the results.
Load Balancing and Network Technologies in the Cloud
GoGrid’s load balancer is unique because it is based on an f5 load balancer that runs on dedicated hardware and not on top of a general purpose machine instance in the cloud . The latter approach must be used with 3rd party load balancers such as HAProxy. Load balancers made by f5 are commonly used in applications outside of the cloud . Other than the hardware brand, little is known about the performance or implementation details of the GoGrid load balancer.
The details of the Elastic Load Balancer (ELB) solution that Amazon offers are even less clear. Amazon has withheld details on the hardware an ELB runs on and the physical placement of an ELB within its network. What is known is that it performs the same basic load balancing functions of distributing HTTP requests among multiple instances and monitoring the operational status of those instances . As the name suggests, there is an “elastic” component capable of scaling to support higher connection rates under circumstances of increasing demand [10,11]. The elastic scaling capability was not examined in this paper.
Networking and Virtualization
The network performance metrics of bandwidth, latency, and jitter are often left unpublished by cloud computing providers, but the understanding of overall application performance in the cloud depends on them. It is uncommon for providers to disclose the networking technology they to interconnect machines for fear of losing a competitive advantage or disclosing a potential shortcoming. Many of the advantages of server virtualization, elasticity and independence of physical location, depend on the network. Virtualization may obscure the concept of physical proximity in cloud computing, but location will have a concrete impact on network latency. The speed of light dictates that network delays will increase as the distance separating end-points increases. A network connection that traverses numerous network devices will exhibit higher network latencies because of queuing and processing delays at each device. Lower quality networking devices can result in inconsistent packet delivery and affect jitter . Additionally, heavy background load from external users has the potential to degrade network performance metrics and significantly increase latency variation . High traffic volumes can cause observed latency variation to increase by building up queues in the network devices as they hold packets waiting for available bandwidth to transmit .
For the most part, we can assume that the common and cost-effective network technologies are the ones used in cloud computing. Due the economics of the cloud computing model, high-end technologies such as InfiniBand, which supports a throughput up to 40 Gb/s and has latencies measured in microseconds , are unlikely to be used by public cloud providers. Ethernet has long been the standard for networking technology and as of January 2011 is available in 100 Mb/s, 1 Gb/s, 10 Gb/s varieties . The network technology used by cloud providers Amazon EC2 and GoGrid will most likely be one of those Ethernet variants.
Amazon’s ELB was the subject of a couple benchmarking efforts [14,15] and its behavior is now better understood even though its specific implementation remains obscured. A comprehensive benchmarking of processing power, memory and disk throughput, and specific application performance was done by CloudHarmony.com . Application specific benchmarks such as CloudStone and MalStone were designed specifically for cloud computing environments [17,18]. Perhaps the most general benchmark in cloud computing is CloudStone, which measures the performance of a social networking application called Olio in an attempt to characterize performance of the typical application . Olio involves several different components including a load balancer, several web servers, and a database backend, all of which interact over the network within the cloud . In evaluating other uses for cloud computing, numerous studies have concluded that the network performance in cloud computing is not yet suitable for HPC applications [19-22]. Also, other studies that measured network performance in EC2 have observed significant variability in the results [22-24].
To compare the performance of proprietary load balancers from Amazon EC2 and GoGrid, a sample workload was designed to test the maximum throughput of the load balancer. It was designed to be representative of a web application in the cloud capable of handling many simultaneous users through the use of load balancing. Since EC2 and GoGrid have different configurations and pricing models for their server instances, the workload aims to place minimal demand on the web server. Thus it is not influenced by differing configurations, and the bottleneck it tests is the maximum number of requests per second the load balancer can handle. The Amazon EC2 m1.small instance type was chosen in the US-East availability zone and the GoGrid instances were chosen with 512 MB of RAM in the California availability zone.
To initiate the measurement process a client instance, also called a load generator, requests a web page from the load balancer through the HTTP protocol. The load balancer then distributes that request among three servers, each identically configured and running the Apache web server application with a copy of the requested 166 byte static web page. This distribution happens in round-robin order. Once the Apache web server receives the request, it will fetch the web page from memory and return it to the load generator. Upon receipt, the load generator will record the time since the request was first sent out, measured as response time. Each load generator sends out many simultaneous requests ranging from 100 to 500 depending on the concurrency level chosen.
One customization that was made to the Apache web server configuration was enabling a feature of HTTP 1.1 called TCP keepalive. This feature allows subsequent requests to be sent over the same TCP session, and helps reduce the amount of protocol overhead involved in establishing separate TCP session for each web request . With TCP keepalive enabled, each request will consist of one network round trip time (RTT) from the client to the load balancer and another RTT from the load balancer to the web server. Those network delays are in addition to processing delays within the load balancer and web server. Other changes to the Apache web server configuration were setting MaxClients to 512, KeepAlive to On, and disabling mod_cache. The Linux kernel version was the same for each provider. The software packages were upgraded to the same version. However, providers can make customizations to the kernel unbeknownst to the customer. This could not be controlled for and is unlikely to have an effect on the results.
The open source HTTP load tester ApacheBench was used to generate requests and collect measurements. Ten repetitions at concurrency levels from 100 to 600 in increments of 100 were run. Command (1) was used to generate 10000 requests with a concurrency level of 100 with the keepalive feature enabled.
ab –k –n 10000 –c 100 –q (1)
All servers were placed within the same availability zone so that the testing had the best chance of being done within the provider’s network. This cannot be guaranteed since the physical placement of instances in cloud computing is done at the provider’s discretion and belonging to the same availability zone does not guarantee physical proximity. To gain some preliminary insight into the network infrastructure, the UNIX traceroute command was used to explore the path between instances. The traceroute results indicated that connections between instances in Go- Grid were done entirely at layer-2, while connections between EC2 instances traversed at least one layer-3 router hop.
To explore the network technologies used within the Amazon EC2 and GoGrid data centers, data was collected on network performance using the UNIX ping utility to measure latency and the Iperf tool to measure bandwidth. Within EC2, network performance data was collected for six system configurations in each of three availability zones (US-East, US-West, EU-West). Four and three system configurations were tested within GoGrid and RackSpace respectively. The ping utility was used to measure round trip times (RTT) for ICMP packets between two instances with identical system configurations and within the same availability zones with 30 replications. Iperf was used to measure average bandwidth at one second intervals for 60 seconds between the same two instances. To measure jitter, the distribution of ICMP RTTs is analyzed directly .
The load balancing measurements of throughput are shown in Table 1. Total throughput for Amazon EC2 regardless of concurrency levels was 2361.6 requests/sec 95% CI [2269.1, 2454.2]. This contrasts to 6293.5 95% CI [6106.6, 6470.3] for GoGrid. Neither provider exhibited increasing throughput with a higher concurrency level.
To explore this significant difference, the number of load generating client instances was increased to three. Table 2 shows that mean aggregate throughput was raised to a maximum of 6537.9 requests/s 95% CI [6430.1, 6645.7] and 2590.4 request/s 95% CI [2362.5, 2818.3] for GoGrid and EC2 respectively. However, the confidence intervals still overlap with the results for a single load generator, indicating there was no statistically significant increase in throughput. As the aggregate concurrency level was increased from 300 to 1200, GoGrid’s load aggregate balancing throughput decreased, while EC2’s aggregate throughput increased. The response time for a single request with a concurrency level of 100 request/sec averaged 78.9 ms for EC2 and 46.93 ms for GoGrid.
Presented in Table 3 are measurements of network bandwidth for multiple configurations in each provider. The measurements of Amazon EC2 m1 and c1 instances were separated because their high variation in latency. However, the Amazon m2 instances (m2.xlarge, m2.2xlarge, m2.4xlarge) all perform very close to 1 Gb/s across all availability zones, so they are shown as a single group. Certain GoGrid instances with over 2 GB of RAM also performed close to the 1 Gb/s mark. The instances with less than 2 GB of RAM measured slightly over 100 Mb/s, while the RackSpace instances measured less than 100 Mb/s.
A BUNCH OF TABLES GO HERE
Based on the bandwidth measurements, the latency results were grouped into instance configurations with similar network bandwidth and presented in Figure 1. This division is likely along lines of different network technologies employed by the provider. The figure shows the median latencies relative to other configuration groups and show quartile ranges, a measure of variation. Notice the seemingly high variability in Amazon m1 instances. In order to compare the measurements in the sub-1ms range among all providers, 28 measurements (out of 767) above 0.8ms are not shown in the figure.
PUT A PICTURE HERE
Figure 2 shows a violin plot of the latency data for the groupings of Amazon EC2 and GoGrid that have similar network characteristics. AWS.m1 and GoGrid <= 1GB are most representative of the networks used for the load balancing measurements. The width of the “violin” for each grouping represents the frequency of a measured value relative to all measured values for that grouping where the “fattest” point is the mode. The “violin” is symmetric about the center only as a visual aid. Figure 1 shows how The frequency distributions show that in addition to having lower mean latencies, GoGrid’s network has less latency variability. The AWS.m1 instances had a significantly more spread out distribution than other instances.
The measurement of load balancing throughput in two cloud computing providers, GoGrid and Amazon EC2, revealed a difference in performance of nearly three-fold. The magnitude of this difference is surprising given the similar pricing between the providers. The CloudHarmony.com results indicate much smaller differences in other performance metrics between similarly priced instances. While the published system configurations for the instances used for testing are different, with the EC2 m1.small instance running on the equivalent of a 1.7 GHz Intel Xeon processor  and GoGrid instance using a 2.26 GHz Intel Xeon E5520 processor , the expected impact of CPU speed is minimal. The EC2 instance does provide 1.7 GB of RAM while the GoGrid instance has 512 MB, but this is not a performance factor with only a 166 byte web page [1,9]. The latency results indicate that although network latency can vary significantly from provider to provider, it still only contributes to a small fraction of the total time spent serving each web request. Even the on the m1 configurations, network latency would only contribute two times the RTT latency, for a total of 0.90 ms, to the total response time. In comparison to the 78.9 ms average total response time measured with the m1.small EC2 instance, the contribution from network latency is insignificant, around one percent. Tests done by Microsoft have confirmed this relationship has a similar magnitude in load balancing web requests outside of the cloud . It also follows that the contribution to response time from network bandwidth is insignificant because each request and each response are contained within a single packet. With the capacity to transmit nearly 1 gigabit every second over the network, the time to send a single packet is on the order of microseconds.
Beyond establishing that network performance in terms of bandwidth or latency does not provide an explanation for the discrepancy in load balancing performance, the results in this paper can be used to infer the type of the networking technology being used and level of network congestion. From the results, it is likely that Gigabit Ethernet is the network technology employed for all but the RackSpace instances. The measured bandwidths of Amazon m2 instances and GoGrid instances with over 2 GB of RAM are extremely close to the 1 Gb/s theoretical limit of 1 Gigabit Ethernet. While it is difficult to achieve throughputs so close to the theoretical limits, 995 Mb/s has been achieved on 1 Gb/s links before with a UDP stream in the absence of competition for bandwidth from other sources . In comparison to TCP, UDP minimizes protocol overhead to achieve close to line rate transmission . The lesser performing configurations of m1 and c1 instances within EC2 and GoGrid instances with less than 2 GB of RAM that had throughputs between 100 Mb/s and 1 Gb/s. They are still likely using Gigabit Ethernet, since the next step down within Ethernet technology is 100 Mb/s and their measured throughputs are above that theoretical limit. The result of network congestion is that packets get queued and if the queues become too long, they are dropped . Both queuing and packet drops negatively affect the network bandwidth. Potential solutions to the problem of external users influencing network performance are the use of Quality of Service (QoS) and traffic shaping mechanisms to control access to network resources .
In addition to affecting network bandwidth, queuing delays will also increase network latency and contribute to the variability in latency. Not only did the Amazon m1 instances have lower network throughput that other Amazon instances, but as a group they had the highest latency and greatest variation in latency. When compared to Amazon EC2 and RackSpace, GoGrid instances had lower mean latencies and also a much tighter distribution of latency.
In this paper we measured the performance of the hardware-based load balancer from GoGrid and the Amazon EC2 ELB. In trying to explain the surprisingly high performance of the GoGrid solution, the null hypothesis of network performance being an influential factor in overall response time was disproven. Instead the source of differentiation must either come from a true difference in the load balancer performance or the testing protocol did not control for differences in hardware configurations. To establish the superiority of one load balancing solution over the other, one more set of tests with different instance configurations in both Amazon EC2 and GoGrid would have to be run. With the CPU and memory benchmarking done by CloudHarmony. com in combination with the network performance comparisons done here, two configurations could be chosen that have been verified to have similar system performance.
Even though the bandwidth and latency measurements were not useful in explaining differences in load balancer performance, they could be used make inferences on the networking technology in use and relative levels of congestion within a provider. Given that the trend of higher latencies in lower priced instances is present in both providers, it would seem the background network demand placed on the economically priced configurations is higher than the demand placed on the more powerful and expensive configurations. A reasonable hypothesis is that instances of the same type, but belonging to different customers, are likely to share the same physical hardware and network interconnects. When there is congestion present for a particular instance type, other instances with the same configuration will be affected the most. If it is true that Amazon uses the same network technology across configurations within EC2, it can be inferred that the m1 instances experience more network congestion than the c1 or m2 configurations, because they exhibit lower throughput, higher latencies, and higher variability in latency. A similar trend exists within GoGrid as well, where the cheaper configurations with 1 GB of RAM or less demonstrate lower throughput and higher latency.
Trends such as these are useful for customers trying to decide which cloud provider and specific configuration will meet their application’s needs and be the most cost effective. A highly congested provider may fall short of the performance expectations based on the published processor and memory specifications. If performance is negatively impacted, then the application may spend more time on the cloud and the customer will incur a higher cost. While latency turns out to be a minor factor in load balancing performance, one factor related to congestion and performance that merits further investigation is the percentage of packets dropped. Due to the retransmission properties of TCP related to congestion avoidance, packet drops can introduce large delays of multiple RTTs and often in unpredictable fashion . Future work should include the measurement of bandwidth with TCP streams to evaluate the impact of packet loss on load balancing HTTP requests.
Dave Jilk of Standing Cloud helped provide background knowledge in cloud computing and suggestions on topics that are currently of interest in the industry. Standing Cloud, Inc. covered the costs incurred to collect measurements for this paper.
Dr. Dirk Grunwald, Professor of Computer Science, University of Colorado at Boulder, advised me on this paper as the professor of a course in computer performance modeling, which was the basis for the measurement techniques I used.
 GoGrid. Cloud Servers. [Online]. Available: http://www.gogrid. com/cloud-hosting/cloud-servers.php
 Salesforce.com. What Is Platform as a Service (PaaS). [Online]. Available: http://www.salesforce.com/paas/
 VMware. Virtualization History, Virtual Machine, Server Consolidation. [Online]. Available: http://www.vmware.com/virtualization/ history.html
 Gourlay, Douglas. (January 8, 2009). What Is NOT Networking for the Cloud [Online]. Available: http://blogs.cisco.com/datacenter/ comments/what_is_not_networking_for_the_cloud/
 Hemsoth, Nicole. (June 17, 2010). Cloud, Wall Street and the Low Latency Imperative. [Online]. Available: http://www.hpcinthecloud. com/blogs/Cloud-Wall-Street-and-the-Low-Latency-Imperative- 96591509.html
 DeCusatis, Dr. Casimer and Bundy, Todd. (October 22, 2009). Limiting Latencies in the Cloud. [Online]. Available: http://www. hpcwire.com/specialfeatures/cloud_computing/news/Limiting- Latencies-in-the-Cloud-65657792.html
 Farber, Rob. (December 1, 2009). HPC Cloud Computing: Pie in the sky? [Online]. Available: http://www.scientificcomputing.com/ articles-HPC-Cloud-Computing-Pie-in-the-sky-120109.aspx
 Harbaugh, Logan G. (May 1, 2006). Load balancers from F5 Networks and Zeus Technology tip the scales. [Online]. Available: http://www.infoworld.com/t/data-management/load-balancers-f5- networks-and-zeus-technology-tip-scales-131
 Amazon Web Services. [Online]. Available: http://aws.amazon. com/ec2/
 MacVittie, Lori. (October 15, 2009). Amazon Elastic Load Balancing Only Simple On the Outside. [Online]. Available: http:// devcentral.f5.com/weblogs/macvittie/archive/2009/10/15/amazonelastic- load-balancing-only-simple-on-the-outside.aspx
 Swidler, Shlomo. (July 12, 2009). The “Elastic” in “Elastic Load Balancing”: ELB Elasticity and How to Test it. [Online]. Available: http://www.shlomoswidler.com/2009/07/elastic-in-elastic-loadbalancing- elb.html
 Barker, Sean “Empirical evaluation of latency-sensitive application performance in the cloud,” in Proceedings of the First Annual ACM SIGMM Conference on Multimedia Systems (Phoenix, Arizona, USA, 2010, pp. 35-36)
 Intel, Inc. 10 Gigabit Ethernet Technology Overview. [Online]. Available: http://www.intel.com/network/connectivity/resources/ doc_library/white_papers/pro10gbe_lr_sa_wp.pdf
 RightScale. (April 1, 2010). Benchmarking Load Balancers in the Cloud. [Online]. Available: http://blog.rightscale.com/2010/04/01/ benchmarking-load-balancers-in-the-cloud/
 Liu, Huan and Wee, Sewook. “Web Server Farm in the Cloud: Performance Evaluation and Dynamic Architecture,” in Lecture Notes in Computer Science: Cloud Computing, Berlin, Germany: Springer Berlin / Heidelberg, vol. 5931/2009, 2009, pp. 369-380.
 CloudHarmony.com. (May 29, 2010). What is an ECU? CPU Benchmarking in the Cloud. [Online]. Available: http://blog.cloudharmony. com/2010/05/what-is-ecu-cpu-benchmarking-in-cloud. html
 Will Sobel, Shanti Subramanyam, Akara Sucharitakul, Jimmy Nguyen, Hubert Wong, Arthur Klepchukov, Sheetal Patil, Armando Fox, David Patterson. (July 2009). Cloudstone: Multi-Platform, Multi-Language Benchmark and Measurement Tools for Web 2.0. [Online]. Available: http://radlab.cs.berkeley.edu/w/upload/2/25/ Cloudstone-Jul09.pdf
 Collin Bennett, Robert Grossman, and Jonathan Seidman. (June 1, 2009). MalStone: A Benchmark for Data Intensive Computing. [Online]. Available: http://malgen.googlecode.com/files/malstone- TR-09-01.pdf
 Napper, Jeffry and Bientinesi, Paolo. “Can cloud computing reach the top500?” in Proceedings of the Combined Workshops on Unconventional High Performance Computing Workshop Plus Memory Access Workshop (Ischia, Italy, 2009, pp. 17-20).
 Cacheiro, Javier Lopez, Fernandez, Carlos and Freire, Esteban. (2009). Providing Grid Services Based on Virtualization. [Online]. Available: https://www.egee.cesga.es/documents/Presentations/ VHPC09/vhpc09.pdf
 Masud, Raihan. High Performance Computing with Clouds. [Online]. Available: http://ix.cs.uoregon.edu/~raihan/HPC_with_ Clouds_Raihan_Masud.pdf
 Walker, Edward. (October 2008). Benchmarking Amazon EC2 for High-Performance Scientific Computing. [Online]. Available: http://www.usenix.org/publications/login/2008-10/openpdfs/ walker.pdf
 Mangot, Dave. (May 13, 2009). EC2 Variability: The numbers revealed. [Online]. Available: http://tech.mangot.com/roller/dave/ entry/ec2_variability_the_numbers_revealed
 Newman, Steve. Three Latency Anomalies.(April 2010). [Online]. Available: http://amistrongeryet.blogspot.com/2010/04/ three-latency-anomalies.html
 The Apache Software Foundation. (2009). Apache HTTP Server Version 2.0 Documentation. [Online] Available: http:// httpd.apache.org/docs/2.0/mod/core.html#keepalive
 Anuskiwicz, Jim. (April 24, 2008). Measuring jitter accurately. [Online]. Available: http://www.lightwaveonline.com/featuredarticles/ measuring-jitter-accurately-54886317.html
 Microsoft. (March 2000). Network Load Balancing Technical Overview. [Online] Available: http://technet.microsoft.com/en-us/ library/bb742455.aspx
 Hughes-Jones, Richard, Clarke, Peter and Dallison, Steven. “Performance of 1 and 10 Gigabit Ethernet cards with server quality motherboards,” in Future Generation Computer Systems. Amsterdam,Netherlands: Elsevier Science Publishers, 2005. vol. 21, issue 4, pp. 469-488.
 Vegt, Arjan van der. (July 16, 1999). UDP stream performance. [Online]. Available: http://www.science.uva.nl/research/air/projects/ old_projects/gigabench/vdvecht/node9.html
 Floyd, S. and Fall, K. 1999. Promoting the use of end-to-end congestion control in the Internet. IEEE/ACM Trans. Netw. 7, 4 (Aug. 1999), 458-472.
 Morris, R. “TCP behavior with many flows,” in Proceedings of the 1997 International Conference on Network Protocols (ICNP ‘97). pp. 205.