Benchmarking pre-production and production OpenStack clouds is not a trivial task. While it's important to find your cloud's limits, you don't want to actually damage your cloud. The OpenStack Rally benchmarking tool aims to make this task as simple as possible.
Since the very beginning Rally has been able to generate enough load for any OpenStack cloud, but generating too big load was the major issue for production clouds, because Rally didn't know how to stop the load until it was too late. Finally, I am happy to say that we have solved this issue.
With the new feature "abort on SLA failure" things are much better.
This feature can be easily tested in real life by running one of the most important and simplest benchmark scenarios, called "Authenticate.keystone". This scenario just tries to authenticate as users that were pre-created by Rally.
In human readable form, this input task means: Create 5 tenants with 10 users in each, then try to authenticate to Keystone 6000 times, performing 50 authentications per second (it run a new authentication request every 20ms). Each authentication request is done by one of the Rally pre-created users. This task passes only if the maximum average duration for authentication takes less than 5 seconds.
Note that this test is quite dangerous because it can subject Keystone to what is essentially a Distributed Denial of Service (DDoS) attack as we run more and more simultaneous authentication requests, and things may go wrong if something is not set properly (for example, on a DevStack deployment in a Small VM on your laptop).
On the duration chart, we can see that the duration for authentication requests reaches 65 seconds at the end of load generation, when Rally stops just before the bad things happened. But why does it run so many attempts tp authenticate? The reason is that we need better "success" criteria. (Note that in this case, "success" means we've managed to bring Keystone to the edge of crashing.) We had to run a lot of iterations to make average duration bigger than 5 seconds. Let's chose better success criteria for this task and run it one more time.
This time load stopped after 1410 iterations (versus 2495), which is much better.
The interesting thing on this chart is that first occurrence of an authentication that took more than 10 seconds was actually the 950th iteration. So why did Rally run 500 more authentication requests then? Let's look at the math: during the execution of that first bad authentication (10 seconds) Rally performed about 50 request/sec * 10 sec = 500 new requests, so as a result we ran just over 1400 iterations, instead of 950.
Rally tries to cover all DevOps and QA use cases, while still being a simple tool for easy and safe OpenStack testing. The "Stop on SLA failure" is crucial, and a huge step forward that enables usto use Rally for testing both pre-production and production OpenStack clouds under load.
P.P.S Kudos to Mike Dubov and everybody who was involved in the development of this feature!