  In the past decade, cloud computing has emerged from a pursuit for a
service-driven information and communication technology (ICT), into a
signifcant fraction of the ICT market. Responding to the growth of the market,
many alternative cloud services and their underlying systems are currently
vying for the attention of cloud users and providers. Thus, benchmarking them
is needed, to enable cloud users to make an informed choice, and to enable
system DevOps to tune, design, and evaluate their systems. This requires
focusing on old and new system properties, possibly leading to the re-design of
classic benchmarking metrics, such as expressing performance as throughput and
latency (response time), and the design of new, cloud-specififc metrics.
Addressing this requirement, in this work we focus on four system properties:
(i) elasticity of the cloud service, to accommodate large variations in the
amount of service requested, (ii) performance isolation between the tenants of
shared cloud systems, (iii) availability of cloud services and systems, and the
(iv) operational risk of running a production system in a cloud
environment.Focusing on key metrics, for each of these properties we review the
state-of-the-art, then select or propose new metrics together with measurement
approaches. We see the presented metrics as a foundation towards upcoming,
industry-standard, cloud benchmarks.
  Keywords: Cloud Computing; Metrics; Measurement; Benchmarking; Elasticity;
Isolation; Performance; Service Level Objective; Availability; Operational
Risk.
