  The massive payment card industry (PCI) involves various entities such as
merchants, issuer banks, acquirer banks, and card brands. Ensuring security for
all entities that process payment card information is a challenging task. The
PCI Security Standards Council requires all entities to be compliant with the
PCI Data Security Standard (DSS), which specifies a series of security
requirements. However, little is known regarding how well PCI DSS is enforced
in practice. In this paper, we take a measurement approach to systematically
evaluate the PCI DSS certification process for e-commerce websites. We develop
an e-commerce web application testbed, BuggyCart, which can flexibly add or
remove 35 PCI DSS related vulnerabilities. Then we use the testbed to examine
the capability and limitations of PCI scanners and the rigor of the
certification process. We find that there is an alarming gap between the
security standard and its real-world enforcement. None of the 6 PCI scanners we
tested are fully compliant with the PCI scanning guidelines, issuing
certificates to merchants that still have major vulnerabilities. To further
examine the compliance status of real-world e-commerce websites, we build a new
lightweight scanning tool named PciCheckerLite and scan 1,203 e-commerce
websites across various business sectors. The results confirm that 86% of the
websites have at least one PCI DSS violation that should have disqualified them
as non-compliant. Our in-depth accuracy analysis also shows that
PciCheckerLite's output is more precise than w3af. We reached out to the PCI
Security Council to share our research results to improve the enforcement in
practice.
