I continue to be impressed by the VZ team. Their latest PCI Compliance Report continues their contribution of data sharing with the industry. Here are a couple cherry picked passages from the exec sum:
"Essentially unchanged from last year, only 21 percent of organizations were fully compliant at the time of their Initial Report on Compliance (IROC). This is interesting, since most were validated to be in compliance during their prior assessment. What causes this erosion over the course of the year?"
Organizations struggled most with the following PCI requirements: 3 (protect stored cardholder data), 10 (track and monitor access), 11 (regularly test systems and processes), and 12 (maintain security policies)."
Here's another passage that surprises no one, from pg. 29:
"The secret to maintaining compliance lies largely in treating it as a daily part of conducting business. To achieve this, the correct mind-set must be instilled across the organization, and this type of integration must come from the top down."
I may be projecting but while I read the report, I kept visualizing the ops and security teams scrambling between the initial and final assessments to hurry up and get compliant. Given the data above, deep down they knew they'll be doing the same thing next year. This got me wondering: how much time, money, and opportunity cost is spent scrambling to get compliant? How can we reduce this cost?
Here's a hypothesis: the cost of running a compliant shop is less than the cost to run annual fire drills. Boy I'd love to test this hypothesis. This also fits into my favorite question about metrics: is control effectiveness increased simply by the fact it's being measured? Mind you I'm talking about measuring, not auditing. This PCI report clearly shows audits don't improve controls throughout the year. What if we applied metrics to key operating controls under the PCI scope? Would a minor investment in spinning up a metrics program be less than the annual effort to get compliant? Is the cost of maintaining a control less than the cost of scrambling to fix it?
I took the liberty to identify a candidate set of metrics that may help answer these questions. I ran through the requirements and selected 23 metrics. I don't think we need a metric for every requirement. My goal was to identify metrics that provide visibility to a group of controls. E.g. % of users with completed role verification per some schedule. This metric may knock off multiple requirements e.g. least privilege, segregation of duties, terminations, vendor, and remote access. I also focused on the operational controls vs. design decisions e.g. metrics aren't a fit to measure your encryption design.
Here are the 23 (feel free to scroll down and read them later if you want more monologue):
Metric Title/ Category/ Description
Post-prod. Vulns/ .Application/ # Post-production applications vulnerabilities Target: 0 Support PCI requirement 6.5
Secure Development/ Application/ % of in scope application development projects that follow secure development practices. Focus on PCI requirement 6.3 and 6.5.
Change Control Security/ Change Control/ Number of changes that reduce or violate security policy. Focus on PCI requirement 6.4.
Production Access/ Change Control/ Number of non-operations personnel with access to production systems. Focus on developer access to address PCI 6.4.
PCI Servers/ Data/ Number of actual in scope servers holding PCI data compared to known target number. Goal is to set a target and verify at <regular intervals>.
Retention/ Data/ Number of in scope records past data retention policy. Frequency: quarterly Target: 0
Device ManagementDevice% of devices managed per PCI requirements. Scanners and/or configuration management agents are good sources to produce this metric. e.g. 1.4 Install personal firewall software on any mobile and/or employee-owned computers 2.2 Develop configuration standards for all system components. 5.1 Antivirus use and management.
Device Vulns/ Device/ Number of unaccepted patch & config vulnerabilities beyond predetermined time
frames e.g. sev 5 within 30 days. target: 0 source: scanner frequency: as frequent as feasible for your org. prerequisites: policy stating patch SLA in days per vuln severity level, identified asset group by business importance. Focus on PCI requirements 6.1, 6.2, 11.2.
Intrusion Detection/ Device/ % of in scope system traffic monitored by IDS/IPS on a <quarterly> basis. target: 100% Focus on PCI requirement 11.4.
Integrity Monitoring/ Device/ % of in scope systems with file integrity monitoring on a <quarterly> basis. target: 100% Focus on PCI requirement 11.5.
Firewall Review/ Firewall/ % of firewall and router rule sets reviewed <at least every 6 months> target: 100% Supports PCI requirement 1.1 "review firewall and router rule sets at least every six months."
Default Credentials/ IAM/ # servers/infrastructure devices with default credentails target: 0 source: scanner, manual A&P frequency: <depends on org e.g. quarterly> Focus on PCI requirement 8.
Terminations/ IAM/ # of Employee terminations outside predetermined time frames Target: 0 Focus on PCI requirement 7.
Role Verification/ IAM/ % of users with completed role verification per schedule target: 100% source: manual or automated frequency: semi-annual Focus on PCI requirement 7.
Credential Strength/ IAM/ % servers/infrastructure devices with 2-factor or password complexity reqs target: 100% source: config review, manual A&P frequency: semi-annual Focus on PCI requirement 8.
Incident Response/ Incident Response/ Number of incidents not handled in accordance to documented incident response procedures. target: 0 Focus on PCI requirement 12.5 and 12.9.
System Monitoring/ Monitoring/ % of in scope systems monitored per PCI requirements. target: 100%
Unauthorized WLAN/ Monitoring/ Number of unauthorized WLANs identified on a quarterly basis. target: 0 Focus on PCI requirement 11.1.
Visitor Badge Return/ Physical Security/ Percentage of visitor badges returned per <time period>. Focus on PCI requirement 9. The goal is to identify an inexpensive metric that provides some attention to physical access. The assumption is measurement in one area will increase the effectiveness of others.
Policy Review/ Policy/ % policies reviewed on an <annual basis>. Focus on PCI requirement 12.1.
Closed Risk Dispositions/ Risk Assessment/ Number of risks identified during the annual assessment without a risk owner and disposition e.g. accept, mitigate, transfer. target: 0 Focus on PCI requirement 12.2.
Vendor Remediation/ Vendor/ # of vendor security findings that have not been addressed within committed time frames. Helps support PCI 2.4 Shared hosting providers must protect each entity’s hosted environment and cardholder data.
Vendor Assessments/ Vendor/ Percent of vendors who receive security assessments within policy e.g. vendors with sensitive data/services must be reviewed semi-annually. Helps support PCI 2.4 and 12.8. Shared hosting providers must protect each entity’s hosted environment and cardholder data.
An interesting observation is that our MoneySec list of metrics only contains 15. Recall the MoneyBall inspired list of metrics is a candidate list to identify key controls that correlate with reducing incidents. If all our hypotheses were correct, we could show that compliance focuses on too many things that don't matter (but that's another post).
I fully understand you're not going to spin up a metrics program for all 23 without some demonstration of value. So I propose you start with one metric, my favorite, that covers many PCI requirements: number of scanner-sourced vulnerabilities past due. This is a great metric because it requires:
You regularly scan
Have a process to rank vulnerabilities
Have a pre-negotiated service level in days to fix vulns based on severity level
Have the ability to age vulnerabilities
This metric knocks off all kinds of PCI reqs e.g. conducting scans, mitigating vulns, default creds, patches, config vulns, everything a scanner catches that's listed in PCI.
Here's an example using our Vuln Tracker tool to age vulns from popular scanners. Note you can use other tools or a spreadsheet (as Brian Keefer demonstrated in our MoneySec presentation).
This example shows a couple hundred vulns overdue, some older than 3 months. From a metrics point of view, you could set a target value for Overdue Vulns at say 10 vulns per month. Here's how that might look using our Metrics Manager (again, any old spreadsheet will suffice).
This shows the history of mitigation performance and how well we performed to our expected target.
What do you think? Does the simple fact of looking at something change its behavior? In my experience, yes. So if you have the annual privilege of meeting your QSA, see if you can save your company money by maintaining compliance vs. getting compliant on an annual basis.
Please provide feedback on the list of metrics above. It was just me and my pint who produced them and I'm sure they can improve!